The robots.txt file is something that WordPress auto generates that tells search engine spiders how to index your site. The file is auto generated for you. You can find your robots.txt by just adding “robots.txt” behind your domain name. The generic WP robots.txt is this.
User-agent: *
Disallow:Sitemap: http://brianlis.com/sitemap.xml.gz
This code tells spiders indexing your site to refer to the sitemap.xml.gz file to index your site. Hopefully your using a sitemap. If not you can download this plugin that makes one for you Google (XML) Sitemap. This is a pretty important step for SEO.
Ok to add some punch to your robots.txt file I add the following code.
# Robots Rule! – Sometimes… #
User-agent: *
Allow: /# Disallow these directories, url types & file-types
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /search/*/feed
Disallow: /search/*/*
Disallow: /readme.html
Disallow: /license.txt
Disallow: /*?*
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Allow: /wp-content/uploads/
Allow: /*?page=*User-agent: Mediapartners-Google
Allow: /User-agent: Adsbot-Google
Allow: /User-agent: Googlebot-Image
Allow: /User-agent: Googlebot-Mobile
Allow: /User-agent: ia_archiver*
Disallow: /User-agent: duggmirror
Disallow: /Sitemap: http://brianlis.com/sitemap.xml.gz
Basically what this does is it helps the possibility of double content on your site. Or content getting indexed twice which is a SEO no-no. For more on WordPress, Robots.txt, and WP SEO check out the WordPress Codex Search Engine Optimization for WordPress.
Keep posting stuff like this i really like it
It’s really a nice and helpful piece of information. I’m glad that you shared this helpful info with us. Please keep us informed like this. Thanks for sharing.