WordPress Consultant

WordPress Design, Developing & Consulting

  • logo
  • Home
  • Services
    • Domain Names
    • Hosting
    • Maintenance
    • Support
    • Search Engine Optimization
    • Website Speed
    • WordPress Training
  • Portfolio
  • Web News
  • Bells & Whistles
  • About
    • Testmonials
  • Contact

WordPress robots.txt file

August 6, 2010 by Brian 2 Comments

The robots.txt file is something that WordPress auto generates that tells search engine spiders how to index your site. The file is auto generated for you. You can find your robots.txt by just adding “robots.txt” behind your domain name. The generic WP robots.txt is this.

User-agent: *
Disallow:

Sitemap: http://brianlis.com/sitemap.xml.gz

This code tells spiders indexing your site to refer to the sitemap.xml.gz file to index your site. Hopefully your using a sitemap. If not you can download this plugin that makes one for you Google (XML) Sitemap. This is a pretty important step for SEO.

Ok to add some punch to your robots.txt file I add the following code.

# Robots Rule! – Sometimes… #
User-agent: *
Allow: /

# Disallow these directories, url types & file-types
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /search/*/feed
Disallow: /search/*/*
Disallow: /readme.html
Disallow: /license.txt
Disallow: /*?*
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Allow: /wp-content/uploads/
Allow: /*?page=*

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: ia_archiver*
Disallow: /

User-agent: duggmirror
Disallow: /

Sitemap: http://brianlis.com/sitemap.xml.gz

Basically what this does is it helps the possibility of double content on your site. Or content getting indexed twice which is a SEO no-no. For more on WordPress, Robots.txt, and WP SEO check out the WordPress Codex Search Engine Optimization for WordPress.

Filed Under: Web News Tagged With: robots.txt, seo, WordPress

Sign up for my Newsletter

  • General WordPress Tips and How-To's
  • Branding & Marketing Suggestions
  • Insider information and offers
  • Internet Tricks of the Trade - Design, Social Media and More

Whatcha Waiting For?

Comments

  1. Registered nurse says

    November 13, 2010 at 4:44 pm

    Keep posting stuff like this i really like it

    Reply
  2. occupational therapy says

    November 17, 2010 at 12:33 am

    It’s really a nice and helpful piece of information. I’m glad that you shared this helpful info with us. Please keep us informed like this. Thanks for sharing.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Client Testimony

Skin of Gold

The design and service exceeded my expectations. Thanks again.

Ariana Daglian

More Testimonies

Logo Design

Logo created by Brian Lis Web Design, Inc.

© 2022 Brian Lis Web Design Inc · Terms of Service · Genesis Framework · WordPress · G+ · Credits