Wordpress users! Are you using ROBOTS.TXT? If not, I encourage you to create one and upload it under your website domain.

What is robots.txt?

From my own definition hehe, ROBOtS.TXT is a piece of file that instructs and acknowledged by major Search engine bots, (Google, Yahoo, MSN, Askjeeves) which area and what files or directories of your website are restricted and don’t need to spider.
For example: If you have an image directory called “products_images” that you don’t want to appear in Google images. All you have to do is to add this line to your robots.txt

User-agent: Googlebot-Image
Disallow: /projects_images/

Benefits of having ROBOTS.TXT

# To avoid duplicate content
Did you know by default, your wordpress display some of your post repeatedly unless if you are using excerpts.
For example:
a copy of your latest post from the homepage, another copy from your date archive pages, another copy from its own post. You can minimized this by using robots.txt. A great robots.txt for seo wordpress sample by AskApache.com

# Minimize Bandwidth consumption
Another reason to keep a robots.txt on your wordpress is to minimized bandwidth usage. Unnecessary wordpress directories to cache like wp-admin, wp-includes, themes, cache, plugins, trackback, comments, and etc. Try to add this on your robots.txt:
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: */trackback
Disallow: */comments
Disallow: /*?*
Disallow: /*?

I tried this trick and minimized almost 30% my monthly bandwidth consumption.
# To hide or to prevent appearing of your private documents in Search Engines:
As a mention above.

How to make a robots.txt

Very easy, no sweat, you may use your friendly Windows Notepad or Wordpad, but not MSWORD. Encode what parts of your site to add in robots.txt. Heres a copy of robots.txt from this site. After encoding the robots.txt, save it as “robots.txt” of corz hehe, then upload it at the root of your website. By the way, If your wordpress is installed under a folder of your domain, you need to adjust my sample command robots.txt lines. Comments are welcome!