In the last SEO post we mentioned about Robots.txt file and how it plays a major role in Search Engine Ranking. It tells the search engine bots which important part of our blog to index and what not to index. This is why you should properly configure a Robots.txt file otherwise any error on your part will result in a poor SERP result.
What is Robots.txt and why should every WP Blog use it?
A Robots.txt is a simple text file in the root directory of the domain. It contains rules for proper indexation of your website. Basically the file gives a search engine information about which parts of your site to index and which part they should not go to.
With the start of Panda Algorithim, duplicate content or low quality sites were being punished and therefore any initial or low quality content should be noindexed with the help of a robots.txt file.
Even if your website is not complete, then you can configure robots.txt to not index your website and once completed you can remove the noindex tag from your development website.
Do I Really Need a Robots.txt File?
It is certainly not necessary to have a Robots.txt file.Your website/blog will be indexed even if you do not have a Robots.txt file. WordPress itself has a virtual robots.txt file created in the root directory.
But having a physical Robots.txt file is beneficial as it provides a way for search engines to index only those portion of site which are deemed important and Bing,Google,Yandex and other high quality search engines will honor the robots.txt file. However low quality search crawlers might just disregard your robots.txt file completely.
Where to Find Robots.txt
It is very easy, you will find robots.txt file in the root of your WordPress installation. If you are still having a static file on your server then you must be having a robots.txt file in the root folder of the domain.
If you do not have a robots.txt file in your WordPress installation then using some of the plugins might help you in creating a Robots.txt file. If you yourself want to create a robots.txt file then you should just follow the syntax of robots.txt. If you have a multiple domain hosting then each of your domain directory should have it’s individual robots.txt. One main domain will not work for all the domains.
The robots.txt has its own syntax to define rules. These rules are also called “directives”.
Search Engine Bots or Spiders
Search engine bots are also known as spiders and some spiders are good while others are bad. Some follow the Robots.txt directives while others do not. Full List of Search Engine Crawlers who do not follow Robots.txt Directives and are called Bad Bots.
As per last report from InCapsula published in 2014 only 27% of search spiders are good bots while 29% of all the website traffic is done by bad bots which might be some impersonater, or hacking tools , content scrapers and spammers.
Out of the 27% traffic which good search engine bots crawl be sure not to block any one of the bot because your traffic will get down if you block any good search engine bots.
How to Create Robots.txt ?
Robots.txt is a simple text file placed in the root directory of your WordPress installation. You can create Robots.txt by following ways
- Using a SEO Plugin like Yoast SEO
- On Your Desktop
Using a SEO Plugin : We will be using Yoast SEO to create a robots.txt file. The default WordPress robots.txt files looks like
Disallow : /wp-includes/
You can copy this and make them spider or search engine specific robots.txt. Like Googlebot-Mobile for Mobile search , Googlebot for desktop search , Googlebot-Image for Image search , Mediapartners-Google for Adsense ads etc. Just place the name of specific robots in the user agent section above.
1.Open up your Yoast SEO Options and Look for Tools and then click File Editor.
2. Enter Spider specific information in the section as pictured below. After editing Click Save Changes to Robots.txt file
Using Desktop PC : You can open a normal text file and rename it as robots.txt. Enter the robots directives and the save the file. After saving the file upload the same file via FTP or any other means to your the WordPress root directory.
Robots.txt Sample File
Here is the sample Robots.txt file which you can use on your WordPress installation.
User-agent: * # disallow all files in these directories Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /archives/ Disallow: /*?* Disallow: *?replytocom Disallow: /wp-* Disallow: /comments/feed/ User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/ #sitemap Sitemap:http://www.example.com/sitemap.xml
Be sure to change the sitemap to your current sitemap otherwise search engines will crawl and go to Example.com/sitemap.xml instead of your WordPress posts.
How to make sure that Robots.txt is not blocking any content on WordPress ?
If you make any changes in your Robots.txt file, make sure that you are not blocking any good bots. You can check this easily by using Google Webmaster Tools.Login Into Google Webmaster Tools
Select your website i.e. Click on the image of the site you added
Click Crawl on left hand side >>Click robots.txt Tester
You can check any url of your website and check whether it is blocked by for Googlebot or not. If you have blocked any url in robots.txt then you can easily update that url.
Are you using right ways of WordPress Robots.txt to optimize your site?
Let us know via comments. Don’t forget to subscribe to our e-mail newsletter to keep receiving more SEO tips.