I have seen many people complaining that they worked hard for many hours on making a good site but they don’t seem to get crawled by search engines. Sometimes, the wrong pages that shouldn’t get crawled, get crawled and then you don’t get many referrals via search engines. To let your sincere effort not go waste, I have a great and easy trick with me.
When any search engine crawls your website, the first thing it searches for is robot.txt.
What is robot.txt?
It is a text file that mentions the pages that the search engine should crawl or skip. If you have your website set up on Wordpress (like mine), the search engine has the permission to crawl all your pages, from admin.php to login.php and so on.
How to set-up robots.txt?
In case you are using a website based on wordpress, just download this.
If you aren’t, just download this text file (already ready, just download) from here.
Now, just upload this text file to the main site database using any FTP client or File Manager from the cPanel.
Done, now what do I do?
Almost done, just register your website here to get crawled by Google and here to get crawled by all other seach engines.
Now, wait for a couple of days while the search engines do their jobs. Once you start getting traffic, enjoy it.
jahanzaib on July 30, 2009
your site is very good
sohail on August 1, 2009
nice article.. but robot.txt file shows sum error..
rishab on August 5, 2009
it should not give any error.
in some servers its already there.
sohail on August 6, 2009
thanq.. its working.. any more tips for ranking increasing.. plz post..
Jolene Blalock Online on October 5, 2009
its also good advice to put in the robots.txt file and dis-allow of your domain’s /cgi-bin/ directory.
admin on August 1, 2009
May I know what the error is because it is working perfectly fine here