How to use the robots.txt file to improve the way search bots crawl your website?

The purpose of the robots.txt file is to tell the search bots which files should and which should not be indexed by them. Most often it is used to specify the files which should not be indexed by search engines.

In order to allow search bots to crawl and index the entire content of your website, you can add the following lines in your robots.txt file:

User-agent: *
Disallow:

On the other hand, if you wish to disallow your website from being indexed entirely, you can use the lines below:

User-agent: *
Disallow: /

For more advanced results you will need to understand the sections in the robots.txt file. The "User-agent:" line specifies for which bots the settings should be valid. You can use "*" as a value to create the rule for all search bots or the name of the bot you wish to make specific rules for.

The "Disallow:" part defines the files and folders that should not be indexed by search engines. Each folder or file must be defined on a new line. For example, the lines below will tell all the search bots not to index the "private" and "security" folders in your public_html folder:

User-agent: *
Disallow: /private
Disallow: /security

Please note that the "Disallow:" statement uses your website root folder as a base directory, therefore the path to your files should be /sample.txt and not  /home/user/public_html/sample.txt for example.

  • 0 Users Found This Useful
Was this answer helpful?

Related Articles

How to decrease the crawl rate of Bing’s search engine bot

The MSN bot or BING bot crawls your website to index your website’s content and show the results...

How to remove a website from Google

There are several options to prevent your web site from being indexed by Google. For best results...

What should I do considering search engine ranking if I need to change my primary domain name?

Changing the primary domain name for your website and preserving the search engine ranking can be...

What should I do if I change my page titles in order to ensure search engines will index the new pages?

There is only one change that needs to take place if you decide to change the page URLs for your...

If I switch hosts, will it affect my search engine ranking?

Switching web hosts will not affect your web site ranking in any way. There is only one thing...