Rule n° 212 - The website’s root contains instructions for web crawlers.

Goal

• Enable targeted referencing.
• Improve guidance for search tools.
• Reduce the energy impact related to the consultation of the site.
• Improve the way content is taken into account by search engines and indexing tools.

Implementation

To define non-indexable directories, files and file types, use the user-agent and disallow instructions in a single text file called robots.txt, located in the root directory of the website.

Alternatively, at a specific page level, use the tag meta name="robots" content="attribut1, attribut2" :

• attribut1 can take the values index (index this page) or noindex (do not index this page);
• attribut2 can take the valuesfollow (follow the links in this page) or nofollow (do not follow the links in this page).

Find out more:

Control

From the URL of your website:

• First, access the address of the robots.txt file, at the root of the website, by typing, for example, http://example.com/robots.txt in the browser's address bar;
• Check that the robots.txt file is in the root directory of the site;
• Check the validity of the syntax of the robots.txt file using the indications given by the search engines

If there is no robots.txt file, check that meta name="robots" content=" attribute1, attribute2 " tag is present and valid in each page.

By Opquast - Read the license