Managing robots.txt for an Odoo website

How to set up a robots.txt file for an Odoo website and allow or disallow crawlers from indexing

What the file "robots.txt" does on a website

As explained in the Google developer documentation Introduction to robots.txt: "A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, you should use "noindex" directives, or password-protect your page".

Odoo versions prior to 14.0 do not have internal tools for editing and managing the robots.txt file, and if you need to change it, it can be tricky without developing. For these reasons, we have developed a module that gives the ability to handle the content of the "robots.txt" file in a simple way.

How the module works

Watch our video tutorial and learn how to work with the "robots.txt" file in Odoo using our module.

Module configuration

To configure the file go to the Website - Configuration - Websites menu and choose a website.

Note that you need to activate the developer mode to see this menu item.

In the section robots.txt select the file content mode.

You can use the one of the module predefined templates for the "robots.txt" content:

  • Allow All to allow search engines (bots) to crawl all your pages, media, and resource files.

  • Custom to use our template that allow to crawl images, style and script files and block the portal user personal page and internal content.

  • Block All to block access for all crawlers in the website if you don't want them to crawl.

You are able to modify the content sample from any template for your purposes and your SEO specialist's advice.

An additioanal information how to create the "robots.txt" file in this documentation from Google.

The Sitemap URL mode allows adding the sitemap URL automatically to the end of the file (mode: Auto) or you can add it manually where you prefer (mode: In Text).

in SEO
Managing robots.txt for an Odoo website
Yurii Razumovskyi

Entrepreneur, Odoo consultant, architect, developer, Ukrainian Odoo association member, OCA Member (2021), Odoo Learning Partner.

Managing robots.txt for an Odoo website
Yurii Razumovskyi 30 March, 2021
Share this post
Is SEO site support a reasonable must or unnecessary expense?