Website robots.txt
Description
Control what you share with Google and other search engines. Allow or block them from retrieving web pages on your site. Customize your robots.txt file for the best website SEO optimization.
Module Features
Contains the predefined templates of a robots.txt file
Allows to manage own "robots.txt" templates
Leaves only one link for the "/robots.txt" (multilang URLs return 404)
This SEO solution was developed in cooperation with a professional marketing agency
Configuration
To configure, follow these steps:
- Go to the Website - Configuration - Settings menu.
- In the section SEO - Robots.txt click on the Edit robots.txt link.
- Select one of the predefined templates or enter your own text.
There are three predefined templates: Allow All, Block All, and Custom.
You can add or edit templates, just click on the Templates link:
- Choose how to operate with the Sitemap URL, let the system add it automatically to the end of the file (default) or add it manually to the text where you prefer.
- Make changes in the "robots.txt" content, if it is necessary.
- Save the changes.
- To get help about creating the "robots.txt" file click on the following link:
