Magento 2 Robots.txt File Configuration
A robots.txt file tells search engine crawlers which URLs the crawler can access on the Magento website. It is useful for SEO optimization for specific pages and informs web crawlers to avoid indexing the disallowed pages.
We look at how to configure the robots.txt file in Magento 2.
Configure Robots.txt File In Magento 2
1. Log in to your Magento 2 Admin Panel.
2. Click on Content, and under Design, choose Configuration.
3. Edit the Global Design Configuration.
4. Expand the Search Engine Robots section.
5. Set the Default Robots option to one of the following:
-
INDEX, FOLLOW- Instruct the search engine crawlers to index the site and check for changes later. -
NOINDEX, FOLLOW- Search engine crawlers avoid indexing the site but check later for changes. -
INDEX, NOFOLLOW- Search engine crawlers index the website once but do not check for changes later. -
NOINDEX, NOFOLLOW- Search engine crawlers avoid indexing the site and do not check for changes.
6. Enter the custom instructions in the Edit Custom Instruction Of Robots.txt File option. For example, you can disallow access to all folders by entering custom instructions.
7. The Reset To Default button will remove the custom instructions and reset the Magento 2 robots.txt file to the system’s default.
8. Click on the Save Configuration button to apply the changes.
Custom Instructions For Magento 2 Robots.txt
You can add custom instructions to the robots.txt file in Magento 2. Below are some examples to consider for Magento 2 Robots.txt file.
Allow Full Access
User-agent:*
Disallow:
Disallow Access to All Folders
User-agent:*
Disallow: /
Default Instructions
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Restrict User Account & Checkout Pages
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow Catalog Search Pages
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Restrict CMS Directories
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/
Disallow URL Filter Searches
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*
Disallow Duplicate Content
Disallow: /tag/
Disallow: /review/
Summary
The Magento 2 robots.txt file is essential to communicate with web crawlers. It informs search engine bots on which pages of your Magento store should be crawled or excluded.
The Robots.txt file allows proper indexation and improves your website's search visibility. By default, Magento 2 includes settings to manage crawlers and bots for store indexing. We have covered how to set up the Robots.txt file in Magento 2.
Get more information on Magento 2 on the MGT Commerce Tutorials.
