Cluster 23: Technical SEO
What Is Robots.Txt?
A robots.txt file is a plain text file placed in the root directory of a website that tells search engine crawlers which URLs they can or cannot access on your site. It uses the Robots Exclusion Protocol (REP) to guide bot traffic, manage server load, and keep crawlers focused on indexable content.
What We’ll Cover
We’ll discuss important aspects of Robots.Txt including:
- Why A Robots.txt Matters
- How A Robots.txt Works
- Example Of A Robots.txt
- Benefits Of A Robots.txt
- Robots.txt Mistakes
- Robots.txt Related Terms
- Robots.txt FAQ
Search Intent
Funnel Stage
Significance
Why Robots.txt Matters
Mechanics
How Robots.txt Works
When a web spider reaches your site, it looks for domain.com/robots.txt before requesting any other URL. The file contains sets of directives targeting specific bots:
- User-agent: Identifies the specific search engine bot the rule applies to (e.g., Googlebot, Bingbot, or * for all bots).
- Disallow: Tells the bot which directory, file path, or URL pattern it is not allowed to crawl.
- Allow: Explicitly permits crawling of a specific subfolder or file within an otherwise disallowed directory.
- Sitemap: Points search engines directly to the location of your XML sitemap.
Application
Robots.txt Example
An e-commerce store needs to keep search crawlers away from account dashboards, shopping carts, and dynamic search filter URLs. Their robots.txt configuration looks like this:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?sort=*
Sitemap: https://www.example.com/sitemap_index.xml
Advantages
Benefits Of A Robots.txt
- Preserves Crawl Budget: Directs bots to revenue-driving pages instead of low-value parameters and admin areas.
- Reduces Server Load: Stops heavy crawling of resource-intensive scripts or large databases.
- Prevents Indexation of Unwanted File Types: Keeps internal PDFs, staging assets, and duplicate pages from being crawled.
- Guides Discovery: Instantly points all compliant crawlers to your official XML sitemaps.
Pitfalls
Robots.txt Mistakes
- Using Robots.txt to Remove Indexed Pages: Disallowing a page in robots.txt only prevents crawling. If the URL has backlinks, Google may still index it without crawling it. Use a
noindextag instead. - Blocking CSS and JavaScript Assets: Googlebot needs stylesheets and scripts to render pages properly and evaluate mobile-friendliness.
- Treating It as a Security Tool: Robots.txt is publicly visible. Listing sensitive admin URLs in plain text exposes those endpoints to anyone.
- Syntax and Casing Errors: File names and URL paths are case-sensitive. Small typos can accidentally block crucial sections of your site.
Vocabulary
Robots.txt Related Terms
Questions
Robots.txt FAQ
Robots.Txt FAQs
Where should the robots.txt file be located?
The robots.txt file must always reside in the top-level root directory of your website (e.g., https://www.example.com/robots.txt). Crawlers will not look for it in subdirectories.
Does robots.txt guarantee a page won’t be indexed?
No. If other websites link to a disallowed URL, search engines can still index the URL based on external signals, displaying it in search results without a snippet. To guarantee de-indexation, allow the page to be crawled and include a ‘noindex’ robots meta tag.
Is the robots.txt file case-sensitive?
Yes. The file name must be entirely lowercase (‘robots.txt’), and directive paths must match the exact case of the URLs on your server.
Take Action
Subscribe to our newsletter.
Subscribe to our newsletter.