7 min read
Also: robots.txt file, Robots Exclusion Protocol, REP, RFC 9309, robots exclusion standard, robots.txt disallow, robots.txt example
robots.txt
Quick answer
robots.txt is a plain-text file at a site's root (/robots.txt) that tells crawlers which paths they may fetch, using User-agent, Disallow and Allow lines defined in RFC 9309. User-agent: * followed by Disallow: / blocks every compliant crawler; it controls crawling, not indexing, and can also point to your sitemap.xml.
robots.txt is the plain-text file at a site's root that tells crawlers which paths they may fetch. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022, and is voluntary: well-behaved bots obey it, others ignore it.
Last verified
What people do with robots.txt
- Add your sitemap to robots.txt
The Sitemap: line, sitemap formats and the 50,000-URL limit.
- See how web crawlers use robots.txt
Crawling vs scraping and crawler etiquette.
- Find the user-agent token to block or allow
How bots identify themselves and how sites check it.
- Crawl politely and handle 429 Too Many Requests
What to do when a site rate-limits your crawler.
- Respect robots.txt in a Python scraper
Requests, BeautifulSoup, Scrapy and Playwright basics.
- Collect web data for AI and RAG responsibly
Where robots.txt and AI crawler tokens fit in.
Key facts
| Standard | RFC 9309, Robots Exclusion Protocol (IETF, September 2022) |
|---|---|
| Originally proposed | 1994, by Martijn Koster |
| Location | Root of each host, e.g. https://example.com/robots.txt |
| Size Google parses | First 500 KiB; content after that is ignored |
| Crawl-delay | Not in RFC 9309; ignored by Google, honoured by Bing |
| Enforcement | Voluntary: a request to crawlers, not access control |
What is robots.txt?
robots.txt is a plain-text file at the root of a website (always https://example.com/robots.txt) that tells automated crawlers which paths they may and may not request. It is the public face of the Robots Exclusion Protocol, an informal convention proposed by Martijn Koster in 1994 that every major search engine followed for almost three decades before it was written down as an official standard.
That standard is RFC 9309, published by the IETF in September 2022 after Google pushed to formalise the protocol in 2019. Nobody "owns" robots.txt: the RFC defines the core syntax, and each crawler operator (Google, Bing, OpenAI, Anthropic and so on) documents how its own bots read the file.
How robots.txt works
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming the crawler it applies to, followed by Allow and Disallow rules that match URL paths:
```
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
```
The key rules from RFC 9309 and Google's documentation:
User-agent matches it most specifically, and ignores the * group if it finds its own. Rules from different groups are not merged.Allow and a Disallow both match a path, the more specific (longer) rule applies; on a tie, Google uses the less restrictive one.* matches any sequence of characters and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks PDF files.What robots.txt does not do
robots.txt is a request, not an access control. It is honoured by well-behaved crawlers and ignored by everything else, and the file is public, so listing a secret path in it advertises that path. Three misunderstandings cause most problems:
noindex meta tag or header instead.noindex, nofollow and crawl-delay lines in robots.txt.robots.txt and AI crawlers
Since 2023, robots.txt has become the main way site owners opt out of AI training and AI search. Each AI company publishes its own user-agent tokens, and they are separate, so blocking one does not block the others:
GPTBot (model training) and OAI-SearchBot (ChatGPT search) are independent. ChatGPT-User covers fetches a user asks for, which OpenAI says robots.txt rules may not apply to.ClaudeBot is the training crawler, with separate tokens for search and user-initiated fetches.Google-Extended is a robots.txt-only token with no crawler of its own. Disallowing it opts content out of use in Gemini model training without affecting Google Search crawling or ranking.Check each vendor's crawler page for the current token list before relying on one, as they have changed several times.
Common problems
https://www. and https:// without www are separate hosts), and paths are case-sensitive.Disallow all, allow all and testing
The two most copied robots.txt files are opposites. To block every compliant crawler from the whole site:
```
User-agent: *
Disallow: /
```
To allow everything, leave the Disallow value empty (or serve no robots.txt at all, which crawlers treat as no restrictions):
```
User-agent: *
Disallow:
```
To test a file, Google Search Console's robots.txt report (under Settings) shows which robots.txt files Google found for a verified property, when it last fetched them and any parse errors; it replaced the older robots.txt Tester, which Google retired in December 2023. Google has also open-sourced the C++ parser Googlebot uses, so you can check a URL against a file locally. For sites you do not own, third-party testers fetch the live file and evaluate a URL and user agent against it.
robots.txt in web automation and scraping
If you build a web crawler or a scraper, robots.txt is the first file you should fetch. It tells you which sections the owner considers off-limits to automated access, and its Sitemap lines often give you a complete list of the URLs you actually want. Respecting it, together with sensible rate limiting, is the baseline of responsible data scraping, and it is one of the factors courts and site owners look at when the legality of scraping is disputed (see is web scraping legal?).
Note the difference between a crawler and a user-directed tool. robots.txt was written for automated bots that discover and fetch pages on their own. An AI agent opening a specific page because a person asked it to is closer to a browser, which is why some vendors treat user-initiated fetches separately. Either way, a site's terms of service still apply.
In Autonoly, the AI agent browses pages you point it at in a real Chromium browser, and you can ask it to read a site's robots.txt and sitemap first and stay out of disallowed paths. Autonoly does not use proxies or rotate IP addresses, so it is not a tool for getting around a site that does not want to be crawled.
Why It Matters
robots.txt is the one machine-readable statement of what a site owner allows bots to fetch. Getting it wrong can deindex a whole site or leave AI crawlers training on content you meant to protect, and anyone building a crawler or scraper needs to read it before sending a single request.
How Autonoly Solves It
Autonoly's AI agent can fetch and read a site's robots.txt and sitemap before it starts, and keep away from disallowed paths when you ask it to. It browses at human pace in a single cloud Chromium session, with no proxy rotation, so it suits respectful, targeted collection rather than mass crawling.
Learn moreExamples
Adding 'User-agent: GPTBot' and 'Disallow: /' to opt a site's content out of OpenAI model training while leaving Googlebot unaffected
A scraper reading robots.txt first, finding the 'Sitemap:' line, and using that sitemap as its list of product URLs instead of crawling every link
Diagnosing a sudden traffic drop to a 'Disallow: /' line copied from a staging server into production
Frequently Asked Questions
What is robots.txt used for?
It tells automated crawlers which parts of a site they may request. Site owners use it to keep search engines out of admin areas, internal search results and duplicate URL variations, to point crawlers at their sitemap, and increasingly to opt out of AI training crawlers such as GPTBot, ClaudeBot and Google-Extended.
Does robots.txt stop a page from appearing in Google?
No. Disallow stops Google crawling a URL, but the URL can still be indexed without a description if other pages link to it. To keep a page out of search results, let Google crawl it and add a noindex meta tag or X-Robots-Tag header. Google cannot see a noindex on a page it is not allowed to fetch.
Does Google support Crawl-delay in robots.txt?
No. Crawl-delay is not part of RFC 9309 and Googlebot ignores it. Google adjusts its crawl rate automatically based on how your server responds; returning 429 or 503 slows it down temporarily. Bing and some other crawlers do honour Crawl-delay.
How do I block AI crawlers with robots.txt?
Add a group for each crawler's token with 'Disallow: /'. Common tokens are GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Gemini training, and CCBot for Common Crawl. They are independent, so list each one, and check the vendors' crawler pages because tokens change.
Is it illegal to ignore robots.txt?
robots.txt is not a law and is not technically enforced. Ignoring it can still matter: courts and site owners consider it alongside terms of service, the type of data and the load you place on the server. Respecting it is the baseline for responsible scraping.
Where does robots.txt have to be?
At the root of each host and protocol, named exactly robots.txt in lowercase. A file at https://example.com/robots.txt does not cover https://shop.example.com or http://example.com. Files in subdirectories are ignored.
Further reading
Related terms, automations and guides
Where this concept shows up in practice.
Stop reading about automation.
Start automating.
Describe what you need in plain English. Autonoly's AI agent builds and runs the automation for you — no code required.