robots.txt

advancedwidespread

A robots.txt is a simple text file located at the root of a website that tells search bots and AI bots which pages they may visit and which they may not.

In plain terms

Imagine a website as a large building with a small sign at the entrance: "Please do not enter the warehouse or offices." A robots.txt file is such a sign. It is not for humans, but for programs that automatically crawl websites, the robots.

The file is written by the website owner. It can specify which parts a Google search robot, an AI company's data collection robot, or any other program may read. This keeps robots away from pages such as the cart page, the internal search results list, or the admin interface.

The robots.txt is a request, not a lock. Respectful robots comply, but technically nothing prevents a program from ignoring it. That is why you should not protect secret or personal data with it, but with a password.

In the AI era, the file has taken on a new role. Many publishers and companies use it to indicate that they do not want their content collected to train language models, or that an AI agent should not open their pages on behalf of the user.

How it works

The owner places a text file named robots.txt in the root of the website, so it is always accessible at the pelda.hu/robots.txt address for anyone. The file only applies to the domain where it is found, so a subdomain (for example, blog.pelda.hu) needs its own file.
Before crawling a site, the bot first requests this file and looks for rules pertaining to its own name (the 'User-agent' line). If it finds none, it follows the rules marked with an asterisk, which apply to everyone.
The rules are simple: the 'Disallow' line forbids a directory or page, while the 'Allow' line permits an exception. The sitemap URL often appears in the file as well, helping the robot find important pages.
A rule-following bot skips prohibited URLs and downloads and processes the rest: a search engine adds them to its index, while an AI crawler uses them as training material or to answer questions.
No one enforces compliance automatically. If a bot does not follow the rules, the owner can only block it by other means, such as a firewall or a bot-filtering service.

An everyday example

A Hungarian news portal does not want its articles used to train AI models. Therefore, it explicitly blocks well-known AI data-collecting bots in its robots.txt file, while still allowing Google's search crawler. All you notice from this is that the articles appear in search results, but some chatbots may respond that they cannot open the given site. At Google, a separate signal (Google-Extended) can be used to request that content not be used to train Gemini models. However, this signal does not exclude articles from Google Search's AI summaries.

Who is affected

marketerA single poorly written line can remove an entire website from search engines and AI answers, which is why it is one of the basic settings for visibility. → Marketing

entrepreneur, executiveIt is a business decision whether to allow your company's content into AI robots: the ban protects the content but may reduce your visibility in AI search engines.

developerYou must maintain the file, and if you write a robot or agent, you must ensure that it respects others' rules. → Code

lawyer, citizenIn the EU, a rights holder can prohibit the use of their content for text and data mining in a machine-readable way. This prohibition does not extend to mining for scientific research purposes. The robots.txt file is one common form of this, which is why it increasingly comes up in copyright disputes. The code of practice drawn up for the EU AI Act also expects it to be respected. → Law

A common misconception

MythWhat I block in robots.txt is safe and cannot appear in search results.

In realityThe file is only a request that robots voluntarily follow, and it is public, meaning anyone can read what you want to hide. The URL of the blocked page may still appear in search results if linked from elsewhere.

Limits and risks

It is not a security tool: protect confidential content with passwords or access restrictions, not with robots.txt.
Compliance is voluntary, and multiple reports indicate that some AI robots ignore or bypass the prohibitions.
Several providers do not necessarily apply robots.txt rules when opening a page at the user's explicit request (for example, when you paste a link into a chatbot).
A single typo, such as a slash placed in the wrong spot, can exclude the entire page from search engines, so every modification is worth checking.
It only affects future crawling: what a robot has already collected is not deleted later from the search engine or from an AI model's training material by the file.

Similar, but not the same

noindex tag – robots.txt blocks crawling. noindex, however, requests that the crawled page not appear in search results. If a page is blocked by robots.txt, the bot does not see the noindex tag on it. Therefore, if you want to remove a page from search results, do not block it in robots.txt.
llms.txt – llms.txt does not block but guides: it offers a concise summary of the page for AI models. For now, it is only a proposal, not a standard.
sitemap – The sitemap lists what should be crawled, while robots.txt specifies what the owner requests to be excluded.

Related concepts

Broader

search engine optimization, web crawling, technikai SEO

Narrower

user-agent, ban on AI robots

On the edge

llms.txt, Search bot, sitemap, data scraping, generative search engine optimization

A brief history

The robots exclusion standard was proposed by Dutch developer Martijn Koster in 1994 after early web robots overloaded servers. For decades, it functioned as an unofficial but widely followed agreement. Google initiated its standardization in 2019, and it became an IETF standard (RFC 9309) in September 2022.

Where things stand

By the end of 2026, most serious websites have a robots.txt file, and major search engines reliably follow it. The debate is around AI robots: many publishers ban them, while compliance remains voluntary. The IETF AI Preferences (AIPREF) working group is working on extensions that would allow separate signaling of whether content can be used for AI training.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

Google now reports AI search impressions in Search ConsoleAug 4, 2026 · Marketing

Also known as

robots.txt · robots.txt file · robot exclusion standard · robots exclusion protocol · Robots Exclusion Protocol · Robots Exclusion Standard · REP · robots.txt file

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.