AI Crawler Controls: Decide What You Mean Before Editing robots.txt

“Block AI” is not a complete instruction. Does the publisher want the page excluded from public search, absent from generated summaries, unavailable for potential model training, or protected from a particular automated fetcher? Those goals can require different signals and have different discovery consequences.

Create a small policy matrix by content type: public marketing pages, licensed editorial work, customer portals, documentation, user-generated content, and sensitive material. Name the owner who can approve a change. A site-wide rule copied from social media can quietly remove a channel the business intended to keep.

OpenAI’s publisher guidance says public sites can appear in ChatGPT search and recommends allowing OAI-SearchBot when a publisher wants content discovered, summarized, cited, and linked. It separately identifies GPTBot as the user agent publishers can disallow for pages they want excluded from potential training.

The same guidance notes that a disallowed page may still have its link and title surfaced when OpenAI learns the URL through another source and has relevance signals. Publishers that do not want this should consider noindex—but a crawler must be allowed to read the meta tag. This is why access and indexing controls need to be designed together rather than layered blindly.

Google’s generative features in Search rely on the Search index, so ordinary crawlability, indexing, and snippet eligibility remain foundational. Google-Extended is a separate robots token for specified use in future Gemini training and grounding in Gemini Apps and Vertex AI. Google explicitly says it has no effect on Google Search inclusion and is not a Search ranking signal.

That distinction matters. Blocking Googlebot can remove a page from Search; changing Google-Extended addresses a different use. Document the reason beside the policy in version control so a later maintainer does not “clean up” a deliberate difference between the two.

Check the public robots.txt response, redirects, status codes, CDN and firewall rules, page-level robots meta tags, and server logs. A robots rule is voluntary crawler guidance; access controls and authentication remain necessary for private information. Never publish confidential material and rely on robots.txt to make it private.

Re-test after platform, proxy, CMS, or domain changes. Keep a dated decision record containing the content scope, intended use allowed or denied, exact rule, validation method, and owner. The goal is not the most restrictive file. It is a policy the business understands and the implementation actually enforces.