Key takeaways
- OpenAI documents OAI-SearchBot for search discovery and GPTBot separately for potential model training controls.
- A robots.txt allowance does not prove that a CDN, WAF, authentication rule, or JavaScript challenge permits access.
- Citation observations and referral visits answer different questions and should be reported separately.
- Optimize the source and landing experience; do not promise a stable “ChatGPT ranking.”
01
Write the policy before the robots rule
Decide which public pages the organization wants discoverable in ChatGPT search, which material should be excluded from potential training, and which pages should not appear in search at all. OpenAI’s publisher guidance distinguishes OAI-SearchBot from GPTBot, so one blanket rule can express the wrong intent.
Record the decision owner and review date. Marketing, security, legal, and engineering may each see a different risk; the final configuration should be a documented publishing policy, not a copied robots.txt snippet whose consequences nobody owns.
02
Test the whole access path
Check robots.txt, meta robots, canonical URLs, status codes, redirects, authentication, CDN rules, WAF policies, bot challenges, geography restrictions, and the rendered source. A crawler allowed by robots.txt can still receive a 403 page or an empty JavaScript shell.
OpenAI notes that a disallowed page may still have its title and link surfaced when the URL is learned elsewhere; noindex is the stronger instruction when the page should not appear. The crawler must be allowed to fetch a page before it can read that meta instruction, so test the combination you actually deployed.
Swipe to compare every column
| Layer | Check | Evidence |
|---|---|---|
| Policy | Search discovery and training intent are separate | Approved crawler matrix |
| Access | Robots, status, CDN, WAF, and challenges agree | Timestamped request tests |
| Page | Canonical source contains the important visible text | Rendered HTML review |
| Observation | Citation and referral are recorded independently | Prompt sample plus analytics |
03
Make the source easy to understand without writing for a bot
Give the page a clear purpose, descriptive title, visible answer, stable facts, and links to the evidence behind material claims. Define the organization, product, location, date, and scope when ambiguity would change the answer. Structured data should agree with what a visitor can see; it is not a hidden substitute for the page.
Keep the useful detail after the click. A thin page may earn a mention and still disappoint the person who wants the method, comparison, price logic, or next step. Search visibility becomes valuable only when the destination helps someone make progress.
04
Report discovery, traffic, and outcomes on separate lines
Observe a documented set of prompts over time and retain the platform, market, date, citation, and answer context. OpenAI says ChatGPT referral links include a utm_source parameter that analytics tools can use. Validate the live parameter before building a permanent report, because product behavior can change.
Join tagged visits to landing-page engagement, useful actions, qualified conversations, and pipeline only where consent and data quality allow it. Do not infer zero visibility from zero referrals: an answer can cite without earning a click. Do not infer commercial value from a citation screenshot either. The honest report shows both.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
- OpenAI: Publishers and Developers FAQ
- OpenAI: Overview of OpenAI crawlers
- Google Search Central: robots meta tag and X-Robots-Tag
- RFC 9309: Robots Exclusion Protocol
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
