Key takeaways
- Treat llms.txt as a voluntary proposal for helping tools navigate useful material—not a universal search or citation standard.
- Point to authoritative, public, canonical sources and keep markdown versions faithful to their HTML counterparts.
- Do not expose private, draft, disallowed, or contradictory material through the file.
- Measure fetches and downstream use separately; the file does not prove ranking, citation, or model training.
01
A map can help an agent without changing the terrain
A documentation site contains hundreds of pages wrapped in navigation and scripts. A concise file can point an agent toward stable overviews, concepts, references, and policies. It does not repair weak evidence, grant crawl permission, override a canonical, or guarantee use by a search or answer system.
The llms.txt site describes the format as a proposal to help agents use a website. Publish it when there is a real collection worth navigating and somebody will own the file as the site changes.
Swipe to compare every column
| Question | Defensible answer | Unsupported leap |
|---|---|---|
| What is it? | A proposed markdown navigation file | A universal web standard |
| What can it improve? | Discovery by supporting clients | Guaranteed indexing or citation |
| What should it list? | Stable authoritative public sources | Every URL on the domain |
| What proves value? | Observed fetch and attributable use | File existence |
02
Curate the file around decisions and authority
Organize links by the jobs an agent needs to complete. Prefer durable overview, concept, reference, pricing, policy, and support pages. Exclude thin variants, tracking URLs, expired campaigns, and pages whose authority is unclear.
If you offer markdown alternates, keep facts, examples, source notes, dates, and canonical identity aligned with the human page. A clean alternate that drops qualifications is easier to process and easier to misuse.
03
Keep access and indexing controls in their own systems
Robots.txt is an access-request protocol, and RFC 9309 says it is not authorization. Authentication protects private material. Meta or header directives address index preferences. llms.txt replaces none of them.
Do not create an attractive directory of sensitive URLs. When a listed page is removed, moved, restricted, or superseded, update the map and underlying URL behavior together.
04
Build evidence without inventing attribution
Log requests for the file and markdown alternates with verified user-agent and network context where practical. Track successful responses, freshness, broken links, and downstream referrals that actually expose a source.
Do not label every AI referral or citation an llms.txt win. Systems also discover pages through ordinary crawling, indexes, links, and feeds. Report controlled tests and their limits.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
