Key takeaways
- Use robots.txt for requested crawl access, noindex for index exclusion, and canonical for duplicate consolidation preference.
- Do not block a URL that a crawler must fetch to see its noindex directive.
- Align redirects, canonicals, sitemaps, internal links, and hreflang with one declared preferred URL.
- Test the delivered response and rendered head for every template state, parameter rule, and environment.
01
A URL cannot follow a strategy the team never wrote down
The CMS adds a canonical to the clean page. A staging rule leaves noindex on the template. Robots.txt blocks the parameter pattern. The sitemap still lists the duplicate. Each setting was reasonable alone; together they make the intended outcome hard to infer.
For each URL pattern, name whether it should be accessible, crawlable, indexable, canonical, discoverable through links, listed in a sitemap, and available to users. Record the reason and owner.
Swipe to compare every column
| Mechanism | Primary job | Common misuse |
|---|---|---|
| robots.txt | Request crawlers not to access paths | Trying to guarantee deindexing or privacy |
| noindex | Request exclusion from the index | Adding it to a blocked URL |
| canonical | Indicate a preferred duplicate URL | Using it as a redirect or removal tool |
| redirect | Send a user-agent to another URL | Mapping unrelated pages to the home page |
02
Let the crawler reach the directive it must read
RFC 9309 standardizes robots.txt as rules crawlers are requested to honor and says those rules are not access authorization. A disallowed URL may still be known from links. If the goal is index exclusion, the crawler generally needs to fetch the page and see noindex.
Protect confidential material with authentication or access control. For public duplicates, prefer a consistent canonical strategy and avoid using noindex merely to force a different canonical selection.
03
Stack signals in the same direction
Google describes redirects and rel=canonical as strong canonicalization signals and sitemap inclusion as weaker. Link to the preferred URL, include it in the sitemap, self-canonicalize it, and redirect when a duplicate should disappear as a destination.
Do not declare one canonical in HTML and another in an HTTP header or rendered script. A canonical is a signal rather than an order; contradictions reduce clarity.
04
Turn indexing configuration into a release test
Capture status, redirect target, robots access, meta and header directives, canonical count and target, sitemap membership, internal references, hreflang, and rendered changes. Fail a release on mutually exclusive states.
After launch, compare declared and selected canonicals, exclusions, and crawler logs by template. The durable fix usually belongs in the CMS or routing policy, not a one-off tag edit.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
- RFC 9309: Robots Exclusion Protocol
- Google Search Central: Canonicalization methods
- Google Search Central: Technical SEO guidance
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
