Key takeaways
- Preserve request time, host, path, status, bytes, latency, user-agent, cache, edge, and request ID with privacy-aware retention.
- Verify important crawler identities; a user-agent string alone can be spoofed.
- Analyze by template, status, purpose, canonical state, depth, freshness, and release window—not only total requests.
- Join logs with crawls, sitemaps, inspection, performance, and change history before declaring a cause.
01
A dashboard total cannot explain which URL returned the wrong thing
Crawl requests fell after a release. Was the crawler blocked, did redirects consolidate paths, did latency rise, or did the site remove duplicates? A total-request chart cannot distinguish those stories. Logs preserve the URL-level sequence.
Collect timestamp, host, method, normalized path, status, bytes, response time, cache outcome, edge or origin, user-agent, and request ID. Document sampling and retention. Exclude personal data and sensitive query values where unnecessary.
Swipe to compare every column
| Pattern | Possible explanation | Evidence to join |
|---|---|---|
| 200 crawl falls | Discovery loss or consolidation | Links, sitemap, redirects |
| 3xx rises | Migration or redirect regression | URL map and chain crawl |
| 5xx rises | Capacity or deployment failure | Origin logs and release timeline |
| Old pages recrawled | External links or stale discovery | Referrers and old sitemap |
02
Verify bots before assigning meaning
Any client can copy a crawler user-agent. Follow the provider’s published verification method where the distinction matters. Keep verified, probable, and unverified traffic separate.
Normalize URLs without erasing meaningful parameters. Classify HTML, assets, feeds, APIs, images, and robots requests because an asset surge is not a page-discovery surge.
03
Read cohorts and sequences, not vanity totals
Group requests by template, directory, status, purpose, canonical state, indexability, depth, sitemap presence, last change, and performance band. Trace a crawler from robots and sitemap through redirects to the destination.
Google’s Crawl Stats report groups by response, file type, purpose, crawler type, host status, size, and time. Logs add URL detail but still show requests—not indexing or ranking decisions.
04
Tie anomalies to changes and test the explanation
Overlay deployments, CDN changes, robots releases, sitemap builds, migrations, outages, and publishing events. Form a falsifiable explanation, inspect representative URLs, reproduce the response, and check search-console evidence.
Monitor verified-crawler 5xx rate, latency, robots availability, redirect loops, unexpected noindex, stale high-value pages, and explosive parameter spaces. The output should be a fixable condition, not a screenshot of bot traffic.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
- Google Search Central: Crawl Stats report
- Google Search Central: Troubleshoot crawling errors
- RFC 9309: Robots Exclusion Protocol
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
