AgentBIAgent with Business Intelligence for DTC sites

Back to the knowledge base

OPS-1

Bucket crawl errors by type

Published:2026-11-18Views:0

404, 5xx, timeouts and robots blocks are four different problems; lumped into one 'crawl failures' number, on-call can't tell whom to call. Bucketing crawl errors by type is the first step from a fuzzy number to actionable queues.

Bucket by status code and cause: 5xx to the server side, 404 to accidental deletion or moved URLs, timeouts to performance and rate limiting, robots blocks to an intentional-check first. Record counts and trends per bucket, attaching context and recent changes to anomalous ones. If numbers don't move, record them as-is — no extra spend, no hunt for growth hacks.

This is a template for a recurring action, not a compliance verdict on any single day's measurement.

Official references

2026-11-22