AgentBIAgent with Business Intelligence for DTC sites

Back to the knowledge base

OPT-1

Crawl budget and per-UA rate rules

Published:2026-11-20Views:0

Crawl budget is not something to minimise at all costs: a blanket Crawl-delay for every user-agent also slows legitimate shopping and comparison agents, so products get discovered later. A steadier approach is to group requests by user-agent, see which paths are expensive, and throttle only those paths while leaving product pages reachable.

Keep the rules explainable in robots.txt: which paths each user-agent group may take, why a limit exists, and when it will be reviewed all belong in a change log. After each edit, re-scan the home page and a few key product pages with JavaScript disabled to confirm products are still in the crawl result.

This article covers crawl reachability and rate configuration only; it makes no compliance judgement about any measured traffic figure. Rate changes are cheap config edits, and precisely because they are easy to change, each one should be dated and traceable.

Official references

2026-11-22