Checklist #11 · P1
抓取频次异常与站点健康告警
Category:抓取与收录
Criteria
爬虫报错激增、soft-404、重定向链、服务器响应变慢
Reference
General explanation and recommendations for this checklist item; no compliance verdict based on a single measurement. Links point to official specifications.
Explanation
Rising crawl errors, soft-404s, longer redirect chains and slower responses usually show up before traffic drops. An agent that hits a 5xx or timeout abandons the visit and does not retry the way a browser does. Typical failures: a redesign creates long 302 chains, discontinued products return a 200 empty page, or peak traffic lets crawlers slow the origin.
Recommendations
- Track the daily change in 5xx rate and TTFB from one vantage; never compare absolute values across vantages.
- Return 410/404 or 301 to a relevant category for discontinued products instead of a 200 empty shell.
- Keep redirect chains to a single hop and remove loops and cross-domain hops.
- Give crawlers their own cache and rate limits so bulk crawling cannot slow the real checkout path.
Official references
Tool:indexability