AgentBIAgent with Business Intelligence for DTC sites

Back to the knowledge base

OPT-6

Privacy terms and the agent-crawl boundary

Published:2026-11-18Views:0

A terms page that only says 'no crawlers' blocks legitimate shopping agents too. You need to separate public product information, private account data and crawl frequency, stating what is readable. A privacy-terms and agent-crawl boundary note turns 'what can be seen' into executable rules rather than one blanket ban.

Practical steps: server-render policy text with version and effective range; state that public product pages are readable while private paths such as account and checkout are excluded via robots and noindex, and give your own definition for crawl frequency. Give key content readable text and clear structure per WCAG principles.

The boundary matters: this article issues no compliance verdict and claims neither compliance nor non-compliance from any single check. Publish only after legal review.

Official references

2026-11-22