AgentBIAgent with Business Intelligence for DTC sites

Back to the knowledge base

OPT-1

Keep staging and production config separate

Published:2026-11-21Views:1

Declaration files such as robots.txt, sitemap and llms.txt most often go wrong when a change is made in staging but never reaches production. Keeping per-environment copies and promoting only the reviewed version to production stops a throttling rule meant for a canary from leaking onto the live site.

Isolation does not mean several independent dialects: all copies should share field names and one template, differing only in environment-specific URLs, rates and canary flags. Review the diff on every sync rather than eyeballing lines; when a declaration is missing in production, first check whether the isolation rules kept it out of the release artifact.

This article covers environment isolation and release discipline only; it makes no compliance judgement about any crawl volume. Config edits are cheap, and precisely because they are cheap, the environment, version and approver belong in the change log.

Official references

2026-11-22