Shopping and search agents usually read robots.txt and the sitemap first. These two files render no pages and cost little bandwidth, yet they decide whether products can be discovered. A wrong rule raises no error — it silently removes products from crawled results, often noticed only after traffic falls.
Check rules per User-agent block: GPTBot, PerplexityBot, Google-Extended and others each need a look. Block only private paths such as checkout, account and internal search. Listing /products/ or /collections/ effectively hides products; declare the sitemap with an absolute URL and re-verify after changes.
robots.txt and the sitemap are the config layer — the cheapest to change, and for that same reason often edited by many hands. Log each change with its date in the definition changelog, and re-crawl the homepage and a few key product pages with JavaScript disabled. This article addresses crawlability only, not a compliance verdict on any measurement.