AgentBIAgent with Business Intelligence for DTC sites

Back to the knowledge base

Checklist #6 · P1

robots.txt 与 sitemap 体检

Category:抓取与收录

Criteria

是否屏蔽关键资源、sitemap 是否可达/时效、结账与账号页是否误开放

Reference

General explanation and recommendations for this checklist item; no compliance verdict based on a single measurement. Links point to official specifications.

Explanation

The first thing an external agent (search engine, shopping assistant, price crawler) does is read robots.txt and the sitemap. Blocking product pages or static assets makes products invisible; an unreachable, stale or never-updated sitemap delays discovery of new listings. Typical failures: /products/ or /collections/ in Disallow, checkout/account pages left open, sub-sitemaps drifting because pagination cursors move.

Recommendations

Official references

Measured examples

Real values from a single crawl of 30 live DTC sites, listed site by site. No aggregation, no verdict, no filling in blanks. — means the field was not retrieved in this run.

Vantage server-KR · 2026-11-09 · 30 sites (single-day cross-section)

Siterobots_statussitemap_declaredsitemap.statussitemap.nsitemap.sub_n
www.dreametech.com200https://www.dreametech.com/sitemap.xml2006217
www.songmics.com200https://www.songmics.com/sitemap.xml20071001
www.ugreen.com200https://www.ugreen.com/sitemap.xml20019964
colourpop.com200https://colourpop.com/sitemap.xml2007995
drinktrade.com200https://www.drinktrade.com/sitemap.xml2007629
fellowproducts.com200https://fellowproducts.com/sitemap.xml20071
www.allbirds.com200https://www.allbirds.com/sitemap.xml2008369
www.blackdiamondequipment.com200https://blackdiamondequipment.com/sitemap.xml200131001
www.brooklinen.com200https://www.brooklinen.com/sitemap.xml2006270
www.cotopaxi.com200https://www.cotopaxi.com/sitemap.xml20061000
www.darngoodyarn.com200https://www.darngoodyarn.com/sitemap.xml2007765
www.everlane.com200https://www.everlane.com/sitemap.xml200151000
www.intelligentsiacoffee.com200https://www.intelligentsia.com/sitemap.xml2007188
www.keenfootwear.com200https://www.keenfootwear.com/sitemap.xml20071001
www.kuiu.com200https://www.kuiu.com/sitemap.xml2007998
www.mvmt.com200https://mvmt.com/sitemap.xml2005461
www.outdoorresearch.com200https://www.outdoorresearch.com/sitemap.xml20011512
www.paragonfitwear.com200https://www.paragonfitwear.com/sitemap.xml2006723
www.taylorstitch.com200https://www.taylorstitch.com/sitemap.xml2009555
www.thirdlove.com200https://www.thirdlove.com/sitemap.xml2007373
www.deathwishcoffee.com200https://www.deathwishcoffee.com/sitemap.xml2006165
www.chubbiesshorts.com200https://www.chubbiesshorts.com/sitemap.xml2002623—
www.beardbrand.com200https://www.beardbrand.com/sitemap.xml200667
www.ombraz.com200—200648
casper.com200https://casper.com/sitemap.xml200690
kyliecosmetics.com200https://kyliecosmetics.com/sitemap.xml200109256
supergoop.com200https://supergoop.com/sitemap.xml200692
floydhome.com200https://floydhome.com/sitemap.xml2006127
counterculturecoffee.com200https://counterculturecoffee.com/sitemap.xml2006182
www.tatcha.com200https://tatcha.com/sitemap.xml200669

Tool:indexability