Checklist #6 · P1
robots.txt 与 sitemap 体检
Category:抓取与收录
Criteria
是否屏蔽关键资源、sitemap 是否可达/时效、结账与账号页是否误开放
Reference
General explanation and recommendations for this checklist item; no compliance verdict based on a single measurement. Links point to official specifications.
Explanation
The first thing an external agent (search engine, shopping assistant, price crawler) does is read robots.txt and the sitemap. Blocking product pages or static assets makes products invisible; an unreachable, stale or never-updated sitemap delays discovery of new listings. Typical failures: /products/ or /collections/ in Disallow, checkout/account pages left open, sub-sitemaps drifting because pagination cursors move.
Recommendations
- Review Disallow rules per agent and block only genuinely private paths; product and collection pages must stay crawlable.
- Declare the absolute Sitemap URL in robots.txt and shard large catalogs with a sitemap index.
- Make lastmod reflect the real modification time, not the generation date.
- After every template or route change, re-crawl the homepage and key product pages with JavaScript disabled.
Official references
Measured examples
Real values from a single crawl of 30 live DTC sites, listed site by site. No aggregation, no verdict, no filling in blanks. — means the field was not retrieved in this run.
Vantage server-KR · 2026-11-09 · 30 sites (single-day cross-section)
| Site | robots_status | sitemap_declared | sitemap.status | sitemap.n | sitemap.sub_n |
|---|---|---|---|---|---|
| www.dreametech.com | 200 | https://www.dreametech.com/sitemap.xml | 200 | 6 | 217 |
| www.songmics.com | 200 | https://www.songmics.com/sitemap.xml | 200 | 7 | 1001 |
| www.ugreen.com | 200 | https://www.ugreen.com/sitemap.xml | 200 | 199 | 64 |
| colourpop.com | 200 | https://colourpop.com/sitemap.xml | 200 | 7 | 995 |
| drinktrade.com | 200 | https://www.drinktrade.com/sitemap.xml | 200 | 7 | 629 |
| fellowproducts.com | 200 | https://fellowproducts.com/sitemap.xml | 200 | 7 | 1 |
| www.allbirds.com | 200 | https://www.allbirds.com/sitemap.xml | 200 | 8 | 369 |
| www.blackdiamondequipment.com | 200 | https://blackdiamondequipment.com/sitemap.xml | 200 | 13 | 1001 |
| www.brooklinen.com | 200 | https://www.brooklinen.com/sitemap.xml | 200 | 6 | 270 |
| www.cotopaxi.com | 200 | https://www.cotopaxi.com/sitemap.xml | 200 | 6 | 1000 |
| www.darngoodyarn.com | 200 | https://www.darngoodyarn.com/sitemap.xml | 200 | 7 | 765 |
| www.everlane.com | 200 | https://www.everlane.com/sitemap.xml | 200 | 15 | 1000 |
| www.intelligentsiacoffee.com | 200 | https://www.intelligentsia.com/sitemap.xml | 200 | 7 | 188 |
| www.keenfootwear.com | 200 | https://www.keenfootwear.com/sitemap.xml | 200 | 7 | 1001 |
| www.kuiu.com | 200 | https://www.kuiu.com/sitemap.xml | 200 | 7 | 998 |
| www.mvmt.com | 200 | https://mvmt.com/sitemap.xml | 200 | 5 | 461 |
| www.outdoorresearch.com | 200 | https://www.outdoorresearch.com/sitemap.xml | 200 | 11 | 512 |
| www.paragonfitwear.com | 200 | https://www.paragonfitwear.com/sitemap.xml | 200 | 6 | 723 |
| www.taylorstitch.com | 200 | https://www.taylorstitch.com/sitemap.xml | 200 | 9 | 555 |
| www.thirdlove.com | 200 | https://www.thirdlove.com/sitemap.xml | 200 | 7 | 373 |
| www.deathwishcoffee.com | 200 | https://www.deathwishcoffee.com/sitemap.xml | 200 | 6 | 165 |
| www.chubbiesshorts.com | 200 | https://www.chubbiesshorts.com/sitemap.xml | 200 | 2623 | — |
| www.beardbrand.com | 200 | https://www.beardbrand.com/sitemap.xml | 200 | 6 | 67 |
| www.ombraz.com | 200 | — | 200 | 6 | 48 |
| casper.com | 200 | https://casper.com/sitemap.xml | 200 | 6 | 90 |
| kyliecosmetics.com | 200 | https://kyliecosmetics.com/sitemap.xml | 200 | 109 | 256 |
| supergoop.com | 200 | https://supergoop.com/sitemap.xml | 200 | 6 | 92 |
| floydhome.com | 200 | https://floydhome.com/sitemap.xml | 200 | 6 | 127 |
| counterculturecoffee.com | 200 | https://counterculturecoffee.com/sitemap.xml | 200 | 6 | 182 |
| www.tatcha.com | 200 | https://tatcha.com/sitemap.xml | 200 | 6 | 69 |
Tool:indexability