Skip to content
GitHub Install

All tests / Crawling

Sitemap index

What this tests. A sitemap index with a plain child and a gzipped one, thirty pages between them, ten of which nothing links to.

The rule. /crawl/sitemaps/sitemap-index.xml is a sitemap index. It names two sitemaps: a plain XML one and a gzipped one served as application/gzip, which you have to decompress yourself. Between them they list 30 pages, each with a <lastmod>. This page links to 20 of them. The other 10 are linked from nowhere, so only a crawler that reads sitemaps finds them.

robots.txt names the index too, which is how a crawler starting from the top of the site would hear about it.

  1. Buying notes: Brindlecote Classic Casserole Dish
  2. Buying notes: Kestleford Travel Saucepan
  3. Buying notes: Vexmoor Classic Stock Pot
  4. Buying notes: Thistlecombe Tall Casserole Dish
  5. Buying notes: Upwold Woven Santoku
  6. Buying notes: Marrowdene Everyday Santoku
  7. Buying notes: Upwold Woven Santoku II
  8. Buying notes: Saltgrove Travel Santoku
  9. Buying notes: Quillmere Twin Santoku
  10. Buying notes: Marrowdene Large Knife Block
  11. Buying notes: Quillmere Classic Knife Block
  12. Buying notes: Pellwick Square Paring Knife
  13. Buying notes: Thistlecombe Matte Paring Knife
  14. Buying notes: Pellwick Travel Bread Knife
  15. Buying notes: Orrin Lane Travel Rice Cooker
  16. Buying notes: Orrin Lane Square Coffee Grinder
  17. Buying notes: Thistlecombe Matte Toaster
  18. Buying notes: Tarrowby Classic Toaster
  19. Buying notes: Saltgrove Travel Rice Cooker
  20. Buying notes: Thistlecombe Brushed Coffee Grinder

Data version 1.0.0. Everything here is made up: products, brands, people, towns. tests.json · robots.txt · Source