Point your scraper here and check its homework.
Every test on this site has an answer key: a JSON file that lists exactly what a correct scraper should come back with. Run Netweir, Scrapy, Playwright, Crawlee, Firecrawl or something you wrote last night, then diff your output against the key. No guessing whether "about 500 products" was right.
The data is fixed. It comes from a seeded generator, so the catalogue you scrape today is the one you'll scrape next month, until the data version changes. You're on 1.0.0; it's in every answer key and at the foot of every page.
Bots are welcome. There are no challenges, no CAPTCHAs, no fingerprinting. The one thing standing between you and the site is a generous abuse limit at the edge, set well above anything a test needs. Keep to the published rules and you'll never meet it.tests.json lists every test for your harness.
Shop & pagination
Plain HTML, nothing clever, and still the place most scrapers lose records: pages they never reach, products they count twice.
| Test | What it tests | Difficulty | Needs | Answer key |
|---|---|---|---|---|
| The catalogue | Categories, subcategories, numbered pages and next links, full product pages with JSON-LD, and the traps real shops set: tracking parameters, missing trailing slashes, products listed twice, items out of stock. | Medium | HTTP only | JSON |
JavaScript pages
The data isn't in the HTML. Fetch these pages without a browser and you get an empty shell, which is the honest result.
| Test | What it tests | Difficulty | Needs | Answer key |
|---|---|---|---|---|
| Infinite scroll | Ninety-six items arrive twelve at a time as you scroll. Stop scrolling early and you miss some. | Medium | Needs JS | JSON |
| Load more button | Sixty items behind a button that has to be clicked five times, waiting each time for the new rows. | Medium | Needs JS | JSON |
| Data from a JSON API | An empty table the page fills from a JSON request after load. Render it, or find the API. | Easy | Needs JS | JSON |
| Shadow DOM | Reviews inside an open shadow root. document.querySelector can't see them; your selectors have to pierce it. | Hard | Needs JS | JSON |
| Same-origin iframe | A store list that lives in an iframe, rendered by script inside the frame. | Medium | Needs JS | JSON |
| Client-rendered app | A single-page app that changes the URL with pushState. Every item has its own address, and none of them exists as HTML. | Hard | Needs JS | JSON |
| Lazy-loaded images | Forty pictures that show a placeholder until they scroll into view. The real address only appears then. | Medium | Needs JS | JSON |
Bad days
What the web does to a crawl at 3 a.m.: limits, delays, errors that clear on retry, redirects that never end, and a redesign overnight.
| Test | What it tests | Difficulty | Needs | Answer key |
|---|---|---|---|---|
| Rate limit | Twenty-one pages that answer 429 with Retry-After once you pass twelve requests in a ten-second window. | Medium | HTTP only | JSON |
| Slow pages | Pages that take one, three and ten seconds to answer. A short timeout loses the slow one. | Easy | HTTP only | JSON |
| Pages that fail once | Every fourth item answers 503 the first time you ask, then works. One page always fails. | Medium | HTTP only | JSON |
| Redirects | A 301, 302, 307, 308 chain, a loop that never ends, and a meta refresh. | Easy | HTTP only | JSON |
| Soft 404s | Ten product links. Three answer 200 but say the product is gone; one is a real 404. | Medium | HTTP only | JSON |
| Redesign switch | The same fifty products in two layouts with every class renamed, plus a crossed-out price and a shipping charge to lure the wrong number. | Hard | HTTP only | JSON |
Forms & logins
Cookies, tokens and state. No database behind any of it; every session is a signed cookie.
| Test | What it tests | Difficulty | Needs | Answer key |
|---|---|---|---|---|
| Login and members area | Log in with the credentials on the page, keep the session cookie, and read three members-only pages that bounce you to the login without it. | Medium | Needs cookies | JSON |
| CSRF token | A form whose hidden token must come back with the cookie it was issued with. Replay it wrong and you get a 403. | Medium | Needs cookies | JSON |
| Search form | A GET search with paged results. Build the query string and follow the pages. | Easy | HTTP only | JSON |
| Multi-step form | Three steps, each a POST, carrying state in a signed cookie, ending in a reference code you can check. | Hard | Needs cookies | JSON |
Manners
One check that has nothing to do with extraction.
| Test | What it tests | Difficulty | Needs | Answer key |
|---|---|---|---|---|
| robots.txt | A linked area that robots.txt disallows. A polite scraper fetches none of it. | Easy | HTTP only | JSON |