All tests / Bad days
Rate limit
What this tests. Twenty-one pages that answer 429 with Retry-After once you pass twelve requests in a ten-second window.
The rule. At most 12 requests in each 10-second window, counted per IP address.
Windows start on whole multiples of 10 seconds since the Unix epoch, so they're the same for everyone.
Request number 13 in a window gets 429 Too Many Requests and a Retry-After header with the
seconds left until the next window. Every page here counts, this one included; the answer key doesn't.
Go one request a second and you'll never see a 429. Go flat out and you'll see plenty; what matters is whether your scraper waits and comes back for every page, or quietly drops them.
The counter lives in the Cloudflare data centre that answers you, so two machines in different cities get a window each, and a burst of parallel requests can slip one or two past before the count catches up. Sequential requests are counted exactly.
- Medium
- HTTP only
- Answer key
/bad/rate-limit/answers.json
Pages
- Page 1
- Page 2
- Page 3
- Page 4
- Page 5
- Page 6
- Page 7
- Page 8
- Page 9
- Page 10
- Page 11
- Page 12
- Page 13
- Page 14
- Page 15
- Page 16
- Page 17
- Page 18
- Page 19
- Page 20
Data version 1.0.0. Everything here is made up: products, brands, people, towns. tests.json · robots.txt · Source