Robots meta and headers

What this tests. Eight pages marked noindex and nofollow in meta tags, X-Robots-Tag headers and rel="nofollow" links. Index the right ones; follow the right links.

The rule. robots.txt lets you in everywhere here; these pages give finer instructions one by one. A <meta name="robots"> tag or an X-Robots-Tag header saying noindex means fetch it but don't keep it. nofollow means keep it if you like, but don't follow its links. A link marked rel="nofollow" is one not to follow. Get it right and you fetch six pages, keep three, and never ask for two of them.

• Medium
• HTTP only
• Answer key /crawl/robots-meta/answers.json