Scored when you loaded this page, not when I deployed it
How often does it accuse
somebody of something untrue?
A test asserts one thing and passes. An eval scores a whole population and gives you a number that can go down. Below is a golden set of 34 pages, each with an answer key written from the definition of every check before anything was run, scored on precision and recall.
Precision is the one that matters. A missed finding costs somebody a small improvement they never hear about. A false finding tells them their working link is broken, they check it, and they stop believing the rest of the report.
Fixture split
15 cases I look at while changing the engine
100.0%
precision
100.0%
recall
15/15
exact
Holdout split
19 cases the engine is never tuned against
100.0%
precision
100.0%
recall
19/19
exact
The only number worth reading twice
Fixture precision minus holdout precision is 0.0 points. The engine is as right on pages it was never tuned against as on the ones it was. A gap opening up here is the tool being fitted to its own answer key.
What it found on the runs before this one
Three defects, all of them in the holdout split, all of them in code that already had a green test suite over it.
- A menu page answering 429 was reported at the highest severity as a page that will not open. This crawler requests eight pages in a row, their server pushes back, and the tool then charged the owner for a status it caused itself. Same for 401 on a members area.
- An image with alt="" was listed as needing a description. An empty alt is the correct way to mark something decorative, so following that advice would have made those pages worse for screen reader users.
- srcset was in the mixed content selector and nothing ever read the attribute, so every responsive image pulling an insecure source passed silently.
All three are written up in full on what broke, with the wrong move I made first.
Every case, and what the engine said about it
Open any row to read the page it was scored against. An empty answer key means the page is correct and the engine is required to stay silent about it.
fixtureA small trades site with nothing wrong with it✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>fixtureA page that never had its head tags filled in✓
answer keymeta-description, title
engine saidmeta-description, title
<html><head><meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/s.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"X"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>fixtureSomebody pasted a paragraph into the description field✓
answer keymeta-long
engine saidmeta-long
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing services across the whole of Calgary and the surrounding area, available seven days a week including evenings and public holidays, with no callout charge. Emergency plumbing services across the whole of Calgary and the surrounding area, available seven days a week including evenings and public holidays, with no callout charge. ">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>fixtureA theme that renders the logo and the page title both as h1✓
answer keyh1-many
engine saidh1-many
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><h1>Northgate Plumbing</h1></body></html>fixtureA page built in a visual editor where every heading is a styled div✓
answer keyh1-missing
engine saidh1-missing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><div class="big">Emergency plumbing in Calgary</div>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>fixtureA gallery page where nobody filled in the alt text✓
answer keyimage-alt
engine saidimage-alt
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Recent work</h1>
<img src="/1.jpg" width="800" height="600">
<img src="/2.jpg" width="800" height="600">
<img src="/3.jpg" width="800" height="600">
<img src="/4.jpg" width="800" height="600">
<a href="tel:4032503230">Call</a></body></html>fixtureFour photos with no width or height, so the page jumps while it loads✓
answer keyimage-size
engine saidimage-size
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Recent work</h1>
<img src="/1.jpg" alt="One">
<img src="/2.jpg" alt="Two">
<img src="/3.jpg" alt="Three">
<img src="/4.jpg" alt="Four">
<a href="tel:4032503230">Call</a></body></html>fixtureOnly two photos lack dimensions, which is below the threshold and must stay quiet✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Recent work</h1>
<img src="/1.jpg" alt="One">
<img src="/2.jpg" alt="Two">
<img src="/3.jpg" alt="Three" width="800" height="600">
<a href="tel:4032503230">Call</a></body></html>fixtureA quote form where every field is a bare input with a placeholder✓
answer keyform-labels
engine saidform-labels
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><form><input name="name" placeholder="Name"><input name="email" placeholder="Email"><textarea name="msg"></textarea><button>Send</button></form></body></html>fixtureA properly labelled form that also carries a hidden anti-spam field✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><form>
<label for="n">Name</label><input id="n" name="name">
<label for="e">Email</label><input id="e" name="email">
<input type="text" name="website" style="display:none" tabindex="-1" autocomplete="off">
<button>Send</button>
</form></body></html>fixtureA local business page with no phone number and no email anywhere on it✓
answer keyno-contact
engine saidno-contact
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1><p>Get in touch through the form on our contact page.</p></body></html>fixtureA staging robots.txt that went live with the site✓
answer keyrobots-blocking
engine saidrobots-blocking
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>fixtureA hand built site that never had either file✓
answer keyrobots, sitemap
engine saidrobots, sitemap
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>fixtureA link to a supplier page that has actually been removed✓
answer keybroken-links
engine saidbroken-links
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><a href="https://supplier.example/old">Our supplier</a></body></html>fixtureA shared host taking over four seconds to answer✓
answer keyslow
engine saidslow
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutA site whose social links all answer 400 to anything that is not a logged-in browser✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><a href="https://facebook.com/northgate">Facebook</a><a href="https://instagram.com/northgate">Instagram</a></body></html>holdoutA menu page that answers 401 because it is a members area, exactly as intended✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutA menu page answering 429 because this tool just requested eight pages in a row✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutA menu page answering 404, which is a real problem and must be reported✓
answer keynav-broken
engine saidnav-broken
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutA menu page that timed out on a slow server✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutAn https page linking out to three ordinary http websites✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><a href="http://oldsupplier.example/">Supplier</a><a href="http://directory.example/">Directory</a><a href="http://forum.example/">Forum</a></body></html>holdoutAn https page loading a logo over http, which browsers really do block✓
answer keymixed-content
engine saidmixed-content
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><img src="http://cdn.example/logo.png" alt="Logo" width="200" height="60"></body></html>holdoutA plugin emitting JSON-LD with a trailing comma✓
answer keystructured-data, structured-data-broken
engine saidstructured-data, structured-data-broken
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing",}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutA site that looks like a grey box when anyone shares it✓
answer keyfavicon, og-image
engine saidfavicon, og-image
<html><head><title>Northgate Plumbing</title>
<meta name="description" content="Emergency plumbing across Calgary.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"X"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutAn older site that was never made to work on a phone✓
answer keyviewport
engine saidviewport
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutThe About page is dead, and it is also an ordinary link in the footer✓
answer keynav-broken
engine saidnav-broken
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><a href="https://example.com/about">About</a></body></html>holdoutA spacer graphic carrying alt="" on purpose, which is exactly what a screen reader wants✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/divider.svg" alt="" width="1200" height="8">
<img src="/van.jpg" alt="Our van" width="800" height="600">
<a href="tel:4032503230">Call</a></body></html>holdoutA search field labelled with aria-label instead of a visible label, which is valid✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><form><input name="q" aria-label="Search the site"><button>Search</button></form></body></html>holdoutA form using the wrapping label pattern, so no for attribute is needed or correct✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><form><label>Your name <input name="name"></label><label>Email <input name="email"></label><button>Send</button></form></body></html>holdoutA labelled form that also carries a CSRF token and a submit button✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><form>
<label for="n">Name</label><input id="n" name="name">
<input type="hidden" name="csrf" value="abc123">
<input type="submit" value="Send">
</form></body></html>holdoutA business whose phone number is in the JSON-LD but not written as a tel: link✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing","telephone":"+1-403-250-3230"}</script></head><body><h1>Emergency plumbing in Calgary</h1><p>Call us on 403 250 3230.</p><img src="/van.jpg" alt="Our van" width="800" height="600"></body></html>holdoutog:image written with name= rather than property=, which most scrapers still accept✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a></body></html>holdoutAn https page whose responsive image pulls its source over http, which browsers block✓
answer keymixed-content
engine saidmixed-content
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><picture><source srcset="http://cdn.example/hero.webp" type="image/webp"><img src="https://example.com/hero.jpg" alt="Hero" width="1200" height="600"></picture></body></html>holdoutA site still served over plain http, where an http image is simply an http image✓
answer keynothing, this page is fine
engine saidnothing
<html><head><title>Northgate Plumbing, emergency callouts in Calgary</title>
<meta name="description" content="Emergency plumbing across Calgary, seven days a week.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta property="og:image" content="https://example.com/share.jpg">
<link rel="icon" href="/favicon.ico">
<script type="application/ld+json">{"@type":"LocalBusiness","name":"Northgate Plumbing"}</script></head><body><h1>Emergency plumbing in Calgary</h1>
<img src="/van.jpg" alt="Our van" width="800" height="600">
<img src="/team.jpg" alt="The team" width="800" height="600">
<img src="/job.jpg" alt="A finished job" width="800" height="600">
<a href="tel:4032503230">403 250 3230</a>
<a href="mailto:hi@northgate.example">Email us</a><img src="http://example.com/logo.png" alt="Logo" width="200" height="60"></body></html>The one rule that decides whether anybody gets accused
Every status the tool can meet, and what it is allowed to conclude from it. This used to be one line inside a 596 line function with eleven network calls in it, so nothing could reach it and nothing tested it.
| status | verdict | why | |
|---|---|---|---|
| 200 | ok | an ordinary page | ✓ |
| 204 | ok | no content is still an answer | ✓ |
| 301 | ok | a redirect is not a dead link | ✓ |
| 308 | ok | a permanent redirect either | ✓ |
| 400 | unverified | what Facebook answers a crawler | ✓ |
| 401 | unverified | a members area, working as intended | ✓ |
| 403 | unverified | a firewall, not a missing page | ✓ |
| 405 | unverified | the method was refused, not the page | ✓ |
| 429 | unverified | rate limiting we caused ourselves | ✓ |
| 404 | gone | the page is not there | ✓ |
| 410 | gone | the page was deliberately removed | ✓ |
| 500 | gone | the server is failing on it | ✓ |
| 503 | gone | the server is failing on it | ✓ |
| no answer | unverified | a timeout is usually a slow server | ✓ |
What this number does not tell you
I wrote both halves of the split. The holdout is protected from tuning and it is not protected from my blind spots, so a case I would not think to write is missing from both sides.
34 cases is small. It reads 100% today, and the first time it did that I took it as a sign the set was too easy rather than a sign the engine was finished, and wrote eight harder pages. Two of them failed immediately.
These are hand written pages. They say nothing at all about how the crawler behaves against a slow server, a redirect loop, a login wall, or markup that only exists after JavaScript has run, and every one of those has caused a defect on this list before.
A false accusation exits non-zero and fails the build. A miss does not. The tool is allowed to be quiet.
Run it yourself with npm run eval. The tool this scores takes your own web address, the guard is a box you can attack, the MCP server scopes what an agent may change, the token measurement prices the same question four ways, and every defect it has had is written down.