Browser automation and scraping at scale, 3 of 26
Every one of these
returns 200.
A scraper does not throw when it breaks. The site ships a redesign, the selector matches nothing, the parse succeeds, and the job writes zero rows and exits zero. Every alarm anybody set up is watching the status code and the exit code, and both are green.
Press each of the three. They fetch real pages, right now, and two of them are broken.
What a field declares
Not just where to look. How many there normally are, and what the value looks like when it is right. Those two declarations are the whole difference between a scraper that fails loudly and one that fails silently.
{ field: 'price', selector: '.price', atLeast: 1, looksLike: 'money' }A field that genuinely can be empty says so with atLeast zero. Inferring it from an empty array is how a real break gets excused as an optional field.
Four decisions
Zero rows is a result, and it should be an alarm
The line that hides every one of these is `?? []`. A field that has never once been empty comes back empty, the default swallows it, and the job writes nothing and exits zero. Declaring how many matches a field normally has turns that silence into a number somebody can watch.
A selector that still matches can still be wrong
When a page is restructured, a selector often lands on the neighbouring element rather than on nothing. The count stays healthy and the values quietly stop being what they were. So each field declares what its value looks like, and a price that stops parsing as money is reported as drift.
The page gets checked as well as the fields
A challenge page answers 200 with perfectly valid HTML. It weighs a tenth of the real page and has no links in it. That is compared against what the page has actually weighed before rather than against a fixed threshold, because "under 5KB is a bot wall" is true for one site and nonsense for the next.
Three fixed targets, not an address box
An endpoint on a public page that fetches whatever a stranger types is a proxy for whatever they want to reach, and somebody else’s website is somebody else’s bandwidth. The two pages here have looked the same for years, which is what makes them usable as a fixture that lives on the internet.
Tested without the internet
32 tests on the rules, none of which fetch anything: the field that matched nothing, the field that matched fewer than usual, the selector that drifted onto a button, the challenge page that weighs a tenth of the real one, and the first run of a new scraper where there is no baseline to compare with and an alarm would be wrong.
npx tsx lib/scrape/health.test.ts
What this is not
These are plain HTTP fetches with the HTML parsed. A page that builds itself with JavaScript returns an empty shell, so every field reads as broken, which is the correct answer and not a useful one. That work needs a real browser, and there is one on the VPS doing exactly that job elsewhere.
And it says nothing about being blocked. Rotation, fingerprints and rate discipline are a separate subject, and the one thing worth saying here is that a scraper which gets blocked and does not know it fails in exactly the way this page is about.
The endpoint lists the three targets. The audit tool is the same fetching with a different question asked of it. The address guard is why a fetcher does not take an address box. The whole list is 41 requirements from 114 job posts.