A web page is mostly not the article

Ask an API for a web page and you get a web page. Ask it for the article and you get something about one twenty-fourth the size.

We measured this properly rather than estimating it, across 74 pages our own search API actually served, running the real extraction path over freshly fetched HTML. Pooled across all of them, what comes back is 4.2% of the raw bytes. The median page is tighter still, at 2.4%.

Where the other 95.8% goes

Nothing exotic. A modern page carries:

• the navigation, twice, once for desktop and once for a mobile menu
• a cookie banner and its consent framework
• analytics and tag-manager payloads
• a newsletter modal
• inline CSS, often a whole framework of it
• the footer, sitemap links, social icons
• structured-data blobs describing the page to machines that are not you
• the article

Only the last item answers the question that was asked.

Why this is a cost problem, not a tidiness problem

If you hand a raw page to a model, it reads all of that and you pay for all of it.

The arithmetic is unkind. At roughly four characters per token, a 500KB page is about 125,000 tokens. Put five of those in a prompt and you are at 625,000 tokens before the model has read a word of your actual question — which does not fit in most context windows at all, and would be expensive if it did.

The same five pages, extracted, are about 26,000 tokens. That fits comfortably, costs a fraction, and contains the same answer.

The measurement is the point

We publish 4.2% rather than “about a tenth” because for a long time this codebase said “roughly a tenth of the tokens” and that figure existed only in a source comment. No benchmark produced it and no test held it. When someone finally measured it, the guess turned out to be conservative by a factor of two.

That is the ordinary failure mode for numbers on a marketing page: somebody estimates once, it gets repeated, and eventually nobody remembers it was a guess. Our rule is that a figure on the site has to come out of a file a script wrote. This one comes from npm run extraction-ratio, which fetches real URLs and runs the real extractor, and it gets re-run rather than remembered.

What extraction cannot do

Two honest limits.

Some pages assemble themselves with JavaScript, and a plain fetch gets a shell. Those need a real browser, which costs seconds rather than milliseconds, so it is a fallback behind a cache rather than the default path.

Some pages will not be read at all — a bot challenge, a paywall, a robots rule we honour. When that happens the result still comes back, marked with the reason, rather than quietly disappearing from the list — the extract endpoint returns every URL you sent, in one list or the other. A shorter list with no explanation is the failure mode that sends you debugging the wrong layer.

Extraction is not a solved problem. It is just a much better default than paying a model to read a cookie banner.