How this works

The page goes through Jina Reader to markdown, then a deterministic clean: bare image embeds, ad markers, horizontal rules and link-only lines are dropped. Site chrome can be a third of a marketing page, and every nav line left in is a candidate the extractor has to read and reject.

Cleaning is conservative on purpose. Anything arguable — author bios, house promos, marketing prose — is left in and rejected later, where you can see what was dropped and why. The result is line-numbered, and those line numbers are the addresses every claim, answer key and grade points back to.

A page is identified by a hash of its cleaned markdown, so re-running an unchanged page reuses everything downstream.

One long call reads the whole document and writes out every factual candidate, each marked accepted or rejected with a reason. Reading it whole matters: the same call returns an entity card — names mapped to disambiguating phrases — which later prevents questions like “How old is Sandy?” on a page where nothing says which Sandy.

Accepted claims are ranked by salience: how much it would matter if someone believed the opposite. That replaced a taxonomy of content types, which was only ever a proxy for the same thing and which models classified badly. Claims past the run's budget are marked capped rather than duplicative — they are testable, and the report should say so.

A question must identify exactly one thing. It does not need to be answerable — answerability is the measurement. Getting the amount of context right is the hard part: too little and nobody could answer, too specific and you have handed over the answer.

The test applied to every qualifier: if you removed this, would the correct answer change or become ambiguous? Keep it only if yes.

Generated questions are then filtered in code, because these are the rules the phrasing model breaks most often, and each one left unchecked manufactures a false finding rather than a plausible null:

• Source pointers — “according to this page” asks the model to read something it has never seen, and its confusion scores as ignorance.
• Unanchored — no proper noun tying the question to the subject. These once manufactured three fake staleness findings.
• Answer leaks — the answer appearing inside the question.
• Yes/no — a coin flip scores 50%. Sent to repair rather than dropped; dropping it once cost us the highest-salience claim on a page.

Survivors go through an ambiguity probe, a repair pass, then a re-probe. Whatever still fails is dropped with a reason. Finally a few controls are written — questions your content genuinely does not answer — to catch a model that invents an answer rather than declining.

Independent of any model: does the page contradict itself? “within minutes” in one place and “within 30 minutes” in another is a real defect, and any agent reading it will pick one of them at random.

This runs after question generation and never feeds back into it. Suppressing a contradictory claim would silently shrink the question set and break comparability between versions.

Candidates are proposed by one model and must then survive a different model whose only job is to knock them down, plus a hedge filter — “potential conflict depending on…” describes an ambiguity, it does not confirm one — plus a deterministic set-intersection check. Models twice invented overlaps between disjoint region lists, and that is arithmetic, not judgement.

This is the measurement, and most of the wall clock. Each question is asked 3 times per condition, because the answer moves between identical calls.

In condition B the model gets a real agent loop: it can search, read what comes back, fetch pages and go again. Every tool call is recorded — the queries it formed, the URLs it opened, how much came back, whether your page was among the results, and whether the sentence containing the answer ever reached its context.

A second model grades each answer against the source span and returns a one-sentence justification, shown beside every answer in the report. Strictness follows the claim type: an exact claim must match the value, a prose claim accepts any wording carrying the same substance.

The 3 samples per condition are collapsed by majority vote into one label per claim, and the disagreement itself is recorded: a claim whose identical samples disagreed sits on the decision boundary, which is where steering has the most leverage.