ThirdOcular
← All notes

July 2026 · AisleLens v3.5

Same commit, two answers

338 captures lost no rows. 78 chosen cases found 11 regressions.

When this was measured
Measured during v3.5; recorded 27 July 2026. The exact run timestamp is not given in the source section.
Historical result, not current accuracy. Note written 1 November 2026.

I had a change that looked ready to ship. In AisleLens, my experiment in testing product pages against written buying standards, a replay over 338 captured store records found 32 clean recall gains and zero rows lost. It compared the status, detail and quoted evidence, not just a green or red result. That is a useful check. It still answered only one question.

On the same commit, 78 deliberately chosen cases found 11 regressions. Those cases were executed across three worktrees, with 234 executions compared mechanically against the parent version. The replay and the chosen cases were not disagreeing about the same evidence. They were asking different questions of the same change.

The change widened where the engine looked for product identifiers. It could now descend into offers and variant lists to find a barcode. That recovered useful-looking answers in the captured pages. But the published standard limited the engine to the product-level node. A barcode belonging to one variant was not necessarily an answer about the product the standard was asking about.

Two regressions were especially easy to miss. Their pass status stayed the same. The quoted evidence changed, acquiring a barcode from a variant list. A comparison of pass counts would have called those rows unchanged, even though the explanation given to a reader had changed. The evidence sentence was part of the result.

I kept the separate rule that rejected a storefront's own object ID as a manufacturer part number. I reverted the widening. The lesson was not that captured examples are useless. They catch damage to cases already encountered. Chosen cases test the boundaries of the rule, including situations a real-store sample never happened to contain.

There is a denominator correction here too: the record calls these 338 stores, then identifies 338 captures representing 334 distinct merchants. Repeated captures do not create independent evidence. These are historical results from July 2026, not a claim about current accuracy. What I carry forward is a release question: did the change stay inside the promise, and did I test an example that could prove it did not?

← Back to the notes