A reviewer who grades their own work grades their own assumptions with it. Before a gallery goes out, a second independent reviewer tries to break it - from the stored artifacts, and where those cannot settle a question, from the app running on a real device. The reviewer is told to assume the findings are wrong until the evidence forces otherwise.
The first question is not whether a finding is true. It is whether the evidence behind it exists and shows what the finding says. Those are different questions, and the second is where public claims usually fail.
Every finding carries an evidence receipt naming a crawl, a screen and a capture. The reviewer re-fetches each one independently of the tool that built the page, and confirms the row exists, belongs to the right crawl, and matches the run's own record of what was found.
Screenshots are content-addressed: a file's name in storage is the SHA-256 of its own bytes. The reviewer downloads each image and hashes what was published. A substituted screenshot is arithmetically impossible to hide rather than merely unlikely.
A finding that says four controls fall below the minimum size is an arithmetic claim, and arithmetic can be redone. The reviewer recovers the interface tree the detector measured - from a different artifact than the finding came from, so the check does not inherit the finding's own errors - and recomputes the geometry from raw element bounds.
This is the step no automation replaces. A finding can resolve perfectly, recompute exactly, and still describe something that is not in the picture. Several did.
Anything still open goes onto a real device.
Every app package is hashed before installation and must match both the cohort lock file and the hash published on the finding's own page. A finding about one version is never checked against another.
The verification device is built to the original profile - same system image, same screen size, same pixel density - so a measurement means the same thing in both places.
Verification runs on its own device, on a reserved port, under its own name, with a watchdog checking continuously that it never appears in the production device registry. The check ran every twenty seconds for the whole session and the result is reported with the findings.
Navigate to the screen, perform the action, and see what happens - repeatedly, with the process watched for death and the system crash log read after every attempt. A crash claim is not accepted because an oracle recorded one. It is accepted when it happens again, or it is marked as not reproducible.
The most important class. An automated reviewer navigates like a program, not a person: it may dismiss a permission dialog to get past it, or kill a process rather than closing it. Both leave the app in a state that looks like a defect. The live pass separates what the app does from what we made it do, by doing the ordinary thing instead and seeing whether the problem survives. Several findings did not.
Apps show screenshots of themselves in onboarding tours. A visual reviewer can analyse that illustration as though it were live interface and report a layout defect inside an image. The interface tree settles it instantly: the whole screen is one non-interactive picture.
A claim that nothing was saved, published beside a screenshot showing the saved item. The easiest to catch and the most damaging to miss, and only caught by someone reading the page against the picture.
Grades are re-derived from what the detector assigned, not from what got stored. Where a storage path changed a grade, the published grade is corrected or the finding is marked unverified rather than restated.
A finding is published when its evidence resolves, its numbers recompute, and its description matches what the screenshot shows. A finding whose screenshot cannot be retrieved is excluded and counted, never quietly dropped. One that survives the artifact pass but fails on the device is removed, and so is one that turns out to describe our own navigation.
It runs in both directions. Claims that could not be checked from artifacts came back exactly right when measured on the device - including for the one app whose run left no measurement trail at all. An odd-looking label we had nearly dismissed turned out to be a real rendering bug once the cause was traced. 7 findings in the current gallery carry a measurement taken twice, on two different machines, and matching.
Findings are about other people’s software. The bar is that a developer receiving one should be able to follow it to the defect without taking anything on trust, and that we should be able to show our working when they ask.