Reach rate
30/34
88.24%threshold 75%
Planted defects Caraxe proved it reached, with mechanical location or control evidence.
Benchmark resultsWe published the standard first.
We published the standard first.
Here is the score.
We hide real bugs in real apps and see how many Caraxe finds. Someone else holds the answers and does the scoring. Our most recent round is the first to clear every bar we set.
Detection when reached
20/30
66.67%threshold 60%
Exact catches among the defects it proved it reached.
Overall catch rate
20/34
58.82%
End to end, including the cost of paths it never reached.
Clean false-credit rate
0/3
0.00%threshold at most 5%
Exact planted-class claims made against clean control builds. Generic suggestions are counted separately.
Our most recent round: 37 builds, 34 with a bug planted and 3 clean, 0 runs that failed to finish. Every number on this page is read out of that run’s own results file and checked against its hash before the page is built. Nobody types these in.
Where it is weakest
One number would hide this.
Caraxe is better on some UI frameworks than others, and that gap is wider than the gap between test rounds. So we show them separately.
| UI framework | Planted | Reached | Caught | Detection when reached | Overall |
|---|---|---|---|---|---|
| Android Views | 8 | 7 | 4 | 57.14% | 50.00% |
| Jetpack Compose | 13 | 11 | 7 | 63.64% | 53.85% |
| React Native | 13 | 12 | 9 | 75.00% | 69.23% |
These three families are what Caraxe supports today. Apps that render their own canvas, such as Flutter and game engines, are not supported yet, and iOS support is planned for a later version. The benchmark covers what the product covers, nothing wider.
The record
Including the round that got worse.
Our last five rounds. Round two went backwards and it stays on the page. Round four beat everything before it and still failed, on one false alarm. A single good score with no history is easy to publish; the whole ladder is here so you do not have to take ours on trust.
| Test round | Overall catch | Outcome |
|---|---|---|
| Round 1 | 26.47% | - |
| Round 2 | 20.59% | went backwards |
| Round 3 | 38.24% | - |
| Round 4 | 55.88% | failed on one false alarm |
| Most recent | 58.82% | passed |
Correction, published with the score
The first score said no.
Scoring marked 1 of 3 clean builds as a false alarm and withheld the pass. Then we read the code. The control Caraxe flagged pops a message and does nothing else – no navigation, no field, no saved state. Caraxe was right and the clean build was not clean.
The claim was reclassified as a real defect in a nominally clean fixture. No scorer, matcher or engine rule was changed to reach that conclusion. Both documents are preserved in our records: the original score that says NOT ISSUED is kept unedited beside the certified one. This is the third time a control built to be clean has turned out to contain a real bug.
A reader who sees only the certificate, or only the report that refuses it, has half the record. Both are available on request:
UNSEAL_REPORT.md and UNSEAL_REPORT.raw.md.Running it again
A second run re-checks what the first one found.
Review the same app again and Caraxe goes back to the issues it reported last time and tells you which are fixed and which are still there. It also tells you when it cannot say. That is what a second review does today, not a plan for later.
We held this back until we could prove it, and the proof is a sealed test. Before either review ran we wrote down exactly which planted defects the second one had to report as fixed, which as still present, and which as new, then locked that expectation away. The run matched it: nothing we expected was missing, and nothing we did not expect was reported. It took ten attempts, and the nine failures are published with the pass in benchmark/results/regression-acceptance/FINAL-REPORT.md, each one named by the defect it exposed.
The code that passed that test is the code that reviews your app. We are not describing a laboratory result and then shipping something else. What we have not done is run that sealed test again on today’s build, so the report is the proof of the method, and the method is what runs.
There is a limit and the report states it. An issue that a model judged rather than measured cannot be re-checked mechanically, so a second run marks those as not re-checkable instead of guessing that they are fixed. Only the issues that can be checked mechanically come back with a fixed-or-still-there answer.
No second run is steered by the first. We tested that and it did not beat simply running again, so we do not do it and do not claim it.
We are still not publishing a number for how much more a second run covers. We had one, and an audit showed the measurement counted the same screen twice whenever its contents changed. The coverage figure stays withdrawn until we can count it properly, and re-checking does not depend on it.
Beyond the planted set
45 findings on 22 apps nobody planted anything in.
Planted bugs show whether Caraxe finds what we hid. Real apps show whether it says anything worth acting on. A second reviewer checked every finding here, and the strongest were re-tested on a real phone.
That review removed our own headline. A crash we had reported as the most serious finding did not happen again in ten attempts on the same build, and no crash record was ever written, so it was withdrawn rather than published with a caveat. Every finding the gallery once called critical is gone the same way. 14 findings were withdrawn in total and 14 are held pending a device test.
The gate
What had to be true before we published this.
We wrote these rules down before we had a score, and we have not loosened them since.
Gate 01
Freeze what is being tested
Lock the build, the apps, the device and the scoring rules before anyone sees a result. Change any of them and it is a new test.
Gate 02
Use more than one kind of app
Test across several apps and UI frameworks, including flows that need navigation, saved state and real data.
Gate 03
Score catches and clean controls
Score against a locked answer key, and include clean builds so a high score cannot hide a noisy one.
Gate 04
Repeat the runs on devices
Run it again on real devices. Say when a run did not finish, and track whether it reached the screen at all, not just whether it reported something.
Gate 05
Prove the operating path
Prove it finishes, costs what we said, keeps its evidence and can be rolled back, on the setup customers actually get.
Gate 06
Publish the misses with the score
Publish the misses, the exclusions and the per-app results next to the headline. Runs that proved nothing stay on the page.
Limits of this result
What this does not say.
We publish what it misses too. In our most recent round Caraxe missed 10 bugs on screens it reached, and never reached 4 more. Clearing our bar means it cleared three at once, not that detection is solved. This is 34 planted bugs across three UI frameworks, not a survey of every Android app, and accessibility findings point at published guidelines rather than legal sign-off.
Publication record
What each result must carry.
Help make the evidence representative.
We are working with a small number of Android teams whose real release workflows can strengthen the multi-app validation set. Participation is scoped before any review.