A catch-rate number is easy to make impressive and hard to make useful. Choose a friendly app, remove incomplete runs, count a noisy signal as a success, and the percentage stops describing the product buyers will actually use.
This post sets out how Caraxe measures current-core, multi-app certification. It deliberately contains no product score: earlier internal campaigns guide engineering, but they are not presented as certification of a current release candidate.
Freeze the candidate before scoring
A result belongs to one exact core build and configuration. Before a certification run starts, the record must identify:
- the code and configuration under review;
- the app binaries and their hashes;
- the device profile and test data;
- the workflows in scope;
- the answer key, denominator, and exclusion rules; and
- the rule for incomplete or inconclusive runs.
If the core or its operating configuration changes, the score does not carry forward. The changed system becomes a new candidate.
Use multiple apps and clean controls
A single reference app is useful for development, but it cannot support a broad reliability claim. Certification must exercise multiple apps across the Android UI frameworks Caraxe says it supports, including deeper navigation, stateful workflows, and data-dependent screens.
Each positive test needs a matched clean control or another hard negative. A finding counts only when it identifies the expected problem without producing the same signal on the healthy comparison. That makes restraint part of the score instead of an afterthought.
Measure reach and detection separately
An automated review cannot identify behavior on a control it never reaches. The run record therefore needs both layers:
- Did the review reach and exercise the intended workflow?
- If it did, did it record the expected finding with usable evidence?
The public product number must still be end to end. Reach misses remain misses; the split exists to explain where the core needs work, not to improve the headline percentage.
Repeat the same work
One successful pass is not reliability. The same candidate should run the same app and workflow repeatedly, on devices, with every incomplete run retained. We will compare completion, workflow reach, findings, evidence, and verdict behavior across those repeats.
The publication threshold will be fixed before scoring. It will not be moved after seeing the result.
Include the operating path
Production readiness is wider than detection. The certification record must also cover:
- review completion and bounded failure behavior;
- wall-clock time and cost by app, not only an average;
- evidence retention and removal checks;
- monitoring and operator recovery; and
- a tested rollback path for the launch configuration.
A strong app result cannot compensate for an operating path that cannot be recovered safely.
Publish enough to challenge the result
When Caraxe publishes a score, it should include the cohort, per-app outcomes, repeated-run results, clean-control results, exclusions, incomplete runs, timing, known misses, and the exact candidate identity. Summary numbers without that record will not be used as sales proof.
Until a core candidate passes this process, it does not receive a public product score. You can read the full evidence standard or ask about a scoped review.