Real apps

Reviewing real Android apps against real reported bugs

We tested real open-source Android apps against bugs their own users reported in public and their own maintainers later fixed. For each bug we take the release that had it and the release that fixed it, run the same review on both, and publish the result whatever it is. If the review reports nothing on the fixed release, that only counts as evidence when the same run shows it actually reached the screen being tested.

We registered ten bugs in advance. None of them has been verified. Three were reviewed and the review could not prove the fix. Two were reviewed and our test equipment could not reproduce the bug at all, so we exclude both from any success rate. Five have not been reviewed yet. The two extra findings lower down came from a different piece of work and do not change these numbers.

The five apps

Element

A private messaging app used by teams and communities.

Signs in with a real account on a real server.

Not run

The button for inviting someone to a room did nothing. You could press it as often as you liked and no invite screen ever opened.

What the project did
The project fixed it and said so in the notes for the next release.
What our review did
Not run
On the build that fixed it
Not run

What we owe you about this one. We have not run this one. There is no campaign result to report: no recorded reproduction and no review of either build. Preparing the sign-in fixture is not the same as running the chain, and we are not counting it as though it were.

Not run

Tapping the round add button twice quickly closed the whole app.

Reported upstream
Reported by a user
What the project did
The project fixed the double-tap crash and named it in the release notes.
What our review did
Not run
On the build that fixed it
Not run

What we owe you about this one. We have not run this one. There is no campaign result to report: no recorded reproduction and no review of either build. Preparing the sign-in fixture is not the same as running the chain, and we are not counting it as though it were.

AnkiDroid

A flashcard app for studying.

No sign-in. It runs on a deck we put there ourselves.

We did not prove this one

Opening a deck's options closed the app.

Reported upstream
Reported by a user
What the project did
The project found the settings were being read before they existed, and fixed it.
What our review did
Our review found it. On the build that had the bug, the review pressed and held a deck, opened its options, and the app died there. Our crash detector recorded the crash, and it is the same crash the user reported.
On the build that fixed it
Nothing to show. On the fixed release the review did not repeat those steps, so it never opened deck options. Nothing crashed, but nothing was attempted either, so this tells us nothing about whether the fix works.

What we owe you about this one. We count this as a miss, and the reason is the half we cannot show. The crash is real and our review caused it; what we never did was open the same screen on the repaired build. Without both halves this is not proof the repair works, so it does not count as a catch, whatever our own detector saw. One problem behind it is ours: when our scoring went looking for the crash it read the short label the record was filed under rather than the detector's own receipt, which had kept the full account all along. Nothing was lost and nothing was mislabelled; our scoring asked the wrong one of the two. The record was correct; the step that read it asked the wrong question. One more thing you should weigh: these four builds are early test releases the project publishes before a finished version, not the stable ones most people install.

On AnkiDroid 2.16alpha52, the crawler reproduced upstream issue #10584 by long-pressing Default, selecting Deck options, and capturing the matching target-process crash. On 2.16alpha53, it did not repeat that route, so this run does not independently demonstrate the fix. The final-rerun chain is a transparent crawler miss, not verified fix evidence.

Independent verifier, quoted in full: docs/ANKIDROID-ANK1-FINAL-RERUN-VERIFICATION-2026-08-31.md, the verifier's report, “Publication-safe result”
We did not prove this one

Opening a deck's options closed the app again, a few releases later, for a different reason.

What the project did
The project traced it to a setting that could be empty when the screen was built, and fixed it.
What our review did
Our review found it, but by a different route than planned. Instead of pressing and holding a deck, it opened deck options from the menu inside the reviewing screen. The app crashed there and our crash detector recorded it. It is the same fault on the same screen, reached a different way.
On the build that fixed it
Nothing to show. On the repaired build the review never reached deck options and stopped making progress, so the comparison could not be made at all. No crash, and nothing attempted to cause one.

What we owe you about this one. Also a miss, and this one carries a second thing we owe you. Our review reached the broken screen by a different route than the one we had committed to in advance: it came through the menu inside the reviewing screen rather than by pressing and holding a deck. It is the same fault and the same screen, but it is a weaker claim than the one we set out to make, and we would rather say so than quietly accept the result. As with the other one, the repaired build was never exercised, so there is no fix proof; and the same fault in our scoring applies, reading the filed label rather than the detector's receipt and so reporting that nothing was found. These four builds are early test releases the project publishes before a finished version, not the stable ones most people install.

On AnkiDroid 2.16alpha54, the crawler opened Deck options from the Reviewer overflow menu and captured the same Kotlin-nullability crash documented in upstream issue #10687. On 2.16alpha55, it never exercised Deck options and the regression comparison became untestable, so this run does not independently demonstrate the fix. The final-rerun chain is a transparent crawler miss, not verified fix evidence.

Independent verifier, quoted in full: docs/ANKIDROID-ANK2-FINAL-RERUN-VERIFICATION-2026-08-31.md, the verifier's report, “Publication-safe result”

Tasks.org

A to-do and reminder app.

No sign-in. It runs on tasks we add ourselves.

Our rig could not produce it

Opening the app's own settings closed the app, when the place it saves backups was not available.

What the project did
The project made it fall back to internal storage instead of failing.
What our review did
The review did reach it. On the release that had the bug it opened the app's own settings screen, stayed on that screen, and nothing crashed or closed.
On the build that fixed it
Nothing to compare against. The bug never appeared on the release that is supposed to have it, so there is no before-and-after here.

What we owe you about this one. Our equipment could not produce this one, so we exclude it from any success rate. Here is what that does and does not show. It does show the review reached the right control, and that the fault did not appear on our rig. It does not show the release is unaffected: the bug needs the app's external storage to be unavailable, and we did not force that condition or measure it independently, so we never put the app into the state the bug requires. Earlier attempts on this chain either failed on our own infrastructure or stopped before reaching the target screen; those are counted separately, and none of them is evidence that the bug does or does not exist.

Our rig could not produce it

Adding a tag or a location to a task left a blank white screen you could not get out of.

What the project did
The project restored the half-finished edit instead of losing it.
What our review did
Our rig could not produce it
On the build that fixed it
Our rig could not produce it

What we owe you about this one. Our equipment could not produce the bug, so this chain proves nothing either way. The review walked the whole path on the build that is supposed to be broken and the fault never appeared, which means the problem is with our test rig rather than with the app or the review. We exclude it from any hit rate rather than scoring it as a pass or a failure. The two findings further down came from the same app; they are separate observations and they do not rescue this chain.

Nextcloud Files

A file app for a server you run yourself.

Signs in with a real account on a real server.

Not run

Tapping the three dots beside a file that had been changed in two places at once, and choosing to download the server's copy, closed the app.

Reported upstream
Reported by a user
What the project did
The project moved the error-and-finish step onto the main screen thread, which is what was crashing.
What our review did
Not run
On the build that fixed it
Not run

What we owe you about this one. We have not run this one. There is no campaign result to report: no recorded reproduction and no review of either build. It is also held back deliberately, because the way we score whether the review reached the right screen is not yet precise enough for this bug: it would currently accept a near miss as a hit, and reject a valid route that arrived a different way. We would rather fix the scoring than collect a number we could not defend.

We did not prove this one

Moving from the media view back to all files left the sorting and view controls missing.

Reported upstream
Reported by a user
What the project did
The project restored the controls on that transition.
What our review did
The review ran on both builds and both runs finished properly. Neither run made the journey this bug is about. On the build with the fault it reached an ordinary file list and used the sorting and view controls there, but it never went to the media view first and came back, which is the step that breaks them. On the repaired build it stopped after three actions and never began that part of the walk at all.
On the build that fixed it
Nothing to compare. Neither run reached the screen the bug is about, so there is no before and after here, and nothing on this chain shows the repair works.

What we owe you about this one. We count this as a miss and we are claiming nothing about the repair. Both runs were valid and both used the same agreed plan, so the comparison itself was set up correctly. The review simply did not go where the check was waiting. Two things need saying plainly. The first is about a label: our own scoring came out of this run marked untestable, and that label must not be read as the result. We already hold a recording from this same equipment showing the affected build reaching that screen with the controls missing, so the equipment can produce this bug. What went wrong is that this run did not go there. The second is about the check itself: on the build with the fault, the part of the review that reads the screen images finished none of its work. It made twelve attempts, completed none, and four of them timed out. So even if the walk had arrived, that check had nothing to report. This was the last run planned for this chain, which makes the miss final rather than pending. It does not turn it into a better result.

NC-2 | Nextcloud Files | RERUN_REQUIRED | final valid, mission-symmetric pair missed the sealed Media -> All Files route and baseline completed zero eligible vision inputs | publish as terminal honest miss; no fix credit; generated untestable is non-authoritative

Independent verifier, quoted in full: docs/NEXTCLOUD-NC2-FINAL-BOUNDARY-PAIR-REVIEW-2026-09-01.md, the verifier's report, “FINAL TABLE LINE”

Thunderbird

An email app.

Signs in with a saved mail account.

Not run

Setting up an account closed the app, when the network was returning nonsense answers.

What the project did
The project fixed the crash during account setup.
What our review did
Not run
On the build that fixed it
Not run

What we owe you about this one. We have not run this one, and it is not ready to run. Reproducing it needs a deliberately broken network set-up that we have not yet proved we can create on demand. Until that fixture is validated the chain does not enter the count at all.

Not run

Renaming an account left the old initials showing on screen.

Reported upstream
Reported by a user
What the project did
The project made the initials follow the name you set.
What our review did
Not run
On the build that fixed it
Not run

What we owe you about this one. We have not run this one, and it is not ready to run. It is about wrong text left on screen rather than a crash or a dead control, and our detector contract does not yet formally cover that. Until it does and is sealed, this chain does not enter the count.

Other problems the review reported

These are separate from the bugs above. The review reported them without being asked to look for them, and we checked each one against the app's published source code before listing it.

There are two findings here, both from the to-do app. The flashcard app produced three candidates and an independent check rejected all three. One was an unlabelled button the project had already reported and was already fixing. One was a menu control below the recommended size, which is a real problem but an example of a type of problem already reported. One was an overlap warning that turned out to be our own detector misreading a screen that layers deliberately. We did not report any of them first, so we do not list them here.

Buttons a screen reader cannot name

Several controls you can tap have nothing a screen reader can announce. Someone using one hears an unlabelled button and has to guess what it does.

How we checked. The review reported it, and we then read the app's own published source at the exact versions we tested and found reachable controls with no label. Android's own guidance says a control you can act on needs one.

What that does not mean. Three things this is not. It is not a release regression: it is present in both versions we looked at, so it is long-standing rather than newly broken. What we retain is the family of unlabelled controls, not a claim about one exact button - mapping the finding onto a specific menu control is our reading of the source and the screen, not something the run proves by itself. And on prior art we can only say this: no matching public Tasks.org issue was found in the searches we recorded, as of 30 August 2026. That is a statement about our search, not a claim that no such issue exists.

A menu control below the minimum touch size

The 'more options' control in the toolbar is 40 by 48 units. The recommended minimum for something you tap is 48 by 48, so it is easier to miss than it should be.

How we checked. Measured in the run at 40 by 48, against a recommended minimum of 48 by 48, and then traced to the published source at the versions we tested, which sets no larger minimum and adds nothing to widen the tappable area.

What that does not mean. Also not a release regression, and not really the app's own doing: the control comes from a shared Android component the app builds on rather than from sizing the app chose. We count it as real and we are saying where it comes from. On prior art, the same limit applies: no matching public Tasks.org issue was found in the searches we recorded, as of 30 August 2026, which is a statement about our search rather than about what exists.

The full working, including the runs that failed and why, is in the evidence pages.