The falsifier returns opposite verdicts on identical code across runs

Measured 2026-08-17 on `main` at `ea8b29a`, with the real `claude` CLI 2.1.224 and a real Linux container. ## What happened The same candidate, on the same unchanged file, was graded **D** by one run and **A** by the next. | Run | Candidate | Falsifier | Grade | |---|---|---|---| | `run-463fdff037` | `orders.py:16` — unguarded dict key access in `order_summary` | **refuted** it | **D** | | `run-f77ae3abb5` | the same candidate, same file, same prompt | **confirmed** it | **A** | Nothing about the input changed between them. The fixture was not edited, the config was not touched, the tier was `deep` both times. ## Why this matters more than a flaky test The falsifier is the product's stated differentiator — "a separate process, denied the discoverer's hypothesis, instructed to kill the finding. What survives is worth your time." A verdict that flips on identical input means: - **The grade is not reproducible.** Two users running the same version against the same commit can see A and D. - **Earned autonomy is calibrated on it.** `acceptance_rate` and `earned_level` are computed from human decisions, but what a human is shown to decide about is the graded finding. Grade noise becomes trust-score noise. - **M1a's Task 10 could not have seen this.** That measurement was n=1 per fixture and explicitly said so: *"n = 1, everywhere. Nothing here supports a rate, a precision, or a recall."* This is the first repeat of the same fixture, and it disagreed. ## What it is not Not a bug in the wiring. Both runs behaved exactly as designed end to end — reproduced in-container, `challenged: true`, graded from the falsifier's verdict, and the re-grade correctly reached the stored row while the human's `rejected` state was preserved (`upsert` never touches `state`). The mechanism is sound; the *input* to it is non-deterministic. ## Worth noting about the candidate itself Both runs surfaced this on the **clean** fixture — the file with the original defect fixed. `order_summary` does index `order["items"]`, `order["id"]` and `item["price"]` without guards, so "unguarded dict key access" is arguably a real observation about code that is merely *not the defect the fixture was built around*. That makes it exactly the kind of borderline finding where a falsifier's judgement is load-bearing, and exactly where instability costs most. ## Suggested next steps, not decided here - Measure it: run one fixture N times and record the verdict distribution. Until that exists the rate is unknown, and "it flipped once" is n=2. - Consider whether a falsify verdict should require agreement across more than one process before it can move a grade — the design already puts falsification in a separate process for anti-anchoring reasons, and a second opinion is a small extension of that. - Consider recording the verdict history per finding, so a flip is visible rather than silently overwriting the grade. Cost of the two runs that produced this: $0.5306 (buggy, grade A) and ~$0.29 x2 (clean).