merge/audit-and-state-map #7

Merged
trevon merged 15 commits from merge/audit-and-state-map into main 2026-09-25 00:07:04 +00:00
2 changed files with 18 additions and 7 deletions
Showing only changes of commit 682be14f97 - Show all commits

fix(adjudicate): fail when uncertified, and state its independence limit

The adjudicator printed "controls FAILED" or "NO SIGNAL" and then exited
0, so a CI step reading the exit status took an uncertified run as a
pass. It now exits 1 unless the controls are meaningful and pass, in both
text and json output.

It was also described as independent. Its evidence is independent (its
own regex over source), but the list of unions, variants, events and
owners it checks comes from pipe's `state` output, so a variant pipe
fails to enumerate is never checked. The docstring and README now say so
instead of "independent adjudicator".
trvon 2026-09-22 13:30:49 -06:00
Signed by: trevon
GPG key ID: 38686EA797D7A8A7

View file

@ -153,7 +153,7 @@ Baselines and reports are JSON, so they diff and review like any other file.
| Rule | What it means | Verified precision (with n) |
| --- | --- | --- |
| `emitted-not-rendered` | A state the bloc emits that no UI code matches, so the user never sees it. | **79/79 (100%)** factual on a CLEAN nuwayze main via an independent adjudicator with passing controls (`tool/adjudicate.py`). Reported once per **root cause**, not once per variant: one finding per unobserved union (dormant feature) and one per marker class when the union is observed. When the UI reads the union's payload through **base-class fields** the finding drops to info, because the data does reach the user. On clean nuwayze main: 30 findings covering 79 variant-level claims. |
| `emitted-not-rendered` | A state the bloc emits that no UI code matches, so the user never sees it. | **79/79 (100%)** factual on a CLEAN nuwayze main via a separate regex adjudicator with passing controls (`tool/adjudicate.py`; see its independence limit below). Reported once per **root cause**, not once per variant: one finding per unobserved union (dormant feature) and one per marker class when the union is observed. When the UI reads the union's payload through **base-class fields** the finding drops to info, because the data does reach the user. On clean nuwayze main: 30 findings covering 79 variant-level claims. |
| `rendered-not-emitted` | A UI branch no state can reach: dead UI. | n=0 on nuwayze (unions whose bloc emits through helper methods are suppressed with a reason). Fires on a fixture with a passing negative control. |
| `loading-not-reset` | An in-flight state whose failure path never resolves, so the spinner sticks. | n=0 on both dogfood repos, both verified to guard correctly; proven able to fire by fixture. |
| `loading-flag-not-reset` | The same consequence for `copyWith` + `bool loading` state, which has no variant to match on. | **7/7 (100%)** on dex after hand adjudication; 4 characterised false positives were eliminated by the current gate. n=0 on nuwayze, verified correct. |
@ -168,8 +168,12 @@ Baselines and reports are JSON, so they diff and review like any other file.
### How precision was measured
`tool/adjudicate.py <repo>` re-derives the two highest-volume claims without
trusting pipe's bookkeeping, and **prints controls first**. It must find evidence
`tool/adjudicate.py <repo>` re-derives the two highest-volume claims with its
own regex evidence instead of pipe's, and **prints controls first**. It is only
partly independent: it takes the list of unions, variants, events and owners to
check from pipe's `state` output, so a variant pipe fails to enumerate is never
checked. It exits non-zero when its controls fail or cannot discriminate, so a
CI step cannot mistake an uncertified run for a pass. It must find evidence
for every variant pipe calls *rendered* and every event pipe calls *dispatched*;
on clean nuwayze main that is 131/131 rendered (type tests, `case`/switch-
expression patterns, or base-class payload reads) and 143/162 dispatched, the

View file

@ -1,9 +1,12 @@
#!/usr/bin/env python3
"""Independent adjudicator for pipe's two highest-volume rules.
"""Adjudicator for pipe's two highest-volume rules.
Usage:
python3 tool/adjudicate.py <repo-path> [--format json]
Exit status: 0 when the controls pass, 1 when they fail or cannot discriminate
(NO SIGNAL). A CI step must not read an uncertified run as a pass.
Precision claims are only meaningful if the thing doing the judging can
demonstrate it discriminates. So this script prints CONTROLS first: it must find
a UI type test for every variant pipe calls *rendered*, and a construction for
@ -18,7 +21,10 @@ dead-handler : event E is never constructed or dispatched in any file
other than the one that declares E.
It classifies every mention of every state/event name by regex on the source
line, so it does not trust pipe's own bookkeeping. It does deliberately NOT
line, so it does not trust pipe's evidence judgements. It is only partly
independent, though: the list of unions, variants, events and owners to check
comes from pipe's own `state` output, so a variant pipe fails to enumerate is
never checked here at all. It does deliberately NOT
judge UX severity: whether an unrendered marker state matters is a judgement,
and whether an unbound handler is worth deleting is a judgement. Only the
factual claims are adjudicated.
@ -400,9 +406,10 @@ def main() -> int:
),
}
certified = result["controls"]["controls_meaningful"] and result["controls"]["controls_pass"]
if args.format == "json":
print(json.dumps(result, indent=2))
return 0
return 0 if certified else 1
c = result["controls"]
print(f"repo {root}")
@ -433,7 +440,7 @@ def main() -> int:
print(f" CONTRADICTED {bad}")
print()
print("caveat: " + result["caveat"])
return 0
return 0 if certified else 1
if __name__ == "__main__":