merge/audit-and-state-map #7
2 changed files with 18 additions and 7 deletions
fix(adjudicate): fail when uncertified, and state its independence limit
The adjudicator printed "controls FAILED" or "NO SIGNAL" and then exited 0, so a CI step reading the exit status took an uncertified run as a pass. It now exits 1 unless the controls are meaningful and pass, in both text and json output. It was also described as independent. Its evidence is independent (its own regex over source), but the list of unions, variants, events and owners it checks comes from pipe's `state` output, so a variant pipe fails to enumerate is never checked. The docstring and README now say so instead of "independent adjudicator".
commit
682be14f97
10
README.md
10
README.md
|
|
@ -153,7 +153,7 @@ Baselines and reports are JSON, so they diff and review like any other file.
|
|||
|
||||
| Rule | What it means | Verified precision (with n) |
|
||||
| --- | --- | --- |
|
||||
| `emitted-not-rendered` | A state the bloc emits that no UI code matches, so the user never sees it. | **79/79 (100%)** factual on a CLEAN nuwayze main via an independent adjudicator with passing controls (`tool/adjudicate.py`). Reported once per **root cause**, not once per variant: one finding per unobserved union (dormant feature) and one per marker class when the union is observed. When the UI reads the union's payload through **base-class fields** the finding drops to info, because the data does reach the user. On clean nuwayze main: 30 findings covering 79 variant-level claims. |
|
||||
| `emitted-not-rendered` | A state the bloc emits that no UI code matches, so the user never sees it. | **79/79 (100%)** factual on a CLEAN nuwayze main via a separate regex adjudicator with passing controls (`tool/adjudicate.py`; see its independence limit below). Reported once per **root cause**, not once per variant: one finding per unobserved union (dormant feature) and one per marker class when the union is observed. When the UI reads the union's payload through **base-class fields** the finding drops to info, because the data does reach the user. On clean nuwayze main: 30 findings covering 79 variant-level claims. |
|
||||
| `rendered-not-emitted` | A UI branch no state can reach: dead UI. | n=0 on nuwayze (unions whose bloc emits through helper methods are suppressed with a reason). Fires on a fixture with a passing negative control. |
|
||||
| `loading-not-reset` | An in-flight state whose failure path never resolves, so the spinner sticks. | n=0 on both dogfood repos, both verified to guard correctly; proven able to fire by fixture. |
|
||||
| `loading-flag-not-reset` | The same consequence for `copyWith` + `bool loading` state, which has no variant to match on. | **7/7 (100%)** on dex after hand adjudication; 4 characterised false positives were eliminated by the current gate. n=0 on nuwayze, verified correct. |
|
||||
|
|
@ -168,8 +168,12 @@ Baselines and reports are JSON, so they diff and review like any other file.
|
|||
|
||||
### How precision was measured
|
||||
|
||||
`tool/adjudicate.py <repo>` re-derives the two highest-volume claims without
|
||||
trusting pipe's bookkeeping, and **prints controls first**. It must find evidence
|
||||
`tool/adjudicate.py <repo>` re-derives the two highest-volume claims with its
|
||||
own regex evidence instead of pipe's, and **prints controls first**. It is only
|
||||
partly independent: it takes the list of unions, variants, events and owners to
|
||||
check from pipe's `state` output, so a variant pipe fails to enumerate is never
|
||||
checked. It exits non-zero when its controls fail or cannot discriminate, so a
|
||||
CI step cannot mistake an uncertified run for a pass. It must find evidence
|
||||
for every variant pipe calls *rendered* and every event pipe calls *dispatched*;
|
||||
on clean nuwayze main that is 131/131 rendered (type tests, `case`/switch-
|
||||
expression patterns, or base-class payload reads) and 143/162 dispatched, the
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Independent adjudicator for pipe's two highest-volume rules.
|
||||
"""Adjudicator for pipe's two highest-volume rules.
|
||||
|
||||
Usage:
|
||||
python3 tool/adjudicate.py <repo-path> [--format json]
|
||||
|
||||
Exit status: 0 when the controls pass, 1 when they fail or cannot discriminate
|
||||
(NO SIGNAL). A CI step must not read an uncertified run as a pass.
|
||||
|
||||
Precision claims are only meaningful if the thing doing the judging can
|
||||
demonstrate it discriminates. So this script prints CONTROLS first: it must find
|
||||
a UI type test for every variant pipe calls *rendered*, and a construction for
|
||||
|
|
@ -18,7 +21,10 @@ dead-handler : event E is never constructed or dispatched in any file
|
|||
other than the one that declares E.
|
||||
|
||||
It classifies every mention of every state/event name by regex on the source
|
||||
line, so it does not trust pipe's own bookkeeping. It does deliberately NOT
|
||||
line, so it does not trust pipe's evidence judgements. It is only partly
|
||||
independent, though: the list of unions, variants, events and owners to check
|
||||
comes from pipe's own `state` output, so a variant pipe fails to enumerate is
|
||||
never checked here at all. It does deliberately NOT
|
||||
judge UX severity: whether an unrendered marker state matters is a judgement,
|
||||
and whether an unbound handler is worth deleting is a judgement. Only the
|
||||
factual claims are adjudicated.
|
||||
|
|
@ -400,9 +406,10 @@ def main() -> int:
|
|||
),
|
||||
}
|
||||
|
||||
certified = result["controls"]["controls_meaningful"] and result["controls"]["controls_pass"]
|
||||
if args.format == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
return 0
|
||||
return 0 if certified else 1
|
||||
|
||||
c = result["controls"]
|
||||
print(f"repo {root}")
|
||||
|
|
@ -433,7 +440,7 @@ def main() -> int:
|
|||
print(f" CONTRADICTED {bad}")
|
||||
print()
|
||||
print("caveat: " + result["caveat"])
|
||||
return 0
|
||||
return 0 if certified else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue