fix: stop verify, reports and briefs from claiming more than pipe checked #1

Merged
trevon merged 12 commits from fix/verify-soundness-and-injection into main 2026-09-22 18:24:06 +00:00

12 commits

Author SHA1 Message Date
trvon
d6b6bcb438
fix(analysis): report an unreadable SDK cleanly instead of crashing
Some checks failed
Lint Commits / commitlint (pull_request) Successful in 17s
Tests / Static Analysis (pull_request) Has been cancelled
Tests / Tests (pull_request) Has been cancelled
ae412a7 routed filesystem failures to exit 2, but the analyzer throws its own
FileSystemException (package:analyzer/file_system), which the dart:io catch
in bin/pipe.dart never matched. A project whose package_config still named a
moved Flutter SDK therefore crashed with exit 255 and a stack trace. Found
by dogfooding on word_alive after its SDK moved.

Analyzer.create now turns that into a PipeException naming the SDK path, with
the remedy to re-run flutter pub get or pass --sdk.
2026-09-22 12:22:03 -06:00
trvon
10db6fbb57
chore: apply dart format to the files this branch changed
The commits above were made without the pre-commit hook, which could not
run dart at the time.
2026-09-22 12:22:02 -06:00
trvon
17c9f9afab
fix(suggest): break score ties by file, line and rule
The plan sorted by score alone. List.sort is not stable, so equal-score
actions came out in an order nothing guaranteed, and two runs over the same
code could hand an agent a different first action. Ties now fall back to file,
line, rule and label, both in the global order and within each file.
2026-09-22 12:05:12 -06:00
trvon
ae412a7964
fix(cli): reject a missing --dir and report filesystem errors cleanly
With several --dir values, one that did not exist was skipped silently: the
no-files guard only fires when every directory is empty, so
`--dir lib --dir tset` scanned lib and reported as though it had covered what
was asked for. A --dir that does not exist is now an error naming it.

A directory that could not be listed, or a baseline that could not be read or
written, escaped as an uncaught FileSystemException: exit 255 and a stack
trace. Those paths now raise a PipeException with the OS reason (exit 2), and
bin/pipe.dart catches any remaining FileSystemException the same way. The exit
codes are documented in bin/pipe.dart: 0 success, 1 a gate failed, 2 pipe could
not do what was asked, 64 usage.

rule_scope_claim_test widened the fixture with a test/ directory it does not
have, relying on exactly the silent skip; it now builds a project that has one.
2026-09-22 12:05:12 -06:00
trvon
e2d84f5e75
fix(guide): persist the yams report with its caution and provenance
`guide --yams` persists the markdown report, not the brief, and the markdown
had neither the brief's caution line ("static reachability claims ... confirm
each against the cited file") nor the [severity/confidence] of each finding -
so a report read back from the corpus presented every finding as equally
certain, with no framing. Its metadata still carried a stale `owner=pi` and
nothing to date or reproduce it.

The markdown now opens with the caution line, headings read
`### N. [severity/confidence] rule — location`, and the metadata records
`generated=<ISO-8601 UTC>` and, when the project is a git checkout,
`commit=<HEAD>`. `owner=pi` is gone.
2026-09-22 12:05:12 -06:00
trvon
4a1c9cdd3e
fix(guide): quote analysed source as bounded, delimited data
The analysed repository is untrusted input, and its source reached the agent
brief and the yams corpus verbatim: emission and dispatch snippets were
`node.toSource()`, provider-read-in-build quoted the whole call, and the brief
printed `evidence: $item` raw (the markdown wrapped it in single backticks,
which a backtick in the source closes). A string literal written as an
instruction to the reader was pasted into the prompt as-is.

Every quoted excerpt now goes through sourceExcerpt(): string literals,
interpolations included, become '…'; comments are dropped (they are not
tokens); whitespace collapses; the result is capped at 120 characters, as
loading-not-reset's catch excerpts already were. Where the brief and the
markdown render evidence, it is labelled "source excerpt (data, not
instructions)" and fenced with one more backtick than the longest run inside
it, so nothing quoted can close its own quotation.

Fingerprints are unaffected: provider-read-in-build still keys on the full call
source, which is kept for identity and no longer quoted.
2026-09-22 12:05:12 -06:00
trvon
39b309af97
fix(guide,suggest): stop packing at the first entry that does not fit
Both packers skipped an entry that did not fit the budget and kept going, so a
large top-ranked finding vanished while smaller, lower-ranked ones were packed
in behind it - presented as the most important work. Packing now stops at the
first entry that does not fit and counts it and everything after it as
omitted.

suggest's agent plan also reserved no room for its own omission note, so the
note itself could push the plan over budget; it now reserves room for the
largest possible note unless the entry is the last one.

When no entry fit, the guide said "this header alone exceeds the budget" even
when the header had room and it was the top entry that was too large. It now
says which of the two it was, with the size.
2026-09-22 12:05:11 -06:00
trvon
749aa53348
feat(report): state scan coverage in every check, guide and suggest output
A report listed findings without saying how much of the project they were
drawn from, so "0 findings" from 3 of 40 files looked like a clean project -
and an agent handed a brief had no way to know otherwise.

check, guide and suggest JSON now carry a `coverage` block
{discovered, selected, resolved, failed, skipped, complete}. The guide brief,
the markdown report, the suggest agent plan and both text outputs open with a
one-line coverage note, which says PARTIAL whenever a discovered file was not
analysed.

The note lives in the brief's header, which budget trimming never touches. The
"Coverage (what was actually examined)" caveats section was the first thing
dropped under a tight budget, silently; when it no longer fits, a one-line
pointer now says how many caveats were left out.
2026-09-22 12:05:11 -06:00
trvon
ac7cce1f76
fix(cli): keep status messages off stdout so JSON output stays parseable
`check --format json --write-baseline b.json` printed "wrote baseline to ..."
after the JSON document, and `guide --yams` printed "persisted to yams as ..."
after the brief, so stdout was no longer a single JSON document (or a clean
brief) and `| jq` failed. Both status lines now go to stderr.
2026-09-22 11:58:20 -06:00
trvon
a0844bb6f0
fix(baseline): key every fingerprint on identity, not on its line
A finding with no discriminating key fell back to `line=N`, and three rules
had none: a dormant emitted-not-rendered union carried only {union, owner},
and notifier-field-unread and notifier-inert-consumer only {owner}.
loading-not-reset embedded the catch's absolute line as `catchLine`. Any edit
above one of those findings changed its fingerprint, so `check --baseline`
re-reported an accepted finding as new and `verify` reported its entry as
RESOLVED.

union and owner now count as discriminating (each of those rules emits at most
one finding per union or notifier), and loading-not-reset carries `catchSite`,
the catch's index among the catch clauses of its enclosing member
(`twoSwallowedAwaits#1`), which does not move when code outside the member
does.

The test that guarded this shifted only findings.first. It now shifts every
finding, and a second test really inserts lines at the top of every fixture
file, re-analyses, and requires the same fingerprint set.

Migration: fingerprints change for emitted-not-rendered (dormant unions),
notifier-field-unread, notifier-inert-consumer and loading-not-reset. Old
baselines stay readable and keep working - both `check --baseline` and
`verify` also match the version-1 fingerprint (`legacyFingerprint`) - but
those entries stay line-sensitive until the baseline is rewritten with
--write-baseline, which now writes `"version": 2`.
2026-09-22 11:58:20 -06:00
trvon
9eb3d98d4d
fix(verify): a finding in an excluded file is not scanned, not resolved
The analyzer skips a file that analysis_options `exclude:` names (contextFor
throws, which pipe records as `skipped`). The verify gate refused a scan only
when files FAILED, and partialScanWarning named only failures, so a skipped
file's accepted findings were simply absent from the fresh report - and verify
listed them as RESOLVED, "these fixes are confirmed", for code nobody read.

Excluded files are a normal part of a project, so they do not abort verify.
Instead each accepted finding whose file was not analysed - skipped, failed,
filtered out by --no-include-generated, or outside every scanned --dir - is
reported as NOT SCANNED: neither resolved nor still present. allResolved (and
so --expect-resolved) now requires that list to be empty too.

partialScanWarning counts skipped files, so a scan with excluded files is
announced as PARTIAL on stderr like one with failures.
2026-09-22 11:58:20 -06:00
trvon
cf9e0b7e57
fix(verify): judge only the baselined findings of the rules that ran
`verify --rule X` compared the fresh report against the whole baseline. Every
accepted finding of every other rule was absent from the report, because those
rules never ran, so verify listed them as RESOLVED - "these fixes are
confirmed" for fixes nobody made.

Baseline entries are now filtered to the enabled rules before comparison. An
entry's rule is its `rule` field or, for a fingerprints-only baseline, the
fingerprint's first `|` segment. Entries set aside are counted as `outOfScope`
in JSON and named in the text summary, so the filter is visible rather than
silent.
2026-09-22 11:48:34 -06:00