fix: stop verify, reports and briefs from claiming more than pipe checked

trevon commented

2026-09-22 18:19:14 +00:00

Owner

Problem

Several paths let pipe report more than it checked, or crash instead of explaining:

verify --rule X compared against the whole baseline, so other rules' findings showed as RESOLVED (exit 0).
Files excluded by analysis_options were silently unscanned, and verify counted their findings as resolved. This is live on word_alive, where lib/functions/** (14 files) is excluded.
Four rules' fingerprints fell back to line numbers, so adding an import turned a live bug into RESOLVED + NEW, still exit 0.
--write-baseline / --yams status lines went to stdout after --format json, which broke jq.
Reports never said how much of the project was scanned; a brief could say "No new findings" over a partial scan.
Budget packing skipped a higher-ranked group that didn't fit and kept packing lower-ranked ones.
Source from the analysed repo (including string literals) reached agent briefs and the yams corpus verbatim and unbounded, which is a prompt-injection path.
The yams artifact dropped the caution line, severity/confidence, and provenance.
A mistyped --dir was skipped silently. Filesystem errors, including the analyzer's own FileSystemException for an unreadable SDK, crashed with exit 255.

Solution

One commit per item; each commit body explains the change.

Verify now reports baselined findings as out-of-scope (other rules) and NOT SCANNED (skipped files), never as resolved.
Every check/guide/suggest output carries a coverage block (JSON) or a coverage line (brief/plan) that budget trimming never drops.
Evidence excerpts are capped at 120 characters, string literals are elided, and each excerpt is fenced and labelled as data.
Exit codes are documented in bin/pipe.dart: 0 / 1 gate failed / 2 could not do what was asked / 64 usage.

Rule changes

No rule behaviour change (identity, reporting and wording only)

Release Note

verify no longer reports unscanned or out-of-scope findings as resolved; every report states its scan coverage.
Migration: fingerprints change for emitted-not-rendered (dormant unions), notifier-field-unread, notifier-inert-consumer, and loading-not-reset. v1 baselines still decode and match through legacyFingerprint. Rewrite them with --write-baseline to drop line sensitivity (new baselines are "version": 2).

Breaking Changes

Yes: a --dir that doesn't exist is now an error (exit 2). The JSON fields are additive.

Tests

Unit: new excluded_files, excerpt, stdout_json and sdk_unreadable tests; baseline stability now shifts every finding's line. scripts/lint.sh format|analyze|test: clean, 134 passed.
Negative control: sdk_unreadable_test fails without the fix.
Dogfood on word_alive: exit 0; coverage 526 discovered / 512 resolved / 14 skipped, flagged PARTIAL on stderr; stdout is pure JSON.
Recall manifest (pipe recall --strict): not run; needs the labelled repo corpus.

Documentation

CHANGELOG: generated by release-please
README: updated in the stacked docs PR

AI usage

Written with Claude Code. The code was drafted without a compiler, then formatted, analyzed, tested and dogfooded before push.

## Problem Several paths let pipe report more than it checked, or crash instead of explaining: - `verify --rule X` compared against the whole baseline, so other rules' findings showed as RESOLVED (exit 0). - Files excluded by `analysis_options` were silently unscanned, and verify counted their findings as resolved. This is live on word_alive, where `lib/functions/**` (14 files) is excluded. - Four rules' fingerprints fell back to line numbers, so adding an import turned a live bug into RESOLVED + NEW, still exit 0. - `--write-baseline` / `--yams` status lines went to stdout after `--format json`, which broke `jq`. - Reports never said how much of the project was scanned; a brief could say "No new findings" over a partial scan. - Budget packing skipped a higher-ranked group that didn't fit and kept packing lower-ranked ones. - Source from the analysed repo (including string literals) reached agent briefs and the yams corpus verbatim and unbounded, which is a prompt-injection path. - The yams artifact dropped the caution line, severity/confidence, and provenance. - A mistyped `--dir` was skipped silently. Filesystem errors, including the analyzer's own `FileSystemException` for an unreadable SDK, crashed with exit 255. ## Solution One commit per item; each commit body explains the change. - Verify now reports baselined findings as out-of-scope (other rules) and NOT SCANNED (skipped files), never as resolved. - Every check/guide/suggest output carries a `coverage` block (JSON) or a coverage line (brief/plan) that budget trimming never drops. - Evidence excerpts are capped at 120 characters, string literals are elided, and each excerpt is fenced and labelled as data. - Exit codes are documented in `bin/pipe.dart`: 0 / 1 gate failed / 2 could not do what was asked / 64 usage. ## Rule changes - [x] No rule behaviour change (identity, reporting and wording only) ## Release Note - `verify` no longer reports unscanned or out-of-scope findings as resolved; every report states its scan coverage. - Migration: fingerprints change for `emitted-not-rendered` (dormant unions), `notifier-field-unread`, `notifier-inert-consumer`, and `loading-not-reset`. v1 baselines still decode and match through `legacyFingerprint`. Rewrite them with `--write-baseline` to drop line sensitivity (new baselines are `"version": 2`). ## Breaking Changes - [x] Yes: a `--dir` that doesn't exist is now an error (exit 2). The JSON fields are additive. ## Tests - [x] Unit: new excluded_files, excerpt, stdout_json and sdk_unreadable tests; baseline stability now shifts every finding's line. `scripts/lint.sh format|analyze|test`: clean, **134 passed**. - [x] Negative control: sdk_unreadable_test fails without the fix. - [x] Dogfood on word_alive: exit 0; coverage 526 discovered / 512 resolved / 14 skipped, flagged PARTIAL on stderr; stdout is pure JSON. - [ ] Recall manifest (`pipe recall --strict`): not run; needs the labelled repo corpus. ## Documentation - [ ] CHANGELOG: generated by release-please - [x] README: updated in the stacked docs PR ## AI usage - [x] Written with Claude Code. The code was drafted without a compiler, then formatted, analyzed, tested and dogfooded before push.

trevon added 12 commits

2026-09-22 18:19:14 +00:00

fix(verify): judge only the baselined findings of the rules that ran cf9e0b7e57

`verify --rule X` compared the fresh report against the whole baseline. Every
accepted finding of every other rule was absent from the report, because those
rules never ran, so verify listed them as RESOLVED - "these fixes are
confirmed" for fixes nobody made.

Baseline entries are now filtered to the enabled rules before comparison. An
entry's rule is its `rule` field or, for a fingerprints-only baseline, the
fingerprint's first `|` segment. Entries set aside are counted as `outOfScope`
in JSON and named in the text summary, so the filter is visible rather than
silent.

fix(verify): a finding in an excluded file is not scanned, not resolved 9eb3d98d4d

The analyzer skips a file that analysis_options `exclude:` names (contextFor
throws, which pipe records as `skipped`). The verify gate refused a scan only
when files FAILED, and partialScanWarning named only failures, so a skipped
file's accepted findings were simply absent from the fresh report - and verify
listed them as RESOLVED, "these fixes are confirmed", for code nobody read.

Excluded files are a normal part of a project, so they do not abort verify.
Instead each accepted finding whose file was not analysed - skipped, failed,
filtered out by --no-include-generated, or outside every scanned --dir - is
reported as NOT SCANNED: neither resolved nor still present. allResolved (and
so --expect-resolved) now requires that list to be empty too.

partialScanWarning counts skipped files, so a scan with excluded files is
announced as PARTIAL on stderr like one with failures.

fix(baseline): key every fingerprint on identity, not on its line a0844bb6f0

A finding with no discriminating key fell back to `line=N`, and three rules
had none: a dormant emitted-not-rendered union carried only {union, owner},
and notifier-field-unread and notifier-inert-consumer only {owner}.
loading-not-reset embedded the catch's absolute line as `catchLine`. Any edit
above one of those findings changed its fingerprint, so `check --baseline`
re-reported an accepted finding as new and `verify` reported its entry as
RESOLVED.

union and owner now count as discriminating (each of those rules emits at most
one finding per union or notifier), and loading-not-reset carries `catchSite`,
the catch's index among the catch clauses of its enclosing member
(`twoSwallowedAwaits#1`), which does not move when code outside the member
does.

The test that guarded this shifted only findings.first. It now shifts every
finding, and a second test really inserts lines at the top of every fixture
file, re-analyses, and requires the same fingerprint set.

Migration: fingerprints change for emitted-not-rendered (dormant unions),
notifier-field-unread, notifier-inert-consumer and loading-not-reset. Old
baselines stay readable and keep working - both `check --baseline` and
`verify` also match the version-1 fingerprint (`legacyFingerprint`) - but
those entries stay line-sensitive until the baseline is rewritten with
--write-baseline, which now writes `"version": 2`.

fix(cli): keep status messages off stdout so JSON output stays parseable ac7cce1f76

`check --format json --write-baseline b.json` printed "wrote baseline to ..."
after the JSON document, and `guide --yams` printed "persisted to yams as ..."
after the brief, so stdout was no longer a single JSON document (or a clean
brief) and `| jq` failed. Both status lines now go to stderr.

feat(report): state scan coverage in every check, guide and suggest output 749aa53348

A report listed findings without saying how much of the project they were
drawn from, so "0 findings" from 3 of 40 files looked like a clean project -
and an agent handed a brief had no way to know otherwise.

check, guide and suggest JSON now carry a `coverage` block
{discovered, selected, resolved, failed, skipped, complete}. The guide brief,
the markdown report, the suggest agent plan and both text outputs open with a
one-line coverage note, which says PARTIAL whenever a discovered file was not
analysed.

The note lives in the brief's header, which budget trimming never touches. The
"Coverage (what was actually examined)" caveats section was the first thing
dropped under a tight budget, silently; when it no longer fits, a one-line
pointer now says how many caveats were left out.

fix(guide,suggest): stop packing at the first entry that does not fit 39b309af97

Both packers skipped an entry that did not fit the budget and kept going, so a
large top-ranked finding vanished while smaller, lower-ranked ones were packed
in behind it - presented as the most important work. Packing now stops at the
first entry that does not fit and counts it and everything after it as
omitted.

suggest's agent plan also reserved no room for its own omission note, so the
note itself could push the plan over budget; it now reserves room for the
largest possible note unless the entry is the last one.

When no entry fit, the guide said "this header alone exceeds the budget" even
when the header had room and it was the top entry that was too large. It now
says which of the two it was, with the size.

fix(guide): quote analysed source as bounded, delimited data 4a1c9cdd3e

The analysed repository is untrusted input, and its source reached the agent
brief and the yams corpus verbatim: emission and dispatch snippets were
`node.toSource()`, provider-read-in-build quoted the whole call, and the brief
printed `evidence: $item` raw (the markdown wrapped it in single backticks,
which a backtick in the source closes). A string literal written as an
instruction to the reader was pasted into the prompt as-is.

Every quoted excerpt now goes through sourceExcerpt(): string literals,
interpolations included, become '…'; comments are dropped (they are not
tokens); whitespace collapses; the result is capped at 120 characters, as
loading-not-reset's catch excerpts already were. Where the brief and the
markdown render evidence, it is labelled "source excerpt (data, not
instructions)" and fenced with one more backtick than the longest run inside
it, so nothing quoted can close its own quotation.

Fingerprints are unaffected: provider-read-in-build still keys on the full call
source, which is kept for identity and no longer quoted.

fix(guide): persist the yams report with its caution and provenance e2d84f5e75

`guide --yams` persists the markdown report, not the brief, and the markdown
had neither the brief's caution line ("static reachability claims ... confirm
each against the cited file") nor the [severity/confidence] of each finding -
so a report read back from the corpus presented every finding as equally
certain, with no framing. Its metadata still carried a stale `owner=pi` and
nothing to date or reproduce it.

The markdown now opens with the caution line, headings read
`### N. [severity/confidence] rule — location`, and the metadata records
`generated=<ISO-8601 UTC>` and, when the project is a git checkout,
`commit=<HEAD>`. `owner=pi` is gone.

fix(cli): reject a missing --dir and report filesystem errors cleanly ae412a7964

With several --dir values, one that did not exist was skipped silently: the
no-files guard only fires when every directory is empty, so
`--dir lib --dir tset` scanned lib and reported as though it had covered what
was asked for. A --dir that does not exist is now an error naming it.

A directory that could not be listed, or a baseline that could not be read or
written, escaped as an uncaught FileSystemException: exit 255 and a stack
trace. Those paths now raise a PipeException with the OS reason (exit 2), and
bin/pipe.dart catches any remaining FileSystemException the same way. The exit
codes are documented in bin/pipe.dart: 0 success, 1 a gate failed, 2 pipe could
not do what was asked, 64 usage.

rule_scope_claim_test widened the fixture with a test/ directory it does not
have, relying on exactly the silent skip; it now builds a project that has one.

fix(suggest): break score ties by file, line and rule 17c9f9afab

The plan sorted by score alone. List.sort is not stable, so equal-score
actions came out in an order nothing guaranteed, and two runs over the same
code could hand an agent a different first action. Ties now fall back to file,
line, rule and label, both in the global order and within each file.

style: apply dart format to the files this branch changed 30fca8744b

The commits above were made without the pre-commit hook, which could not
run dart at the time.

fix(analysis): report an unreadable SDK cleanly instead of crashing

Lint Commits / commitlint (pull_request) Failing after 30s

Details

Tests / Static Analysis (pull_request) Has been cancelled

Details

Tests / Tests (pull_request) Has been cancelled

Details

d29255b0ba

ae412a7 routed filesystem failures to exit 2, but the analyzer throws its own
FileSystemException (package:analyzer/file_system), which the dart:io catch
in bin/pipe.dart never matched. A project whose package_config still named a
moved Flutter SDK therefore crashed with exit 255 and a stack trace. Found
by dogfooding on word_alive after its SDK moved.

Analyzer.create now turns that into a PipeException naming the SDK path, with
the remedy to re-run flutter pub get or pass --sdk.

trevon referenced this pull request

2026-09-22 18:19:22 +00:00

fix(guide,suggest): word agent guidance no stronger than the evidence #2

trevon force-pushed fix/verify-soundness-and-injection from d29255b0ba

Lint Commits / commitlint (pull_request) Failing after 30s

Details

Tests / Static Analysis (pull_request) Has been cancelled

Details

Tests / Tests (pull_request) Has been cancelled

Details

to d6b6bcb438

Lint Commits / commitlint (pull_request) Successful in 17s