With several --dir values, one that did not exist was skipped silently: the
no-files guard only fires when every directory is empty, so
`--dir lib --dir tset` scanned lib and reported as though it had covered what
was asked for. A --dir that does not exist is now an error naming it.
A directory that could not be listed, or a baseline that could not be read or
written, escaped as an uncaught FileSystemException: exit 255 and a stack
trace. Those paths now raise a PipeException with the OS reason (exit 2), and
bin/pipe.dart catches any remaining FileSystemException the same way. The exit
codes are documented in bin/pipe.dart: 0 success, 1 a gate failed, 2 pipe could
not do what was asked, 64 usage.
rule_scope_claim_test widened the fixture with a test/ directory it does not
have, relying on exactly the silent skip; it now builds a project that has one.
The analyzer skips a file that analysis_options `exclude:` names (contextFor
throws, which pipe records as `skipped`). The verify gate refused a scan only
when files FAILED, and partialScanWarning named only failures, so a skipped
file's accepted findings were simply absent from the fresh report - and verify
listed them as RESOLVED, "these fixes are confirmed", for code nobody read.
Excluded files are a normal part of a project, so they do not abort verify.
Instead each accepted finding whose file was not analysed - skipped, failed,
filtered out by --no-include-generated, or outside every scanned --dir - is
reported as NOT SCANNED: neither resolved nor still present. allResolved (and
so --expect-resolved) now requires that list to be empty too.
partialScanWarning counts skipped files, so a scan with excluded files is
announced as PARTIAL on stderr like one with failures.
The previous commit over-corrected. It skipped EVERY case for a repo that had an
unreadable file, so a repo with one unreadable file silently dropped the
regression guard for all of its cases - a worse failure than the false MISSED it
replaced, because a skipped case guards nothing:
$ pipe recall --repo-with-one-unreadable-file
pipe: recall evaluated none of its 2 case(s): ... # both cases lost
Only a NEGATIVE verdict is unsafe when files could not be read. A hit stands,
because evidence that was actually read is evidence; a miss becomes "unknown"
rather than a false regression. So:
recall 1/1 known bugs accounted for (1 skipped)
by rule {dead-handler: 1/1}
skipped: evidence-in-UNREADABLE-item_bloc: 1 file(s) could not be analysed in /repo
The case with readable evidence is still evaluated and still found; only the case
whose evidence lives in the unreadable file is skipped, with the reason.
Also hardened, from the same round's probes: the "analysed nothing" guard tested
only `resolved == 0 && failed > 0`, so a run where every file was SKIPPED
(outside any analysis context) could still report success having analysed
nothing. It is not reachable through --dir (a path that does not resolve under
the root now hits the no-files guard first), but it was open by construction, so
the test is now simply "nothing was resolved".
Probes that came back clean this round, so they are recorded rather than
re-tested: all six commands on a healthy project produce ZERO bytes on stderr and
the same 20 findings (no false positives from the last three fixes); `contexts 0`
is not reachable; and a baseline written from a partially readable project does
warn on stderr at write time, which is the mitigation for the reverse
cross-baseline direction (`new 6 not in the baseline` is then factually correct
rather than a spurious regression claim).
109 tests (was 108), dart analyze clean.
An unreadable file is handed back by the analyzer as a ResolvedUnitResult with
EMPTY content and zero declarations - not an error, not a failure. pipe counted
that as a successful resolution, so a project whose files could not be read
produced:
$ pipe index /tmp/unreadable
files 12/12 resolved (12 discovered, 0 generated, 0 skipped, 0 failed)
$ pipe check /tmp/unreadable
findings 0 exit 0
Verified directly: the same 12 files give `contentLen=0 decls=0` when
unreadable and `contentLen=722 decls=2` when readable, with `failure=null` in
both cases. So `check` reported a clean bill of health for a project it had
extracted nothing from - the same silent-success family as the empty scan and
the truncated scan.
Two changes, both at points where the information exists exactly once:
- the resolution loop detects a unit with empty content for a file that has bytes
on disk (a genuinely empty file is still fine) and records it as a failure, so
`filesFailed` becomes reachable and `index` accounts for it:
`11/12 resolved (12 discovered, 0 generated, 0 skipped, 1 failed)`;
- the outcome summary warns when some files failed and REFUSES when every file
failed, so a partial result can never be mistaken for a complete one:
pipe: WARNING 1 file(s) could not be analysed (11 resolved), so results are PARTIAL.
pipe: no file could be analysed: all 12 file(s) failed to resolve
The normal fixture is byte-for-byte unaffected (no warning, same 20 findings),
so this adds no false positive on readable projects.
Found by probing whether an all-files-fail run was reachable; a chmod-000 corpus
made it reachable and exposed that it had been silently scored as success.
106 tests (was 104), dart analyze clean.
The committed versions were not in canonical format; they were reformatted after
the commit, which would have made the pre-commit hook rewrite them on the next
commit and block it.
Continuing the "where else can pipe report success without doing the work?"
line, which produced the --limit false verification last iteration.
1. The empty-scan guard from 51929e2 had a hole at a different layer. It fires in
loadProject, which counts DISCOVERED files, so a project whose only source is
generated still had one file:
$ pipe check . --no-include-generated
files 0/0 resolved (1 discovered, 0 selected, 1 generated, ...)
findings 0 exit 0
Files were discovered, then every one was filtered out of the selection, and
the analysis ran over nothing. The guard now also fires where selection
happens - the single place a file can be dropped - naming the cause:
"nothing to analyse: all 1 discovered file(s) were excluded by
--no-include-generated".
2. `recall` reported "0/0 known bugs accounted for" and exited 0 - including
under --strict - when it had verified nothing: an empty manifest, or a --root
under which none of the repos exist. "0/0 accounted for" reads as a pass, so
both are now failures that name why:
recall evaluated none of its 5 case(s): no repo was present under /corpus
recall manifest has no cases: recall/manifest.json
The per-case `skipped:` lines are kept; this only changes the case where there
is no result at all to report.
Verified: the two new tests fail without these changes and pass with them; a
healthy recall run still reports 3/5 with its notes, a project with generated
files included still exits 0, and `check --baseline <unrelated>` is still clearly
distinguishable from a clean project.
Also probed and CLEAN, so they are not re-tested blindly: `--budget 1` discloses
omissions in both text and JSON ("All 20 findings omitted ... Raise --budget"),
and a baseline matching nothing prints `findings 20 new=20` against a clean
project's `findings 0 new=0`.
104 tests (was 102), dart analyze clean.
`--limit` is documented as a debugging aid, but a truncated analysis looked
exactly like a complete one, and `verify` interpreted the truncation as work that
had been done:
$ pipe verify . --baseline b.json --limit 2
baseline 20 accepted findings
resolved 19 no longer reported — these fixes are confirmed
still 1 still reported
RESOLVED
19 findings were reported as FIXED because they were never scanned. That is a
false verification of fixes nobody made - the worst output this tool can produce,
because someone could use it to claim a defect was repaired.
Two changes:
- `verify` refuses a partial scan outright: "verify needs a complete scan: with
--limit, findings that were never scanned are reported as resolved", with a
remedy pointing at `check --limit` for debugging.
- Every other command now warns, because a truncated `check --limit 2` reporting
1 finding instead of 20 was equally mistakable for a clean result:
pipe: WARNING --limit 2 is a debugging aid: 2 of 12 file(s) were analysed,
so every result below is PARTIAL.
The warning is emitted at the single place truncation can happen (the file
selection in resolveProject) rather than in each command, so a future command
cannot forget it - forgetting to wire something up is precisely the failure mode
this whole iteration kept finding.
Found by asking whether any other path could report success without doing the
work, after the empty-scan fix. 102 tests (was 100), dart analyze clean.
Analysing zero files exited 0 with no message, so the tool passed while having
checked nothing:
$ pipe check . --dir libs # typo: analysed 0 files, exit 0
$ pipe check . --dir <empty dir> # 0 files, exit 0
That is the same defect class as a test that cannot fail, at the level of the
whole tool: a CI job running `pipe check` "passed" having inspected no code at
all. A scan that finds no Dart files is now an actionable error:
pipe: no Dart files found under libs in /project; pass --dir to choose which
directories to scan
This also covers the nonexistent `--dir` case, which was previously skipped
silently by the directory walk.
Second defect: `--no-include-generated` printed
files 12/12 resolved (13 discovered, 1 generated, 0 skipped, 0 failed)
The 13-versus-12 gap could not be reconciled from the line - it read as though a
file had vanished or failed silently. `filesSelected` existed in the model and in
the JSON all along; the text line simply omitted it. It now prints `12 selected`
whenever it differs from the discovered count, so the arithmetic resolves, and
stays quiet when nothing was filtered.
Found by running the remaining unexercised flags (`--dir`, `--include-generated`)
end to end, as the execution-first pass did for the rest.
100 tests (was 95), dart analyze clean.
