Two more ways a partial scan produced a false claim, the same hazard class
as the --limit and unreadable-file cases verify already refuses:
- `verify --rule X` compared the fresh X-only report against the whole
baseline, so every accepted finding from any other rule was printed as
RESOLVED ("these fixes are confirmed") when it had merely not been
checked. The baseline is now filtered to the selected rules.
- `--write-baseline` (check and guide) wrote whatever the run produced.
Under --rule or --limit, or with files that could not be analysed, the
baseline silently omitted every finding outside that subset, and the
next full `check --baseline` reported them all as new. It now refuses:
--rule/--limit before analysis, unreadable files after it.
Each is covered by a test that fails against the previous cli.dart
(subset_scan_test.dart, and the unreadable-file case in
unreadable_test.dart). README documents the refusal.
The previous commit over-corrected. It skipped EVERY case for a repo that had an
unreadable file, so a repo with one unreadable file silently dropped the
regression guard for all of its cases - a worse failure than the false MISSED it
replaced, because a skipped case guards nothing:
$ pipe recall --repo-with-one-unreadable-file
pipe: recall evaluated none of its 2 case(s): ... # both cases lost
Only a NEGATIVE verdict is unsafe when files could not be read. A hit stands,
because evidence that was actually read is evidence; a miss becomes "unknown"
rather than a false regression. So:
recall 1/1 known bugs accounted for (1 skipped)
by rule {dead-handler: 1/1}
skipped: evidence-in-UNREADABLE-item_bloc: 1 file(s) could not be analysed in /repo
The case with readable evidence is still evaluated and still found; only the case
whose evidence lives in the unreadable file is skipped, with the reason.
Also hardened, from the same round's probes: the "analysed nothing" guard tested
only `resolved == 0 && failed > 0`, so a run where every file was SKIPPED
(outside any analysis context) could still report success having analysed
nothing. It is not reachable through --dir (a path that does not resolve under
the root now hits the no-files guard first), but it was open by construction, so
the test is now simply "nothing was resolved".
Probes that came back clean this round, so they are recorded rather than
re-tested: all six commands on a healthy project produce ZERO bytes on stderr and
the same 20 findings (no false positives from the last three fixes); `contexts 0`
is not reachable; and a baseline written from a partially readable project does
warn on stderr at write time, which is the mitigation for the reverse
cross-baseline direction (`new 6 not in the baseline` is then factually correct
rather than a spurious regression claim).
109 tests (was 108), dart analyze clean.
The all-unreadable case was fixed last commit; the PARTIAL case still produced
two false claims.
1. `verify` reported unread findings as fixes. With two unreadable files it
printed:
baseline 20 accepted findings
resolved 8 no longer reported — these fixes are confirmed
RESOLVED
+ lib/item_bloc.dart:48 LoadItems [dead-handler]
item_bloc.dart was one of the files it could not read. Those 8 findings were
absent from the fresh report because they were never read, not because they
were fixed. `verify` now refuses a scan in which any file failed, mirroring
the --limit guard: a claim about someone's fixes must rest on a complete scan.
2. `recall` reported such a case as MISSED, a false regression claim that also
drove `--strict` to exit non-zero. A case whose evidence lives in a file that
could not be read cannot be judged either way, so it is now recorded as
skipped with the reason, exactly like a repo that is not present:
"1 file(s) could not be analysed in /repo".
3. Consequential accuracy fix: the vacuous-run guard added two commits ago said
"no repo was present under <root>" whenever nothing was evaluated. With the
new skip reason that diagnosis was wrong - the repo WAS present, its files
were unreadable - and the remedy pointed at the wrong problem. The message now
reports the actual skip reasons.
Verified: verify exits 2 on a partially readable project and still works on a
fully readable one; recall skips the unreadable case, keeps evaluating the
others, and reports no MISSED. The normal fixture is unaffected. 108 tests (was
106); the new tests fail without these changes (+0 -2) and skip honestly if
chmod cannot restrict the process.
Found by trying to FALSIFY the claim that the silent-success family was closed.
It was not - for the second iteration running, the falsification attempt found a
real defect.
