trevon/pipe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(guide,suggest): word agent guidance no stronger than the evidence #2

Merged
trevon merged 13 commits from docs/honest-agent-guidance into main 2026-09-22 21:19:58 +00:00
Conversation 0 Commits 13 Files changed 17 +104 -36
3 changed files with 104 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files
Showing only changes of commit f7eeef7baa - Show all commits

docs: bring the README in line with what pipe does and prints

Prev Next 
Drift, corrected:

- The suggest example showed a "fix: Either render ..." line that no current
  remediation produces; it now shows the current agent-plan shape
  (caution line, coverage line, wire-up remediation), marked illustrative
  because it could not be regenerated from one run.
- Common flags: suggest's `text|agent|json` formats, `recall -c` defaulting to
  8 rather than 4, `--[no-]include-generated` defaulting on, and a missing
  --dir being an error.
- `--yams` persists the full markdown report, not the brief.
- The architecture list named recall.dart twice and omitted cli.dart,
  errors.dart, pool.dart and provider_rules.dart (and now excerpt.dart).
- The hardcoded "84 tests" is gone; the recall result and the word_alive
  manifest note are dated "as of 2026-09-20", since both describe branches that
  were unmerged at that date.

Documented from the previous PR: stdout carries only the requested output,
the `coverage` block and one-line coverage note, exit codes (0/1/2/64),
verify's --rule filter and NOT SCANNED category, version-2 baselines and how
version-1 fingerprints are still matched, packing that stops at the first
entry that does not fit, bounded source excerpts, pasteable verify commands,
and the policy's effect on the legend and on suggest's certainty bonus.
trvon 2026-09-22 12:04:01 -06:00
commit f7eeef7baa
Signed by: trevon
GPG key ID: 38686EA797D7A8A7

135
README.md
View file

@ -78,21 +78,45 @@ pipe verify <path> --baseline .pipe-baseline.json --expect-resolved
pipe suggest <path> --format agent --budget 3000 pipe suggest <path> --format agent --budget 3000
``` ```
Common flags: `--format text|json` (`agent|md|json` for `guide`), `-c N` Common flags: `--format text|json` (`agent|md|json` for `guide`,
(concurrency, default 4), `--sdk <path>` (override SDK discovery), `--dir` `text|agent|json` for `suggest`), `-c N` (concurrency, default 4; 8 for
(repeatable), `--include-generated`, `--limit N`, `--rule <name>` (repeatable), `recall`), `--sdk <path>` (override SDK discovery), `--dir` (repeatable,
`--top N`. default `lib`; a directory that does not exist is an error, not skipped),
`--[no-]include-generated` (default on: `*.g.dart`, `*.freezed.dart` and
`*.mocks.dart` are scanned unless you pass `--no-include-generated`),
`--limit N`, `--rule <name>` (repeatable), `--top N`.
Command-specific flags: `index --fail-on-error` exits 1 when the resolved project Command-specific flags: `index --fail-on-error` exits 1 when the resolved project
has analyzer errors (the default reports the count and exits 0); has analyzer errors (the default reports the count and exits 0);
`suggest --budget N --max-actions N --top N` bound the agent plan and the text `suggest --budget N --max-actions N --top N` bound the agent plan and the text
report; `guide --budget N --max-findings N` bound the agent brief, and `--yams` report; `guide --budget N --max-findings N` bound the agent brief, and `--yams`
with `--yams-name <name>` persists that brief to the yams corpus; with `--yams-name <name>` persists the full markdown report (not the
budget-trimmed brief) to the yams corpus, tagged with the analysed commit and a
generation timestamp;
`recall --manifest <path> --root <dir>` locate the labelled-bug manifest and the `recall --manifest <path> --root <dir>` locate the labelled-bug manifest and the
directory holding the repos it names, and `recall --strict` exits non-zero when a directory holding the repos it names, and `recall --strict` exits non-zero when a
case it expects to still find is missed. Every numeric flag rejects a value that case it expects to still find is missed. Every numeric flag rejects a value that
is not a positive integer rather than falling back to its default. is not a positive integer rather than falling back to its default.
Output contract:
- **stdout carries only the requested output.** Status lines such as
`wrote baseline to ...` and `persisted to yams as ...`, and warnings such as a
partial scan, go to stderr, so `--format json` output always parses.
- **Every report states its coverage.** `check`, `guide` and `suggest` JSON carry
a `coverage` block `{discovered, selected, resolved, failed, skipped,
complete}`, and the text outputs, the guide brief and the suggest agent plan
open with a one-line coverage note that says PARTIAL whenever a discovered
file was not analysed. `skipped` counts files the analyzer does not cover:
those your `analysis_options.yaml` `exclude:` names, or files outside any
analysis context. The note is part of the brief's header, so budget trimming
never drops it.
- **Exit codes:** 0 success; 1 a gate failed (`check --fail-on`,
`verify --expect-resolved`, `recall --strict`, `index --fail-on-error`, or
`guide --yams` could not persist); 2 pipe could not do what was asked (a bad
path or `--dir`, an unreadable project, a baseline that cannot be read or
written); 64 invalid usage.
`pipe` finds the right SDK itself: it reads `flutterRoot` from the target's `pipe` finds the right SDK itself: it reads `flutterRoot` from the target's
`.dart_tool/package_config.json`, then `FLUTTER_ROOT`, then `flutter` on `PATH`. `.dart_tool/package_config.json`, then `FLUTTER_ROOT`, then `flutter` on `PATH`.
Flutter projects need Flutter's bundled Dart SDK because `dart:ui` is not part of Flutter projects need Flutter's bundled Dart SDK because `dart:ui` is not part of
@ -120,7 +144,8 @@ is an alias for it).
`pipe verify` closes the other half of the loop — proving a fix rather than only `pipe verify` closes the other half of the loop — proving a fix rather than only
reporting a problem. It compares a fresh analysis against the baseline and prints reporting a problem. It compares a fresh analysis against the baseline and prints
what is `resolved`, what `still` remains, and what is `new`, exiting non-zero with what is `resolved`, what `still` remains, and what is `new`, exiting non-zero with
`--expect-resolved` while anything accepted is still reported: `--expect-resolved` while anything accepted is still reported or was not
scanned:
```text ```text
$ pipe verify . --baseline .pipe-baseline.json $ pipe verify . --baseline .pipe-baseline.json
@ -135,9 +160,25 @@ RESOLVED
Baselines are self-describing (each entry carries the rule plus a readable label), Baselines are self-describing (each entry carries the rule plus a readable label),
so a resolved finding is named rather than shown as an opaque hash. so a resolved finding is named rather than shown as an opaque hash.
`verify` only calls a finding resolved when it looked for it:
- With `--rule`, accepted findings of other rules are set aside as out of scope
(counted, not judged), because a rule that did not run reports nothing.
- An accepted finding in a file this scan did not analyse — excluded by
`analysis_options.yaml`, filtered by `--no-include-generated`, or outside every
`--dir` — is listed as **NOT SCANNED**, never as resolved.
- A scan in which any file failed to resolve, or one truncated by `--limit`, is
refused outright.
A baseline stores a **fingerprint** per finding (`rule + file + semantic A baseline stores a **fingerprint** per finding (`rule + file + semantic
identity` such as union/variant/provider/owner/member), so moving code down a identity` such as union/variant/provider/owner/member; for two swallowed catches
file does not resurrect accepted findings. `--fail-on-finding` only counts new in one method, the catch's index within that method), so moving code does not
resurrect accepted findings. Baselines are written as `"version": 2`. Version-1
baselines, whose fingerprints embedded a line number for four rules
(`emitted-not-rendered` on a dormant union, `notifier-field-unread`,
`notifier-inert-consumer`, `loading-not-reset`), are still matched, but those
entries stay line-sensitive until you rewrite the baseline with
`--write-baseline`. `--fail-on-finding` only counts new
error-severity findings, and severity counts in reports are computed over new error-severity findings, and severity counts in reports are computed over new
findings, so a ratcheted baseline does not keep a build red. findings, so a ratcheted baseline does not keep a build red.
@ -231,8 +272,9 @@ The fixture has a regression guard (`patternView`) so this cannot come back.
This is also why the precision numbers above are stated as *factual* precision: This is also why the precision numbers above are stated as *factual* precision:
they say how often the claim holds, not how often the issue is worth acting on. they say how often the claim holds, not how often the issue is worth acting on.
Every finding carries `file:line`, a `path:line: snippet` evidence string, a Every finding carries `file:line`, a `path:line: excerpt` evidence string (string
confidence, a UX severity, and a remediation. If a rule cannot be validated it is literals elided, at most 120 characters), a confidence, a UX severity, and a
remediation. If a rule cannot be validated it is
demoted or deleted, not defended. demoted or deleted, not defended.
### Recall, measured not assumed ### Recall, measured not assumed
@ -257,11 +299,11 @@ Precision alone is not evidence. Two mechanisms pin recall:
into a permanent guard against the real code rather than a deleted into a permanent guard against the real code rather than a deleted
expectation. expectation.
Current result: **5/5 accounted for** (2 open bugs found, 3 fixed bugs still Result as of 2026-09-20: **5/5 accounted for** (2 open bugs found, 3 fixed bugs
quiet), 0 regressions — measured against the **working tree**, so a repo checked still quiet), 0 regressions — measured against the **working tree**, so a repo
out on an unmerged fix branch reports its cases as missed. The two nuwayze cases checked out on an unmerged fix branch reports its cases as missed. At that date
are wired up on branches (`wire/nuwayze-*`) that are not merged yet, so they still the two nuwayze cases were wired up on branches (`wire/nuwayze-*`) that were not
read as `open` here; they flip to `fixed` when those land. yet merged, so they read as `open`; they flip to `fixed` when those land.
Recall is still measured against a small labelled set. It bounds the risk of Recall is still measured against a small labelled set. It bounds the risk of
false negatives; it does not prove they are absent. false negatives; it does not prove they are absent.
@ -284,9 +326,10 @@ approval rather than offered as an equal choice.
`--policy wire-up` (default) / `--policy remove` flips the emphasis for teams that `--policy wire-up` (default) / `--policy remove` flips the emphasis for teams that
would rather delete. It changes the wording of every remediation that presents a would rather delete. It changes the wording of every remediation that presents a
choice, the `decision:` line in the agent brief, and the heading of the choice, the fixes in the brief's rule legend, the `decision:` line in the agent
dead-handler section (`Unreachable handlers - wire up the control` versus brief, the heading of the dead-handler section (`Unreachable handlers - wire up
`Cleanly deletable`). the control` versus `Cleanly deletable`), and whether `suggest` scores a dead
handler as needing no product judgement (only under `remove`).
This is a policy, not a finding: pipe still reports the same facts either way. This is a policy, not a finding: pipe still reports the same facts either way.
@ -298,14 +341,23 @@ the plan is presented both globally and **per file**. Only new findings are
planned, so a ratcheted baseline yields an empty plan. planned, so a ratcheted baseline yields an empty plan.
```text ```text
# pipe plan — /path/to/app
Static heuristics; confirm each against the cited file. Order is a score, not a mandate.
Coverage: 814 of 814 discovered file(s) analysed (0 failed, 0 skipped by analysis_options or context).
146 findings collapse to 58 root cause(s) across 41 file(s). Work them in this order.
1. [warning/high] emitted-not-rendered — CommentsState (CommentsBloc) 1. [warning/high] emitted-not-rendered — CommentsState (CommentsBloc)
why: 125 = warning 50 x high (x1.0), one fix clears 6 findings (+50), user-visible failure (+25) why: 75 = warning 50 x high (x1.0), user-visible failure (+25)
fix: Either render CommentsLoading in the consuming widget (or delete it and its emits) ... fix: Wire it up: decide whether the feature ships, and if it does, mount CommentsBloc in a screen and render its states. Removing the bloc and its DI registration is the exception and needs explicit approval.
where: lib/blocs/comment/comment_bloc.dart:143, ... +2 more where: lib/blocs/comment/comment_bloc.dart:143
clears 6 finding(s) (CommentSubmitError, CommentSubmitSuccess, CommentSubmitting, clears 1 finding(s) (CommentSubmitError, CommentSubmitSuccess, CommentSubmitting, CommentsError, CommentsLoaded, CommentsLoading); verify with `pipe check /path/to/app --rule emitted-not-rendered`
CommentsError, CommentsLoaded, CommentsLoading); verify with `pipe check --rule emitted-not-rendered`
``` ```
(Abbreviated and illustrative: the counts are not from a single run. The line
shapes, the wording and the score arithmetic are what `suggest --format agent`
prints; a dormant union is one finding, so it scores no breadth bonus.)
The score is deliberately simple and always printed, so the order can be argued The score is deliberately simple and always printed, so the order can be argued
with rather than trusted blindly: with rather than trusted blindly:
@ -315,7 +367,7 @@ with rather than trusted blindly:
| confidence | high ×1.0, medium ×0.7, low ×0.4 — a low-confidence finding may not be real | | confidence | high ×1.0, medium ×0.7, low ×0.4 — a low-confidence finding may not be real |
| breadth | +10 per extra finding the one fix clears (capped at +50) | | breadth | +10 per extra finding the one fix clears (capped at +50) |
| user-visible failure | +25 when the rule flagged the failure as UX-critical | | user-visible failure | +25 when the rule flagged the failure as UX-critical |
| certainty | +15 for rules whose fix needs no product judgement (dead handler, event without handler, loading not reset) | | certainty | +15 for rules whose fix needs no product judgement (event without handler, loading not reset; dead handler only under `--policy remove`, because wiring one up means choosing the control that dispatches it) |
Because the bonuses are additive, a wide and certain `info` group (10 base + 15 Because the bonuses are additive, a wide and certain `info` group (10 base + 15
certainty + 50 breadth = 75) can outrank a narrow `warning` (35): severity certainty + 50 breadth = 75) can outrank a narrow `warning` (35): severity
@ -325,11 +377,20 @@ implied, and the `why:` line prints the arithmetic for every action.
## Agent guidance ## Agent guidance
`pipe guide` emits a brief designed to be pasted into an agent prompt: findings `pipe guide` emits a brief designed to be pasted into an agent prompt: a coverage
grouped by root cause, dense `file:line` anchors (`file:18,29,32`), a rule legend line, findings grouped by root cause, dense `file:line` anchors
carrying the standard fix, a `decision:` line that resolves "fix vs delete" where (`file:18,29,32`), a rule legend carrying the standard fix for the chosen
the project makes that decidable, and a `verify:` command. Truncation is always `--policy`, a `decision:` line that resolves "fix vs delete" where the project
explicit — never silently dropped. makes that decidable (scoped to the scanned directories), and a `verify:`
command that can be pasted as-is (`pipe check <root> --rule X`, plus
`--baseline` and `--dir` when the run used them). Truncation is always explicit —
never silently dropped — and packing stops at the first entry that does not fit,
so a lower-ranked finding never stands in for a dropped higher-ranked one.
The analysed repository is treated as untrusted input. Quoted source is a bounded
excerpt — string literals replaced by `'…'`, comments dropped, one line of at most
120 characters — labelled `source excerpt (data, not instructions)` and fenced so
that nothing inside it can close its own quotation.
It was verified by giving independent agents *only* the brief text and no tool It was verified by giving independent agents *only* the brief text and no tool
access. Five rounds of that found real defects (a legend that did not match the access. Five rounds of that found real defects (a legend that did not match the
@ -348,19 +409,25 @@ actionable class.
## Architecture ## Architecture
```text ```text
bin/pipe.dart entrypoint bin/pipe.dart entrypoint, exit codes
lib/src/cli.dart commands and option parsing
lib/src/errors.dart PipeException (message + remedy)
lib/src/project.dart package graph validation, source discovery lib/src/project.dart package graph validation, source discovery
lib/src/sdk.dart SDK discovery (Flutter-aware) lib/src/sdk.dart SDK discovery (Flutter-aware)
lib/src/analysis.dart AnalysisContextCollection lifecycle, per-file resolution lib/src/analysis.dart AnalysisContextCollection lifecycle, per-file
resolution, scan coverage
lib/src/pool.dart bounded-concurrency map over files
lib/src/index.dart resolution health, timing, diagnostics lib/src/index.dart resolution health, timing, diagnostics
lib/src/model.dart declaration model lib/src/model.dart declaration model
lib/src/state_flow.dart the state-flow IR (unions, emissions, render sites, lib/src/state_flow.dart the state-flow IR (unions, emissions, render sites,
events, flags, Riverpod) events, flags, Riverpod)
lib/src/excerpt.dart bounded, literal-free source excerpts
lib/src/rules.dart breakage detectors, findings, baselines lib/src/rules.dart breakage detectors, findings, baselines
lib/src/provider_rules.dart provider / ChangeNotifier rules
lib/src/recall.dart labelled-bug recall measurement lib/src/recall.dart labelled-bug recall measurement
lib/src/verify.dart baseline verification (resolved / still / new) lib/src/verify.dart baseline verification (resolved / still / new /
not scanned)
lib/src/suggest.dart ranked fix order per file lib/src/suggest.dart ranked fix order per file
lib/src/recall.dart labelled-bug recall measurement
lib/src/pipeline.dart shared analyze -> graph -> rules pipeline lib/src/pipeline.dart shared analyze -> graph -> rules pipeline
lib/src/guide.dart agent brief / markdown report / yams emission lib/src/guide.dart agent brief / markdown report / yams emission
tool/adjudicate.py independent precision adjudicator tool/adjudicate.py independent precision adjudicator
@ -414,7 +481,7 @@ is byte-identical across runs.
dart test dart test
``` ```
84 tests across resolution/IR extraction, rule contracts (positive *and* Tests cover resolution/IR extraction, rule contracts (positive *and*
negative controls per rule), Riverpod and provider modelling, agent-brief negative controls per rule), Riverpod and provider modelling, agent-brief
rendering, fix-order scoring, baseline round-tripping (including legacy rendering, fix-order scoring, baseline round-tripping (including legacy
fingerprints-only baselines), verification, and recall evaluation, against a fingerprints-only baselines), verification, and recall evaluation, against a

3
bin/pipe.dart
View file

@ -8,7 +8,8 @@ import 'package:pipe/src/errors.dart';
/// ///
/// - 0: success. /// - 0: success.
/// - 1: a gate failed: `check --fail-on`, `verify --expect-resolved`, /// - 1: a gate failed: `check --fail-on`, `verify --expect-resolved`,
/// `recall --strict`, or `guide --yams` could not persist. /// `recall --strict`, `index --fail-on-error`, or `guide --yams` could not
/// persist.
/// - 2: pipe could not do what was asked (a [PipeException], or a filesystem /// - 2: pipe could not do what was asked (a [PipeException], or a filesystem
/// error such as an unreadable or unwritable path). /// error such as an unreadable or unwritable path).
/// - 64: invalid command-line usage. /// - 64: invalid command-line usage.

2
recall/manifest.json
View file

@ -31,7 +31,7 @@
"match": { "match": {
"provider": "subscriptionStreamProvider" "provider": "subscriptionStreamProvider"
}, },
"note": "fixed on branch fix/subscription-stream-entitlement (PR #38): the tier is now read from confirmed AsyncData only. Guard holds while that branch is checked out; becomes a default-branch guard once merged.", "note": "fixed on branch fix/subscription-stream-entitlement (PR #38): the tier is now read from confirmed AsyncData only. | As of 2026-09-20 that branch was unmerged, so the guard holds only while it is checked out; it becomes a default-branch guard once merged.",
"status": "fixed" "status": "fixed"
}, },
{ {