The rarity-greedy comparator acted on every evaluated window, so it spent
each hour's budget on the first ~15 windows and scored 0 hits in every
cell. Criteria 2 and 3 of the Stage 4 gate were void against it.
rarity_calibrated keeps the same rarity value, chooser, costs and budget,
and adds a per-node threshold frozen before the evaluated span: the m-th
largest value over the trailing 24 h of label-free history between the
warm-up and the --eval-after bound, with m set so expected spend matches
the hourly budget. Without that history the comparator is uncalibrated
and the gate fails by name; it never falls back to another rule.
rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour
ordering, not causal) are reported as diagnostics and never judged.
fleet gate pools counts, keeps thresholds per node, and fails a pooled
cell when any node is uncalibrated or lacks the comparator.
Contract change: in `fleet gate --json`, rarity_hits and
rarity_cpu_seconds now describe the calibrated comparator; the old
numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds.
Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md,
sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95,
indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is
dated after the registration time in that document (2026-11-21T00:00Z);
the YAMS record is what predates it.
Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the
strict analyzer on the changed TUs. Not run: fold on the changed TUs,
make fuzz, the live Docker tests and the perf KPI. The comparator has
not been run on validation-range data.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.
- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
is now enforced (previous commit) and the text says which criterion the tool
applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
is NOT enough (evaluation still replays the whole store), a bounded evaluation
range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
now asserts >= 5, requires the provenance fields, and adds the honest-evidence
invariant: today every cell is in-sample, so at least one cell must report
in_sample true, none may pass, and each must carry the out-of-sample failure.
A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
it collects gate INPUT, and a passing verdict requires out-of-sample windows.
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:
- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
evaluates every model on every equal time slice of the ranked history; each
(model, period) cell simulates the model and rarity-greedy with their own
hourly budgets exactly as policy_eval does (policy_eval.cells). With one
model equal to --model and one period the cell equals the single-model
evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
positives, inspections, CPU-s, hits, refusals) per (model, period) and
applies learn::promotion_gate unchanged to each; passes only if all pass.
Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
cycle now also publishes all three seed models to dist/gate/; strawberry.gate
(sensors) evaluates them with the pre-registered evaluation traces; the
operator-run orch.strawberry_gate pushes the reports back and pools them;
evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
criteria and numbers unchanged).
Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fleet train advertised --json but printed text, so the central cycle's
train-seedN.json evidence was not JSON; it now emits one object incl.
duplicate_windows. fleet overlay gains --json (bundles_superseded,
overlays_skipped). Both commands read only *.sbx (no dotfiles) from the
bundle dir, so a cursor or stray file is never fed to the merge.
scripts/fleet_e2e.sh drives the real binary + central cycle script over N
synthetic sensors: --since auto export chain, Salt-pull simulation, 3-seed
train + overlay, sha256 publish, determinism, overlay distribution and a
round-2 export carrying fleet rarity features. 6/6 checks pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>