The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.
- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
is now enforced (previous commit) and the text says which criterion the tool
applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
is NOT enough (evaluation still replays the whole store), a bounded evaluation
range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
now asserts >= 5, requires the provenance fields, and adds the honest-evidence
invariant: today every cell is in-sample, so at least one cell must report
in_sample true, none may pass, and each must carry the out-of-sample failure.
A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
it collects gate INPUT, and a passing verdict requires out-of-sample windows.
The in-sample check landed without any way to bound an evaluation, so every cell
was in-sample and every promotion path refused. Cutting the training bundles
earlier does not help: training spans the bundles, evaluation replays the whole
store with no lower bound, and the store keeps everything (retention 0), so
eval_first <= training_last always held.
replay --policy-eval gains --eval-after NS|auto (0 = unbounded). The bound is
applied where windows are ranked and where injected traces are planned, so the
periods partition only the out-of-sample span; auto derives it as the latest
training-range end over the loaded --eval-model files. Comparisons are written
without first + warm-up arithmetic so INT64_MAX timestamps cannot overflow. A
bound past the end of the store does not produce a silent empty cell: the cell
carries eval_error and promotion_gate refuses naming it.
Provenance also gets honest about width: training_last_ns is the last training
window's END (a window spans window_ns, so using its start left up to 60 s of
trained-on data looking disjoint), and the single-model path (model promote) now
uses the ranked span instead of the whole store, which had guaranteed a refusal.
Verified: R11c passes with 128 out-of-sample windows and a derived bound, and
unbounded returns to in-sample. Refuses an explicit bound later than all history.
- scripts/threat_traces.py: enforces the mechanical rules of the 2026-10-28
threat-trace pre-registration: required # meta keys; eval families first
reported 2025-07-01..2026-10-28 with a new-family/new-variant bucket and
write-up sources only; training families before the cutoff with every
fifth family (first-report order) in dev; no family in both files;
RFC 5737 addresses and .example domains only; size caps; and that every
file compiles with replay's parser. --sha256 prints the freeze digest.
- tests/trace_check.cpp: strawberry-trace-check dev tool (replay's parser).
- strawberry.gate: optional gate.warmup_hours (unset = the pre-registered
12 h; TEST pillars only).
- deploy/salt/test/fleet_run.sh: gate phase - tops sensor history past the
10 min injection floor, runs orch.strawberry_gate, asserts 3 gate models
were published, 9 pooled cells over 3 sensors with positives, sensors keep
their reports, and still no promoted model anywhere.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:
- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
evaluates every model on every equal time slice of the ranked history; each
(model, period) cell simulates the model and rarity-greedy with their own
hourly budgets exactly as policy_eval does (policy_eval.cells). With one
model equal to --model and one period the cell equals the single-model
evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
positives, inspections, CPU-s, hits, refusals) per (model, period) and
applies learn::promotion_gate unchanged to each; passes only if all pass.
Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
cycle now also publishes all three seed models to dist/gate/; strawberry.gate
(sensors) evaluates them with the pre-registered evaluation traces; the
operator-run orch.strawberry_gate pushes the reports back and pools them;
evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
criteria and numbers unchanged).
Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-1 bundles are not a fair baseline: the 120 s cursor overlap is a large
share of a few-minute test history. Verified on the live fleet: incremental
192/206/194 windows vs full 230/244/232.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deploy/salt/test/: builds a release tarball from HEAD in ubuntu:24.04 (CI path),
boots systemd containers (rootless, no --privileged), master==central + 3
sensors with the host's Salt onedir mounted read-only, applies the real states,
runs a workload with planted YARA detections, then two orchestrated cycles.
43 checks (install/config/idempotence, real daemons + telemetry, R2 cache
miss/hit, cp.push collect, 3-seed cycle, canary hold then fleet rollout, no
promoted model anywhere, fleet bits in round 2, --since cursor, dedup).
State fixes found by running them for real:
- init.sls no longer includes strawberry.distribute: a plain state.apply on
sensors shipped models fleet-wide and bypassed the canary; only the
orchestration distributes now.
- install.sls: CI tarballs carry a +g<sha> suffix that 'strawberry version'
does not print, so install.sh re-ran on every apply; record the installed
release and compare base version + marker.
- orch collect: mv -f (re-collected bundles must not be skipped).
- central cycle: builds the overlay first; a training refusal for too few
teacher labels (normal on a young fleet) publishes the overlay only and
keeps the sensors' shadow, any other error still fails.
- distribute.sls: serves only what the cycle published (cp.hash_file; the
cp.list_master file-list cache can miss a just-published file).
Render check gains overlay-only and full publish cases (20 checks).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fleet train advertised --json but printed text, so the central cycle's
train-seedN.json evidence was not JSON; it now emits one object incl.
duplicate_windows. fleet overlay gains --json (bundles_superseded,
overlays_skipped). Both commands read only *.sbx (no dotfiles) from the
bundle dir, so a cursor or stray file is never fed to the merge.
scripts/fleet_e2e.sh drives the real binary + central cycle script over N
synthetic sensors: --since auto export chain, Salt-pull simulation, 3-seed
train + overlay, sha256 publish, determinism, overlay distribution and a
round-2 export carrying fleet rarity features. 6/6 checks pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pillar data (deploy/salt/pillar/) + README for the deploying agent: hub-and-spoke
(sensor export -> central fleet train + overlay -> distribute), role assignment,
CLI surface, security invariants, and the honest gaps (no `fleet overlay` CLI yet,
node_id not wired, no multi-node/docker-swarm validation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>