The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.
- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
is now enforced (previous commit) and the text says which criterion the tool
applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
is NOT enough (evaluation still replays the whole store), a bounded evaluation
range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
now asserts >= 5, requires the provenance fields, and adds the honest-evidence
invariant: today every cell is in-sample, so at least one cell must report
in_sample true, none may pass, and each must carry the out-of-sample failure.
A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
it collects gate INPUT, and a passing verdict requires out-of-sample windows.
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.
In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.
Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.
Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.
3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).
Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:
- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
evaluates every model on every equal time slice of the ranked history; each
(model, period) cell simulates the model and rarity-greedy with their own
hourly budgets exactly as policy_eval does (policy_eval.cells). With one
model equal to --model and one period the cell equals the single-model
evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
positives, inspections, CPU-s, hits, refusals) per (model, period) and
applies learn::promotion_gate unchanged to each; passes only if all pass.
Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
cycle now also publishes all three seed models to dist/gate/; strawberry.gate
(sensors) evaluates them with the pre-registered evaluation traces; the
operator-run orch.strawberry_gate pushes the reports back and pools them;
evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
criteria and numbers unchanged).
Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written before any threat trace is collected or evaluated. Train on
families first reported before 2025-07-01, evaluate on families first
reported after it (new-family bucket primary), from public write-ups and
Atomic Red Team only: Triage and MalwareBazaar terms forbid ML training.
Adds gate criterion 5 (model recall >= rarity-greedy on new families).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A promoted model (promoted.sbm, written only by `model promote` after the
gate) can now request real inspections. Only two, both additive:
- memory_scan: through the daemon's memory-scan queue;
- deep_rescan: the root's image re-scanned on demand; new matches are
recorded as indicators of kind model_inspection (findings titled "YARA
match found by a deeper rescan the inspection model requested") and
positive labels.
watch_subtree has no mechanism and rescan_on_next_rules is the retro-hunt's
job, so both are masked out of acting decisions (kExecutableActions), and
replay's policy evaluation uses the same action set.
Guardrails:
- InspectionExecutor (bounded queue, one low-priority worker, submit()
never blocks); WorkClass::model_inspect lets the governor defer it, and
a deferred decision is dropped.
- Each action rechecks that the promoted model is still active at the
decision's generation (demoting or replacing it revokes queued work),
that the root's node id and raw image path still match (decisions carry
root_image, not the feature template), and skips protected processes.
A deep rescan discards its result if the process, image or rules changed
during the scan.
- An acting decision is durably written before it is dispatched; every
attempt becomes an inspection_outcome record (queued, scanned, skipped,
deferred, revoked, queue_full, failed).
- Source invariant R10: no file may couple learning types to the
exec-guard, OpenSnitch or policy APIs (daemon_app.cpp and main.cpp, the
wiring points, are reviewed by hand). Models never block, contain, kill
or create findings directly.
- Config model_act and model_lambda; status model block reports acting
stats; timings.model_inspection.
docs/evaluation/stage4-execution-gate.md amends the gate's method (not its
thresholds): executable action set, greedy promoted decisions, budgets as
estimated inspection seconds by scan profile.
Also: TerminalSafeBuf's destructor no longer lets an exception escape;
the health message clamps huge durations; replay timestamp overflow; the
Docker test build runs with -j2.
Tests: test_inspection_executor (every skip and execute path, revocation,
queue limits); shadow and replay acting cases; R10 fixture; fault injection
and invariant updates. Validation reported by the session that wrote it:
48/48 local, 46/46 Docker plus privileged live checks, 20 fuzz targets
(9.5M runs), three paired KPI runs without regressions (no acting model
was loaded, so active-inspection cost is not measured yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- replay --policy-eval: over ranked windows, simulates the model and
rarity-greedy (the same chooser, rarity score / 48 as the value), each
with its own hourly budget, and tallies the shadow decisions the daemon
logged. Hits are inspected "extra" positives: late labels or injected
evaluation traces on windows rules did not flag at exec time. Labels are
known after the fact, so every action's payoff is computable without
propensity weighting. Reports CPU-seconds, inspections, hits, recall,
hits per CPU-second and budget refusals.
- docs/evaluation/stage4-promotion-gate.md (pre-registered): at least 10
extra positives, model efficiency >= 1.5x rarity-greedy, model recall
not lower. promotion_gate() checks it.
- strawberry model promote [--inject FILE] [--warmup H] [--yes]: replays
with the shadow model, applies the gate and only then writes
promoted.sbm; asks for confirmation.
- learn/inspection.hpp: options, the greedy chooser and the
inspection_action record, shared by shadow mode and the evaluation.
- Fix: the hourly budget reset whenever a window closed with an earlier
timestamp than the hour's start (windows close roughly, not strictly,
in time order), letting spend exceed the budget; in both shadow mode
and the evaluation.
Tests: test_replay R11 (gate criteria; extra positives exclude rule-flagged
windows; spend within the budget; logged shadow decisions joined to their
windows); CLI translations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/learn/trace.cpp: a small trace format (exec, connect, write,
ptrace, module with +offsets and process aliases) compiled to
telemetry records with pids above any pid_max and node ids that name
the trace and copy.
- --replay --inject FILE [--inject-copies N]: copies are spread evenly
over the ranked history (a first pass reads the time range), scored
but never learned from, and checked against alert budgets of 1, 3 and
10 per day: is the copy's best window above the (budget x days)-th best
real window? Ties count against. Injection leaves real windows and
their ranking unchanged.
- config/traces/standard.trace: the six plan traces (browser to curl|sh,
systemd user persistence, memfd loader, ptrace injection, download
then exec, python from a PDF viewer).
- docs/evaluation/stage3-rarity-preregistration.md: method and pass
thresholds, written before any run on host data.
Tests: test_trace (standard traces, error messages with line numbers,
records); test_replay R8 (every copy found, real ranking unchanged, a
novel chain caught at 10/day and a bare shell not). fuzz_trace_dsl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>