The gate's own documentation claimed more than the tool did, in ways that mattered once the integrity checks landed. - "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it is now enforced (previous commit) and the text says which criterion the tool applies: complete evidence, refused false, every pooled cell passing. - The model digest proves agreement ACROSS reports. It cannot prove that the file on disk now is the one that was evaluated. Corrected everywhere. - The remedy was described inconsistently: cutting the training bundles earlier is NOT enough (evaluation still replays the whole store), a bounded evaluation range is required. Written consistently, with --eval-after NS|auto. - scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It now asserts >= 5, requires the provenance fields, and adds the honest-evidence invariant: today every cell is in-sample, so at least one cell must report in_sample true, none may pass, and each must carry the out-of-sample failure. A pass over in-sample cells is itself a failure. - The Salt/range runbooks advertised the flow as producing promotion evidence; it collects gate INPUT, and a passing verdict requires out-of-sample windows.
5.2 KiB
Stage 4: threat-grounded traces, pre-registered
Written 2026-10-28, before any threat trace was collected and before any
model was evaluated on one. It amends the evaluation set of
the promotion gate and adds one criterion; the
gate's existing thresholds do not change. A changed method gets a new dated
document. (Also in YAMS as source=decision, task=strawberry-data-engine.)
Why
The hand-written traces (config/traces/training.trace, standard.trace)
are a few shapes written by the same author. A model trained on 7 of them
ranked held-out copies of the same 7 perfectly (2026-10-28: test AUC 1.000),
which shows it memorized them, not that it generalizes. The question this
experiment answers: does the model steer inspection toward threats that
appeared after everything it learned from?
Design
- Cutoff D = 2025-07-01. A family's date is the date of its first public report (vendor or researcher write-up), not a sample's first-seen date.
- Unit: family or campaign, not sample. Variants and shared loaders
would leak across a per-sample split.
- First reported before D: training (
threats-train.trace), except a dev set (every fifth family in first-report order) used for any tuning. - First reported on or after D: evaluation (
threats-eval.trace). - Eval traces are bucketed: new family (no report of the family before D) and new variant (a family also reported before D). The new-family bucket is the primary result; variants are reported separately.
- First reported before D: training (
- Sources (all committable):
- Public vendor and researcher write-ups, with the URL and publication date recorded. Facts are transcribed; no text is copied.
- Atomic Red Team (MIT) for technique variety, training split only.
- Not used: Triage / Recorded Future and abuse.ch / MalwareBazaar (their terms, checked 2026-10-28, forbid training ML models on their data and redistributing it), VirusTotal, or any sample executed here.
- Inclusion (each trace): Linux; the write-up states an entry exec chain (at least two execs with a parent relation, or one exec with its stated parent such as sshd, a web worker or cron) and at least one network connection or one persistence or drop write.
- Transcription rules, the same for both splits:
- Only events the write-up states. An event that has to be assumed to
connect them (e.g. the shell that ran a stated command) is allowed and
listed under
inferred. - Sandbox artifacts are rewritten to a plausible host entry point and the
rewrite is listed under
normalized. - IPs become RFC 5737 documentation addresses (192.0.2.0/24,
198.51.100.0/24, 203.0.113.0/24), domains
.example; ports stay as reported. No wallets, usernames or keys. - Write labels are the ones
src/file_watch.cppproduces.
- Only events the write-up states. An event that has to be assumed to
connect them (e.g. the shell that ran a stated command) is allowed and
listed under
- Per-trace metadata (
# metalines, checked byscripts/threat_traces.py):family,campaign,first_reported,source_url,source_type(writeup / atomic),attack(ATT&CK technique ids),split(train / dev / eval),bucket(eval only),normalized,inferred,license. - Collection stopping rule: training, up to 40 families; evaluation,
every qualifying family or campaign first reported between D and
2026-10-28 that a search of the listed sources finds, up to 30. The eval
list is collected before any model is evaluated on it and then frozen:
threats-eval.trace's sha256 is recorded in an amendment and in YAMS. It is not edited after the first evaluation (a fix gets a new dated file and the old result stays reported).
Gate amendment
- Injected evaluation positives become
standard.traceplusthreats-eval.trace. Training may usetraining.traceandthreats-train.trace, never either evaluation file. - New criterion 5: on the new-family bucket of
threats-eval.trace, model recall at the budget >= rarity-greedy recall. A model that only learned the shapes it was shown fails here. - Criteria 1-4 and their thresholds are unchanged.
Criterion 5 is a check on threat families, not on evaluation windows: the injected eval traces land
in windows the candidate was trained on unless the evaluation is bounded, so a cell can still fail
the gate's out-of-sample check (FAIL: out-of-sample: in-sample evaluation ..., see
the promotion gate). New families in new time do not make the windows
new. A criterion-5 result is only out-of-sample evidence once replay --policy-eval --eval-after NS|auto restricts the evaluated windows to the model's post-training history: the bound, the
excluded-window logic and the provenance eval_error are in the tree; the CLI flag is landing in
parallel (API_CONTRACT.md).
Known limits
- The traces are transcribed by hand by the same project that builds the model. The rules above (stated events only, logged inferences, frozen eval list) limit, but do not remove, that bias.
- Write-ups describe what made a threat notable; quiet behavior is under-represented. Passive implants (BPFDoor-style) contribute little.
- Injected traces measure targeting of reported attack shapes on this host's background, not detection of attacks that were never reported.