Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Written before any threat trace is collected or evaluated. Train on families first reported before 2025-07-01, evaluate on families first reported after it (new-family bucket primary), from public write-ups and Atomic Red Team only: Triage and MalwareBazaar terms forbid ML training. Adds gate criterion 5 (model recall >= rarity-greedy on new families). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.5 KiB
4.5 KiB
Stage 4: threat-grounded traces, pre-registered
Written 2026-10-28, before any threat trace was collected and before any
model was evaluated on one. It amends the evaluation set of
the promotion gate and adds one criterion; the
gate's existing thresholds do not change. A changed method gets a new dated
document. (Also in YAMS as source=decision, task=strawberry-data-engine.)
Why
The hand-written traces (config/traces/training.trace, standard.trace)
are a few shapes written by the same author. A model trained on 7 of them
ranked held-out copies of the same 7 perfectly (2026-10-28: test AUC 1.000),
which shows it memorized them, not that it generalizes. The question this
experiment answers: does the model steer inspection toward threats that
appeared after everything it learned from?
Design
- Cutoff D = 2025-07-01. A family's date is the date of its first public report (vendor or researcher write-up), not a sample's first-seen date.
- Unit: family or campaign, not sample. Variants and shared loaders
would leak across a per-sample split.
- First reported before D: training (
threats-train.trace), except a dev set (every fifth family in first-report order) used for any tuning. - First reported on or after D: evaluation (
threats-eval.trace). - Eval traces are bucketed: new family (no report of the family before D) and new variant (a family also reported before D). The new-family bucket is the primary result; variants are reported separately.
- First reported before D: training (
- Sources (all committable):
- Public vendor and researcher write-ups, with the URL and publication date recorded. Facts are transcribed; no text is copied.
- Atomic Red Team (MIT) for technique variety, training split only.
- Not used: Triage / Recorded Future and abuse.ch / MalwareBazaar (their terms, checked 2026-10-28, forbid training ML models on their data and redistributing it), VirusTotal, or any sample executed here.
- Inclusion (each trace): Linux; the write-up states an entry exec chain (at least two execs with a parent relation, or one exec with its stated parent such as sshd, a web worker or cron) and at least one network connection or one persistence or drop write.
- Transcription rules, the same for both splits:
- Only events the write-up states. An event that has to be assumed to
connect them (e.g. the shell that ran a stated command) is allowed and
listed under
inferred. - Sandbox artifacts are rewritten to a plausible host entry point and the
rewrite is listed under
normalized. - IPs become RFC 5737 documentation addresses (192.0.2.0/24,
198.51.100.0/24, 203.0.113.0/24), domains
.example; ports stay as reported. No wallets, usernames or keys. - Write labels are the ones
src/file_watch.cppproduces.
- Only events the write-up states. An event that has to be assumed to
connect them (e.g. the shell that ran a stated command) is allowed and
listed under
- Per-trace metadata (
# metalines, checked byscripts/threat_traces.py):family,campaign,first_reported,source_url,source_type(writeup / atomic),attack(ATT&CK technique ids),split(train / dev / eval),bucket(eval only),normalized,inferred,license. - Collection stopping rule: training, up to 40 families; evaluation,
every qualifying family or campaign first reported between D and
2026-10-28 that a search of the listed sources finds, up to 30. The eval
list is collected before any model is evaluated on it and then frozen:
threats-eval.trace's sha256 is recorded in an amendment and in YAMS. It is not edited after the first evaluation (a fix gets a new dated file and the old result stays reported).
Gate amendment
- Injected evaluation positives become
standard.traceplusthreats-eval.trace. Training may usetraining.traceandthreats-train.trace, never either evaluation file. - New criterion 5: on the new-family bucket of
threats-eval.trace, model recall at the budget >= rarity-greedy recall. A model that only learned the shapes it was shown fails here. - Criteria 1-4 and their thresholds are unchanged.
Known limits
- The traces are transcribed by hand by the same project that builds the model. The rules above (stated events only, logged inferences, frozen eval list) limit, but do not remove, that bias.
- Write-ups describe what made a threat notable; quiet behavior is under-represented. Passive implants (BPFDoor-style) contribute little.
- Injected traces measure targeting of reported attack shapes on this host's background, not detection of attacks that were never reported.