strawberry/docs/evaluation/stage4-threat-traces-preregistration.md
trvon 085e8bdd2d
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Stage 4: pre-register threat-grounded traces (cutoff 2025-07-01, family split)
Written before any threat trace is collected or evaluated. Train on
families first reported before 2025-07-01, evaluate on families first
reported after it (new-family bucket primary), from public write-ups and
Atomic Red Team only: Triage and MalwareBazaar terms forbid ML training.
Adds gate criterion 5 (model recall >= rarity-greedy on new families).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 05:05:30 -06:00

4.5 KiB

Stage 4: threat-grounded traces, pre-registered

Written 2026-10-28, before any threat trace was collected and before any model was evaluated on one. It amends the evaluation set of the promotion gate and adds one criterion; the gate's existing thresholds do not change. A changed method gets a new dated document. (Also in YAMS as source=decision, task=strawberry-data-engine.)

Why

The hand-written traces (config/traces/training.trace, standard.trace) are a few shapes written by the same author. A model trained on 7 of them ranked held-out copies of the same 7 perfectly (2026-10-28: test AUC 1.000), which shows it memorized them, not that it generalizes. The question this experiment answers: does the model steer inspection toward threats that appeared after everything it learned from?

Design

  • Cutoff D = 2025-07-01. A family's date is the date of its first public report (vendor or researcher write-up), not a sample's first-seen date.
  • Unit: family or campaign, not sample. Variants and shared loaders would leak across a per-sample split.
    • First reported before D: training (threats-train.trace), except a dev set (every fifth family in first-report order) used for any tuning.
    • First reported on or after D: evaluation (threats-eval.trace).
    • Eval traces are bucketed: new family (no report of the family before D) and new variant (a family also reported before D). The new-family bucket is the primary result; variants are reported separately.
  • Sources (all committable):
    • Public vendor and researcher write-ups, with the URL and publication date recorded. Facts are transcribed; no text is copied.
    • Atomic Red Team (MIT) for technique variety, training split only.
    • Not used: Triage / Recorded Future and abuse.ch / MalwareBazaar (their terms, checked 2026-10-28, forbid training ML models on their data and redistributing it), VirusTotal, or any sample executed here.
  • Inclusion (each trace): Linux; the write-up states an entry exec chain (at least two execs with a parent relation, or one exec with its stated parent such as sshd, a web worker or cron) and at least one network connection or one persistence or drop write.
  • Transcription rules, the same for both splits:
    • Only events the write-up states. An event that has to be assumed to connect them (e.g. the shell that ran a stated command) is allowed and listed under inferred.
    • Sandbox artifacts are rewritten to a plausible host entry point and the rewrite is listed under normalized.
    • IPs become RFC 5737 documentation addresses (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24), domains .example; ports stay as reported. No wallets, usernames or keys.
    • Write labels are the ones src/file_watch.cpp produces.
  • Per-trace metadata (# meta lines, checked by scripts/threat_traces.py): family, campaign, first_reported, source_url, source_type (writeup / atomic), attack (ATT&CK technique ids), split (train / dev / eval), bucket (eval only), normalized, inferred, license.
  • Collection stopping rule: training, up to 40 families; evaluation, every qualifying family or campaign first reported between D and 2026-10-28 that a search of the listed sources finds, up to 30. The eval list is collected before any model is evaluated on it and then frozen: threats-eval.trace's sha256 is recorded in an amendment and in YAMS. It is not edited after the first evaluation (a fix gets a new dated file and the old result stays reported).

Gate amendment

  • Injected evaluation positives become standard.trace plus threats-eval.trace. Training may use training.trace and threats-train.trace, never either evaluation file.
  • New criterion 5: on the new-family bucket of threats-eval.trace, model recall at the budget >= rarity-greedy recall. A model that only learned the shapes it was shown fails here.
  • Criteria 1-4 and their thresholds are unchanged.

Known limits

  • The traces are transcribed by hand by the same project that builds the model. The rules above (stated events only, logged inferences, frozen eval list) limit, but do not remove, that bias.
  • Write-ups describe what made a threat notable; quiet behavior is under-represented. Passive implants (BPFDoor-style) contribute little.
  • Injected traces measure targeting of reported attack shapes on this host's background, not detection of attacks that were never reported.