# Stage 3 rarity baseline: pre-registered evaluation Written 2026-10-26, before any evaluation run on host data. The thresholds below do not change after results are seen; a changed method gets a new dated document. (Also recorded in YAMS as `source=decision`, `task=strawberry-data-engine`.) ## Question Does the rarity baseline (`src/learn/rarity.cpp`, windows from `src/learn/replay.cpp`) put realistic attack activity in front of an operator at an alert volume an operator would accept? ## Method - Data: this host's telemetry stream, at least 7 days of history after a 24 h warm-up (`--warmup 24`). - Traces: `config/traces/standard.trace` (6 traces: browser to curl|sh, systemd user persistence, memfd loader, ptrace injection, download then exec, python from a PDF viewer), 5 copies each (`--inject-copies 5`), spread evenly over the ranked history. Injected events are scored but never learned from. - A copy is caught at budget B (alerts per day) if its best window scores strictly above the (B x days)-th best real window. Ties count against. - Command: `sudo strawberry --replay --inject config/traces/standard.trace --inject-copies 5 --warmup 24 --json`. - Repeat on three non-overlapping 7-day periods when available. ## Thresholds | Outcome | Criterion (each of the 3 periods) | |---|---| | Rarity is useful as an advisory signal (Stage 5 candidate) | caught at 3/day >= 60% of copies, and every trace caught at least once at 10/day | | Rarity only steers inspection (Stage 4 input, never shown) | caught at 10/day >= 40% of copies | | Rarity is not useful here | anything less | Stage 5 additionally requires beating rules-only detection by >= 20% relative recall at <= 3 alerts/day (plan, Stage 5 gate). ## Known limits - Traces are synthetic; their parents are trace processes, not real ones (a real browser's parent chain is not modeled). - Rarity measures novelty, not maliciousness: an attack that looks like routine activity on this host is invisible to it by design. - Copies are scored against a model that has seen real history only, so a repeated real attack would score lower on later copies than here.