Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
- src/learn/trace.cpp: a small trace format (exec, connect, write, ptrace, module with +offsets and process aliases) compiled to telemetry records with pids above any pid_max and node ids that name the trace and copy. - --replay --inject FILE [--inject-copies N]: copies are spread evenly over the ranked history (a first pass reads the time range), scored but never learned from, and checked against alert budgets of 1, 3 and 10 per day: is the copy's best window above the (budget x days)-th best real window? Ties count against. Injection leaves real windows and their ranking unchanged. - config/traces/standard.trace: the six plan traces (browser to curl|sh, systemd user persistence, memfd loader, ptrace injection, download then exec, python from a PDF viewer). - docs/evaluation/stage3-rarity-preregistration.md: method and pass thresholds, written before any run on host data. Tests: test_trace (standard traces, error messages with line numbers, records); test_replay R8 (every copy found, real ranking unchanged, a novel chain caught at 10/day and a bare shell not). fuzz_trace_dsl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.1 KiB
2.1 KiB
Stage 3 rarity baseline: pre-registered evaluation
Written 2026-10-26, before any evaluation run on host data. The thresholds
below do not change after results are seen; a changed method gets a new
dated document. (Also recorded in YAMS as source=decision,
task=strawberry-data-engine.)
Question
Does the rarity baseline (src/learn/rarity.cpp, windows from
src/learn/replay.cpp) put realistic attack activity in front of an
operator at an alert volume an operator would accept?
Method
- Data: this host's telemetry stream, at least 7 days of history after a
24 h warm-up (
--warmup 24). - Traces:
config/traces/standard.trace(6 traces: browser to curl|sh, systemd user persistence, memfd loader, ptrace injection, download then exec, python from a PDF viewer), 5 copies each (--inject-copies 5), spread evenly over the ranked history. Injected events are scored but never learned from. - A copy is caught at budget B (alerts per day) if its best window scores strictly above the (B x days)-th best real window. Ties count against.
- Command:
sudo strawberry --replay --inject config/traces/standard.trace --inject-copies 5 --warmup 24 --json. - Repeat on three non-overlapping 7-day periods when available.
Thresholds
| Outcome | Criterion (each of the 3 periods) |
|---|---|
| Rarity is useful as an advisory signal (Stage 5 candidate) | caught at 3/day >= 60% of copies, and every trace caught at least once at 10/day |
| Rarity only steers inspection (Stage 4 input, never shown) | caught at 10/day >= 40% of copies |
| Rarity is not useful here | anything less |
Stage 5 additionally requires beating rules-only detection by >= 20% relative recall at <= 3 alerts/day (plan, Stage 5 gate).
Known limits
- Traces are synthetic; their parents are trace processes, not real ones (a real browser's parent chain is not modeled).
- Rarity measures novelty, not maliciousness: an attack that looks like routine activity on this host is invisible to it by design.
- Copies are scored against a model that has seen real history only, so a repeated real attack would score lower on later copies than here.