strawberry/docs/evaluation/stage3-rarity-preregistration.md
trvon 4e25666a70
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Inject scripted attack traces into replay and score them
- src/learn/trace.cpp: a small trace format (exec, connect, write,
  ptrace, module with +offsets and process aliases) compiled to
  telemetry records with pids above any pid_max and node ids that name
  the trace and copy.
- --replay --inject FILE [--inject-copies N]: copies are spread evenly
  over the ranked history (a first pass reads the time range), scored
  but never learned from, and checked against alert budgets of 1, 3 and
  10 per day: is the copy's best window above the (budget x days)-th best
  real window? Ties count against. Injection leaves real windows and
  their ranking unchanged.
- config/traces/standard.trace: the six plan traces (browser to curl|sh,
  systemd user persistence, memfd loader, ptrace injection, download
  then exec, python from a PDF viewer).
- docs/evaluation/stage3-rarity-preregistration.md: method and pass
  thresholds, written before any run on host data.

Tests: test_trace (standard traces, error messages with line numbers,
records); test_replay R8 (every copy found, real ranking unchanged, a
novel chain caught at 10/day and a bare shell not). fuzz_trace_dsl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 22:37:44 -06:00

2.1 KiB

Stage 3 rarity baseline: pre-registered evaluation

Written 2026-10-26, before any evaluation run on host data. The thresholds below do not change after results are seen; a changed method gets a new dated document. (Also recorded in YAMS as source=decision, task=strawberry-data-engine.)

Question

Does the rarity baseline (src/learn/rarity.cpp, windows from src/learn/replay.cpp) put realistic attack activity in front of an operator at an alert volume an operator would accept?

Method

  • Data: this host's telemetry stream, at least 7 days of history after a 24 h warm-up (--warmup 24).
  • Traces: config/traces/standard.trace (6 traces: browser to curl|sh, systemd user persistence, memfd loader, ptrace injection, download then exec, python from a PDF viewer), 5 copies each (--inject-copies 5), spread evenly over the ranked history. Injected events are scored but never learned from.
  • A copy is caught at budget B (alerts per day) if its best window scores strictly above the (B x days)-th best real window. Ties count against.
  • Command: sudo strawberry --replay --inject config/traces/standard.trace --inject-copies 5 --warmup 24 --json.
  • Repeat on three non-overlapping 7-day periods when available.

Thresholds

Outcome Criterion (each of the 3 periods)
Rarity is useful as an advisory signal (Stage 5 candidate) caught at 3/day >= 60% of copies, and every trace caught at least once at 10/day
Rarity only steers inspection (Stage 4 input, never shown) caught at 10/day >= 40% of copies
Rarity is not useful here anything less

Stage 5 additionally requires beating rules-only detection by >= 20% relative recall at <= 3 alerts/day (plan, Stage 5 gate).

Known limits

  • Traces are synthetic; their parents are trace processes, not real ones (a real browser's parent chain is not modeled).
  • Rarity measures novelty, not maliciousness: an attack that looks like routine activity on this host is invisible to it by design.
  • Copies are scored against a model that has seen real history only, so a repeated real attack would score lower on later copies than here.