strawberry/tests
trvon f4adc12a74
gate: judge the model against a calibrated rarity comparator
The rarity-greedy comparator acted on every evaluated window, so it spent
each hour's budget on the first ~15 windows and scored 0 hits in every
cell. Criteria 2 and 3 of the Stage 4 gate were void against it.

rarity_calibrated keeps the same rarity value, chooser, costs and budget,
and adds a per-node threshold frozen before the evaluated span: the m-th
largest value over the trailing 24 h of label-free history between the
warm-up and the --eval-after bound, with m set so expected spend matches
the hourly budget. Without that history the comparator is uncalibrated
and the gate fails by name; it never falls back to another rule.

rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour
ordering, not causal) are reported as diagnostics and never judged.
fleet gate pools counts, keeps thresholds per node, and fails a pooled
cell when any node is uncalibrated or lacks the comparator.

Contract change: in `fleet gate --json`, rarity_hits and
rarity_cpu_seconds now describe the calibrated comparator; the old
numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds.

Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md,
sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95,
indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is
dated after the registration time in that document (2026-11-21T00:00Z);
the YAMS record is what predates it.

Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the
strict analyzer on the changed TUs. Not run: fold on the changed TUs,
make fuzz, the live Docker tests and the perf KPI. The comparator has
not been run on validation-range data.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 19:20:55 -07:00
..
fixtures Rules R1: ship pinned yara-forge-core source snapshot in the release (no per-node fetch) 2026-11-11 13:21:12 -07:00
behavior_live.cpp Add exec prevention, behavior telemetry and persistence watch 2026-10-25 18:17:16 -06:00
cli_replay_fixture.cpp Replay: add strict read-only training summaries 2026-11-03 06:54:17 -07:00
es_live_payload.cpp Monitor execs and file writes on macOS with Endpoint Security 2026-10-27 09:55:21 -06:00
exec_guard_live.cpp KPI: bound collection and reject incomplete measurements 2026-11-03 06:57:00 -07:00
file_watch_live.cpp Add exec prevention, behavior telemetry and persistence watch 2026-10-25 18:17:16 -06:00
fleet_inspect.cpp fleet-inspect: print manifest time range and window start min/max 2026-11-14 12:16:32 -07:00
fleet_seed.cpp fleet e2e: add out-of-sample evaluation check via --eval-after auto 2026-11-19 18:22:58 -07:00
live_bpf_daemon_smoke.cpp Integrate features into the daemon, build and docs 2026-10-25 18:17:30 -06:00
live_bpf_pipeline_smoke.cpp Integrate features into the daemon, build and docs 2026-10-25 18:17:30 -06:00
macos_es_live.cpp Monitor execs and file writes on macOS with Endpoint Security 2026-10-27 09:55:21 -06:00
meson.build threat traces checker + fleet gate in the dockerized Salt test 2026-11-15 09:36:24 -07:00
model_score.cpp Review fixes: write->exec feature constant, accurate comments, tag-safe CI 2026-11-10 16:09:03 -07:00
process_monitor_live_smoke.cpp Attach the fork tracepoint on current kernels 2026-10-25 21:33:39 -06:00
scan_bench.cpp yara: rule partitions — scan each file only with the rules that can match it 2026-11-15 06:58:39 -07:00
source_invariants.py Stage 6: export rarity-delta record (per-TupleKind count-min, bounded decode) + harden R13 for braced initializers 2026-11-07 19:39:22 -07:00
store_compare.hpp Drop unqlite-cpp and import legacy databases 2026-10-25 18:46:21 -06:00
test_behavior_engine.cpp Review fixes #4/#5: ptrace medium middle-ground + reject fleet-train flags elsewhere 2026-11-10 16:17:01 -07:00
test_cli.cpp cli: honor <group> --help, and add fleet run for a standalone central node 2026-11-18 12:11:34 -07:00
test_cli_replay.py Findings: reconcile evidence and harden policy rebuilds 2026-11-03 06:55:47 -07:00
test_config.cpp yara: rule partitions — scan each file only with the rules that can match it 2026-11-15 06:58:39 -07:00
test_control_server.cpp Privacy: attribute tracker contacts to programs via OpenSnitch events 2026-10-28 06:15:28 -06:00
test_correlation_engine.cpp Findings: reconcile evidence and harden policy rebuilds 2026-11-03 06:55:47 -07:00
test_correlation_incremental.cpp Findings: reconcile evidence and harden policy rebuilds 2026-11-03 06:55:47 -07:00
test_crash_safety.cpp Build on macOS: portable syscalls, sockets and tray 2026-10-27 09:54:54 -06:00
test_daemon_runtime.cpp Findings: reconcile evidence and harden policy rebuilds 2026-11-03 06:55:47 -07:00
test_database.cpp Integrate features into the daemon, build and docs 2026-10-25 18:17:30 -06:00
test_doctor.cpp Daemon memory: budget scan buffers, stream large images on the launch path, report throttling 2026-11-14 15:36:14 -07:00
test_evidence_store.cpp Replace web UI with a Qt tray app 2026-10-25 18:17:03 -06:00
test_exec_guard.cpp exec guard: answer a cache hit without holding the cache lock 2026-11-19 17:29:38 -07:00
test_fault_injection.cpp Pipeline: keep oversized cache refusals unscanned 2026-11-03 06:57:35 -07:00
test_features.cpp Stage 3.5b2: replace exec-lineage with write->exec data-flow feature 2026-11-08 16:37:06 -07:00
test_fleet.cpp Stage 6: node identity (fleet.hpp) — pure node_uid + wrapper 2026-11-07 18:59:51 -07:00
test_fleet_export.cpp Fleet F1: merge dedup per node+window, overlay newest-per-node, export --since 2026-11-12 06:50:03 -07:00
test_fleet_gate.cpp gate: judge the model against a calibrated rarity comparator 2026-11-20 19:20:55 -07:00
test_fleet_merge.cpp fleet: node-share cap never below the fair share 1/N 2026-11-14 11:29:06 -07:00
test_fleet_overlay.cpp Stage 6: export populates fleet_bits from a local overlay (export_store fleet_surprisal callback; run_export loads fleet-overlay.sbo) 2026-11-08 08:09:47 -07:00
test_hash_utils.cpp Daemon memory: budget scan buffers, stream large images on the launch path, report throttling 2026-11-14 15:36:14 -07:00
test_incident_report.cpp Integrate features into the daemon, build and docs 2026-10-25 18:17:30 -06:00
test_incremental_view.cpp store: incremental read-only refresh (fixes the tray spinning a core) 2026-11-15 08:46:00 -07:00
test_inspection_executor.cpp Stage 4: execute a promoted model's inspections, under guardrails 2026-10-27 21:03:57 -06:00
test_installed_files.cpp Record routine indicators as evidence only 2026-10-26 05:37:58 -06:00
test_invariants.cpp governor: fail-safe host pressure, diagnostic diskstats, capped scan buffers 2026-11-17 13:45:07 -07:00
test_learn.cpp Learn: handle extreme restored rarity clocks 2026-11-03 06:52:20 -07:00
test_model.cpp gate: refuse in-sample and degenerate evidence, and identify the model 2026-11-19 16:20:40 -07:00
test_opensnitch.cpp Monitor execs and file writes on macOS with Endpoint Security 2026-10-27 09:55:21 -06:00
test_opensnitch_events.cpp Privacy: macOS tracker source via Little Snitch log-traffic (WS2 macOS) 2026-10-28 18:54:27 -06:00
test_perf_kpi.py kpi: budget the per-phase RSS peak, not only the last sample 2026-11-19 16:19:41 -07:00
test_platform_process.cpp Read process facts on macOS without procfs 2026-10-27 09:55:09 -06:00
test_policy.cpp Findings: reconcile evidence and harden policy rebuilds 2026-11-03 06:55:47 -07:00
test_process_monitor.cpp Integrate features into the daemon, build and docs 2026-10-25 18:17:30 -06:00
test_process_query.cpp Integrate features into the daemon, build and docs 2026-10-25 18:17:30 -06:00
test_redact.cpp Unify file ids, record uid/comm, redact command lines 2026-10-26 17:58:48 -06:00
test_repeat_aggregator.cpp Detection: unify the two repeat-aggregation sinks + IOC severity resolution 2026-10-28 17:39:48 -06:00
test_replay.cpp gate: judge the model against a calibrated rarity comparator 2026-11-20 19:20:55 -07:00
test_resource_governor.cpp governor: release a memory episode on the memory hold, not the current RSS 2026-11-17 13:52:37 -07:00
test_retro_hunt.cpp Retro-hunt: re-scan binaries seen before when the rules change 2026-10-27 08:59:38 -06:00
test_rule_partition.cpp yara: rule partitions — scan each file only with the rules that can match it 2026-11-15 06:58:39 -07:00
test_rule_updater.cpp Rules R1: ship pinned yara-forge-core source snapshot in the release (no per-node fetch) 2026-11-11 13:21:12 -07:00
test_runtime_pipeline.cpp detection: mark a timed-out scan incomplete without bypassing quarantine 2026-11-19 16:48:19 -07:00
test_shadow.cpp Stage 4: keep the live rarity model across daemon restarts 2026-10-27 23:25:01 -06:00
test_store_codec.cpp Add compressed event store 2026-10-25 18:17:17 -06:00
test_store_differential.cpp Drop unqlite-cpp and import legacy databases 2026-10-25 18:46:21 -06:00
test_store_import.cpp Drop unqlite-cpp and import legacy databases 2026-10-25 18:46:21 -06:00
test_store_log.cpp Add the store's disk floor, retention, verify and compact 2026-10-26 06:19:29 -06:00
test_store_maintenance.cpp Add the store's disk floor, retention, verify and compact 2026-10-26 06:19:29 -06:00
test_store_manifest.cpp Add compressed event store 2026-10-25 18:17:17 -06:00
test_store_reduce.cpp Store 7c/7d: tunable merge horizon + CPR interleave-aware merge mode 2026-11-12 06:50:09 -07:00
test_store_segment.cpp Bound finding size and say why a store write is refused 2026-10-26 06:43:46 -06:00
test_telemetry.cpp Unify file ids, record uid/comm, redact command lines 2026-10-26 17:58:48 -06:00
test_trace.cpp Stage 4: ship a training trace set; install traces 2026-10-28 04:50:04 -06:00
test_tracker_set.cpp Privacy: tracker list feeds (advisory only, never block) 2026-10-28 05:25:53 -06:00
test_train.cpp test_train: skip gradient samples on a kink inside both steps 2026-11-20 10:32:12 -07:00
test_yara_scanner.cpp macOS: build and test fixes found re-validating on the Mac (57/57 green) 2026-11-15 08:35:34 -07:00
trace_check.cpp threat traces checker + fleet gate in the dockerized Salt test 2026-11-15 09:36:24 -07:00