Both hit paths (fanotify and Endpoint Security) called finish() while holding
cache_mutex_. finish() is the answer: it writes the kernel response, runs the
observation callback, bumps stats under stats_mutex_ and pushes to
report_mutex_. Holding the cache lock across that serialised every cache-hit
exec behind one mutex even though the daemon runs two workers, so on Linux a
slow answer was a slow exec - in the path whose whole job is to keep launches
fast.
detail::answer_cached_assessment copies the entry under the lock, releases it,
and only then answers. cache_mutex_ is now a leaf lock (also at the work-loop
insert), so the old edge cache_mutex_ -> {pending, stats, report} is gone and no
inversion can be introduced. Answer-exactly-once is unchanged: respond() still
checks pending->responded under the pending mutex, and the hit path still
answers once and returns.
The seam is declared in exec_guard.hpp so the tests no longer redeclare an
internal function in their own translation unit.
Two ways a verdict could be built on evidence the tool did not actually have.
Pooling only ever upgraded provenance: one node that knew its training range set
the pooled flag, so a cell whose other nodes could not prove a range was still
reported as evaluated, and a cell could pass with no evaluated range at all. A
node with an unknown or unusable range now marks the pooled cell incomplete,
zeroes the displayed range (no range is shown that only some nodes could prove),
and appends "provenance incomplete: N of M contributing node report(s)". A known
range can no longer upgrade an unknown cell.
The documented "passes only if all 9 cells pass" was not enforced: passed was
just "some cells and not refused", so a one-seed, one-period report set exited 0.
A completeness gate now requires 3 seeds, 3 periods contiguous from 0, the full
seed x period cross product, and per-report consistency, and names every reason
("need 3 training seeds...", "missing: seed3 period 2", "inconsistent
--eval-model / --periods across node reports") before any verdict is computed.
The in-sample check landed without any way to bound an evaluation, so every cell
was in-sample and every promotion path refused. Cutting the training bundles
earlier does not help: training spans the bundles, evaluation replays the whole
store with no lower bound, and the store keeps everything (retention 0), so
eval_first <= training_last always held.
replay --policy-eval gains --eval-after NS|auto (0 = unbounded). The bound is
applied where windows are ranked and where injected traces are planned, so the
periods partition only the out-of-sample span; auto derives it as the latest
training-range end over the loaded --eval-model files. Comparisons are written
without first + warm-up arithmetic so INT64_MAX timestamps cannot overflow. A
bound past the end of the store does not produce a silent empty cell: the cell
carries eval_error and promotion_gate refuses naming it.
Provenance also gets honest about width: training_last_ns is the last training
window's END (a window spans window_ns, so using its start left up to 60 s of
trained-on data looking disjoint), and the single-model path (model promote) now
uses the ranked span instead of the whole store, which had guaranteed a refusal.
Verified: R11c passes with 128 out-of-sample windows and a derived bound, and
unbounded returns to in-sample. Refuses an explicit bound later than all history.
Review of 893127c found that reporting `scanned = false` for a timed-out
guard scan was wrong in a way that mattered: decide_exec answers `allow` on
`!scanned` BEFORE it looks at the operator quarantine, so a quarantined binary
whose scan overran its deadline would have been allowed to run. It also dropped
the only `scanned = true` on the normal assessment path, so every ordinary
assessment became "unscanned".
The assessment now keeps `scanned = true` (quarantine, allowlist and IOC
decisions still apply, and the guard still fails open on a thin result) and
carries a separate `incomplete` flag, which alone prevents caching:
guard_assessment_cacheable() and the pipeline's cache_put both exclude it.
Also fixes the same verdict being lost one caller up: scan_on_demand set
ok = false with an explanatory error, then unconditionally reset ok = true after
the match loop, so control scan-fd, MCP and retro-hunt all reported a timed-out
scan as a completed one with no error.
Tests: a timed-out assessment keeps scanned true and top_rule evidence, is
marked incomplete, and a second exec re-scans instead of hitting the cache; a
quarantined incomplete assessment still blocks in both enforce and audit, while
an incomplete clean one is allowed and not cacheable.
Three launch-path defects from the audit.
The scan cache key was dev/ino/size/mtime, and mtime is settable via utimensat,
so a rewrite that restored size and mtime (cp -p, install -p, package restore, or
a deliberate attacker) served the old verdict with no TTL. The key now includes
ctime, and the fanotify path calls the shared detail::image_cache_key instead of
re-deriving the fields, so the guard and the pipeline cannot drift apart again.
recent_images_ was inserted before the scan and never removed, so an image that
was unavailable or whose buffer grant was refused was never retried for that
process instance - contradicting scan_image's own comment. The key is now
dropped when the outcome is unavailable, read_error or yara_error, while
completed scans stay deduped.
Live exec scans were deferred for up to ~5 s per emergency episode. The live
queue is bounded at 4096 and full means dropped exec events, so a fork storm
during an emergency could lose telemetry the guard depends on. The live lane now
takes a single non-blocking probe (kLiveExecHold is 0, pinned by a
static_assert) and scans anyway, fail-open; background inventory keeps the
bounded hold.
Tests: a same-size, mtime-restored rewrite is re-scanned rather than served from
cache; a transient scan failure is retried on the next event while a completed
scan is not; the live-lane probe never blocks.
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.
In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.
Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.
Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.
3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).
Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
Two launch-path defects let a weaker-than-complete scan become the cached
answer for an executable until the next rules reload. In enforce mode that
means the exec guard silently stops consulting YARA for that binary.
1. A YARA deadline overrun was reported as success. timed_yara_scan/
timed_yara_scan_ranges returned `ok || timed_out`, so assess_exec set
have_matches and cached the image with an EMPTY match list; the guard cached
that assessment too. The guard's YARA budget is exactly prevention_timeout_ms
(1 s by default), so one slow scan was enough. They now return a
BoundedYaraScan {ok, timed_out} and only `completed() == ok && !timed_out`
may set have_matches or reach cache_put; scan_image returns yara_error and
caches nothing; the assessment reports scanned == false so the guard fails
open instead of inheriting a clean answer. Matches found before the deadline
are still reported as evidence, and a timeout is counted in yara_timeouts.
2. The guard cached degraded and partial assessments, contradicting
exec_guard.hpp ("not cached") and the pipeline, which already excluded them.
A degraded result is produced by a transient condition (the scan-buffer lease
is momentarily held), so one collision pinned the image to hash/IOC-only for
the whole rules+policy generation. The rule now lives in one place,
guard_assessment_cacheable(), used at both the Linux and macOS cache stores.
Tests: a timed-out assess_exec is not scanned, still reports the partial match,
and a second exec re-scans instead of hitting a cache; the cache predicate is
covered for clean, degraded, partial and unscanned assessments.
compute_level picked the release hold from the RSS *after* the drop, so an
episode raised by the daemon's own memory released in 150 ms as soon as RSS
fell below the warning ratio — ignoring the documented 1000 ms memory hold.
A warning-level memory episode could therefore clear almost immediately.
The episode is now classified when it is raised: memory_driven_episode_ is set
when memory drove the level, and that decides the release hold. Critically this
does not clear when the level steps down (critical -> warning is the same
episode); the next escalation re-classifies it.
Tests: a memory warning -> calm transition is still warning at 200 ms and
999 ms and clears at 1000 ms, while a CPU-driven episode still clears in 150 ms.
Host pressure stays fail-safe: any of CPU/I/O/memory PSI above its profile
threshold, or low MemAvailable, still defers background work and can degrade
the exec guard. That is deliberate — global I/O PSI also counts waits with no
block-device counter (FUSE/network-backed I/O, display page flips), so idle
disks never prove pressure is benign. An audit of an earlier version that
vetoed I/O PSI below 5% block utilization rejected it on exactly that basis.
/proc/diskstats is therefore diagnostic telemetry only, exposed as
host.block_io_util_pct. It is keyed by major:minor so a reused device name
cannot compare unrelated counters, and reports unknown (-1) for the first
sample, a new/replaced/reset device, or an oversized read, instead of a false
0%.
scan_buffer_budget() now derives from the same effective budget that pressure
uses — the configured/profile budget capped by cgroup memory.max — with the
existing 32 MiB floor, so a cgroup-capped daemon cannot be granted scan
buffers larger than its own limit. Lock order stays mutex_ -> buffer_mutex_.
Tests: deterministic fail-safe cases across every diskstat value, the
new-device regression, scanner budget under a 100 MiB cgroup cap, and a seeded
property asserting severe I/O PSI always holds work whatever diskstats says.
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:
- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
evaluates every model on every equal time slice of the ranked history; each
(model, period) cell simulates the model and rarity-greedy with their own
hourly budgets exactly as policy_eval does (policy_eval.cells). With one
model equal to --model and one period the cell equals the single-model
evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
positives, inspections, CPU-s, hits, refusals) per (model, period) and
applies learn::promotion_gate unchanged to each; passes only if all pass.
Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
cycle now also publishes all three seed models to dist/gate/; strawberry.gate
(sensors) evaluates them with the pre-registered evaluation traces; the
operator-run orch.strawberry_gate pushes the reports back and pools them;
evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
criteria and numbers unchanged).
Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root cause, from the Tracy capture the CPU watcher took when the tray spun
(2026-11-15 08:15, 67% for 15 s): database.reload_read_only rebuilt the whole
evidence view from every segment — mean 1.4 s, max 1.7 s, 4x in 20 s — because
every 3 s poll that saw the daemon append evidence triggered a full reload.
- store::scan_store_read_only_from + ReadCursor (per segment: id, sealed,
bytes, records delivered). When the previously seen segments are an
unchanged prefix (same ids, sealed ones byte-identical), only records
appended since are delivered: the rest of the previously last segment
(skipping what was applied) and new segments. Any other change —
compaction, retention, reduction, a rewritten or shrunk segment — delivers
everything and the caller rebuilds.
- Database read-only refresh uses it; records apply in the same order a full
rebuild uses, so the view is identical. read_only_reload_counts() reports
full vs incremental refreshes.
Live, on the real evidence store: the change-driven reload went from ~1.4 s
to 2.8 ms; the tray did ~97 ms of work in a 60 s window.
Test (test_incremental_view, seeded property): a long-lived reader
refreshing across random appends, upserts, seals and a compaction always
equals a freshly opened reader; appends refresh incrementally (42-49 per run),
the compaction forces a rebuild, and no change reports no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.
Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
priority order (big non-PE first) under rule_partition_budget_pct (default
100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
partition, full_scans); config rule_partitions (default true) and
rule_partition_budget_pct.
Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
differences, but the corpus has no rule matches, so parity is not proven.
- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
zero-copy blocks; file offsets and filesize keep file semantics) with
EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
loaded segments of ELF images >= 4 MiB at launch; such results are marked
`partial`, never cached, and the pipeline still scans the whole file. Status
pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).
Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.
- Exec guard (assess_exec) no longer copies an image before deciding: a cached
scan answers with no read; an image over guard_full_scan_max_bytes, or when
the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
(degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
ScanBufferLease): in-memory image copies bounded to memory budget / 4
(balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
memory_full_pct / throttled, and resources.scan_buffers; doctor check
resources.memory (problem when throttled at MemoryHigh, with the fix).
Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seen in the dockerized 3-sensor run: max_node_share=0.25 (pillar default) with
3 similar-sized nodes cut every node to 25% of the total, dropping 158/468
windows (34%) while correcting no imbalance. The cap is now
max(max_node_share, 1/N) * total, so a balanced fleet is never trimmed and
only nodes above an equal share are cut. Test: balanced 3-node fleet keeps
all windows; a dominant 4th node is still trimmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ship 3.5c's provenance attribution to the central learner. ExportLabel gains two
additive uint64 tags (8 origin_node_hash, 9 origin_exe_hash); a positive window's
label now carries token_hash(dropper node) + token_hash(path_template(dropper
exe)) from ReplayWindow.origin_*. fleet_merge counts windows_with_origin.
R12/R13: a FIELD addition, not a new record type -- no codec.hpp/enum change
(R12 untouched), both fields are uint64 hashes so no raw path/node string crosses
the export boundary (R13 green, no name-loophole). Back-compat additive:
kExportSchemaVersion stays 1; old bundles decode with origins 0; old readers
ignore tags 8/9. Positive labels only -- suppressed/negative carry no origin.
Tests: round-trip of the hash fields + defaults; extraction proves a dropper->exec
detected window's positive label carries a non-zero origin while a no-write
detected window and suppressed labels carry 0. source_invariants green.
A learned ranking head + TrainingExample plumbing are phase-2 (out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reuse the global write->exec linkage (recent_writes) to attribute a detection to
its ROOT CAUSE. recent_writes now maps path -> {writer_node, writer_exe}; when a
window is late_detected (retro-hunt flags a binary after it ran) or detected
(YARA), set ReplayWindow.origin_node/origin_exe from whoever WROTE the flagged
binary -- usually an earlier node than the exec leaf that ran it. Surfaced in
window_json. Deterministic provenance walk (first writer wins); replay-level only,
no export-schema/record-type/enum change (R12/R13 intact), self_written_execs
unchanged. A learned ranking head over ambiguous origins is the follow-up (3.5d).
Test R10: a dropper writes /tmp/evil in one window, a separate process execs it
and is flagged late -> the window's origin_node is the dropper, not the exec leaf;
no in-set write -> empty origin; deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address code-review findings on this session's changes (no behavior change):
- Share the dense-slot index for the write->exec feature as
learn::kWriteExecFeatureIndex; featurize() and the model-score dev tool now
reference it instead of a duplicated literal 42 (prevents silent column drift
if the feature layout shifts).
- Fix the self_written_execs comments: the linkage is a GLOBAL bounded FIFO
(recent_writes), not strictly "this window" — the docs now match replay.cpp.
- release.yml: never cancel a tag/release run (cancel-in-progress only for
non-tag refs); push ordering doesn't guarantee the tag event arrives after the
branch event, so a same-sha branch push must not abort an in-progress release.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.5b's ordered exec-lineage proved inert in clean A/Bs; replace it with a general
write->exec data-flow feature: WindowSummary.self_written_execs (exec of a file
written earlier in the same window) at x[42], tracked via OpenWindow.written_paths.
Honest status: this is the conceptually-correct provenance feature (it separates
download->exec-the-same-file from download->exec-unrelated), but it ALSO shows no
AUC gain on detector-labeled data -- for the same structural reason as 3.5b: the
teacher labels (DownloadThenExecute etc.) correlate with existing count/location
features or re-derive the detector. Validating provenance LEARNING requires the
novel-chain generalization benchmark (train on detector-covered, hold out
detector-evading attacks); committed as that benchmark's feature foundation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feed ordered parent->child exe-template bigrams into the window feature vector
(WindowSummary.lineage_edges, note_lineage_edge, a salted hash loop in featurize,
wired at replay on_exec). The model already got per-kind lineage surprisal
(x[25..29]); this adds path ORDER, which the unordered token bag discarded.
Clean A/B on a fixed corpus (same 697 windows, same flags, only the binary
differs): model AUC 0.734 -> 0.737 (rarity 0.520 unchanged). Null on this
shallow-lineage data (marker execs are 1-process, synthetic data flat) but the
feature is correct and order-sensitive (test: A>B != B>A). Kept as the path
representation that Stage 3.5c's retro-hunt attribution head needs; its payoff is
expected on real multi-stage attack telemetry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose the merge cap and train/test split on `strawberry fleet train` so a
single-node real-attack bundle can pre-seed a model. Default stays temporal +
cap 0.25 (behavior-preserving); `--split random` + `--max-node-share 1` lets the
Caldera-attack export (15 positive windows) train. Verified: real-only train AUC
1.0 vs rarity 0.563; default still refuses identically.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Do not turn an empty cached digest into an unbounded descriptor hash. Distinguish size refusals from YARA results so the exec guard applies its own bounded scan/degrade policy. Cover cached read budgets across size boundaries, known hashes, inventory/live events and real guard-cache reuse. Fold/live/KPI and post-rollout CPU acceptance remain pending; OpenSnitch checksum work is separate.
Preserve independent support, typed rule namespaces and stable process lifetimes. Share authoritative managed-set reconciliation across daemon and offline correlation, load offline policy without seeding/migration, and release resources after shutdown-flush exceptions. Keep retirement history, transaction ownership and fault/allocation regressions; no whole-set crash-atomicity or live acceptance claim.
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
Represent full-range ordered clock differences without signed overflow and allow complete decay. Add seeded restore-clock regressions and a dedicated fuzz seed.
The teacher-label join made a window a positive if it was detected
(replay.cpp), but it never applied the allowlist, so a window running a cleared
false positive (e.g. /usr/bin/ss) was labelled a teacher positive — poisoning
Stage 4 training. Negative labels were written to the stream but replay dropped
them.
- replay_store now collects negative labels into ReplayOptions.negative_labels
(it used to ignore any non-positive label).
- A window whose exec hash is negatively labelled is marked `suppressed`; the
example/policy-eval positive is gated on `!suppressed`, so such a window is
kept as a *negative* training example (useful signal) and never a positive,
even if a cleared rule "detected" it.
- DetectionPipeline seeds a negative label for every allowlisted sha256 at init,
so evidence stored before the allowlist (the historical ss storm) is covered
without waiting for a re-exec.
Test: a detected window with a stored negative label for its hash is a positive
before the label and a kept-negative after. Suite 51/51; analyzer + clang-tidy
clean on replay.cpp and detection_pipeline.cpp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correlation runs over the whole store and never consulted the allowlist, so an
allowlisted hash (e.g. the /usr/bin/ss feed false positive the operator cleared)
regenerated the same findings on every pass from already-stored indicators —
`doctor fix retire-allowlisted-findings` cleared them but they came straight back
(~9,400 findings observed again). This also meant the findings were a poisoned
teacher-label source for training.
- classify() now skips an indicator whose (hash, path, rule) is allowlisted, the
single chokepoint shared by CorrelationEngine::build_findings (the oracle) and
IncrementalCorrelator::update, so both agree. New AllowlistFn; build_findings /
correlate take an optional predicate (default = no filtering, existing tests and
the incremental==full oracle unchanged); IncrementalCorrelator::set_allowlist.
- DetectionPipeline wires the predicate to PolicyStore::match, so the live
incremental path skips allowlisted evidence. Retiring the findings now sticks.
Test: build_findings with an allowlist predicate emits nothing for the matched
hash and is unaffected for others. Suite 51/51; analyzer + clang-tidy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>