Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
The scan path (handle_process) and the exec-guard recording path
(record_exec_decision) had grown two near-identical repeat-aggregation maps
(repeats_ / exec_repeats_) plus three copies of the IOC-feed severity idiom.
Refactor to one implementation; no behavior change (the existing aggregation and
IOC-severity tests are the oracle and pass unchanged).
- New include/strawberry/repeat_aggregator.hpp: RepeatAggregator owns the
per-key window map, the 20-full-then-summary cadence and the aggregated count;
observe(key, now, seen_ns, pid) returns {full, at_threshold, aggregated,
emit_summary, notify, state}. Each caller keeps its own key, summary-id prefix,
summary shape and notify behaviour. Both pipeline sinks now hold one instance
(scan notifies at most hourly; exec passes notify_ns=0). status
repeats_aggregated sums both totals (atomic, lock-free read).
- ioc_hit_severity(feed): one source for the three IOC match sites (scan_image,
assess_exec, handle_process) that was the policy_->ioc_feed_severity ternary.
- Leaves the latency-critical assess_exec/decide path untouched; both sinks still
run under mutex_, off the FAN response.
Unit test tests/test_repeat_aggregator.cpp (window reset, threshold, summary and
notify cadence, per-key independence). Suite 51/51; clang static analyzer and
clang-tidy clean on detection_pipeline.cpp; fold's 7 int-overflow are the accepted
monotonic-timestamp FP class (unchanged by the move); perf-kpi PASS, no regression
(exec_decision p50 unchanged, exec_assess/db_commit within noise or faster).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second piece of tracker attribution (docs/tracker-attribution.md): which
programs reach domains on a trackers list. Advisory only: never a block,
never a finding (source invariant R11).
- OpenSnitch event listener: OpenSnitch's JSON "remote" logger is the
hostname source (Strawberry's eBPF connects have none). It streams over
TCP and cannot leave fields out, so events carry command lines and
environments. The listener binds 127.0.0.1 at a port below 1024 (only
root can), accepts only root-owned peers (/proc/net/tcp), bounds
clients/object size/buffering, and reads only pid, uid, path,
destination and action; arguments and environment are never copied.
- TrackerContacts: per program x domain, bounded at 20000 keys. A root-only
tracker_contact telemetry record (type 25) on the first contact, then at
most hourly with the count since.
- Status `privacy` block; control verb privacy-trackers; CLI
`strawberry privacy trackers [--json]`.
- `strawberry opensnitch enable-events` adds the logger to OpenSnitch's
config (with a .bak); the installer runs it when OpenSnitch is present,
unless privacy_trackers=false. Config: privacy_trackers (default true),
opensnitch_events (default 127.0.0.1:979).
- EasyPrivacy is a default trackers feed (GPL-3.0+ or CC-BY-SA-3.0,
downloaded at run time).
Tests: E1-E7 (test_opensnitch_events: event fields, stream splitting,
peer uid, address rules, listener end to end, contacts and record, config
editor), the control verb, CLI translation and help, daemon status. New
fuzz_opensnitch_events (90 s clean).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First piece of tracker attribution (docs/tracker-attribution.md): which
programs reach advertising and tracking domains, built with the
measurement rules of the CarolinaCon 2026 privacy talk (bridgescope):
a listed domain is a circumstantial label, never proof.
- Feed kind "trackers": hosts lines, plain domains and adblock
whole-domain rules (||host^, ||host^$third-party). Cosmetic rules,
exceptions, paths and other options are skipped; the comment cut used
to turn "site.example##.ad" into a domain.
- TrackerSet: suffix matching (a.b.t.example matches t.example, never
nott.example, an IP literal or a bare TLD); first feed reported.
- List parsing moves from rule_updater.cpp to ioc_list.cpp (no network
code) so the query-side core can use it.
- Source invariant R11: tracker lists never meet exec-guard or OpenSnitch
code. They are not in the OpenSnitch blocklist sync.
Tests: T1-T3 (test_tracker_set), trackers in the list-parsing property
and the list fuzzer (60 s clean), R11 self-test fixture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shadow scorer's rarity model started empty at every daemon start, so
each restart cost an hour of warm-up with no decisions. The daemon now
saves it to model_dir/rarity.state (checksummed, atomic write, hourly and
at stop) and restores it at start. A restored state younger than the
7-day half-life skips the warm-up; decay catches up on the first event.
A file made with other rarity settings, a bad checksum, or a negative or
non-finite count is refused with a reason (status model.rarity_state,
doctor model.rarity_state) and the model starts empty. The file must be
owned by the daemon user and not group/world writable, like model files.
Also fixes status model.warmup_windows, which was never filled in.
Tests: R1/R2 in test_learn (round trip, decay clock, refusals), S8 in
test_shadow (warm-up skipped only when recent), daemon restart restores
the saved state, doctor warning; new fuzz_rarity_state target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A promoted model (promoted.sbm, written only by `model promote` after the
gate) can now request real inspections. Only two, both additive:
- memory_scan: through the daemon's memory-scan queue;
- deep_rescan: the root's image re-scanned on demand; new matches are
recorded as indicators of kind model_inspection (findings titled "YARA
match found by a deeper rescan the inspection model requested") and
positive labels.
watch_subtree has no mechanism and rescan_on_next_rules is the retro-hunt's
job, so both are masked out of acting decisions (kExecutableActions), and
replay's policy evaluation uses the same action set.
Guardrails:
- InspectionExecutor (bounded queue, one low-priority worker, submit()
never blocks); WorkClass::model_inspect lets the governor defer it, and
a deferred decision is dropped.
- Each action rechecks that the promoted model is still active at the
decision's generation (demoting or replacing it revokes queued work),
that the root's node id and raw image path still match (decisions carry
root_image, not the feature template), and skips protected processes.
A deep rescan discards its result if the process, image or rules changed
during the scan.
- An acting decision is durably written before it is dispatched; every
attempt becomes an inspection_outcome record (queued, scanned, skipped,
deferred, revoked, queue_full, failed).
- Source invariant R10: no file may couple learning types to the
exec-guard, OpenSnitch or policy APIs (daemon_app.cpp and main.cpp, the
wiring points, are reviewed by hand). Models never block, contain, kill
or create findings directly.
- Config model_act and model_lambda; status model block reports acting
stats; timings.model_inspection.
docs/evaluation/stage4-execution-gate.md amends the gate's method (not its
thresholds): executable action set, greedy promoted decisions, budgets as
estimated inspection seconds by scan profile.
Also: TerminalSafeBuf's destructor no longer lets an exception escape;
the health message clamps huge durations; replay timestamp overflow; the
Docker test build runs with -j2.
Tests: test_inspection_executor (every skip and execute path, revocation,
queue limits); shadow and replay acting cases; R10 fixture; fault injection
and invariant updates. Validation reported by the session that wrote it:
48/48 local, 46/46 Docker plus privileged live checks, 20 fuzz targets
(9.5M runs), three paired KPI runs without regressions (no acting model
was loaded, so active-inspection cost is not measured yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon now scores live windows with the shadow model and logs the
inspection it would request. Nothing is executed.
- learn/shadow: a live replay engine (same windows and rarity as
`strawberry replay`) fed every telemetry record on the writer thread.
When a window closes, the model's value v gives each action a benefit
v x reach - lambda x CPU cost; the best positive one is chosen if the
hour's budget (10/30/120 CPU-s by scan profile) allows, else none.
With probability 0.02 a uniformly random other action is taken instead,
and every decision records its propensity for off-policy evaluation.
Decisions are inspection_action telemetry records.
- Replay gains a live mode (no ranking bookkeeping, so memory stays
bounded) and a window-closed callback.
- Daemon: model_shadow (default on) loads <model_dir>/shadow.sbm when it
appears or changes, checked every minute; an invalid file is reported
and not used. Status gains a `model` block (kernel, generation,
decisions by action, exploration, would-be spend, budget refusals).
- learn_sources in src/meson.build: tests that compile the daemon from
sources take the learning library whole.
Limits, documented: decisions come when a window closes (a few minutes
after its first exec), and the live rarity model starts empty at each
daemon start (the first hour is learned, not decided).
Tests: test_shadow (no model, greedy choices, budget never exceeded,
determinism, exploration rate and propensities, record codec); daemon
status; fuzz_telemetry_records covers inspection_action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Replay builds each window's WindowSummary as it goes (exec depth and
locations, connections, writes, ptrace, module loads, scan outcomes,
surprisal) and can collect training examples: features, a label (a
detection or late label touched the window), time, root program and the
rarity score. Sampled to at most 50k: positives up to a fifth,
negatives by reservoir with at most 1% per root program.
--inject-train traces become positive examples and are not evaluated.
- learn/train: the value head (and hidden layers) learn P(teacher
positive) with a class-weighted log loss and Adam; positive-unlabeled
data, so unlabeled is "not known bad", documented. Split by time
(newest 20% tested); reports test loss, top-k hits and ROC AUC for the
model and for the rarity baseline. Refuses with fewer than 5 positives.
Deterministic for a seed. The policy head waits for shadow-mode
inspection outcomes.
- CLI: `strawberry model train [store_dir] [--inject-train FILE]
[--epochs N] [--seed N] [--out FILE] [--json]` writes the shadow model
(generation after the newest existing one, a note saying what it was
trained on); `strawberry model show [file]`; `strawberry replay --model
FILE` ranks windows (and evaluates injections) by the model's value.
Config model_dir (default /var/lib/strawberry/model).
- doctor: the model check loads promoted.sbm and shadow.sbm; an invalid
file is a problem (it is never used).
Tests: test_train (gradients against finite differences, skipping steps
that cross a ReLU kink; a planted pattern learned to high AUC; determinism;
refusal without labels; roc_auc); test_replay R10 (examples, labels,
per-program cap, training traces, model ranking); CLI and doctor cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New meson option enable_endpoint_security (auto, macOS only) links
EndpointSecurity and bsm and builds three ES backends behind the
existing interfaces. Helpers are in src/platform/es_support.hpp.
- ProcessMonitor: NOTIFY_EXEC events (pid, parent, path, dev:ino,
argv, start time) plus NOTIFY_TRACE and NOTIFY_KEXTLOAD for the
behavior rules. Events go to a bounded queue (4096, overflow counted
in events_dropped) drained on the monitor thread, so callbacks stay
on one thread. Without an ES client the monitor polls libproc
(backend "libproc"). backend_name() reports "endpoint_security".
- ExecGuard: AUTH_EXEC with the fanotify guard's rules. It fails open
at min(timeout, ES deadline - 500 ms) and on a full queue, and on
macOS 27+ ES_DEADLINE_MISS_MODE_FAIL_OPEN makes the kernel allow a
missed deadline too. Platform binaries and SIP paths are never
denied, and our own execs are muted. ES does not cache verdicts, so
rule changes apply on the next exec. Our own cache keys on
dev:ino:size:mtime:generation. The image is opened by path and its
dev:ino checked against the event; a swapped file counts as
unscanned.
- FileWatcher: NOTIFY_CREATE/RENAME/CLOSE(modified) with inverted
target-path muting: named entries selected literally, whole
directories by prefix. macos_watch_specs() covers launchd
agents/daemons, periodic jobs, cron tabs, sudo drop-ins, shell rc
files, authorized_keys, Downloads and the temp directories.
match_watch_spec() (direct children, as with fanotify directory
marks) is portable and tested. Home dirs come from the directory
service (local_home_dirs).
Every ES handler is noexcept and records swallowed exceptions. Client
errors name the missing requirement (root, entitlement, Full Disk
Access), and the daemon keeps running detection-only.
macos_es_live (root, ES entitlement) checks deny/allow/cache/
platform-binary/fail-open/audit for the guard, exec telemetry and
the file watcher, using small signed payload binaries. It has not run
yet: the development Mac has SIP enabled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
include/strawberry/platform_process.hpp answers what /proc answers on
Linux: running PIDs (proc_listallpids), executable path
(proc_pidpath), parent and start time (sysctl KERN_PROC_PID, readable
for every process), argv (KERN_PROCARGS2) and a boot id
(kern.bootsessionuuid). A process is identified by its start time in
microseconds since the epoch plus the boot id, so node ids survive PID
reuse as on Linux. It is header-only, so the per-test source lists stay
as they are.
Users on macOS: resolve_process_ref (default root), the pipeline's
image hashing and exec-decision actor, and YaraScanner::scan_process
(scans the image by path). A synthetic procfs root still takes the
procfs code, so the existing procfs tests run on macOS too.
The resource governor samples Mach/sysctl on macOS: own CPU from
getrusage, RSS from task_info, load from getloadavg, available memory
from host_statistics64 (free + inactive + purgeable + speculative).
The behavior engine knows macOS debuggers (debugserver, dtrace) and
kext loaders (kernelmanagerd, kmutil), and macOS temp and Downloads
paths count as drop locations. The Linux lists are unchanged.
The KERN_PROCARGS2 parser is portable: a property test (exact argv
recovery, truncation never invents data, total on junk) and the
fuzz_procargs target run on every platform. test_platform_process
checks the live helpers against this process; the daemon test reads
the stopped state through sysctl on macOS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tree now compiles on macOS. Each Linux-only call gets a Darwin
branch, and the Linux branch stays as it was:
- io::fdatasync uses fcntl(F_FULLFSYNC): fsync alone only reaches the
drive cache on APFS, and the store's crash atomicity relies on it.
- Control socket: FD_CLOEXEC set after socket/accept/recvmsg (no
SOCK_CLOEXEC, accept4 or MSG_CMSG_CLOEXEC), SO_NOSIGPIPE, peer
credentials from LOCAL_PEERCRED + LOCAL_PEERPID, getgrouplist with
int groups and a growing buffer. launchd has no RuntimeDirectory, so
the socket directory is created (0750, access group) when missing.
Default socket path is /var/run/strawberry/control.sock.
- CRC32C detects ARMv8 CRC with hw.optional.armv8_crc32 (no auxv).
- stat mtime via st_mtimespec; the BPF shared header uses stdint
typedefs off Linux.
- Background threads join the PRIO_DARWIN_BG band (lower CPU and I/O);
crash safety zeroes RLIMIT_CORE in place of PR_SET_DUMPABLE.
- The desktop app drops Qt DBus off Linux and shows notifications as
tray messages (app/notifier_tray.cpp).
- make setup passes Homebrew's keg-only libarchive to meson.
Tests: behavior_live (Linux ptrace API) builds on Linux only, the
control-server fd count reads /dev/fd, the crash test checks the core
limit on macOS, and the runtime pipeline test no longer references a
YARA-only helper when YARA is absent (that also broke Linux builds
without libyara).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing runs in the daemon yet; these are the pieces the next increments
train and run in shadow.
- learn/features: a WindowSummary accumulated from a window's events
(exec locations, memfd/deleted/uid0 execs, depth, connection classes
and ports, persistence and drop writes, ptrace, module loads, scan
outcomes, capped surprisal per tuple kind, bounded tokens) and
featurize(): 48 explicit features scaled to about [0, 1] plus signed
hashing of tokens into 208 dims, with the governor level and budget
left as context. A window's own detections are never features: they
are the teacher labels.
- learn/mlp: 256 -> 128 -> 64 ReLU with a policy head over the
inspection actions (none, memory_scan, deep_rescan, watch_subtree,
rescan_on_next_rules) and a value head. Scalar reference, AVX2+FMA
chosen at runtime, NEON on aarch64; safe default (no inspection) on a
wrong input size or an inconsistent model. Deterministic init per seed.
- learn/model_file: "SBMD" format with bounded dimensions, exact length,
a sha256 trailer and finite weights checked on load, each failure
named; atomic save into a 0700 directory.
Tests: test_model (kernels against scalar for arbitrary sizes, forward
pass invariants, deterministic init, round-trip, every flipped byte and
truncation rejected, non-finite weights rejected, features bounded and
deterministic). fuzz_model_file. NEON is built and tested only on aarch64.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
strawberry [--config PATH] [--json] <command> [<subcommand>] [args] [--flags]
status, doctor, doctor fix <action>, daemon run, findings list/show/
correlate, process search/trace/trace-node/suspend/resume/kill, scan
file/processes/memory, rules reload/update/check/fetch, policy show/set,
opensnitch enable-checksums, store verify/compact, replay, demo seed,
help [command], version.
One table in src/cli_commands.cpp defines each command (words,
positionals, flags, help) and how it maps onto the existing flag form,
which main() still parses; help is generated from the same table.
Unknown options and wrong arity print the command's usage. Process
actions validate node ids before building their JSON payload.
The older flag form (strawberry --status, strawberry <config>) still
works. The systemd units now run `strawberry daemon run --config ...` and
`strawberry rules fetch --config ...`. README, AGENTS.md and the
false-positive guide use the command form.
doctor renders the daemon's checks with their fixes; doctor fix runs a
maintenance action (with --dry-run). Against an older daemon the CLI says
to reinstall rather than "unknown command". replay explains that
telemetry is root-only instead of reporting zeros, and keeps config
output out of --json.
tests/test_cli.cpp: translations, flag-form passthrough, errors, help.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- doctor (control verb, read-only): store health and disk floor,
telemetry growth per day and days until the floor when retention is
unlimited, leftover store files, superseded findings, retro-hunt state,
thread/write errors, logs (journald rotates them), model (none yet).
Each check carries a maintenance action or advice. Pure checks over
gathered inputs (src/doctor.cpp), so they test without a daemon.
- maintain (control payload verb, privileged, rate limited, audited):
retire-superseded-findings and clean-store, each with dry_run.
- finding_retired evidence record: removes a finding from view; it stays
in the stream's history. CorrelationEngine::superseded_findings finds
old hash/rule findings by recomputing the stable id, and old lineage
findings only when the current one's parent process is among their
nodes (a reused pid on another boot is not a match). Lineage findings
now record parent_process_node_id.
- store::inspect_store_files / clean_store_leftovers: orphan/ files older
than N days and *.compact / MANIFEST.tmp temp files; never a listed
segment.
Tests: test_doctor (check statuses, superseded detection including pid
reuse and no-replacement cases, retirement surviving a reopen, cleanup
keeping recent orphans and listed segments); daemon test exercises both
verbs over the socket. fuzz_evidence_records covers finding_retired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The main source of teacher labels over time: rules improve daily, while
most malware runs once.
- The pipeline remembers every binary that ran (image_seen evidence
record: sha256, path, first/last seen, size; at most one write per hash
a day). Deleted and memfd images are skipped: nothing to find again.
- Each rules load computes a digest of the rule files and IOC hash lists
(paths after resolving symlinks, sizes, mtimes).
- RetroHunter (src/retro_hunt.cpp, own thread, idle priority, paced at
retro_hunt_pace_ms, deferred by the governor as WorkClass::retro_hunt)
re-scans when the digest differs from the last completed pass, most
recently seen first. Only paths that still hold the same content are
scanned (O_NOFOLLOW, regular files, hash compared); others count as
changed or missing. A completed pass is recorded (retro_hunt record) so
a restart does not repeat it; an interrupted one starts over.
- New matches (rules not recorded before for that hash, not allowlisted)
become indicators with scan_kind retro_hunt, a file/indicator graph
link, a positive label and a notification. Correlation turns them into
findings titled "YARA match on a binary that ran here (found after a
rules update)". Nothing is blocked or contained from here.
- Replay reads positive labels from the evidence stream and marks windows
that ran a binary labelled bad later (LATE in the CLI).
- Status: a retro_hunt block (API_CONTRACT.md); config: retro_hunt,
retro_hunt_pace_ms.
Tests: test_retro_hunt (remembered once; passes only on a digest change
and survive a restart; a clean-at-exec binary matching a new rule becomes
an indicator, label, finding and notification; a replaced binary is
counted as changed, never scanned as the old one; a stop abandons a pass
unrecorded); test_replay R9 (late labels, from a real store).
fuzz_evidence_records covers the new records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/learn/trace.cpp: a small trace format (exec, connect, write,
ptrace, module with +offsets and process aliases) compiled to
telemetry records with pids above any pid_max and node ids that name
the trace and copy.
- --replay --inject FILE [--inject-copies N]: copies are spread evenly
over the ranked history (a first pass reads the time range), scored
but never learned from, and checked against alert budgets of 1, 3 and
10 per day: is the copy's best window above the (budget x days)-th best
real window? Ties count against. Injection leaves real windows and
their ranking unchanged.
- config/traces/standard.trace: the six plan traces (browser to curl|sh,
systemd user persistence, memfd loader, ptrace injection, download
then exec, python from a PDF viewer).
- docs/evaluation/stage3-rarity-preregistration.md: method and pass
thresholds, written before any run on host data.
Tests: test_trace (standard traces, error messages with line numbers,
records); test_replay R8 (every copy found, real ranking unchanged, a
novel chain caught at 10/day and a bare shell not). fuzz_trace_dsl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replay (src/learn/replay.cpp, strawberry --replay [dir] [--top N] [--json]):
reads the telemetry stream read-only, orders events by time (a 30 s
reorder buffer absorbs the writer's batching), joins behavior events to
the latest exec of their pid at that time (keeps PID reuse apart), builds
60 s windows (a root exec, its descendants and their behavior, at most 64
events), scores each exec, connection and persistence write with the
rarity model before observing it, and marks windows whose processes had
detections. Prints the most unusual windows with the tuples that made
them unusual. Deterministic. tests/test_replay.cpp: chain windows, PID
reuse, order insensitivity within the horizon, determinism, ranking,
detection marking, a real store round trip. fuzz_replay feeds arbitrary
record streams.
UTF-8: file names are bytes, and the fuzzer showed nlohmann's dump()
throwing on a path that is not UTF-8. The same applied to the daemon's
control replies (kill/suspend/quarantine name the exe path) and to the
OpenSnitch containment rule, so a binary with such a name could not be
contained, and json_escape wrote the raw bytes into status JSON. Now:
- every .dump() in src/ uses error_handler_t::replace (U+FFFD), enforced
by source invariant R9;
- json_escape writes invalid sequences as � (valid_utf8 and
utf8_sequence_length in json_util.hpp);
- containment refuses such a path with a clear error (a rule could never
match the real bytes); quarantine still blocks execution by hash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
src/learn/ (pure libraries; nothing in the daemon calls them yet, Stage 3
uses them in replay):
- tokenize: path templates (user homes, per-uid dirs, pids, digit and hex
runs, UUIDs, mktemp-style names abstracted) and command-line shapes
(flags kept, values, paths, URLs, IPs, numbers and encoded blobs
classed; the first plain word kept as a subcommand; long argument lists
summarized). Bounded and noexcept.
- rarity: decayed count-min sketches over parent>child, grandparent>
parent>child, exe+command-line shape, exe+connection class and
exe+persistence location, scored as surprisal in bits; a bounded
first-seen table; deterministic hashing for replay. About 1.3 MiB.
tests/test_learn.cpp: tokenizer cases and idempotence; count-min never
underestimates (also after decay); determinism; rare scores above common;
half-life decay; bounded first-seen. fuzz_tokenizer found two
idempotence bugs (truncation before templating, and a cut that left a
" (deleted)" suffix), fixed here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every finding is a function of one group of indicators (a process, a
parent, a hash or a rule). IncrementalCorrelator keeps group membership
(ids and keys, not indicator contents) and rebuilds only the groups that
new or changed indicators touch. Each pass used to copy every indicator
and rebuild every finding; it now costs about the size of what changed.
- correlation_engine.cpp: the four group loops become per-group
functions (re-indented; logic unchanged), shared by the full pass and
the incremental one.
- EvidenceView keeps a bounded change log of written indicator ids
(resets after clear(), at first use, or past 65536 entries);
Database::take_indicator_changes / find_indicators expose it, with
rolled-back writes reported as removed.
- DetectionPipeline holds its lock only to read changes and members and
to store findings, as before.
Fix found by the oracle test: an indicator naming a process node but no
pid formed a second group for the same process with the same finding id,
and the two findings overwrote each other. The group pid now comes from
the node id when the pid is missing (finding ids are unchanged).
tests/test_correlation_incremental.cpp: random batches of new and updated
indicators give exactly the findings the full pass after every batch
stored (I1, I2); a batch touching one process rebuilds only its groups
(I3); the store's change log, including a rolled-back write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File node ids: include/strawberry/node_ids.hpp is now the only place
they are built (file_<sha256> or file_path_<sha256 of path>). The three
modules disagreed: correlation used a 16-digit path hash and behavior
detections the raw path, so path-keyed evidence never met the rest. An
exec detection with a known hash also writes a same_path edge from the
path id to the content id. Source invariant R8 rejects hand-built file
ids in src/ and include/.
Processes carry the real uid and kernel comm: from the eBPF exec event
(which now includes the uid) or /proc. Stored on process nodes and in
process_exec telemetry (fields 14, 15).
Command lines are redacted before any storage (capture_redact_cmdline,
default on): sensitive flags and KEY=value pairs, Authorization/Bearer
credentials, URL passwords, tokens used as URL user names, well-known
token shapes (GitHub, GitLab, Slack, AWS key ids, sk-, JWTs) and mysql
-p<password>. The rules repeat to a fixed point, so redaction is
idempotent by construction. tests/test_redact.cpp covers the rules plus
idempotence, size and planted-secret properties; fuzz_cmdline_redactor
found two idempotence bugs and a leak (--password --token X) that are
fixed here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 increment 1. Until now only detections were stored, so the
learning stages would never see a normal example.
- TelemetrySink: bounded lock-free multi-producer queue (8192); try_emit
never blocks, allocates or throws; a full queue drops and counts.
- A guarded writer thread drains it every 2 s (or at a quarter full)
into the store's root-only telemetry stream through the pipeline,
writing a gap record whenever events were dropped.
- The pipeline emits one process_exec (pid, parent, node ids, exe,
capped cmdline, sha256, deleted/memfd, live/inventory) and one
scan_result (outcome: scanned, cache_hit, too_large, unavailable,
read_error, no_rules, yara_error, pid_scan; bytes, scan time, rules
generation, matched rules) for every handled exec, detection or not.
- Config telemetry=true (default); status gains a telemetry block
(emitted, dropped, queued, written, gaps).
Tests: C1 (never blocks; exactly once and in order under concurrency,
clean under ThreadSanitizer), record round-trips, C2 (one process_exec
and one scan_result per handled exec, stored in the telemetry stream);
fuzz_telemetry_records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 1 increment 7.
- Disk floor: below max(store_min_free_mb, store_min_free_pct) free,
telemetry is refused and counted (a gap record with the count and
time range is written once space returns) or, with
store_floor_action=evict_oldest_telemetry, the oldest sealed
telemetry is deleted; evidence continues until store_hard_min_free_mb.
- Retention (telemetry only, off by default): store_retention_days and
store_max_mb, applied when a segment is sealed. Evidence is never
deleted by retention.
- Status gains a store block (floor, free space, per-stream sizes,
compression ratio, drops, deletions); health reports a low disk,
a failed evidence store and unsaved detections.
- strawberry --store-verify checks the manifest and every segment
(read-only, safe while the daemon runs; root-only telemetry is noted,
not flagged). strawberry --store-compact rewrites sealed segments at
zstd level 19 with the daemon stopped, never touching a damaged one.
- The manifest records each sealed segment's uncompressed size.
Tests L7-L11 cover the floor, gaps, retention by age and size,
eviction, compaction and verify.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per the triage decisions:
- DeletedExecutable when a new file exists at the executable's path (an
update or reinstall) is stored as evidence only. A binary with nothing
left at its path still raises a low finding.
- scripts/install.sh writes /var/lib/strawberry/installed-files.sha256
(hashes of the units it installs, root-owned, written first). A
PersistenceWrite whose file matches its listed hash is evidence only;
other content at those paths, or a suspicious writer, is reported as
before. The daemon ignores a manifest that is not root-owned or is
group/world-writable.
Evidence-only indicators make no finding, alert or containment on their
own; correlation counts them when the same process has another signal.
Status reports pipeline.evidence_only. Tests cover the manifest, the
behavior hook, correlation and the pipeline (replaced vs deleted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing shipped used the library after the event store landed. A small
reader/writer for its file format (src/store/legacy_kv) replaces it for
the legacy code, and the submodule, subproject and library test are
removed.
A writer open now imports a legacy database.db file into the event
store: indicators are replayed in an order consistent with the old
hash/rule indexes, the result is verified against the old file, the
old file is kept as database.db.migrated-<ts>, and an import
interrupted between its final renames is finished on the next open.
Read-only handles read a legacy file directly and switch to the store
once it is imported. Ids re-stored under another hash are reported
(their old index entries cannot be carried over).
Tests: legacy import properties I1-I6 (locked and corrupt files are
left untouched), the differential test now shares its comparisons, and
fuzz_legacy_kv covers the file parser. A copy of a real 137 KB legacy
database imported cleanly into 7 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wire the policy, feeds, containment, exec guard, behavior, governor,
crash safety and event store into the daemon, detection pipeline, CLI
and meson build (zstd now required); Dockerfile, conan, systemd unit
(LimitCORE=0, resource limits) and installer updates; existing tests
adapted (store paths are directories now). Detections are counted and
notified even when evidence cannot be persisted (persist_failures in
status). prevention_timeout_ms is clamped and behavior dedupe handles
out-of-order events. Docs: API contract, developer guide, plan, UX,
agent rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>