Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.
Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
priority order (big non-PE first) under rule_partition_budget_pct (default
100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
partition, full_scans); config rule_partitions (default true) and
rule_partition_budget_pct.
Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.
- Exec guard (assess_exec) no longer copies an image before deciding: a cached
scan answers with no read; an image over guard_full_scan_max_bytes, or when
the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
(degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
ScanBufferLease): in-memory image copies bounded to memory budget / 4
(balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
memory_full_pct / throttled, and resources.scan_buffers; doctor check
resources.memory (problem when throttled at MemoryHigh, with the fix).
Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve independent support, typed rule namespaces and stable process lifetimes. Share authoritative managed-set reconciliation across daemon and offline correlation, load offline policy without seeding/migration, and release resources after shutdown-flush exceptions. Keep retirement history, transaction ownership and fault/allocation regressions; no whole-set crash-atomicity or live acceptance claim.
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
Second half of the sandbox-safe tracker transport. A small relay in the host
network namespace bridges OpenSnitch to the sandboxed daemon, so the daemon keeps
PrivateNetwork=true + RestrictAddressFamilies=AF_UNIX and never sees argv/env.
- `strawberry opensnitch relay` (new CLI verb, Map::flag -> --opensnitch-relay):
binds the OpenSnitch loopback port via the existing OpenSnitchEventListener
(root-peer check, argv/env dropped at parse), and forwards each redacted tuple
(encode_connection) to the daemon's AF_UNIX ingest socket, reconnecting on
failure. Foreground loop until SIGTERM/SIGINT; drops (not blocks) when the
socket is down. Linux-only. The listener callback runs on one thread, so the
sink needs no lock.
- scripts/strawberry-trackerd.service: runs the relay in the host namespace with
only CAP_NET_BIND_SERVICE, no PrivateNetwork, and /proc/net kept readable for
the peer check (no ProcSubset=pid); otherwise the same tight sandbox as
strawberry-update.service. install.sh installs+manifests+starts it when
OpenSnitch is present and privacy_trackers is not false.
- docs/tracker-attribution.md: replace the "known limitation" with the relay
architecture (rejected alternatives noted). API_CONTRACT: privacy.listener now
describes the ingest socket + relay; also documents feed-add max_severity (B1).
Suite 51/51; clang static analyzer and clang-tidy clean on main.cpp and
cli_commands.cpp. Full relay->OpenSnitch end-to-end needs host validation
(sudo + a real OpenSnitch); the socket path is covered by E9.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allowlisting a hash suppresses new detections but does not delete findings
already stored for it (e.g. the /usr/bin/ss false-positive storm). That cleanup
now lives in doctor instead of a one-off compaction.
- DetectionPipeline::retire_allowlisted_findings(dry_run): a finding is stale
when it references at least one hash and every hash it references (from
metadata "hashes" and file_<sha256> nodes) is allowlisted, so a finding that
also implicates a non-allowlisted binary is never dropped. Mirrors
retire_superseded_findings (retire_finding + transaction).
- node_ids.hpp gains content_hash_of_file_node_id() (the inverse of
file_node_id for the content case) so the pipeline parses file ids through the
canonical helper, not a "file_" literal (source invariant R8).
- Doctor check findings.allowlisted (warn) with maintenance action
retire-allowlisted-findings; DoctorInputs.allowlisted_findings gathered by the
daemon; run_maintenance handles the action; CLI `doctor fix` help + completion
and API_CONTRACT updated.
Tests: test_retire_allowlisted_findings (two execs of a flagged binary -> a
finding; allowlist the hash -> dry-run lists it, apply retires it, idempotent);
doctor check ok/warn cases; CLI completion string. Suite 50/50; clang static
analyzer, clang-tidy and source invariants clean on the changed TUs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second piece of tracker attribution (docs/tracker-attribution.md): which
programs reach domains on a trackers list. Advisory only: never a block,
never a finding (source invariant R11).
- OpenSnitch event listener: OpenSnitch's JSON "remote" logger is the
hostname source (Strawberry's eBPF connects have none). It streams over
TCP and cannot leave fields out, so events carry command lines and
environments. The listener binds 127.0.0.1 at a port below 1024 (only
root can), accepts only root-owned peers (/proc/net/tcp), bounds
clients/object size/buffering, and reads only pid, uid, path,
destination and action; arguments and environment are never copied.
- TrackerContacts: per program x domain, bounded at 20000 keys. A root-only
tracker_contact telemetry record (type 25) on the first contact, then at
most hourly with the count since.
- Status `privacy` block; control verb privacy-trackers; CLI
`strawberry privacy trackers [--json]`.
- `strawberry opensnitch enable-events` adds the logger to OpenSnitch's
config (with a .bak); the installer runs it when OpenSnitch is present,
unless privacy_trackers=false. Config: privacy_trackers (default true),
opensnitch_events (default 127.0.0.1:979).
- EasyPrivacy is a default trackers feed (GPL-3.0+ or CC-BY-SA-3.0,
downloaded at run time).
Tests: E1-E7 (test_opensnitch_events: event fields, stream splitting,
peer uid, address rules, listener end to end, contacts and record, config
editor), the control verb, CLI translation and help, daemon status. New
fuzz_opensnitch_events (90 s clean).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First piece of tracker attribution (docs/tracker-attribution.md): which
programs reach advertising and tracking domains, built with the
measurement rules of the CarolinaCon 2026 privacy talk (bridgescope):
a listed domain is a circumstantial label, never proof.
- Feed kind "trackers": hosts lines, plain domains and adblock
whole-domain rules (||host^, ||host^$third-party). Cosmetic rules,
exceptions, paths and other options are skipped; the comment cut used
to turn "site.example##.ad" into a domain.
- TrackerSet: suffix matching (a.b.t.example matches t.example, never
nott.example, an IP literal or a bare TLD); first feed reported.
- List parsing moves from rule_updater.cpp to ioc_list.cpp (no network
code) so the query-side core can use it.
- Source invariant R11: tracker lists never meet exec-guard or OpenSnitch
code. They are not in the OpenSnitch blocklist sync.
Tests: T1-T3 (test_tracker_set), trackers in the list-parsing property
and the list fuzzer (60 s clean), R11 self-test fixture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shadow scorer's rarity model started empty at every daemon start, so
each restart cost an hour of warm-up with no decisions. The daemon now
saves it to model_dir/rarity.state (checksummed, atomic write, hourly and
at stop) and restores it at start. A restored state younger than the
7-day half-life skips the warm-up; decay catches up on the first event.
A file made with other rarity settings, a bad checksum, or a negative or
non-finite count is refused with a reason (status model.rarity_state,
doctor model.rarity_state) and the model starts empty. The file must be
owned by the daemon user and not group/world writable, like model files.
Also fixes status model.warmup_windows, which was never filled in.
Tests: R1/R2 in test_learn (round trip, decay clock, refusals), S8 in
test_shadow (warm-up skipped only when recent), daemon restart restores
the saved state, doctor warning; new fuzz_rarity_state target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A promoted model (promoted.sbm, written only by `model promote` after the
gate) can now request real inspections. Only two, both additive:
- memory_scan: through the daemon's memory-scan queue;
- deep_rescan: the root's image re-scanned on demand; new matches are
recorded as indicators of kind model_inspection (findings titled "YARA
match found by a deeper rescan the inspection model requested") and
positive labels.
watch_subtree has no mechanism and rescan_on_next_rules is the retro-hunt's
job, so both are masked out of acting decisions (kExecutableActions), and
replay's policy evaluation uses the same action set.
Guardrails:
- InspectionExecutor (bounded queue, one low-priority worker, submit()
never blocks); WorkClass::model_inspect lets the governor defer it, and
a deferred decision is dropped.
- Each action rechecks that the promoted model is still active at the
decision's generation (demoting or replacing it revokes queued work),
that the root's node id and raw image path still match (decisions carry
root_image, not the feature template), and skips protected processes.
A deep rescan discards its result if the process, image or rules changed
during the scan.
- An acting decision is durably written before it is dispatched; every
attempt becomes an inspection_outcome record (queued, scanned, skipped,
deferred, revoked, queue_full, failed).
- Source invariant R10: no file may couple learning types to the
exec-guard, OpenSnitch or policy APIs (daemon_app.cpp and main.cpp, the
wiring points, are reviewed by hand). Models never block, contain, kill
or create findings directly.
- Config model_act and model_lambda; status model block reports acting
stats; timings.model_inspection.
docs/evaluation/stage4-execution-gate.md amends the gate's method (not its
thresholds): executable action set, greedy promoted decisions, budgets as
estimated inspection seconds by scan profile.
Also: TerminalSafeBuf's destructor no longer lets an exception escape;
the health message clamps huge durations; replay timestamp overflow; the
Docker test build runs with -j2.
Tests: test_inspection_executor (every skip and execute path, revocation,
queue limits); shadow and replay acting cases; R10 fixture; fault injection
and invariant updates. Validation reported by the session that wrote it:
48/48 local, 46/46 Docker plus privileged live checks, 20 fuzz targets
(9.5M runs), three paired KPI runs without regressions (no acting model
was loaded, so active-inspection cost is not measured yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon now scores live windows with the shadow model and logs the
inspection it would request. Nothing is executed.
- learn/shadow: a live replay engine (same windows and rarity as
`strawberry replay`) fed every telemetry record on the writer thread.
When a window closes, the model's value v gives each action a benefit
v x reach - lambda x CPU cost; the best positive one is chosen if the
hour's budget (10/30/120 CPU-s by scan profile) allows, else none.
With probability 0.02 a uniformly random other action is taken instead,
and every decision records its propensity for off-policy evaluation.
Decisions are inspection_action telemetry records.
- Replay gains a live mode (no ranking bookkeeping, so memory stays
bounded) and a window-closed callback.
- Daemon: model_shadow (default on) loads <model_dir>/shadow.sbm when it
appears or changes, checked every minute; an invalid file is reported
and not used. Status gains a `model` block (kernel, generation,
decisions by action, exploration, would-be spend, budget refusals).
- learn_sources in src/meson.build: tests that compile the daemon from
sources take the learning library whole.
Limits, documented: decisions come when a window closes (a few minutes
after its first exec), and the live rarity model starts empty at each
daemon start (the first hour is learned, not decided).
Tests: test_shadow (no model, greedy choices, budget never exceeded,
determinism, exploration rate and propensities, record codec); daemon
status; fuzz_telemetry_records covers inspection_action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The macos job (macos-15, Homebrew deps, Qt app skipped) builds and
runs meson test. It only runs on GitHub: Forgejo also reads
.github/workflows and has no macOS runner.
README: macOS build, the Linux to macOS backend map, ES requirements
and installed files. API_CONTRACT: macOS socket path and peer
credentials, the endpoint_security and libproc backend values, and
the macOS status fields. DEVELOPER: platform map and the GitHub +
Forgejo remotes. AGENTS: current state of the port, including that
the ES backends have not run live yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- doctor (control verb, read-only): store health and disk floor,
telemetry growth per day and days until the floor when retention is
unlimited, leftover store files, superseded findings, retro-hunt state,
thread/write errors, logs (journald rotates them), model (none yet).
Each check carries a maintenance action or advice. Pure checks over
gathered inputs (src/doctor.cpp), so they test without a daemon.
- maintain (control payload verb, privileged, rate limited, audited):
retire-superseded-findings and clean-store, each with dry_run.
- finding_retired evidence record: removes a finding from view; it stays
in the stream's history. CorrelationEngine::superseded_findings finds
old hash/rule findings by recomputing the stable id, and old lineage
findings only when the current one's parent process is among their
nodes (a reused pid on another boot is not a match). Lineage findings
now record parent_process_node_id.
- store::inspect_store_files / clean_store_leftovers: orphan/ files older
than N days and *.compact / MANIFEST.tmp temp files; never a listed
segment.
Tests: test_doctor (check statuses, superseded detection including pid
reuse and no-replacement cases, retirement surviving a reopen, cleanup
keeping recent orphans and listed segments); daemon test exercises both
verbs over the socket. fuzz_evidence_records covers finding_retired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The main source of teacher labels over time: rules improve daily, while
most malware runs once.
- The pipeline remembers every binary that ran (image_seen evidence
record: sha256, path, first/last seen, size; at most one write per hash
a day). Deleted and memfd images are skipped: nothing to find again.
- Each rules load computes a digest of the rule files and IOC hash lists
(paths after resolving symlinks, sizes, mtimes).
- RetroHunter (src/retro_hunt.cpp, own thread, idle priority, paced at
retro_hunt_pace_ms, deferred by the governor as WorkClass::retro_hunt)
re-scans when the digest differs from the last completed pass, most
recently seen first. Only paths that still hold the same content are
scanned (O_NOFOLLOW, regular files, hash compared); others count as
changed or missing. A completed pass is recorded (retro_hunt record) so
a restart does not repeat it; an interrupted one starts over.
- New matches (rules not recorded before for that hash, not allowlisted)
become indicators with scan_kind retro_hunt, a file/indicator graph
link, a positive label and a notification. Correlation turns them into
findings titled "YARA match on a binary that ran here (found after a
rules update)". Nothing is blocked or contained from here.
- Replay reads positive labels from the evidence stream and marks windows
that ran a binary labelled bad later (LATE in the CLI).
- Status: a retro_hunt block (API_CONTRACT.md); config: retro_hunt,
retro_hunt_pace_ms.
Tests: test_retro_hunt (remembered once; passes only on a digest change
and survive a restart; a clean-at-exec binary matching a new rule becomes
an indicator, label, finding and notification; a replaced binary is
counted as changed, never scanned as the old one; a stop abandons a pass
unrecorded); test_replay R9 (late labels, from a real store).
fuzz_evidence_records covers the new records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The KPI detect phase still dropped events after the correlation fix.
Cause: every detection kept ~12 KB of evidence in memory, 20k detections
pushed RSS past the 512 MB budget, the governor went to emergency, and in
emergency every live exec scan waited 5 s, so the queue overflowed and
stayed full (memory that is evidence never comes back by waiting).
- The governor holds live exec scans for at most 5 s per emergency
episode, then lets them run (counted as forced); background work still
waits.
- Repeated detections of the same image and rules: the first 20 per
10-minute window are stored in full, then one summary indicator per
key keeps the count and first/last pid and time (rewritten at most
every 30 s). All detections are counted; alerts repeat at most once a
minute per key. Status reports pipeline.repeats_aggregated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 1 increment 7.
- Disk floor: below max(store_min_free_mb, store_min_free_pct) free,
telemetry is refused and counted (a gap record with the count and
time range is written once space returns) or, with
store_floor_action=evict_oldest_telemetry, the oldest sealed
telemetry is deleted; evidence continues until store_hard_min_free_mb.
- Retention (telemetry only, off by default): store_retention_days and
store_max_mb, applied when a segment is sealed. Evidence is never
deleted by retention.
- Status gains a store block (floor, free space, per-stream sizes,
compression ratio, drops, deletions); health reports a low disk,
a failed evidence store and unsaved detections.
- strawberry --store-verify checks the manifest and every segment
(read-only, safe while the daemon runs; root-only telemetry is noted,
not flagged). strawberry --store-compact rewrites sealed segments at
zstd level 19 with the daemon stopped, never touching a damaged one.
- The manifest records each sealed segment's uncompressed size.
Tests L7-L11 cover the floor, gaps, retention by age and size,
eviction, compaction and verify.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per the triage decisions:
- DeletedExecutable when a new file exists at the executable's path (an
update or reinstall) is stored as evidence only. A binary with nothing
left at its path still raises a low finding.
- scripts/install.sh writes /var/lib/strawberry/installed-files.sha256
(hashes of the units it installs, root-owned, written first). A
PersistenceWrite whose file matches its listed hash is evidence only;
other content at those paths, or a suspicious writer, is reported as
before. The daemon ignores a manifest that is not root-owned or is
group/world-writable.
Evidence-only indicators make no finding, alert or containment on their
own; correlation counts them when the same process has another signal.
Status reports pipeline.evidence_only. Tests cover the manifest, the
behavior hook, correlation and the pipeline (replaced vs deleted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every catch now handles the exception or records it with
STRAWBERRY_NOTE_SWALLOWED() (strawberry/swallowed.hpp): an atomic count,
exported in status as resources.swallowed_exceptions, plus a file:line
log line written with write(2), so it is safe out of memory and in
destructors. The 16 empty catches (store error fallbacks, destructors,
YARA compiler callback, thread entry logging) now use it. Status also
reports thread_errors, which was counted but never exported.
Enforced by source invariant R7 (with a self-test fixture) and by
clang-tidy bugprone-empty-catch in a new .clang-tidy (errors; inherits
parent config). AGENTS.md and CLAUDE.md state the rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wire the policy, feeds, containment, exec guard, behavior, governor,
crash safety and event store into the daemon, detection pipeline, CLI
and meson build (zstd now required); Dockerfile, conan, systemd unit
(LimitCORE=0, resource limits) and installer updates; existing tests
adapted (store paths are directories now). Detections are counted and
notified even when evidence cannot be persisted (persist_failures in
status). prevention_timeout_ms is clamped and behavior dedupe handles
out-of-order events. Docs: API contract, developer guide, plan, UX,
agent rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>