Commit graph

26 commits

Author SHA1 Message Date
trvon
7149a90893
gate: refuse in-sample and degenerate evidence, and identify the model
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.

In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.

Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.

Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.

3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).

Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
2026-11-19 16:20:40 -07:00
trvon
169a05cd2a
fuzz: target the YARA rule partitioner
rule_partition.cpp parses rule sources (splitting rules, extracting top-level
condition identifiers and facts), but had no fuzz target, contradicting the
repository rule that every parser of attacker-influenced input gets one.
fuzz_rule_partition asserts the splitter is a total, exact partition of its
input, that rule/condition slicing stays within the rule, and that partition
selection agrees with magic bytes and the size bucket boundaries.

It is registered in fuzz/meson.build (with rule_partition.cpp), swept by
scripts/run_fuzzers.sh, and documented next to the other targets. Seeds cover
comments, strings, regexes, hex blocks, an include and a truncated source.
2026-11-17 13:45:12 -07:00
trvon
a1d77dda9a
governor: fail-safe host pressure, diagnostic diskstats, capped scan buffers
Host pressure stays fail-safe: any of CPU/I/O/memory PSI above its profile
threshold, or low MemAvailable, still defers background work and can degrade
the exec guard. That is deliberate — global I/O PSI also counts waits with no
block-device counter (FUSE/network-backed I/O, display page flips), so idle
disks never prove pressure is benign. An audit of an earlier version that
vetoed I/O PSI below 5% block utilization rejected it on exactly that basis.

/proc/diskstats is therefore diagnostic telemetry only, exposed as
host.block_io_util_pct. It is keyed by major:minor so a reused device name
cannot compare unrelated counters, and reports unknown (-1) for the first
sample, a new/replaced/reset device, or an oversized read, instead of a false
0%.

scan_buffer_budget() now derives from the same effective budget that pressure
uses — the configured/profile budget capped by cgroup memory.max — with the
existing 32 MiB floor, so a cgroup-capped daemon cannot be granted scan
buffers larger than its own limit. Lock order stays mutex_ -> buffer_mutex_.

Tests: deterministic fail-safe cases across every diskstat value, the
new-device regression, scanner budget under a 100 MiB cgroup cap, and a seeded
property asserting severe I/O PSI always holds work whatever diskstats says.
2026-11-17 13:45:07 -07:00
trvon
29e490dcab
Replay: add strict read-only training summaries
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
2026-11-03 06:54:17 -07:00
trvon
52b02f1dc7
Learn: preserve label provenance and bound replay sampling
Resolve allowlist and quarantine retractions without erasing independent support. Refresh capped entity reservoirs and bound allocated slot bookkeeping; retain deterministic seeded mutation regressions.
2026-11-03 06:53:28 -07:00
trvon
c565cb5aee
Learn: handle extreme restored rarity clocks
Represent full-range ordered clock differences without signed overflow and allow complete decay. Add seeded restore-clock regressions and a dedicated fuzz seed.
2026-11-03 06:52:20 -07:00
trvon
8f6bd1d4e0
Privacy: attribute tracker contacts to programs via OpenSnitch events
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Second piece of tracker attribution (docs/tracker-attribution.md): which
programs reach domains on a trackers list. Advisory only: never a block,
never a finding (source invariant R11).

- OpenSnitch event listener: OpenSnitch's JSON "remote" logger is the
  hostname source (Strawberry's eBPF connects have none). It streams over
  TCP and cannot leave fields out, so events carry command lines and
  environments. The listener binds 127.0.0.1 at a port below 1024 (only
  root can), accepts only root-owned peers (/proc/net/tcp), bounds
  clients/object size/buffering, and reads only pid, uid, path,
  destination and action; arguments and environment are never copied.
- TrackerContacts: per program x domain, bounded at 20000 keys. A root-only
  tracker_contact telemetry record (type 25) on the first contact, then at
  most hourly with the count since.
- Status `privacy` block; control verb privacy-trackers; CLI
  `strawberry privacy trackers [--json]`.
- `strawberry opensnitch enable-events` adds the logger to OpenSnitch's
  config (with a .bak); the installer runs it when OpenSnitch is present,
  unless privacy_trackers=false. Config: privacy_trackers (default true),
  opensnitch_events (default 127.0.0.1:979).
- EasyPrivacy is a default trackers feed (GPL-3.0+ or CC-BY-SA-3.0,
  downloaded at run time).

Tests: E1-E7 (test_opensnitch_events: event fields, stream splitting,
peer uid, address rules, listener end to end, contacts and record, config
editor), the control verb, CLI translation and help, daemon status. New
fuzz_opensnitch_events (90 s clean).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 06:15:28 -06:00
trvon
2e94ae1485
Privacy: tracker list feeds (advisory only, never block)
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
First piece of tracker attribution (docs/tracker-attribution.md): which
programs reach advertising and tracking domains, built with the
measurement rules of the CarolinaCon 2026 privacy talk (bridgescope):
a listed domain is a circumstantial label, never proof.

- Feed kind "trackers": hosts lines, plain domains and adblock
  whole-domain rules (||host^, ||host^$third-party). Cosmetic rules,
  exceptions, paths and other options are skipped; the comment cut used
  to turn "site.example##.ad" into a domain.
- TrackerSet: suffix matching (a.b.t.example matches t.example, never
  nott.example, an IP literal or a bare TLD); first feed reported.
- List parsing moves from rule_updater.cpp to ioc_list.cpp (no network
  code) so the query-side core can use it.
- Source invariant R11: tracker lists never meet exec-guard or OpenSnitch
  code. They are not in the OpenSnitch blocklist sync.

Tests: T1-T3 (test_tracker_set), trackers in the list-parsing property
and the list fuzzer (60 s clean), R11 self-test fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 05:25:53 -06:00
trvon
f76db466eb
Stage 4: keep the live rarity model across daemon restarts
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The shadow scorer's rarity model started empty at every daemon start, so
each restart cost an hour of warm-up with no decisions. The daemon now
saves it to model_dir/rarity.state (checksummed, atomic write, hourly and
at stop) and restores it at start. A restored state younger than the
7-day half-life skips the warm-up; decay catches up on the first event.

A file made with other rarity settings, a bad checksum, or a negative or
non-finite count is refused with a reason (status model.rarity_state,
doctor model.rarity_state) and the model starts empty. The file must be
owned by the daemon user and not group/world writable, like model files.

Also fixes status model.warmup_windows, which was never filled in.

Tests: R1/R2 in test_learn (round trip, decay clock, refusals), S8 in
test_shadow (warm-up skipped only when recent), daemon restart restores
the saved state, doctor warning; new fuzz_rarity_state target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 23:25:01 -06:00
trvon
bbfda149a5
Stage 4: execute a promoted model's inspections, under guardrails
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
A promoted model (promoted.sbm, written only by `model promote` after the
gate) can now request real inspections. Only two, both additive:

- memory_scan: through the daemon's memory-scan queue;
- deep_rescan: the root's image re-scanned on demand; new matches are
  recorded as indicators of kind model_inspection (findings titled "YARA
  match found by a deeper rescan the inspection model requested") and
  positive labels.

watch_subtree has no mechanism and rescan_on_next_rules is the retro-hunt's
job, so both are masked out of acting decisions (kExecutableActions), and
replay's policy evaluation uses the same action set.

Guardrails:
- InspectionExecutor (bounded queue, one low-priority worker, submit()
  never blocks); WorkClass::model_inspect lets the governor defer it, and
  a deferred decision is dropped.
- Each action rechecks that the promoted model is still active at the
  decision's generation (demoting or replacing it revokes queued work),
  that the root's node id and raw image path still match (decisions carry
  root_image, not the feature template), and skips protected processes.
  A deep rescan discards its result if the process, image or rules changed
  during the scan.
- An acting decision is durably written before it is dispatched; every
  attempt becomes an inspection_outcome record (queued, scanned, skipped,
  deferred, revoked, queue_full, failed).
- Source invariant R10: no file may couple learning types to the
  exec-guard, OpenSnitch or policy APIs (daemon_app.cpp and main.cpp, the
  wiring points, are reviewed by hand). Models never block, contain, kill
  or create findings directly.
- Config model_act and model_lambda; status model block reports acting
  stats; timings.model_inspection.

docs/evaluation/stage4-execution-gate.md amends the gate's method (not its
thresholds): executable action set, greedy promoted decisions, budgets as
estimated inspection seconds by scan profile.

Also: TerminalSafeBuf's destructor no longer lets an exception escape;
the health message clamps huge durations; replay timestamp overflow; the
Docker test build runs with -j2.

Tests: test_inspection_executor (every skip and execute path, revocation,
queue limits); shadow and replay acting cases; R10 fixture; fault injection
and invariant updates. Validation reported by the session that wrote it:
48/48 local, 46/46 Docker plus privileged live checks, 20 fuzz targets
(9.5M runs), three paired KPI runs without regressions (no acting model
was loaded, so active-inspection cost is not measured yet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 21:03:57 -06:00
trvon
0006a86ff4
Stage 4: shadow mode for the inspection model
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The daemon now scores live windows with the shadow model and logs the
inspection it would request. Nothing is executed.

- learn/shadow: a live replay engine (same windows and rarity as
  `strawberry replay`) fed every telemetry record on the writer thread.
  When a window closes, the model's value v gives each action a benefit
  v x reach - lambda x CPU cost; the best positive one is chosen if the
  hour's budget (10/30/120 CPU-s by scan profile) allows, else none.
  With probability 0.02 a uniformly random other action is taken instead,
  and every decision records its propensity for off-policy evaluation.
  Decisions are inspection_action telemetry records.
- Replay gains a live mode (no ranking bookkeeping, so memory stays
  bounded) and a window-closed callback.
- Daemon: model_shadow (default on) loads <model_dir>/shadow.sbm when it
  appears or changes, checked every minute; an invalid file is reported
  and not used. Status gains a `model` block (kernel, generation,
  decisions by action, exploration, would-be spend, budget refusals).
- learn_sources in src/meson.build: tests that compile the daemon from
  sources take the learning library whole.

Limits, documented: decisions come when a window closes (a few minutes
after its first exec), and the live rarity model starts empty at each
daemon start (the first hour is learned, not decided).

Tests: test_shadow (no model, greedy choices, budget never exceeded,
determinism, exploration rate and propensities, record codec); daemon
status; fuzz_telemetry_records covers inspection_action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 12:04:40 -06:00
trvon
713122f845
Read process facts on macOS without procfs
include/strawberry/platform_process.hpp answers what /proc answers on
Linux: running PIDs (proc_listallpids), executable path
(proc_pidpath), parent and start time (sysctl KERN_PROC_PID, readable
for every process), argv (KERN_PROCARGS2) and a boot id
(kern.bootsessionuuid). A process is identified by its start time in
microseconds since the epoch plus the boot id, so node ids survive PID
reuse as on Linux. It is header-only, so the per-test source lists stay
as they are.

Users on macOS: resolve_process_ref (default root), the pipeline's
image hashing and exec-decision actor, and YaraScanner::scan_process
(scans the image by path). A synthetic procfs root still takes the
procfs code, so the existing procfs tests run on macOS too.

The resource governor samples Mach/sysctl on macOS: own CPU from
getrusage, RSS from task_info, load from getloadavg, available memory
from host_statistics64 (free + inactive + purgeable + speculative).

The behavior engine knows macOS debuggers (debugserver, dtrace) and
kext loaders (kernelmanagerd, kmutil), and macOS temp and Downloads
paths count as drop locations. The Linux lists are unchanged.

The KERN_PROCARGS2 parser is portable: a property test (exact argv
recovery, truncation never invents data, total on junk) and the
fuzz_procargs target run on every platform. test_platform_process
checks the live helpers against this process; the daemon test reads
the stopped state through sysctl on macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:55:09 -06:00
trvon
fdd66e8bab
Stage 4 groundwork: window features, inspection MLP, model files
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Nothing runs in the daemon yet; these are the pieces the next increments
train and run in shadow.

- learn/features: a WindowSummary accumulated from a window's events
  (exec locations, memfd/deleted/uid0 execs, depth, connection classes
  and ports, persistence and drop writes, ptrace, module loads, scan
  outcomes, capped surprisal per tuple kind, bounded tokens) and
  featurize(): 48 explicit features scaled to about [0, 1] plus signed
  hashing of tokens into 208 dims, with the governor level and budget
  left as context. A window's own detections are never features: they
  are the teacher labels.
- learn/mlp: 256 -> 128 -> 64 ReLU with a policy head over the
  inspection actions (none, memory_scan, deep_rescan, watch_subtree,
  rescan_on_next_rules) and a value head. Scalar reference, AVX2+FMA
  chosen at runtime, NEON on aarch64; safe default (no inspection) on a
  wrong input size or an inconsistent model. Deterministic init per seed.
- learn/model_file: "SBMD" format with bounded dimensions, exact length,
  a sha256 trailer and finite weights checked on load, each failure
  named; atomic save into a 0700 directory.

Tests: test_model (kernels against scalar for arbitrary sizes, forward
pass invariants, deterministic init, round-trip, every flipped byte and
truncation rejected, non-finite weights rejected, features bounded and
deterministic). fuzz_model_file. NEON is built and tested only on aarch64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:53:25 -06:00
trvon
b55723547d
Add doctor checks and maintenance actions to the daemon
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
- doctor (control verb, read-only): store health and disk floor,
  telemetry growth per day and days until the floor when retention is
  unlimited, leftover store files, superseded findings, retro-hunt state,
  thread/write errors, logs (journald rotates them), model (none yet).
  Each check carries a maintenance action or advice. Pure checks over
  gathered inputs (src/doctor.cpp), so they test without a daemon.
- maintain (control payload verb, privileged, rate limited, audited):
  retire-superseded-findings and clean-store, each with dry_run.
- finding_retired evidence record: removes a finding from view; it stays
  in the stream's history. CorrelationEngine::superseded_findings finds
  old hash/rule findings by recomputing the stable id, and old lineage
  findings only when the current one's parent process is among their
  nodes (a reused pid on another boot is not a match). Lineage findings
  now record parent_process_node_id.
- store::inspect_store_files / clean_store_leftovers: orphan/ files older
  than N days and *.compact / MANIFEST.tmp temp files; never a listed
  segment.

Tests: test_doctor (check statuses, superseded detection including pid
reuse and no-replacement cases, retirement surviving a reopen, cleanup
keeping recent orphans and listed segments); daemon test exercises both
verbs over the socket. fuzz_evidence_records covers finding_retired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:27:23 -06:00
trvon
d54d987b3a
Retro-hunt: re-scan binaries seen before when the rules change
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The main source of teacher labels over time: rules improve daily, while
most malware runs once.

- The pipeline remembers every binary that ran (image_seen evidence
  record: sha256, path, first/last seen, size; at most one write per hash
  a day). Deleted and memfd images are skipped: nothing to find again.
- Each rules load computes a digest of the rule files and IOC hash lists
  (paths after resolving symlinks, sizes, mtimes).
- RetroHunter (src/retro_hunt.cpp, own thread, idle priority, paced at
  retro_hunt_pace_ms, deferred by the governor as WorkClass::retro_hunt)
  re-scans when the digest differs from the last completed pass, most
  recently seen first. Only paths that still hold the same content are
  scanned (O_NOFOLLOW, regular files, hash compared); others count as
  changed or missing. A completed pass is recorded (retro_hunt record) so
  a restart does not repeat it; an interrupted one starts over.
- New matches (rules not recorded before for that hash, not allowlisted)
  become indicators with scan_kind retro_hunt, a file/indicator graph
  link, a positive label and a notification. Correlation turns them into
  findings titled "YARA match on a binary that ran here (found after a
  rules update)". Nothing is blocked or contained from here.
- Replay reads positive labels from the evidence stream and marks windows
  that ran a binary labelled bad later (LATE in the CLI).
- Status: a retro_hunt block (API_CONTRACT.md); config: retro_hunt,
  retro_hunt_pace_ms.

Tests: test_retro_hunt (remembered once; passes only on a digest change
and survive a restart; a clean-at-exec binary matching a new rule becomes
an indicator, label, finding and notification; a replaced binary is
counted as changed, never scanned as the old one; a stop abandons a pass
unrecorded); test_replay R9 (late labels, from a real store).
fuzz_evidence_records covers the new records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 08:59:38 -06:00
trvon
4e25666a70
Inject scripted attack traces into replay and score them
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
- src/learn/trace.cpp: a small trace format (exec, connect, write,
  ptrace, module with +offsets and process aliases) compiled to
  telemetry records with pids above any pid_max and node ids that name
  the trace and copy.
- --replay --inject FILE [--inject-copies N]: copies are spread evenly
  over the ranked history (a first pass reads the time range), scored
  but never learned from, and checked against alert budgets of 1, 3 and
  10 per day: is the copy's best window above the (budget x days)-th best
  real window? Ties count against. Injection leaves real windows and
  their ranking unchanged.
- config/traces/standard.trace: the six plan traces (browser to curl|sh,
  systemd user persistence, memfd loader, ptrace injection, download
  then exec, python from a PDF viewer).
- docs/evaluation/stage3-rarity-preregistration.md: method and pass
  thresholds, written before any run on host data.

Tests: test_trace (standard traces, error messages with line numbers,
records); test_replay R8 (every copy found, real ranking unchanged, a
novel chain caught at 10/day and a bare shell not). fuzz_trace_dsl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 22:37:44 -06:00
trvon
92cb4e9ba8
Add --replay, and keep non-UTF-8 host strings from breaking JSON
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Replay (src/learn/replay.cpp, strawberry --replay [dir] [--top N] [--json]):
reads the telemetry stream read-only, orders events by time (a 30 s
reorder buffer absorbs the writer's batching), joins behavior events to
the latest exec of their pid at that time (keeps PID reuse apart), builds
60 s windows (a root exec, its descendants and their behavior, at most 64
events), scores each exec, connection and persistence write with the
rarity model before observing it, and marks windows whose processes had
detections. Prints the most unusual windows with the tuples that made
them unusual. Deterministic. tests/test_replay.cpp: chain windows, PID
reuse, order insensitivity within the horizon, determinism, ranking,
detection marking, a real store round trip. fuzz_replay feeds arbitrary
record streams.

UTF-8: file names are bytes, and the fuzzer showed nlohmann's dump()
throwing on a path that is not UTF-8. The same applied to the daemon's
control replies (kill/suspend/quarantine name the exe path) and to the
OpenSnitch containment rule, so a binary with such a name could not be
contained, and json_escape wrote the raw bytes into status JSON. Now:
- every .dump() in src/ uses error_handler_t::replace (U+FFFD), enforced
  by source invariant R9;
- json_escape writes invalid sequences as � (valid_utf8 and
  utf8_sequence_length in json_util.hpp);
- containment refuses such a path with a clear error (a rule could never
  match the real bytes); quarantine still blocks execution by hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:49:04 -06:00
trvon
134aebf7a9
Add the tokenizer and rarity model for the learning stages
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
src/learn/ (pure libraries; nothing in the daemon calls them yet, Stage 3
uses them in replay):

- tokenize: path templates (user homes, per-uid dirs, pids, digit and hex
  runs, UUIDs, mktemp-style names abstracted) and command-line shapes
  (flags kept, values, paths, URLs, IPs, numbers and encoded blobs
  classed; the first plain word kept as a subcommand; long argument lists
  summarized). Bounded and noexcept.
- rarity: decayed count-min sketches over parent>child, grandparent>
  parent>child, exe+command-line shape, exe+connection class and
  exe+persistence location, scored as surprisal in bits; a bounded
  first-seen table; deterministic hashing for replay. About 1.3 MiB.

tests/test_learn.cpp: tokenizer cases and idempotence; count-min never
underestimates (also after decay); determinism; rare scores above common;
half-life decay; bounded first-seen. fuzz_tokenizer found two
idempotence bugs (truncation before templating, and a cut that left a
" (deleted)" suffix), fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:21:28 -06:00
trvon
c452be93d4
Unify file ids, record uid/comm, redact command lines
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
File node ids: include/strawberry/node_ids.hpp is now the only place
they are built (file_<sha256> or file_path_<sha256 of path>). The three
modules disagreed: correlation used a 16-digit path hash and behavior
detections the raw path, so path-keyed evidence never met the rest. An
exec detection with a known hash also writes a same_path edge from the
path id to the content id. Source invariant R8 rejects hand-built file
ids in src/ and include/.

Processes carry the real uid and kernel comm: from the eBPF exec event
(which now includes the uid) or /proc. Stored on process nodes and in
process_exec telemetry (fields 14, 15).

Command lines are redacted before any storage (capture_redact_cmdline,
default on): sensitive flags and KEY=value pairs, Authorization/Bearer
credentials, URL passwords, tokens used as URL user names, well-known
token shapes (GitHub, GitLab, Slack, AWS key ids, sk-, JWTs) and mysql
-p<password>. The rules repeat to a fixed point, so redaction is
idempotent by construction. tests/test_redact.cpp covers the rules plus
idempotence, size and planted-secret properties; fuzz_cmdline_redactor
found two idempotence bugs and a leak (--password --token X) that are
fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 17:58:48 -06:00
trvon
2044873b95
Record labels, generations, lifecycle and governor samples
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Teacher signals and context for the learning stages, written to the
evidence stream (retention never removes it):

- label: operator actions from the control socket (quarantine, contain,
  kill, suspend: positive; allowlist-add: negative; allowlist-remove and
  quarantine-remove: retracted) and allowlist suppressions (negative,
  once per entry, subject and rule). Never from the MCP.
- rules_generation: every rules load attempt, with sources, counts and
  the error of a failed reload.
- policy_generation: each policy change, with a sha256 of the policy.
- lifecycle: daemon start and stop.

governor_sample (pressure level, CPU/IO/memory, per-class admissions) goes
to the telemetry stream once a minute.

Database::append_evidence writes records the evidence view ignores.

Tests: record round-trips; suppression labels and dedupe and rules
generations in the pipeline test; operator labels, policy generations and
lifecycle in the daemon test; the record fuzzer covers the new decoders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 17:23:09 -06:00
trvon
b46e367e6d
Record YARA string hits as rule_match telemetry
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Each rule match from a fresh scan now writes a rule_match record: the
rule, namespace, severity, rules generation and its matched strings
(identifier, offset, length, 64-bit FNV-1a digest of the captured bytes).
The matched bytes themselves are never stored. Cache hits write no
rule_match, since their scan_result already names the rules.

Hits are collected in the libyara callback (libyara 4+) and capped at
16 strings per rule, 4 hits per string and 256 per scan. The callback
also tolerates a null user_data now.

Tests: caps, offsets and digests against a real scan; record round-trip
with the encode cap; the telemetry record fuzzer covers rule_match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 11:08:58 -06:00
trvon
8ab8c9b353
Record raw behavior events and every exec-guard verdict
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Stage 2 increment 2.
- Connect, ptrace and module-load events go to telemetry raw, before the
  behavior engine filters them (pid, uid, comm, address/port or target);
  the join to processes happens when the data is read.
- Persistence and drop-location writes go to telemetry before any rule
  decides on them.
- The exec guard reports every verdict through a new on_observation hook
  (allow verdicts and cache hits included; ExecDecision still reports
  only non-allow): verdict, answered, scanned, cache hit, degraded, top
  rule/severity and latency, as guard_decision records.

Tests: record round-trips; the live exec-guard test checks one
observation per checked exec, including allows and cache hits;
fuzz_telemetry_records covers the new decoders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 09:57:17 -06:00
trvon
a32e045412
Record every exec and scan result as telemetry
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Stage 2 increment 1. Until now only detections were stored, so the
learning stages would never see a normal example.

- TelemetrySink: bounded lock-free multi-producer queue (8192); try_emit
  never blocks, allocates or throws; a full queue drops and counts.
- A guarded writer thread drains it every 2 s (or at a quarter full)
  into the store's root-only telemetry stream through the pipeline,
  writing a gap record whenever events were dropped.
- The pipeline emits one process_exec (pid, parent, node ids, exe,
  capped cmdline, sha256, deleted/memfd, live/inventory) and one
  scan_result (outcome: scanned, cache_hit, too_large, unavailable,
  read_error, no_rules, yara_error, pid_scan; bytes, scan time, rules
  generation, matched rules) for every handled exec, detection or not.
- Config telemetry=true (default); status gains a telemetry block
  (emitted, dropped, queued, written, gaps).

Tests: C1 (never blocks; exactly once and in order under concurrency,
clean under ThreadSanitizer), record round-trips, C2 (one process_exec
and one scan_result per handled exec, stored in the telemetry stream);
fuzz_telemetry_records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 09:16:26 -06:00
trvon
dc50f06f8d
Drop unqlite-cpp and import legacy databases
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Nothing shipped used the library after the event store landed. A small
reader/writer for its file format (src/store/legacy_kv) replaces it for
the legacy code, and the submodule, subproject and library test are
removed.

A writer open now imports a legacy database.db file into the event
store: indicators are replayed in an order consistent with the old
hash/rule indexes, the result is verified against the old file, the
old file is kept as database.db.migrated-<ts>, and an import
interrupted between its final renames is finished on the next open.
Read-only handles read a legacy file directly and switch to the store
once it is imported. Ids re-stored under another hash are reported
(their old index entries cannot be carried over).

Tests: legacy import properties I1-I6 (locked and corrupt files are
left untouched), the differential test now shares its comparisons, and
fuzz_legacy_kv covers the file parser. A copy of a real 137 KB legacy
database imported cleanly into 7 KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:46:21 -06:00
trvon
adb40e5b62
Add compressed event store
zstd block codec (CRC32C, varint tagged records), segment files with
sealed footers and block index, crash-atomic MANIFEST, LogStore
(lock, recovery, rolls, orphaning, heal with backoff), the evidence
view, and the legacy unqlite store kept for import. Property tests
under injected faults and crashes, a differential test against the old
store, and fuzz targets for every decoder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:17 -06:00
trvon
6628ea9aea
Harden against crashes and I/O faults
Fix for the 2026-10-25 host freeze (an uncaught iostream exception on a
/proc read aborted the daemon; its core dump deadlocked with the exec
guard): non-throwing /proc reads, crash handlers that release fanotify
permission fds, guarded thread entries, from_chars parsing.

Adds the io fault-injection seam with property tests, source invariant
checks (R1-R6) with self-test fixtures, libFuzzer targets and seeds
(including I/O-fault fuzzing), the fold changed-files scan scripts and
the agent workflow docs (CLAUDE.md, prompts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:17 -06:00