Commit graph

97 commits

Author SHA1 Message Date
trvon
1c9b005645
store: incremental read-only refresh (fixes the tray spinning a core)
Root cause, from the Tracy capture the CPU watcher took when the tray spun
(2026-11-15 08:15, 67% for 15 s): database.reload_read_only rebuilt the whole
evidence view from every segment — mean 1.4 s, max 1.7 s, 4x in 20 s — because
every 3 s poll that saw the daemon append evidence triggered a full reload.

- store::scan_store_read_only_from + ReadCursor (per segment: id, sealed,
  bytes, records delivered). When the previously seen segments are an
  unchanged prefix (same ids, sealed ones byte-identical), only records
  appended since are delivered: the rest of the previously last segment
  (skipping what was applied) and new segments. Any other change —
  compaction, retention, reduction, a rewritten or shrunk segment — delivers
  everything and the caller rebuilds.
- Database read-only refresh uses it; records apply in the same order a full
  rebuild uses, so the view is identical. read_only_reload_counts() reports
  full vs incremental refreshes.

Live, on the real evidence store: the change-driven reload went from ~1.4 s
to 2.8 ms; the tray did ~97 ms of work in a 60 s window.

Test (test_incremental_view, seeded property): a long-lived reader
refreshing across random appends, upserts, seals and a compaction always
equals a freshly opened reader; appends refresh incrementally (42-49 per run),
the compaction forces a rebuild, and no change reports no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:46:00 -07:00
trvon
db4c49155f
yara: rule partitions — scan each file only with the rules that can match it
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.

Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
  after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
  pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
  size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
  kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
  priority order (big non-PE first) under rule_partition_budget_pct (default
  100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
  would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
  and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
  differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
  partition, full_scans); config rule_partitions (default true) and
  rule_partition_budget_pct.

Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:58:39 -07:00
trvon
4586eed50d
yara engine: range/fast-mode scanning API, ELF load ranges, engine A/B bench, opt-in guard partial scan
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
  full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
  builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
  differences, but the corpus has no rule matches, so parity is not proven.

- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
  zero-copy blocks; file offsets and filesize keep file semantics) with
  EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
  bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
  loaded segments of ELF images >= 4 MiB at launch; such results are marked
  `partial`, never cached, and the pipeline still scans the whole file. Status
  pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
  not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).

Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:51:10 -07:00
trvon
5685600e57
Daemon memory: budget scan buffers, stream large images on the launch path, report throttling
2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.

- Exec guard (assess_exec) no longer copies an image before deciding: a cached
  scan answers with no read; an image over guard_full_scan_max_bytes, or when
  the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
  new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
  (degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
  ScanBufferLease): in-memory image copies bounded to memory budget / 4
  (balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
  an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
  checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
  memory_full_pct / throttled, and resources.scan_buffers; doctor check
  resources.memory (problem when throttled at MemoryHigh, with the fix).

Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:36:14 -07:00
trvon
758ef9db07
fleet: node-share cap never below the fair share 1/N
Seen in the dockerized 3-sensor run: max_node_share=0.25 (pillar default) with
3 similar-sized nodes cut every node to 25% of the total, dropping 158/468
windows (34%) while correcting no imbalance. The cap is now
max(max_node_share, 1/N) * total, so a balanced fleet is never trimmed and
only nodes above an equal share are cut. Test: balanced 3-node fleet keeps
all windows; a dominant 4th node is still trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:29:06 -07:00
trvon
8f4cb586b8
Rules R2: node-local compiled-rules cache keyed on libyara version + sources
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:54:28 -07:00
trvon
ebade1ba80
Store 7c/7d: tunable merge horizon + CPR interleave-aware merge mode
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:09 -07:00
trvon
4436bee457
Fleet F1: merge dedup per node+window, overlay newest-per-node, export --since
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:03 -07:00
trvon
00af731088
Fleet E1: export egress dir + age/size retention (produce + prune synced bundles)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:14 -07:00
trvon
1ee208fb4b
Stage 7e: scheduled stop->reduce->start telemetry-reduction maintenance (opt-in)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
cbcb9896cd
Stage 7c: NodeMerge collapse redundant repeated benign accesses on live nodes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
9aa9c9bb46
Stage 7b: minimal reachable skeleton (skeletonize dead-benign nodes far from labels)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
69d86bbbc3
Rules R1: ship pinned yara-forge-core source snapshot in the release (no per-node fetch)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:12 -07:00
trvon
c679a03561
Stage 7a: LogGC dead benign subtree GC (causality-preserving telemetry reduction)
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Successful in 53m38s
release / publish (push) Has been skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-10 19:32:12 -07:00
trvon
58907eb45e
Stage 3.5d: export attribution signal (detection->dropper origin hashes)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 56m28s
release / build (ubuntu:24.04, noble) (push) Successful in 59m8s
release / publish (push) Has been skipped
Ship 3.5c's provenance attribution to the central learner. ExportLabel gains two
additive uint64 tags (8 origin_node_hash, 9 origin_exe_hash); a positive window's
label now carries token_hash(dropper node) + token_hash(path_template(dropper
exe)) from ReplayWindow.origin_*. fleet_merge counts windows_with_origin.

R12/R13: a FIELD addition, not a new record type -- no codec.hpp/enum change
(R12 untouched), both fields are uint64 hashes so no raw path/node string crosses
the export boundary (R13 green, no name-loophole). Back-compat additive:
kExportSchemaVersion stays 1; old bundles decode with origins 0; old readers
ignore tags 8/9. Positive labels only -- suppressed/negative carry no origin.

Tests: round-trip of the hash fields + defaults; extraction proves a dropper->exec
detected window's positive label carries a non-zero origin while a no-write
detected window and suppressed labels carry 0. source_invariants green.
A learned ranking head + TrainingExample plumbing are phase-2 (out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 18:04:15 -07:00
trvon
b448c51c1d
Stage 3.5c: provenance attribution (trace a detection to the dropper node)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 55m28s
release / build (ubuntu:24.04, noble) (push) Successful in 51m16s
release / publish (push) Has been skipped
Reuse the global write->exec linkage (recent_writes) to attribute a detection to
its ROOT CAUSE. recent_writes now maps path -> {writer_node, writer_exe}; when a
window is late_detected (retro-hunt flags a binary after it ran) or detected
(YARA), set ReplayWindow.origin_node/origin_exe from whoever WROTE the flagged
binary -- usually an earlier node than the exec leaf that ran it. Surfaced in
window_json. Deterministic provenance walk (first writer wins); replay-level only,
no export-schema/record-type/enum change (R12/R13 intact), self_written_execs
unchanged. A learned ranking head over ambiguous origins is the follow-up (3.5d).

Test R10: a dropper writes /tmp/evil in one window, a separate process execs it
and is flagged late -> the window's origin_node is the dropper, not the exec leaf;
no in-set write -> empty origin; deterministic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 17:21:55 -07:00
trvon
e5ee53cf18
Review fixes: write->exec feature constant, accurate comments, tag-safe CI
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 56m31s
release / build (ubuntu:24.04, noble) (push) Successful in 51m33s
release / publish (push) Has been skipped
Address code-review findings on this session's changes (no behavior change):
- Share the dense-slot index for the write->exec feature as
  learn::kWriteExecFeatureIndex; featurize() and the model-score dev tool now
  reference it instead of a duplicated literal 42 (prevents silent column drift
  if the feature layout shifts).
- Fix the self_written_execs comments: the linkage is a GLOBAL bounded FIFO
  (recent_writes), not strictly "this window" — the docs now match replay.cpp.
- release.yml: never cancel a tag/release run (cancel-in-progress only for
  non-tag refs); push ordering doesn't guarantee the tag event arrives after the
  branch event, so a same-sha branch push must not abort an in-progress release.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 16:09:03 -07:00
trvon
0282066340
Stage 3.5b2: replace exec-lineage with write->exec data-flow feature
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been cancelled
3.5b's ordered exec-lineage proved inert in clean A/Bs; replace it with a general
write->exec data-flow feature: WindowSummary.self_written_execs (exec of a file
written earlier in the same window) at x[42], tracked via OpenWindow.written_paths.

Honest status: this is the conceptually-correct provenance feature (it separates
download->exec-the-same-file from download->exec-unrelated), but it ALSO shows no
AUC gain on detector-labeled data -- for the same structural reason as 3.5b: the
teacher labels (DownloadThenExecute etc.) correlate with existing count/location
features or re-derive the detector. Validating provenance LEARNING requires the
novel-chain generalization benchmark (train on detector-covered, hold out
detector-evading attacks); committed as that benchmark's feature foundation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 16:37:06 -07:00
trvon
8544e921a2
Stage 3.5b: ordered lineage-path bigram features
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 28m43s
release / build (ubuntu:24.04, noble) (push) Successful in 26m22s
release / publish (push) Has been skipped
Feed ordered parent->child exe-template bigrams into the window feature vector
(WindowSummary.lineage_edges, note_lineage_edge, a salted hash loop in featurize,
wired at replay on_exec). The model already got per-kind lineage surprisal
(x[25..29]); this adds path ORDER, which the unordered token bag discarded.

Clean A/B on a fixed corpus (same 697 windows, same flags, only the binary
differs): model AUC 0.734 -> 0.737 (rarity 0.520 unchanged). Null on this
shallow-lineage data (marker execs are 1-process, synthetic data flat) but the
feature is correct and order-sensitive (test: A>B != B>A). Kept as the path
representation that Stage 3.5c's retro-hunt attribution head needs; its payoff is
expected on real multi-stage attack telemetry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 14:22:14 -07:00
trvon
2cc137b8e7
Stage 3.5a: fleet-train --max-node-share/--test-fraction/--split
Expose the merge cap and train/test split on `strawberry fleet train` so a
single-node real-attack bundle can pre-seed a model. Default stays temporal +
cap 0.25 (behavior-preserving); `--split random` + `--max-node-share 1` lets the
Caldera-attack export (15 positive windows) train. Verified: real-only train AUC
1.0 vs rarity 0.563; default still refuses identically.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 12:18:32 -07:00
trvon
39912f6e64
Stage 6: wire config node_id to export (stable node identity; gap #2) 2026-11-08 09:05:25 -07:00
trvon
446e5c3afe
Stage 6: export populates fleet_bits from a local overlay (export_store fleet_surprisal callback; run_export loads fleet-overlay.sbo) 2026-11-08 08:09:47 -07:00
trvon
9313134aee
Stage 6: populate WindowSummary.fleet_bits from a fleet overlay during scoring (ReplayOptions.fleet_surprisal callback) 2026-11-08 08:01:40 -07:00
trvon
97374a16d2
Stage 6: fleet-overlay feature — per-kind fleet_bits[5] in featurize (reserved slots x[37..41], back-compatible) 2026-11-08 07:55:40 -07:00
trvon
7ad7596bda
Stage 6: load fleet overlay + sketch_surprisal_bits (query fleet-wide rarity for a tuple) 2026-11-08 05:38:00 -07:00
trvon
c939c3536b
Stage 6: fleet rarity overlay — sum per-node count-min sketches and serialize as an overlay bundle 2026-11-07 21:28:35 -07:00
trvon
ae4555ec38
Stage 6: central fleet merge core — export bundles to capped TrainingExamples (max_node_share) 2026-11-07 21:12:42 -07:00
trvon
6a028810da
Stage 6: export rarity sketches (export_rarity_delta) from the saved RarityModel state; CLI attaches them when present 2026-11-07 20:47:53 -07:00
trvon
45f13f7f16
Stage 6: extract windows/features/labels from the store via the replay pipeline into an export bundle 2026-11-07 20:20:41 -07:00
trvon
b5bb1689f9
Stage 6: export rarity-delta record (per-TupleKind count-min, bounded decode) + harden R13 for braced initializers 2026-11-07 19:39:22 -07:00
trvon
fdd699f037
Stage 6: export payload records (window/feature/label, hybrid token hashes) + R13 redaction invariant 2026-11-07 19:32:47 -07:00
trvon
23e70fe823
Stage 6: export bundle format (fleet_export) — sealed segment reusing the store codec; RecordType export_* + R12 2026-11-07 19:12:40 -07:00
trvon
b7336de261
Stage 6: node identity (fleet.hpp) — pure node_uid + wrapper 2026-11-07 18:59:51 -07:00
trvon
74aa9664ad
Pipeline: keep oversized cache refusals unscanned
Do not turn an empty cached digest into an unbounded descriptor hash. Distinguish size refusals from YARA results so the exec guard applies its own bounded scan/degrade policy. Cover cached read budgets across size boundaries, known hashes, inventory/live events and real guard-cache reuse. Fold/live/KPI and post-rollout CPU acceptance remain pending; OpenSnitch checksum work is separate.
2026-11-03 06:57:35 -07:00
trvon
be237d6040
Findings: reconcile evidence and harden policy rebuilds
Preserve independent support, typed rule namespaces and stable process lifetimes. Share authoritative managed-set reconciliation across daemon and offline correlation, load offline policy without seeding/migration, and release resources after shutdown-flush exceptions. Keep retirement history, transaction ownership and fault/allocation regressions; no whole-set crash-atomicity or live acceptance claim.
2026-11-03 06:55:47 -07:00
trvon
29e490dcab
Replay: add strict read-only training summaries
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
2026-11-03 06:54:17 -07:00
trvon
52b02f1dc7
Learn: preserve label provenance and bound replay sampling
Resolve allowlist and quarantine retractions without erasing independent support. Refresh capped entity reservoirs and bound allocated slot bookkeeping; retain deterministic seeded mutation regressions.
2026-11-03 06:53:28 -07:00
trvon
c565cb5aee
Learn: handle extreme restored rarity clocks
Represent full-range ordered clock differences without signed overflow and allow complete decay. Add seeded restore-clock regressions and a dedicated fuzz seed.
2026-11-03 06:52:20 -07:00
trvon
4ade69ec7e
Learn: allowlisted binaries are negative labels, not teacher positives
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The teacher-label join made a window a positive if it was detected
(replay.cpp), but it never applied the allowlist, so a window running a cleared
false positive (e.g. /usr/bin/ss) was labelled a teacher positive — poisoning
Stage 4 training. Negative labels were written to the stream but replay dropped
them.

- replay_store now collects negative labels into ReplayOptions.negative_labels
  (it used to ignore any non-positive label).
- A window whose exec hash is negatively labelled is marked `suppressed`; the
  example/policy-eval positive is gated on `!suppressed`, so such a window is
  kept as a *negative* training example (useful signal) and never a positive,
  even if a cleared rule "detected" it.
- DetectionPipeline seeds a negative label for every allowlisted sha256 at init,
  so evidence stored before the allowlist (the historical ss storm) is covered
  without waiting for a re-exec.

Test: a detected window with a stored negative label for its hash is a positive
before the label and a kept-negative after. Suite 51/51; analyzer + clang-tidy
clean on replay.cpp and detection_pipeline.cpp.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 22:02:56 -06:00
trvon
db654cc217
Correlation honors the allowlist, so cleared false positives stop regenerating
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Correlation runs over the whole store and never consulted the allowlist, so an
allowlisted hash (e.g. the /usr/bin/ss feed false positive the operator cleared)
regenerated the same findings on every pass from already-stored indicators —
`doctor fix retire-allowlisted-findings` cleared them but they came straight back
(~9,400 findings observed again). This also meant the findings were a poisoned
teacher-label source for training.

- classify() now skips an indicator whose (hash, path, rule) is allowlisted, the
  single chokepoint shared by CorrelationEngine::build_findings (the oracle) and
  IncrementalCorrelator::update, so both agree. New AllowlistFn; build_findings /
  correlate take an optional predicate (default = no filtering, existing tests and
  the incremental==full oracle unchanged); IncrementalCorrelator::set_allowlist.
- DetectionPipeline wires the predicate to PolicyStore::match, so the live
  incremental path skips allowlisted evidence. Retiring the findings now sticks.

Test: build_findings with an allowlist predicate emits nothing for the matched
hash and is unaffected for others. Suite 51/51; analyzer + clang-tidy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 19:52:36 -06:00
trvon
c77bc79dc7
Privacy: macOS tracker source via Little Snitch log-traffic (WS2 macOS)
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Completes the tracker-attribution transport on macOS. There is no network
sandbox to work around there, so the daemon (root) reads Little Snitch's traffic
stream directly and feeds the same on_opensnitch_connection consumer — no relay
or ingest socket (those are the Linux sandbox answer).

- LittleSnitchParser (cross-platform, unit-tested): parses `littlesnitch
  log-traffic` CSV via a header-name->index map (tolerates column reordering),
  yielding the shared redacted OpenSnitchConnection for outbound rows only
  (program, uid, host/ip/port, protocol 6->tcp/17->udp). Little Snitch's CSV has
  no argv/env, so redaction is structural. split_csv_line handles quoted fields.
- LittleSnitchSource (macOS only): posix_spawns `littlesnitch log-traffic
  --stream`, reads the pipe on a guarded thread, parses and calls back; stop()
  SIGTERMs the CLI (EOF -> loop ends) then reaps it.
- Daemon: start_tracker_attribution gains an __APPLE__ branch that starts the
  source; shutdown/status handle it. New config littlesnitch_cli (default CLI
  path). status privacy.listener source "little-snitch".
- docs: macOS section rewritten (needs Little Snitch "Allow access via Terminal").

Test E10 covers the parser (header, outbound/inbound, deny, quoted exe, no-dest).
Linux suite 51/51; analyzer + clang-tidy clean (Linux view). The __APPLE__ code
compiles/validates on the Mac next (ssh); live spawn needs root + LS Terminal access.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 18:54:27 -06:00
trvon
b09c416b32
Privacy: daemon-side AF_UNIX ingest socket for the tracker relay (WS2a)
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
First half of the sandbox-safe tracker transport (architecture deep dive). The
daemon's old loopback TCP listener could not bind under its own sandbox
(PrivateNetwork=true + RestrictAddressFamilies=AF_UNIX). Replace it with an
AF_UNIX ingest socket the daemon *can* bind under the sandbox; a host-namespace
relay (next change) will bind the OpenSnitch port, strip argv/env, and forward
redacted tuples here.

- opensnitch_events.hpp/cpp: encode_connection / decode_connection serialize only
  the redacted fields (never argv/env, which the parser already drops) as one
  JSON object — the relay->daemon wire form. New TrackerIngestServer: an AF_UNIX
  stream listener (Linux-only, like the TCP listener) that accepts only a root
  peer (SO_PEERCRED), frames with take_json_objects, decodes and calls back;
  bounded clients/object/buffer as before.
- daemon: start_tracker_attribution now opens the ingest socket
  (opensnitch_ingest_socket, default /run/strawberry/trackers.sock) instead of
  the in-sandbox TCP listener, so it works with the sandbox unchanged and is
  simply idle until a relay connects. status privacy.listener reports the ingest
  socket + relay_events; shutdown stops it. New config key opensnitch_ingest_socket.
- Tests: E8 encode/decode round-trip + redaction (no args/env on the wire);
  E9 TrackerIngestServer end to end (connect, two tuples, callback, stats);
  daemon-runtime configs set privacy_trackers=false (not under test).

Suite 51/51; clang static analyzer and clang-tidy clean on the changed TUs. The
relay run-mode + systemd unit + install wiring + docs follow in WS2b/WS2c.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 17:50:16 -06:00
trvon
4664ae4458
Detection: unify the two repeat-aggregation sinks + IOC severity resolution
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The scan path (handle_process) and the exec-guard recording path
(record_exec_decision) had grown two near-identical repeat-aggregation maps
(repeats_ / exec_repeats_) plus three copies of the IOC-feed severity idiom.
Refactor to one implementation; no behavior change (the existing aggregation and
IOC-severity tests are the oracle and pass unchanged).

- New include/strawberry/repeat_aggregator.hpp: RepeatAggregator owns the
  per-key window map, the 20-full-then-summary cadence and the aggregated count;
  observe(key, now, seen_ns, pid) returns {full, at_threshold, aggregated,
  emit_summary, notify, state}. Each caller keeps its own key, summary-id prefix,
  summary shape and notify behaviour. Both pipeline sinks now hold one instance
  (scan notifies at most hourly; exec passes notify_ns=0). status
  repeats_aggregated sums both totals (atomic, lock-free read).
- ioc_hit_severity(feed): one source for the three IOC match sites (scan_image,
  assess_exec, handle_process) that was the policy_->ioc_feed_severity ternary.
- Leaves the latency-critical assess_exec/decide path untouched; both sinks still
  run under mutex_, off the FAN response.

Unit test tests/test_repeat_aggregator.cpp (window reset, threshold, summary and
notify cadence, per-key independence). Suite 51/51; clang static analyzer and
clang-tidy clean on detection_pipeline.cpp; fold's 7 int-overflow are the accepted
monotonic-timestamp FP class (unchanged by the move); perf-kpi PASS, no regression
(exec_decision p50 unchanged, exec_assess/db_commit within noise or faster).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 17:39:48 -06:00
trvon
36982c4b56
Doctor: detect and retire findings for hashes you have since allowlisted
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Allowlisting a hash suppresses new detections but does not delete findings
already stored for it (e.g. the /usr/bin/ss false-positive storm). That cleanup
now lives in doctor instead of a one-off compaction.

- DetectionPipeline::retire_allowlisted_findings(dry_run): a finding is stale
  when it references at least one hash and every hash it references (from
  metadata "hashes" and file_<sha256> nodes) is allowlisted, so a finding that
  also implicates a non-allowlisted binary is never dropped. Mirrors
  retire_superseded_findings (retire_finding + transaction).
- node_ids.hpp gains content_hash_of_file_node_id() (the inverse of
  file_node_id for the content case) so the pipeline parses file ids through the
  canonical helper, not a "file_" literal (source invariant R8).
- Doctor check findings.allowlisted (warn) with maintenance action
  retire-allowlisted-findings; DoctorInputs.allowlisted_findings gathered by the
  daemon; run_maintenance handles the action; CLI `doctor fix` help + completion
  and API_CONTRACT updated.

Tests: test_retire_allowlisted_findings (two execs of a flagged binary -> a
finding; allowlist the hash -> dry-run lists it, apply retires it, idempotent);
doctor check ok/warn cases; CLI completion string. Suite 50/50; clang static
analyzer, clang-tidy and source invariants clean on the changed TUs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 11:37:52 -06:00
trvon
e93f8e37da
Detection: aggregate repeated exec-guard decisions; read /proc outside the lock
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The exec-guard would-block recording path stored one indicator (and, via
correlation's per-process grouping, one finding) per exec, with no aggregation,
unlike the scan path. A critical feed or rule matching a frequently-run binary
would flood evidence the way /usr/bin/ss did.

- record_exec_decision now collapses repeated identical decisions (keyed by
  verdict|hash|rule) with the same 10-min window / 20-full-then-summary scheme
  as handle_process (new exec_repeats_ map; counts into repeats_aggregated_).
  Beyond the window it refreshes one "exec-repeat" summary indicator and returns
  before the per-exec graph write.
- The actor's /proc identity reads are now done before taking mutex_, not under
  it. The clang static analyzer (new strict gate) flagged ::read in a critical
  section (unix.BlockInCriticalSection) here; this also honors the AGENTS rule
  that scans/reads stay off DetectionPipeline::mutex_. Analyzer now clean.

Test: test_repeated_would_block_is_summarized drives 30 identical would-block
decisions and asserts exactly 20 full exec-audit indicators + 1 exec-repeat
summary and repeats_aggregated == 10. Suite 50/50; analyzer and clang-tidy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 10:31:04 -06:00
trvon
6453d14589
Detection: IOC hash severity comes from per-feed trust, not a hardcoded critical
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
A benign system binary (/usr/bin/ss) whose SHA-256 was on a noisy community hash
feed produced thousands of critical "would have been blocked" findings: IOC hash
matches were hardcoded severity "critical" at every site, and in exec_guard an
ioc_match forces a prevention hit regardless of the severity threshold.

- FeedSpec gains `max_severity` (append-only; serialize/parse/validate, control
  socket `feed-add` accepts it). Empty = default for the kind: hash feeds are
  advisory "high", not "critical". PolicyStore::ioc_feed_severity() resolves it.
- All three IOC severity sites now use the feed's severity
  (detection_pipeline.cpp scan_image, assess_exec, handle_process), replacing the
  hardcoded "critical".
- Trust gating: ExecAssessment::ioc_match is set only when the matched feed
  resolves to "critical" (a curated feed that should force prevention). A
  community feed stays advisory — its "high" is subject to prevention_min_severity
  (default "critical"), so it no longer would-blocks system tools per exec.

Tests: test_ioc_hash_detection uses an explicit critical feed (trusted path) and
asserts assess_exec forces the hit; new test_ioc_community_feed_is_advisory
asserts a default feed is "high", ioc_match is not forced, and decide_exec allows
it in audit and enforce; test_policy covers max_severity validation, the advisory
default, and a critical round-trip. Suite 50/50; clang static analyzer and
clang-tidy clean on the changed TUs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 10:18:02 -06:00
trvon
8f6bd1d4e0
Privacy: attribute tracker contacts to programs via OpenSnitch events
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Second piece of tracker attribution (docs/tracker-attribution.md): which
programs reach domains on a trackers list. Advisory only: never a block,
never a finding (source invariant R11).

- OpenSnitch event listener: OpenSnitch's JSON "remote" logger is the
  hostname source (Strawberry's eBPF connects have none). It streams over
  TCP and cannot leave fields out, so events carry command lines and
  environments. The listener binds 127.0.0.1 at a port below 1024 (only
  root can), accepts only root-owned peers (/proc/net/tcp), bounds
  clients/object size/buffering, and reads only pid, uid, path,
  destination and action; arguments and environment are never copied.
- TrackerContacts: per program x domain, bounded at 20000 keys. A root-only
  tracker_contact telemetry record (type 25) on the first contact, then at
  most hourly with the count since.
- Status `privacy` block; control verb privacy-trackers; CLI
  `strawberry privacy trackers [--json]`.
- `strawberry opensnitch enable-events` adds the logger to OpenSnitch's
  config (with a .bak); the installer runs it when OpenSnitch is present,
  unless privacy_trackers=false. Config: privacy_trackers (default true),
  opensnitch_events (default 127.0.0.1:979).
- EasyPrivacy is a default trackers feed (GPL-3.0+ or CC-BY-SA-3.0,
  downloaded at run time).

Tests: E1-E7 (test_opensnitch_events: event fields, stream splitting,
peer uid, address rules, listener end to end, contacts and record, config
editor), the control verb, CLI translation and help, daemon status. New
fuzz_opensnitch_events (90 s clean).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 06:15:28 -06:00
trvon
2e94ae1485
Privacy: tracker list feeds (advisory only, never block)
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
First piece of tracker attribution (docs/tracker-attribution.md): which
programs reach advertising and tracking domains, built with the
measurement rules of the CarolinaCon 2026 privacy talk (bridgescope):
a listed domain is a circumstantial label, never proof.

- Feed kind "trackers": hosts lines, plain domains and adblock
  whole-domain rules (||host^, ||host^$third-party). Cosmetic rules,
  exceptions, paths and other options are skipped; the comment cut used
  to turn "site.example##.ad" into a domain.
- TrackerSet: suffix matching (a.b.t.example matches t.example, never
  nott.example, an IP literal or a bare TLD); first feed reported.
- List parsing moves from rule_updater.cpp to ioc_list.cpp (no network
  code) so the query-side core can use it.
- Source invariant R11: tracker lists never meet exec-guard or OpenSnitch
  code. They are not in the OpenSnitch blocklist sync.

Tests: T1-T3 (test_tracker_set), trackers in the list-parsing property
and the list fuzzer (60 s clean), R11 self-test fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 05:25:53 -06:00
trvon
f172bc1710
Replay and training take several trace files; fix an autostart label
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
--inject and --inject-train are now repeatable (training.trace plus
threats-train.trace, standard.trace plus threats-eval.trace). Trace names
must be unique across the files.

standard.trace labelled its autostart write "XDG autostart", but the file
watcher emits "desktop autostart", so that write never matched what a
real host records. No gate evaluation has used standard.trace yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 05:09:41 -06:00
trvon
2a4812829f
Use macOS store paths and launchd hints in the CLI, app and MCP
The CLI fallback, the tray app and strawberry-mcp looked for evidence
in /var/lib/strawberry; kSystemStorePath (config.hpp) is
/var/db/strawberry on macOS. User-facing hints quoted systemctl and
journalctl; they come from include/strawberry/service_hints.hpp
(launchctl and /var/log/strawberry.log on macOS), including doctor's
log-rotation note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 04:39:57 -06:00