The finite-difference vs analytic gradient check in test_gradients used a
2e-3 + 2e-2*|n2| tolerance that ubuntu-noble's FP codegen (FMA/libm) pushed a
borderline sample past, while debian-trixie and Arch passed. Deterministic
(fixed seed), so it blocked the v0.2.4 release build. Loosen to 5e-3 + 5e-2*|n2|
(still catches a real gradient bug, which misses by orders of magnitude).
Verified on ubuntu:24.04 in Docker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ship 3.5c's provenance attribution to the central learner. ExportLabel gains two
additive uint64 tags (8 origin_node_hash, 9 origin_exe_hash); a positive window's
label now carries token_hash(dropper node) + token_hash(path_template(dropper
exe)) from ReplayWindow.origin_*. fleet_merge counts windows_with_origin.
R12/R13: a FIELD addition, not a new record type -- no codec.hpp/enum change
(R12 untouched), both fields are uint64 hashes so no raw path/node string crosses
the export boundary (R13 green, no name-loophole). Back-compat additive:
kExportSchemaVersion stays 1; old bundles decode with origins 0; old readers
ignore tags 8/9. Positive labels only -- suppressed/negative carry no origin.
Tests: round-trip of the hash fields + defaults; extraction proves a dropper->exec
detected window's positive label carries a non-zero origin while a no-write
detected window and suppressed labels carry 0. source_invariants green.
A learned ranking head + TrainingExample plumbing are phase-2 (out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reuse the global write->exec linkage (recent_writes) to attribute a detection to
its ROOT CAUSE. recent_writes now maps path -> {writer_node, writer_exe}; when a
window is late_detected (retro-hunt flags a binary after it ran) or detected
(YARA), set ReplayWindow.origin_node/origin_exe from whoever WROTE the flagged
binary -- usually an earlier node than the exec leaf that ran it. Surfaced in
window_json. Deterministic provenance walk (first writer wins); replay-level only,
no export-schema/record-type/enum change (R12/R13 intact), self_written_execs
unchanged. A learned ranking head over ambiguous origins is the follow-up (3.5d).
Test R10: a dropper writes /tmp/evil in one window, a separate process execs it
and is flagged late -> the window's origin_node is the dropper, not the exec leaf;
no in-set write -> empty origin; deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#4 (guard retention): same-binary parent/child self-ptrace never rates "high"
(it's self-debugging, not injection), but from a drop location (/tmp) it now
stays "medium" rather than "low" -- a same-binary anti-debug/injection staged in
/tmp still surfaces, while sanitizer builds no longer flood high-severity
PtraceAttach. Injection into another program is unchanged.
#5 (CLI-UX contract): --max-node-share/--test-fraction/--split now error when
passed to any command other than 'fleet train' instead of being silently
ignored (noun-verb commands already reject unknown options; this covers the
flag-form and other fleet subcommands).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address code-review findings on this session's changes (no behavior change):
- Share the dense-slot index for the write->exec feature as
learn::kWriteExecFeatureIndex; featurize() and the model-score dev tool now
reference it instead of a duplicated literal 42 (prevents silent column drift
if the feature layout shifts).
- Fix the self_written_execs comments: the linkage is a GLOBAL bounded FIFO
(recent_writes), not strictly "this window" — the docs now match replay.cpp.
- release.yml: never cancel a tag/release run (cancel-in-progress only for
non-tag refs); push ordering doesn't guarantee the tag event arrives after the
branch event, so a same-sha branch push must not abort an in-progress release.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The version lived in three hard-coded copies (CLI `version`, daemon status,
banner); v0.2.0 shipped with the CLI still saying 0.1.0. meson.build is now
the only source: it passes STRAWBERRY_VERSION to every target, and the CLI
test requires the exact version.
packaging/ci/build-release.sh refuses to publish a tag build unless
meson.build, the tag and the built binary's `strawberry version` agree; CI
never rewrites the version.
scripts/release.sh X.Y.Z is the one manual release step: on a clean main
equal to forgejo/main it bumps meson.build, makes a signed commit and a
signed tag, and pushes both (--dry-run shows the plan). The Security Hub
release channel deploys signed tags only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A parent/child of the SAME binary tracing each other is self-debugging
(sanitizers' StopTheWorld, crash handlers), not injection. The code already
scored that `low`, but `suspicious_image` (which flags drop locations like /tmp)
took precedence and forced `high` -- so sanitizer test binaries built in /tmp
flooded findings as high-severity PtraceAttach (observed live: ~160 FPs on
/tmp/toolcheck-*/*_san). Reorder so the strong structural self-debug signal beats
the weak location heuristic: same-program-family -> low regardless of location;
ptrace into ANOTHER program stays high (suspicious image) or medium. Injection
detection is unchanged. Test added for the /tmp self-ptrace case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build-only tool: loads a trained model (.sbm) and an export bundle, applies
forward(model, features) per window, prints TSV window_id/value/label/x[42] plus
a summary partitioned by the write->exec feature slot. Unblocks the novel-chain
benchmark: replay --model ranks by rarity (model_value null), so it could not
measure a model's score on holdout windows; this does. On the detector-covered
train bundle the data-flow model scores write->exec windows 0.634 vs 0.0003 for
the rest -- it clearly learned the x[42] feature.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.5b's ordered exec-lineage proved inert in clean A/Bs; replace it with a general
write->exec data-flow feature: WindowSummary.self_written_execs (exec of a file
written earlier in the same window) at x[42], tracked via OpenWindow.written_paths.
Honest status: this is the conceptually-correct provenance feature (it separates
download->exec-the-same-file from download->exec-unrelated), but it ALSO shows no
AUC gain on detector-labeled data -- for the same structural reason as 3.5b: the
teacher labels (DownloadThenExecute etc.) correlate with existing count/location
features or re-derive the detector. Validating provenance LEARNING requires the
novel-chain generalization benchmark (train on detector-covered, hold out
detector-evading attacks); committed as that benchmark's feature foundation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feed ordered parent->child exe-template bigrams into the window feature vector
(WindowSummary.lineage_edges, note_lineage_edge, a salted hash loop in featurize,
wired at replay on_exec). The model already got per-kind lineage surprisal
(x[25..29]); this adds path ORDER, which the unordered token bag discarded.
Clean A/B on a fixed corpus (same 697 windows, same flags, only the binary
differs): model AUC 0.734 -> 0.737 (rarity 0.520 unchanged). Null on this
shallow-lineage data (marker execs are 1-process, synthetic data flat) but the
feature is correct and order-sensitive (test: A>B != B>A). Kept as the path
representation that Stage 3.5c's retro-hunt attribution head needs; its payoff is
expected on real multi-stage attack telemetry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose the merge cap and train/test split on `strawberry fleet train` so a
single-node real-attack bundle can pre-seed a model. Default stays temporal +
cap 0.25 (behavior-preserving); `--split random` + `--max-node-share 1` lets the
Caldera-attack export (15 positive windows) train. Verified: real-only train AUC
1.0 vs rarity 0.563; default still refuses identically.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Do not turn an empty cached digest into an unbounded descriptor hash. Distinguish size refusals from YARA results so the exec guard applies its own bounded scan/degrade policy. Cover cached read budgets across size boundaries, known hashes, inventory/live events and real guard-cache reuse. Fold/live/KPI and post-rollout CPU acceptance remain pending; OpenSnitch checksum work is separate.
Use suspend-aware BOOTTIME waits and owned-container cleanup without changing the workload. Reject missing coverage and preserve producer/report failures. Include runtime/helper regressions, constrained build-job support and live exec-guard crash safety. Document the failed live/idle gates without relaxing thresholds or claiming production acceptance.
Preserve independent support, typed rule namespaces and stable process lifetimes. Share authoritative managed-set reconciliation across daemon and offline correlation, load offline policy without seeding/migration, and release resources after shutdown-flush exceptions. Keep retirement history, transaction ownership and fault/allocation regressions; no whole-set crash-atomicity or live acceptance claim.
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
Represent full-range ordered clock differences without signed overflow and allow complete decay. Add seeded restore-clock regressions and a dedicated fuzz seed.
Allowlisting a binary suppressed only *new* detections: the findings
already stored for it stayed in the evidence store, and every reload
re-materialised them. That is why the ~9k iproute2 `ss` false positives
kept "coming back" after the hash was allowlisted a day earlier -- nothing
retracted them, and the only cleanup was `doctor fix
retire-allowlisted-findings`, which operators do not know to run.
allowlist-add now retires the now-allowlisted findings at the moment of
the action (alongside the existing containment release) and reports the
count as `retired_findings` in its reply. retire_allowlisted_findings is
hash-based and self-filtering, so a path/rule entry that matches no
fully-allowlisted finding retires nothing.
Regression test (test_daemon_runtime): a detected binary's finding is
materialised (forced correlation), allowlist-add reports
retired_findings >= 1, and after correlation re-runs the finding does not
come back (A already makes correlation skip allowlisted entities).
Also refresh the doctor findings.allowlisted message, which claimed
allowlisting does not delete stored findings -- it now does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On a host that sits at critical/emergency for a long time (its own apps holding
most of RAM, or the daemon slightly over its soft budget), the starvation guard
was disabled (`!hard`), so correlation, retro-hunt and inventory deferred
forever — observed deferred 153 min with forced=0, which also starves the
retro-hunt delayed labels that Stage 4 training needs.
- Tiered starvation guard: it now fires under critical/emergency too, but after a
level-scaled window (1x warning, 4x critical, 12x emergency) so background work
is rare under load, not zero. A true host-memory crunch (host_severe_,
MemAvailable below the severe floor) still holds everything, since forcing work
then could tip the host toward OOM. exec guard/scan paths unchanged.
- Raise the balanced own-memory soft budget 512 -> 768 MiB: the daemon's steady
footprint grew (telemetry buffers, the ingest server, the shadow model), so 512
left it permanently over budget -> pinned critical/emergency. 768 fits the real
baseline, still well under the 1.5 GiB cgroup cap.
Tests updated for the 768 budget and the tiered guard (critical forces after the
4x window; severe holds it). Suite 51/51; analyzer + clang-tidy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The teacher-label join made a window a positive if it was detected
(replay.cpp), but it never applied the allowlist, so a window running a cleared
false positive (e.g. /usr/bin/ss) was labelled a teacher positive — poisoning
Stage 4 training. Negative labels were written to the stream but replay dropped
them.
- replay_store now collects negative labels into ReplayOptions.negative_labels
(it used to ignore any non-positive label).
- A window whose exec hash is negatively labelled is marked `suppressed`; the
example/policy-eval positive is gated on `!suppressed`, so such a window is
kept as a *negative* training example (useful signal) and never a positive,
even if a cleared rule "detected" it.
- DetectionPipeline seeds a negative label for every allowlisted sha256 at init,
so evidence stored before the allowlist (the historical ss storm) is covered
without waiting for a re-exec.
Test: a detected window with a stored negative label for its hash is a positive
before the label and a kept-negative after. Suite 51/51; analyzer + clang-tidy
clean on replay.cpp and detection_pipeline.cpp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correlation runs over the whole store and never consulted the allowlist, so an
allowlisted hash (e.g. the /usr/bin/ss feed false positive the operator cleared)
regenerated the same findings on every pass from already-stored indicators —
`doctor fix retire-allowlisted-findings` cleared them but they came straight back
(~9,400 findings observed again). This also meant the findings were a poisoned
teacher-label source for training.
- classify() now skips an indicator whose (hash, path, rule) is allowlisted, the
single chokepoint shared by CorrelationEngine::build_findings (the oracle) and
IncrementalCorrelator::update, so both agree. New AllowlistFn; build_findings /
correlate take an optional predicate (default = no filtering, existing tests and
the incremental==full oracle unchanged); IncrementalCorrelator::set_allowlist.
- DetectionPipeline wires the predicate to PolicyStore::match, so the live
incremental path skips allowlisted evidence. Retiring the findings now sticks.
Test: build_findings with an allowlist predicate emits nothing for the matched
hash and is unaffected for others. Suite 51/51; analyzer + clang-tidy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the tracker-attribution transport on macOS. There is no network
sandbox to work around there, so the daemon (root) reads Little Snitch's traffic
stream directly and feeds the same on_opensnitch_connection consumer — no relay
or ingest socket (those are the Linux sandbox answer).
- LittleSnitchParser (cross-platform, unit-tested): parses `littlesnitch
log-traffic` CSV via a header-name->index map (tolerates column reordering),
yielding the shared redacted OpenSnitchConnection for outbound rows only
(program, uid, host/ip/port, protocol 6->tcp/17->udp). Little Snitch's CSV has
no argv/env, so redaction is structural. split_csv_line handles quoted fields.
- LittleSnitchSource (macOS only): posix_spawns `littlesnitch log-traffic
--stream`, reads the pipe on a guarded thread, parses and calls back; stop()
SIGTERMs the CLI (EOF -> loop ends) then reaps it.
- Daemon: start_tracker_attribution gains an __APPLE__ branch that starts the
source; shutdown/status handle it. New config littlesnitch_cli (default CLI
path). status privacy.listener source "little-snitch".
- docs: macOS section rewritten (needs Little Snitch "Allow access via Terminal").
Test E10 covers the parser (header, outbound/inbound, deny, quoted exe, no-dest).
Linux suite 51/51; analyzer + clang-tidy clean (Linux view). The __APPLE__ code
compiles/validates on the Mac next (ssh); live spawn needs root + LS Terminal access.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First half of the sandbox-safe tracker transport (architecture deep dive). The
daemon's old loopback TCP listener could not bind under its own sandbox
(PrivateNetwork=true + RestrictAddressFamilies=AF_UNIX). Replace it with an
AF_UNIX ingest socket the daemon *can* bind under the sandbox; a host-namespace
relay (next change) will bind the OpenSnitch port, strip argv/env, and forward
redacted tuples here.
- opensnitch_events.hpp/cpp: encode_connection / decode_connection serialize only
the redacted fields (never argv/env, which the parser already drops) as one
JSON object — the relay->daemon wire form. New TrackerIngestServer: an AF_UNIX
stream listener (Linux-only, like the TCP listener) that accepts only a root
peer (SO_PEERCRED), frames with take_json_objects, decodes and calls back;
bounded clients/object/buffer as before.
- daemon: start_tracker_attribution now opens the ingest socket
(opensnitch_ingest_socket, default /run/strawberry/trackers.sock) instead of
the in-sandbox TCP listener, so it works with the sandbox unchanged and is
simply idle until a relay connects. status privacy.listener reports the ingest
socket + relay_events; shutdown stops it. New config key opensnitch_ingest_socket.
- Tests: E8 encode/decode round-trip + redaction (no args/env on the wire);
E9 TrackerIngestServer end to end (connect, two tuples, callback, stats);
daemon-runtime configs set privacy_trackers=false (not under test).
Suite 51/51; clang static analyzer and clang-tidy clean on the changed TUs. The
relay run-mode + systemd unit + install wiring + docs follow in WS2b/WS2c.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The scan path (handle_process) and the exec-guard recording path
(record_exec_decision) had grown two near-identical repeat-aggregation maps
(repeats_ / exec_repeats_) plus three copies of the IOC-feed severity idiom.
Refactor to one implementation; no behavior change (the existing aggregation and
IOC-severity tests are the oracle and pass unchanged).
- New include/strawberry/repeat_aggregator.hpp: RepeatAggregator owns the
per-key window map, the 20-full-then-summary cadence and the aggregated count;
observe(key, now, seen_ns, pid) returns {full, at_threshold, aggregated,
emit_summary, notify, state}. Each caller keeps its own key, summary-id prefix,
summary shape and notify behaviour. Both pipeline sinks now hold one instance
(scan notifies at most hourly; exec passes notify_ns=0). status
repeats_aggregated sums both totals (atomic, lock-free read).
- ioc_hit_severity(feed): one source for the three IOC match sites (scan_image,
assess_exec, handle_process) that was the policy_->ioc_feed_severity ternary.
- Leaves the latency-critical assess_exec/decide path untouched; both sinks still
run under mutex_, off the FAN response.
Unit test tests/test_repeat_aggregator.cpp (window reset, threshold, summary and
notify cadence, per-key independence). Suite 51/51; clang static analyzer and
clang-tidy clean on detection_pipeline.cpp; fold's 7 int-overflow are the accepted
monotonic-timestamp FP class (unchanged by the move); perf-kpi PASS, no regression
(exec_decision p50 unchanged, exec_assess/db_commit within noise or faster).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allowlisting a hash suppresses new detections but does not delete findings
already stored for it (e.g. the /usr/bin/ss false-positive storm). That cleanup
now lives in doctor instead of a one-off compaction.
- DetectionPipeline::retire_allowlisted_findings(dry_run): a finding is stale
when it references at least one hash and every hash it references (from
metadata "hashes" and file_<sha256> nodes) is allowlisted, so a finding that
also implicates a non-allowlisted binary is never dropped. Mirrors
retire_superseded_findings (retire_finding + transaction).
- node_ids.hpp gains content_hash_of_file_node_id() (the inverse of
file_node_id for the content case) so the pipeline parses file ids through the
canonical helper, not a "file_" literal (source invariant R8).
- Doctor check findings.allowlisted (warn) with maintenance action
retire-allowlisted-findings; DoctorInputs.allowlisted_findings gathered by the
daemon; run_maintenance handles the action; CLI `doctor fix` help + completion
and API_CONTRACT updated.
Tests: test_retire_allowlisted_findings (two execs of a flagged binary -> a
finding; allowlist the hash -> dry-run lists it, apply retires it, idempotent);
doctor check ok/warn cases; CLI completion string. Suite 50/50; clang static
analyzer, clang-tidy and source invariants clean on the changed TUs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The exec-guard would-block recording path stored one indicator (and, via
correlation's per-process grouping, one finding) per exec, with no aggregation,
unlike the scan path. A critical feed or rule matching a frequently-run binary
would flood evidence the way /usr/bin/ss did.
- record_exec_decision now collapses repeated identical decisions (keyed by
verdict|hash|rule) with the same 10-min window / 20-full-then-summary scheme
as handle_process (new exec_repeats_ map; counts into repeats_aggregated_).
Beyond the window it refreshes one "exec-repeat" summary indicator and returns
before the per-exec graph write.
- The actor's /proc identity reads are now done before taking mutex_, not under
it. The clang static analyzer (new strict gate) flagged ::read in a critical
section (unix.BlockInCriticalSection) here; this also honors the AGENTS rule
that scans/reads stay off DetectionPipeline::mutex_. Analyzer now clean.
Test: test_repeated_would_block_is_summarized drives 30 identical would-block
decisions and asserts exactly 20 full exec-audit indicators + 1 exec-repeat
summary and repeats_aggregated == 10. Suite 50/50; analyzer and clang-tidy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>