Commit graph

161 commits

Author SHA1 Message Date
trvon
110facb12f
Release v0.2.3
All checks were successful
release / draft (push) Successful in 19s
release / build (debian:trixie, trixie) (push) Successful in 58m6s
release / build (ubuntu:24.04, noble) (push) Successful in 53m39s
release / publish (push) Successful in 13s
v0.2.3
2026-11-10 19:34:20 -07:00
trvon
c679a03561
Stage 7a: LogGC dead benign subtree GC (causality-preserving telemetry reduction)
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Successful in 53m38s
release / publish (push) Has been skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-10 19:32:12 -07:00
trvon
58907eb45e
Stage 3.5d: export attribution signal (detection->dropper origin hashes)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 56m28s
release / build (ubuntu:24.04, noble) (push) Successful in 59m8s
release / publish (push) Has been skipped
Ship 3.5c's provenance attribution to the central learner. ExportLabel gains two
additive uint64 tags (8 origin_node_hash, 9 origin_exe_hash); a positive window's
label now carries token_hash(dropper node) + token_hash(path_template(dropper
exe)) from ReplayWindow.origin_*. fleet_merge counts windows_with_origin.

R12/R13: a FIELD addition, not a new record type -- no codec.hpp/enum change
(R12 untouched), both fields are uint64 hashes so no raw path/node string crosses
the export boundary (R13 green, no name-loophole). Back-compat additive:
kExportSchemaVersion stays 1; old bundles decode with origins 0; old readers
ignore tags 8/9. Positive labels only -- suppressed/negative carry no origin.

Tests: round-trip of the hash fields + defaults; extraction proves a dropper->exec
detected window's positive label carries a non-zero origin while a no-write
detected window and suppressed labels carry 0. source_invariants green.
A learned ranking head + TrainingExample plumbing are phase-2 (out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 18:04:15 -07:00
trvon
b448c51c1d
Stage 3.5c: provenance attribution (trace a detection to the dropper node)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 55m28s
release / build (ubuntu:24.04, noble) (push) Successful in 51m16s
release / publish (push) Has been skipped
Reuse the global write->exec linkage (recent_writes) to attribute a detection to
its ROOT CAUSE. recent_writes now maps path -> {writer_node, writer_exe}; when a
window is late_detected (retro-hunt flags a binary after it ran) or detected
(YARA), set ReplayWindow.origin_node/origin_exe from whoever WROTE the flagged
binary -- usually an earlier node than the exec leaf that ran it. Surfaced in
window_json. Deterministic provenance walk (first writer wins); replay-level only,
no export-schema/record-type/enum change (R12/R13 intact), self_written_execs
unchanged. A learned ranking head over ambiguous origins is the follow-up (3.5d).

Test R10: a dropper writes /tmp/evil in one window, a separate process execs it
and is flagged late -> the window's origin_node is the dropper, not the exec leaf;
no in-set write -> empty origin; deterministic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 17:21:55 -07:00
trvon
70652b5fb0
Review fixes #4/#5: ptrace medium middle-ground + reject fleet-train flags elsewhere
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 55m51s
release / build (ubuntu:24.04, noble) (push) Successful in 51m11s
release / publish (push) Has been skipped
#4 (guard retention): same-binary parent/child self-ptrace never rates "high"
(it's self-debugging, not injection), but from a drop location (/tmp) it now
stays "medium" rather than "low" -- a same-binary anti-debug/injection staged in
/tmp still surfaces, while sanitizer builds no longer flood high-severity
PtraceAttach. Injection into another program is unchanged.

#5 (CLI-UX contract): --max-node-share/--test-fraction/--split now error when
passed to any command other than 'fleet train' instead of being silently
ignored (noun-verb commands already reject unknown options; this covers the
flag-form and other fleet subcommands).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 16:17:01 -07:00
trvon
e5ee53cf18
Review fixes: write->exec feature constant, accurate comments, tag-safe CI
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 56m31s
release / build (ubuntu:24.04, noble) (push) Successful in 51m33s
release / publish (push) Has been skipped
Address code-review findings on this session's changes (no behavior change):
- Share the dense-slot index for the write->exec feature as
  learn::kWriteExecFeatureIndex; featurize() and the model-score dev tool now
  reference it instead of a duplicated literal 42 (prevents silent column drift
  if the feature layout shifts).
- Fix the self_written_execs comments: the linkage is a GLOBAL bounded FIFO
  (recent_writes), not strictly "this window" — the docs now match replay.cpp.
- release.yml: never cancel a tag/release run (cancel-in-progress only for
  non-tag refs); push ordering doesn't guarantee the tag event arrives after the
  branch event, so a same-sha branch push must not abort an in-progress release.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 16:09:03 -07:00
trvon
43efeac44c
Release: one version source, CI version contract, scripts/release.sh
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been skipped
The version lived in three hard-coded copies (CLI `version`, daemon status,
banner); v0.2.0 shipped with the CLI still saying 0.1.0. meson.build is now
the only source: it passes STRAWBERRY_VERSION to every target, and the CLI
test requires the exact version.

packaging/ci/build-release.sh refuses to publish a tag build unless
meson.build, the tag and the built binary's `strawberry version` agree; CI
never rewrites the version.

scripts/release.sh X.Y.Z is the one manual release step: on a clean main
equal to forgejo/main it bumps meson.build, makes a signed commit and a
signed tag, and pushes both (--dry-run shows the plan). The Security Hub
release channel deploys signed tags only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 08:55:05 -07:00
trvon
7f3cbeca71
Detection quality: self-ptrace of the same binary stays low even from /tmp
All checks were successful
release / draft (push) Successful in 19s
release / build (debian:trixie, trixie) (push) Successful in 58m7s
release / build (ubuntu:24.04, noble) (push) Successful in 58m54s
release / publish (push) Successful in 14s
v0.2.2
A parent/child of the SAME binary tracing each other is self-debugging
(sanitizers' StopTheWorld, crash handlers), not injection. The code already
scored that `low`, but `suspicious_image` (which flags drop locations like /tmp)
took precedence and forced `high` -- so sanitizer test binaries built in /tmp
flooded findings as high-severity PtraceAttach (observed live: ~160 FPs on
/tmp/toolcheck-*/*_san). Reorder so the strong structural self-debug signal beats
the weak location heuristic: same-program-family -> low regardless of location;
ptrace into ANOTHER program stays high (suspicious image) or medium. Injection
detection is unchanged. Test added for the /tmp self-ptrace case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-09 08:06:36 -07:00
trvon
42db81dc9e
CI: fix release-build contention (concurrency by sha, timeout, ccache)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 58m22s
release / build (ubuntu:24.04, noble) (push) Successful in 55m6s
release / publish (push) Has been skipped
The single self-hosted runner ran the full matrix for BOTH a branch push and its
tag, so a commit's two runs fought for the one CPU (debian build failed on run 75
under that contention). Add:
- concurrency group keyed by sha + cancel-in-progress: a commit's tag run cancels
  its branch run instead of competing;
- timeout-minutes: 60 on the build job so a wedged build fails cleanly;
- a best-effort ccache cache (CCACHE_DIR + actions/cache, continue-on-error) so
  the cold single-CPU build goes warm/incremental when the runner has a cache
  service (meson uses ccache automatically); harmless cold otherwise.
No daemon code changed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-09 06:29:48 -07:00
trvon
1733bc5656
Stage 3.5 bench: make write->exec linkage global (fixes self_written_execs)
All checks were successful
release / draft (push) Successful in 21s
release / build (debian:trixie, trixie) (push) Successful in 1h1m23s
release / build (ubuntu:24.04, noble) (push) Successful in 56m1s
release / publish (push) Successful in 14s
v0.2.1
The write->exec data-flow feature used a PER-WINDOW written_paths set, but a
dropper/download and the exec of the dropped file routinely land in different
replay windows (writer's window vs the exec's parent window, or a window that hit
max_window_events) -- so self_written_execs fired for only ~1/15 real download
chains. Mirror the live behavior engine's drop_writes_: record written paths in a
global bounded FIFO (recent_writes) and match on exec. Now a base64-decode evasion
that DownloadThenExecute misses (detections=0) is scored 0.594 by a data-flow
model vs 0.021 by a no-provenance model and ~0 by the detectors -- the data-flow
model generalizes to the detector-evading attack it never trained on. Benign ~0 for
both (no FP).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 18:26:51 -07:00
trvon
0b730eace2
Stage 3.5 bench: strawberry-model-score dev tool (per-window model scoring)
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been cancelled
Build-only tool: loads a trained model (.sbm) and an export bundle, applies
forward(model, features) per window, prints TSV window_id/value/label/x[42] plus
a summary partitioned by the write->exec feature slot. Unblocks the novel-chain
benchmark: replay --model ranks by rarity (model_value null), so it could not
measure a model's score on holdout windows; this does. On the detector-covered
train bundle the data-flow model scores write->exec windows 0.634 vs 0.0003 for
the rest -- it clearly learned the x[42] feature.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 16:59:30 -07:00
trvon
fdd96ed64f
Release v0.2.0: bump version + fix license to GPL-3.0
All checks were successful
release / draft (push) Successful in 19s
release / build (debian:trixie, trixie) (push) Successful in 54m43s
release / build (ubuntu:24.04, noble) (push) Successful in 50m32s
release / publish (push) Successful in 13s
v0.2.0
Bump project + daemon version 0.1.0 -> 0.2.0. Correct meson `license` from the
stale `MIT` to `GPL-3.0-or-later` to match the GPLv3 LICENSE file and README.
First tagged release carrying Stage 6 fleet learning and Stage 3.5 provenance
work (3.5a fleet-train flags, 3.5b2 write->exec data-flow feature).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 16:40:04 -07:00
trvon
0282066340
Stage 3.5b2: replace exec-lineage with write->exec data-flow feature
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been cancelled
3.5b's ordered exec-lineage proved inert in clean A/Bs; replace it with a general
write->exec data-flow feature: WindowSummary.self_written_execs (exec of a file
written earlier in the same window) at x[42], tracked via OpenWindow.written_paths.

Honest status: this is the conceptually-correct provenance feature (it separates
download->exec-the-same-file from download->exec-unrelated), but it ALSO shows no
AUC gain on detector-labeled data -- for the same structural reason as 3.5b: the
teacher labels (DownloadThenExecute etc.) correlate with existing count/location
features or re-derive the detector. Validating provenance LEARNING requires the
novel-chain generalization benchmark (train on detector-covered, hold out
detector-evading attacks); committed as that benchmark's feature foundation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 16:37:06 -07:00
trvon
e1a619f44d
CI: one compile job per build (runner caps jobs at one CPU)
Some checks failed
release / draft (push) Has been skipped
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been cancelled
release / build (debian:trixie, trixie) (push) Has been cancelled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:34:07 -07:00
trvon
8544e921a2
Stage 3.5b: ordered lineage-path bigram features
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 28m43s
release / build (ubuntu:24.04, noble) (push) Successful in 26m22s
release / publish (push) Has been skipped
Feed ordered parent->child exe-template bigrams into the window feature vector
(WindowSummary.lineage_edges, note_lineage_edge, a salted hash loop in featurize,
wired at replay on_exec). The model already got per-kind lineage surprisal
(x[25..29]); this adds path ORDER, which the unordered token bag discarded.

Clean A/B on a fixed corpus (same 697 windows, same flags, only the binary
differs): model AUC 0.734 -> 0.737 (rarity 0.520 unchanged). Null on this
shallow-lineage data (marker execs are 1-process, synthetic data flat) but the
feature is correct and order-sensitive (test: A>B != B>A). Kept as the path
representation that Stage 3.5c's retro-hunt attribution head needs; its payoff is
expected on real multi-stage attack telemetry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 14:22:14 -07:00
trvon
2cc137b8e7
Stage 3.5a: fleet-train --max-node-share/--test-fraction/--split
Expose the merge cap and train/test split on `strawberry fleet train` so a
single-node real-attack bundle can pre-seed a model. Default stays temporal +
cap 0.25 (behavior-preserving); `--split random` + `--max-node-share 1` lets the
Caldera-attack export (15 positive windows) train. Verified: real-only train AUC
1.0 vs rarity 0.563; default still refuses identically.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 12:18:32 -07:00
trvon
50254dc8cc
Docs: codify the system + lean AGENTS/CLAUDE for open-source navigation
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 31m52s
release / build (ubuntu:24.04, noble) (push) Successful in 29m30s
release / publish (push) Has been skipped
- README: front door states the full system (core + experimental shadow-only
  inspection model + distributed fleet learning); new "Fleet learning" usage
  section; learn/ + fleet_export/merge + governor in Architecture; a proper
  "Documentation" map; license corrected MIT -> GPLv3 (matches LICENSE).
- AGENTS.md: crisp YAMS work-loop (retrieve/inspect/persist/report), replaced
  the stale ~64-line dated "Current Project Reality" with an accurate concise
  state + pointers (PLAN/DEVELOPER/git/YAMS), trimmed stale work-order/milestone,
  added "Keep This File Lean".
- CLAUDE.md: work-loop + evidence rule up top.
- gitignore the local validation/attack-range harness (deploy/validation/{docker,vm}).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 10:41:05 -07:00
trvon
8f061ad902
Merge ci/release-workflow: Forgejo release workflow, test lane and runner probe
Some checks failed
release / draft (push) Has been cancelled
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been cancelled
Forgejo now reads .forgejo/workflows and ignores .github/workflows, whose
ubuntu-24.04/macos-15 labels no runner here provides. The test lane runs on
the ghost-ci runner (label docker).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 10:21:42 -07:00
trvon
120e296acb
Stage 6: local multi-node fleet-loop validation (gap #4)
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
run-fleet-loop.sh + README: proves the distributed loop end to end with real binaries
(seed N nodes -> round-1 export -> central fleet train + fleet overlay -> distribute ->
round-2 export), asserting the fleet signal propagates (round-1 fleet_feat=0 ->
round-2 fleet_feat>0, N distinct nodes, non-empty overlay). Verified: 2 nodes PASS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 09:32:32 -07:00
trvon
cd93ec4da3
Stage 6: strawberry-fleet-inspect dev tool (bundle record + fleet_bits stats) 2026-11-08 09:29:24 -07:00
trvon
0bc61ac714
Stage 6: strawberry-fleet-seed dev tool (deterministic synthetic telemetry store for fleet validation; gap #4a) 2026-11-08 09:27:09 -07:00
trvon
39912f6e64
Stage 6: wire config node_id to export (stable node identity; gap #2) 2026-11-08 09:05:25 -07:00
trvon
b95bdb04de
Stage 6: strawberry fleet overlay CLI — produce the fleet overlay from a bundle directory (gap #1) 2026-11-08 08:59:52 -07:00
trvon
621d00b86e
Stage 6: Salt pillar + deployment contract for distributed mode
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Pillar data (deploy/salt/pillar/) + README for the deploying agent: hub-and-spoke
(sensor export -> central fleet train + overlay -> distribute), role assignment,
CLI surface, security invariants, and the honest gaps (no `fleet overlay` CLI yet,
node_id not wired, no multi-node/docker-swarm validation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 08:21:44 -07:00
trvon
446e5c3afe
Stage 6: export populates fleet_bits from a local overlay (export_store fleet_surprisal callback; run_export loads fleet-overlay.sbo) 2026-11-08 08:09:47 -07:00
trvon
9313134aee
Stage 6: populate WindowSummary.fleet_bits from a fleet overlay during scoring (ReplayOptions.fleet_surprisal callback) 2026-11-08 08:01:40 -07:00
trvon
97374a16d2
Stage 6: fleet-overlay feature — per-kind fleet_bits[5] in featurize (reserved slots x[37..41], back-compatible) 2026-11-08 07:55:40 -07:00
trvon
7ad7596bda
Stage 6: load fleet overlay + sketch_surprisal_bits (query fleet-wide rarity for a tuple) 2026-11-08 05:38:00 -07:00
trvon
c939c3536b
Stage 6: fleet rarity overlay — sum per-node count-min sketches and serialize as an overlay bundle 2026-11-07 21:28:35 -07:00
trvon
31e4833169
Stage 6: strawberry fleet train — merge a bundle directory and train the shadow model; fleet_merge populates rarity_bits 2026-11-07 21:22:14 -07:00
trvon
ae4555ec38
Stage 6: central fleet merge core — export bundles to capped TrainingExamples (max_node_share) 2026-11-07 21:12:42 -07:00
trvon
6a028810da
Stage 6: export rarity sketches (export_rarity_delta) from the saved RarityModel state; CLI attaches them when present 2026-11-07 20:47:53 -07:00
trvon
caa7740265
Stage 6: strawberry export CLI verb + node_uid manifest + store-dir e2e (closes 1e deferred validation) 2026-11-07 20:34:33 -07:00
trvon
45f13f7f16
Stage 6: extract windows/features/labels from the store via the replay pipeline into an export bundle 2026-11-07 20:20:41 -07:00
trvon
b5bb1689f9
Stage 6: export rarity-delta record (per-TupleKind count-min, bounded decode) + harden R13 for braced initializers 2026-11-07 19:39:22 -07:00
trvon
fdd699f037
Stage 6: export payload records (window/feature/label, hybrid token hashes) + R13 redaction invariant 2026-11-07 19:32:47 -07:00
trvon
23e70fe823
Stage 6: export bundle format (fleet_export) — sealed segment reusing the store codec; RecordType export_* + R12 2026-11-07 19:12:40 -07:00
trvon
b7336de261
Stage 6: node identity (fleet.hpp) — pure node_uid + wrapper 2026-11-07 18:59:51 -07:00
trvon
f58a765033
CI: Forgejo release workflow, test lane and runner probe
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 31m13s
release / build (ubuntu:24.04, noble) (push) Successful in 26m59s
release / publish (push) Has been skipped
release.yml builds, tests and packs a Linux install tree per distro
(debian:trixie, ubuntu:24.04), one at a time with JOBS=2 and nice.
Branches (main, ci/**) and PRs run the test lane and print each sha256;
a v* tag uploads the tarballs and .sha256 files to a draft release and
publishes it only when every build passed.

runner-probe.yml checks that a runner with the docker label picks up
jobs, can clone this repo with the job token and reach the API.

packaging/ci/build-release.sh is the shared build step. It refuses to
run outside CI because it installs packages.

Forgejo now reads .forgejo/workflows and ignores .github/workflows,
whose ubuntu-24.04/macos-15 labels no runner here provides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:26:02 -07:00
trvon
253c84111f
Ignore local tool output (.pi/, .clangd) and the brick.html audit artifact
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
brick.html is a ~9.5 MB brick (structure audit) report; .pi/ and .clangd are
local tool state. None belong in the tree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-05 16:24:54 -07:00
trvon
74aa9664ad
Pipeline: keep oversized cache refusals unscanned
Do not turn an empty cached digest into an unbounded descriptor hash. Distinguish size refusals from YARA results so the exec guard applies its own bounded scan/degrade policy. Cover cached read budgets across size boundaries, known hashes, inventory/live events and real guard-cache reuse. Fold/live/KPI and post-rollout CPU acceptance remain pending; OpenSnitch checksum work is separate.
2026-11-03 06:57:35 -07:00
trvon
a5f2dbedfc
KPI: bound collection and reject incomplete measurements
Use suspend-aware BOOTTIME waits and owned-container cleanup without changing the workload. Reject missing coverage and preserve producer/report failures. Include runtime/helper regressions, constrained build-job support and live exec-guard crash safety. Document the failed live/idle gates without relaxing thresholds or claiming production acceptance.
2026-11-03 06:57:00 -07:00
trvon
be237d6040
Findings: reconcile evidence and harden policy rebuilds
Preserve independent support, typed rule namespaces and stable process lifetimes. Share authoritative managed-set reconciliation across daemon and offline correlation, load offline policy without seeding/migration, and release resources after shutdown-flush exceptions. Keep retirement history, transaction ownership and fault/allocation regressions; no whole-set crash-atomicity or live acceptance claim.
2026-11-03 06:55:47 -07:00
trvon
29e490dcab
Replay: add strict read-only training summaries
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
2026-11-03 06:54:17 -07:00
trvon
52b02f1dc7
Learn: preserve label provenance and bound replay sampling
Resolve allowlist and quarantine retractions without erasing independent support. Refresh capped entity reservoirs and bound allocated slot bookkeeping; retain deterministic seeded mutation regressions.
2026-11-03 06:53:28 -07:00
trvon
c565cb5aee
Learn: handle extreme restored rarity clocks
Represent full-range ordered clock differences without signed overflow and allow complete decay. Add seeded restore-clock regressions and a dedicated fuzz seed.
2026-11-03 06:52:20 -07:00
trvon
2488423fbd
Allowlist: retire a binary's stored findings when it is allowlisted
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Allowlisting a binary suppressed only *new* detections: the findings
already stored for it stayed in the evidence store, and every reload
re-materialised them. That is why the ~9k iproute2 `ss` false positives
kept "coming back" after the hash was allowlisted a day earlier -- nothing
retracted them, and the only cleanup was `doctor fix
retire-allowlisted-findings`, which operators do not know to run.

allowlist-add now retires the now-allowlisted findings at the moment of
the action (alongside the existing containment release) and reports the
count as `retired_findings` in its reply. retire_allowlisted_findings is
hash-based and self-filtering, so a path/rule entry that matches no
fully-allowlisted finding retires nothing.

Regression test (test_daemon_runtime): a detected binary's finding is
materialised (forced correlation), allowlist-add reports
retired_findings >= 1, and after correlation re-runs the finding does not
come back (A already makes correlation skip allowlisted entities).

Also refresh the doctor findings.allowlisted message, which claimed
allowlisting does not delete stored findings -- it now does.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-29 13:53:21 -06:00
trvon
182e49b4c0
Build: keep LTO temporaries off tmpfs so a relink can't OOM the host
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
/tmp is tmpfs (RAM) on the dev host; a large GCC LTO relink writes its
ltrans files to $TMPDIR and, when that is /tmp, can exhaust memory and
freeze the machine (observed this session). Point the build's TMPDIR at a
disk-backed build/tmp, created as an order-only prerequisite of the
compiling targets (build, conan-build, fuzz). Override with BUILD_TMPDIR=.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 22:38:02 -06:00
trvon
728b2462e7
Governor: background work no longer starves under sustained pressure
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
On a host that sits at critical/emergency for a long time (its own apps holding
most of RAM, or the daemon slightly over its soft budget), the starvation guard
was disabled (`!hard`), so correlation, retro-hunt and inventory deferred
forever — observed deferred 153 min with forced=0, which also starves the
retro-hunt delayed labels that Stage 4 training needs.

- Tiered starvation guard: it now fires under critical/emergency too, but after a
  level-scaled window (1x warning, 4x critical, 12x emergency) so background work
  is rare under load, not zero. A true host-memory crunch (host_severe_,
  MemAvailable below the severe floor) still holds everything, since forcing work
  then could tip the host toward OOM. exec guard/scan paths unchanged.
- Raise the balanced own-memory soft budget 512 -> 768 MiB: the daemon's steady
  footprint grew (telemetry buffers, the ingest server, the shadow model), so 512
  left it permanently over budget -> pinned critical/emergency. 768 fits the real
  baseline, still well under the 1.5 GiB cgroup cap.

Tests updated for the 768 budget and the tiered guard (critical forces after the
4x window; severe holds it). Suite 51/51; analyzer + clang-tidy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 22:33:25 -06:00
trvon
4ade69ec7e
Learn: allowlisted binaries are negative labels, not teacher positives
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The teacher-label join made a window a positive if it was detected
(replay.cpp), but it never applied the allowlist, so a window running a cleared
false positive (e.g. /usr/bin/ss) was labelled a teacher positive — poisoning
Stage 4 training. Negative labels were written to the stream but replay dropped
them.

- replay_store now collects negative labels into ReplayOptions.negative_labels
  (it used to ignore any non-positive label).
- A window whose exec hash is negatively labelled is marked `suppressed`; the
  example/policy-eval positive is gated on `!suppressed`, so such a window is
  kept as a *negative* training example (useful signal) and never a positive,
  even if a cleared rule "detected" it.
- DetectionPipeline seeds a negative label for every allowlisted sha256 at init,
  so evidence stored before the allowlist (the historical ss storm) is covered
  without waiting for a re-exec.

Test: a detected window with a stored negative label for its hash is a positive
before the label and a kept-negative after. Suite 51/51; analyzer + clang-tidy
clean on replay.cpp and detection_pipeline.cpp.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 22:02:56 -06:00