Commit graph

2 commits

Author SHA1 Message Date
trvon
7149a90893
gate: refuse in-sample and degenerate evidence, and identify the model
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.

In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.

Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.

Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.

3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).

Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
2026-11-19 16:20:40 -07:00
trvon
fdd66e8bab
Stage 4 groundwork: window features, inspection MLP, model files
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Nothing runs in the daemon yet; these are the pieces the next increments
train and run in shadow.

- learn/features: a WindowSummary accumulated from a window's events
  (exec locations, memfd/deleted/uid0 execs, depth, connection classes
  and ports, persistence and drop writes, ptrace, module loads, scan
  outcomes, capped surprisal per tuple kind, bounded tokens) and
  featurize(): 48 explicit features scaled to about [0, 1] plus signed
  hashing of tokens into 208 dims, with the governor level and budget
  left as context. A window's own detections are never features: they
  are the teacher labels.
- learn/mlp: 256 -> 128 -> 64 ReLU with a policy head over the
  inspection actions (none, memory_scan, deep_rescan, watch_subtree,
  rescan_on_next_rules) and a value head. Scalar reference, AVX2+FMA
  chosen at runtime, NEON on aarch64; safe default (no inspection) on a
  wrong input size or an inconsistent model. Deterministic init per seed.
- learn/model_file: "SBMD" format with bounded dimensions, exact length,
  a sha256 trailer and finite weights checked on load, each failure
  named; atomic save into a 0700 directory.

Tests: test_model (kernels against scalar for arbitrary sizes, forward
pass invariants, deterministic init, round-trip, every flipped byte and
truncation rejected, non-finite weights rejected, features bounded and
deterministic). fuzz_model_file. NEON is built and tested only on aarch64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:53:25 -06:00