Commit graph

3 commits

Author SHA1 Message Date
trvon
c565cb5aee
Learn: handle extreme restored rarity clocks
Represent full-range ordered clock differences without signed overflow and allow complete decay. Add seeded restore-clock regressions and a dedicated fuzz seed.
2026-11-03 06:52:20 -07:00
trvon
f76db466eb
Stage 4: keep the live rarity model across daemon restarts
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The shadow scorer's rarity model started empty at every daemon start, so
each restart cost an hour of warm-up with no decisions. The daemon now
saves it to model_dir/rarity.state (checksummed, atomic write, hourly and
at stop) and restores it at start. A restored state younger than the
7-day half-life skips the warm-up; decay catches up on the first event.

A file made with other rarity settings, a bad checksum, or a negative or
non-finite count is refused with a reason (status model.rarity_state,
doctor model.rarity_state) and the model starts empty. The file must be
owned by the daemon user and not group/world writable, like model files.

Also fixes status model.warmup_windows, which was never filled in.

Tests: R1/R2 in test_learn (round trip, decay clock, refusals), S8 in
test_shadow (warm-up skipped only when recent), daemon restart restores
the saved state, doctor warning; new fuzz_rarity_state target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 23:25:01 -06:00
trvon
134aebf7a9
Add the tokenizer and rarity model for the learning stages
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
src/learn/ (pure libraries; nothing in the daemon calls them yet, Stage 3
uses them in replay):

- tokenize: path templates (user homes, per-uid dirs, pids, digit and hex
  runs, UUIDs, mktemp-style names abstracted) and command-line shapes
  (flags kept, values, paths, URLs, IPs, numbers and encoded blobs
  classed; the first plain word kept as a subcommand; long argument lists
  summarized). Bounded and noexcept.
- rarity: decayed count-min sketches over parent>child, grandparent>
  parent>child, exe+command-line shape, exe+connection class and
  exe+persistence location, scored as surprisal in bits; a bounded
  first-seen table; deterministic hashing for replay. About 1.3 MiB.

tests/test_learn.cpp: tokenizer cases and idempotence; count-min never
underestimates (also after decay); determinism; rare scores above common;
half-life decay; bounded first-seen. fuzz_tokenizer found two
idempotence bugs (truncation before templating, and a cut that left a
" (deleted)" suffix), fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:21:28 -06:00