Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
src/learn/ (pure libraries; nothing in the daemon calls them yet, Stage 3 uses them in replay): - tokenize: path templates (user homes, per-uid dirs, pids, digit and hex runs, UUIDs, mktemp-style names abstracted) and command-line shapes (flags kept, values, paths, URLs, IPs, numbers and encoded blobs classed; the first plain word kept as a subcommand; long argument lists summarized). Bounded and noexcept. - rarity: decayed count-min sketches over parent>child, grandparent> parent>child, exe+command-line shape, exe+connection class and exe+persistence location, scored as surprisal in bits; a bounded first-seen table; deterministic hashing for replay. About 1.3 MiB. tests/test_learn.cpp: tokenizer cases and idempotence; count-min never underestimates (also after decay); determinism; rare scores above common; half-life decay; bounded first-seen. fuzz_tokenizer found two idempotence bugs (truncation before templating, and a cut that left a " (deleted)" suffix), fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
18 lines
628 B
C++
18 lines
628 B
C++
// Path and command-line shapes over arbitrary bytes: never crash, bounded,
|
|
// path_template idempotent (learn/tokenize.hpp).
|
|
|
|
#include "strawberry/learn/tokenize.hpp"
|
|
|
|
#include <cstdint>
|
|
#include <string>
|
|
|
|
using namespace strawberry::learn;
|
|
|
|
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
|
|
const std::string_view in(reinterpret_cast<const char*>(data), size);
|
|
const auto t = path_template(in);
|
|
if (t.size() > kMaxShapeBytes || path_template(t) != t) __builtin_trap();
|
|
if (cmdline_shape(in).size() > kMaxShapeBytes + 16) __builtin_trap();
|
|
(void) location_class(in);
|
|
return 0;
|
|
}
|