strawberry/fuzz/fuzz_model_file.cpp
trvon 7149a90893
gate: refuse in-sample and degenerate evidence, and identify the model
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.

In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.

Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.

Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.

3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).

Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
2026-11-19 16:20:40 -07:00

35 lines
1.4 KiB
C++

// Model file decoder over untrusted bytes: never crashes; whatever it
// accepts re-encodes to a model that decodes to the same model, and runs a
// forward pass safely. Version 1 files are upgraded on re-encode, so byte
// equality is only required for the current version.
#include "strawberry/learn/model_file.hpp"
#include <cstdint>
#include <cstring>
#include <string>
using namespace strawberry::learn;
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
const std::string_view in(reinterpret_cast<const char*>(data), size);
Model m;
std::string error;
if (decode_model(in, m, error)) {
const auto bytes = encode_model(m);
Model again;
std::string again_error;
if (!decode_model(bytes, again, again_error) || !again.consistent()) __builtin_trap();
if (again.shape != m.shape || again.generation != m.generation || again.note != m.note ||
again.training_first_ns != m.training_first_ns || again.training_last_ns != m.training_last_ns)
__builtin_trap();
uint16_t version = 0;
if (size >= 6) std::memcpy(&version, data + 4, sizeof(version));
if (version == kModelVersion && bytes != in) __builtin_trap();
std::vector<float> x(m.shape.in, 0.5f);
(void) forward(m, x);
} else if (error.empty()) {
__builtin_trap();
}
return 0;
}