Four findings from a four-lane audit made the promotion gate unable to support its own claims. In-sample evaluation: training walks the whole post-warm-up store and the gate evaluates the same windows, with nothing checking provenance - replay.cpp itself listed point_in_time_splits_and_overlap_not_assessed. Models now record the window range they learned from (file format v2; v1 still loads as unknown) and a content fingerprint, evaluation reports its own range plus in_sample, and promotion_gate fails with the reason: an overlapping range, no recorded range, or no digest. Unprovable provenance never counts as out-of-sample. Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns 0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check was vacuous. A cell whose baseline scored no hits or spent no CPU now fails, naming the budget refusals that starved it. Stale models: cells keyed on a bare filename, so a later cycle that refused to train left an older cycle's dist/gate models to be judged, and nothing bound a verdict to the file it judged. Cells and reports carry the model digest, and fleet gate sets refused (non-zero exit) for a cell without one or for a (model, period) whose digests disagree across nodes instead of pooling them. 3 seeds are not generalization: they vary initialisation and the per-node shuffle over identical windows. The procedure document now says so and states what would test generalization (disjoint periods, models trained on an earlier cut, out-of-sample ranges). Tests: in-sample flagged, disjoint range not flagged, unprovable provenance fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report refused, range round-trip and fingerprint, v1 compatibility.
35 lines
1.4 KiB
C++
35 lines
1.4 KiB
C++
// Model file decoder over untrusted bytes: never crashes; whatever it
|
|
// accepts re-encodes to a model that decodes to the same model, and runs a
|
|
// forward pass safely. Version 1 files are upgraded on re-encode, so byte
|
|
// equality is only required for the current version.
|
|
|
|
#include "strawberry/learn/model_file.hpp"
|
|
|
|
#include <cstdint>
|
|
#include <cstring>
|
|
#include <string>
|
|
|
|
using namespace strawberry::learn;
|
|
|
|
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
|
|
const std::string_view in(reinterpret_cast<const char*>(data), size);
|
|
Model m;
|
|
std::string error;
|
|
if (decode_model(in, m, error)) {
|
|
const auto bytes = encode_model(m);
|
|
Model again;
|
|
std::string again_error;
|
|
if (!decode_model(bytes, again, again_error) || !again.consistent()) __builtin_trap();
|
|
if (again.shape != m.shape || again.generation != m.generation || again.note != m.note ||
|
|
again.training_first_ns != m.training_first_ns || again.training_last_ns != m.training_last_ns)
|
|
__builtin_trap();
|
|
uint16_t version = 0;
|
|
if (size >= 6) std::memcpy(&version, data + 4, sizeof(version));
|
|
if (version == kModelVersion && bytes != in) __builtin_trap();
|
|
std::vector<float> x(m.shape.in, 0.5f);
|
|
(void) forward(m, x);
|
|
} else if (error.empty()) {
|
|
__builtin_trap();
|
|
}
|
|
return 0;
|
|
}
|