Brick analysis and measurement flow

Design seam

Before editing, state:

observable graph or renderer behavior;
owning seam: extractor, identity, model, metric, importer, or renderer;
expected node/edge/diagnostic delta;
invariants: deterministic output, provenance, and unrelated identities.

Evidence lanes

Keep lexical/syntactic extraction independent from imported Fold symbolic or binary evidence. Merge them only in the canonical graph, retaining lane, backend, confidence, and provenance on every edge.

Test loop

Add a minimal fixture demonstrating the desired relationship.
Add a clean-negative fixture for likely lexical false positives.
Assert public model behavior rather than private regex details.
Implement, run the focused test, then the full suite.
Add a regression fixture when a bug is discovered.

Property tests are appropriate for stable ID determinism, JSON round trips, and graph invariants. Fuzzing is appropriate for report ingestion and source text; the invariant is no panic, runaway allocation, or path escape.

Measurement loop

For an extractor or score change, write an experiment contract:

{
  "hypothesis": "what should improve",
  "owner": "one extractor or metric",
  "fixtures": ["labeled corpus names"],
  "expected": {
    "increases": ["precision or recall metric"],
    "preserves": ["stable IDs", "schema", "unrelated edges"]
  },
  "decision": "pending"
}

Use at least three repeats for performance claims. Compare raw artifacts and report median latency plus observed range. A count on an unlabeled repository is coverage information, not precision or recall.

Promotion rule

Keep a behavior change when labeled positives improve without clean-negative regressions and stable invariants hold.
Keep an unproven strategy explicit and disabled by default.
Reject changes within measurement noise or changes that silently lose coverage/provenance.

1.9 KiB Raw Permalink Blame History