14 lines
No EOL
2.5 KiB
JSON
14 lines
No EOL
2.5 KiB
JSON
{
|
|
"nextId": 2,
|
|
"tasks": [
|
|
{
|
|
"id": "1",
|
|
"subject": "12B serving: Q8_0 projection support + config/detection router (engine feature)",
|
|
"description": "GOAL: make gem serve a correct Gemma-4-12B by adding Q8_0 projection support plus a config/detection router so both Q4_0 (fast/31B) and Q8_0 (correct/12B) coexist. DONE: gem generates sane 12B text with no 236xxx degenerate signature, 31B Q4_0 stays oracle-identical (logits cosine 0.99999, full ctest green, -Werror clean), and the router auto-selects the servable artifact grade by name.\n\nCONTEXT (proven): engine is correct; Q4_0 quantization of the 12B genuinely flips next-token argmax into the 236xxx band (verified with an oracle-validated independent numpy reference on ggml Q4_0 AND on a faithful self re-quant; same weights at Q8_0 give sane argmax 575). gem's manifest is Q4_0-only; correct 12B serving requires Q8_0 projections. Proof artifacts: artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md; tools /private/tmp/ref12.py (oracle-validated reference) and /private/tmp/requant12.py.\n\nROUTER DESIGN to implement alongside the kernels:\n1) Capability-aware acceptance: model.c EXPECT_LAYER takes an accepted-quant SET and records each tensor's actual quant (per-tensor, config profile per model size: 12B projects q8_0, 31B projects q4_0; embeddings q8_0).\n2) Runtime servability detector: cheap startup probe (deterministic logits on a fixed prompt) + signature rule keyed on argmax-in-degenerate-band (FIXED in scripts/remediate-12b.sh --verify: sane = argmax < GARBAGE_FLOOR, not top-10 tail count); outcome feeds accept/refuse/auto-promote.\n3) Auto-promote via the existing model resolver (GEM_MODEL_DIR / HF cache) so `gem --model gemma-4-12B-it` picks the servable grade once Q8_0 loads.\n4) Execution-path routing: surface each tensor's quant into the fixed-graph kernel selector so the same graph runs Metal-Q4_0 (31B) or Metal-Q8_0 (new projection kernels) paths.\n\nGATES: per AGENTS.md, kernel correctness lands first as a backend-op test in the adjacent llama.cpp checkout, then the local benchmark gate with raw JSON + print-config + 3 reps. 31B must not regress (recorded oracle). Then promote Q8_0 as the 12B serving path in README/docs.\n\nDEPENDENCIES: this is a standalone engine-feature task (independent of docs-restructure). No blocker other than scope; the timed Metal/Q8_0 kernel work is the bulk.",
|
|
"status": "pending",
|
|
"createdAt": 1788216711007,
|
|
"updatedAt": 1788216711007,
|
|
"revision": 0
|
|
}
|
|
]
|
|
} |