45 lines
2.5 KiB
Markdown
45 lines
2.5 KiB
Markdown
# gem
|
|
|
|
`gem` is a small, fast, **native C inference engine for dense Gemma 4 12B and 31B** on Apple
|
|
Silicon (M4 Max), built to be an offline OpenAI-compatible server for hax. It owns GGUF loading,
|
|
the fixed Gemma 4 graph, tokenizer, KV cache, sampling, Metal dispatch, and the minimal
|
|
HTTP/SSE chat/completions endpoint. It **never links libllama/ggml and never launches
|
|
`llama-server`**; the adjacent llama.cpp checkout is a reference and benchmark oracle only.
|
|
|
|
Highlights:
|
|
|
|
- strict mmap-backed GGUF parsing and 667/833-tensor manifests for 12B/31B;
|
|
- native Gemma 4 BPE, canonical turn/thinking/tool prompt formatting;
|
|
- full Metal decode/prefill for Q4_0 projections with Q8_0 tied embeddings;
|
|
- auto-routed long-context attention (1K SWA wrap → 4K BQ8/BK64 prefill → 8K split16 decode);
|
|
- exact-prefix KV reuse (2nd-turn ~70x faster), native sampler, speculative MTP sidecars;
|
|
- OpenAI-compatible `/v1/chat/completions` (SSE + non-stream) at 127.0.0.1, for hax (provider label
|
|
`llamacpp` is only a protocol selector).
|
|
|
|
> **12B note:** the engine structurally supports 12B, but Q4_0 quantization of the 12B checkpoint
|
|
> provably flips next-token logits into a degenerate band (oracle-validated; see
|
|
> `artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md`). Correct 12B serving requires Q8_0 projection
|
|
> support (tracked engine feature). 12B at Q4_0 is caught by the `gem.12b_servability_gate` ctest and
|
|
> documented-unsupported; **31B Q4_0 is the validated serving path**.
|
|
|
|
## Quickstart
|
|
|
|
```sh
|
|
cmake -S . -B build && cmake --build build -j # build
|
|
ctest --test-dir build --output-on-failure # 70-test suite
|
|
build/gem generate --model gemma-4-31B-it-Q4_0 --prompt Hello # name-resolved (GEM_MODEL_DIR / HF cache)
|
|
build/gem agent --model gemma-4-31B-it-Q4_0 -- -p 'inspect this project' # offline hax (preferred)
|
|
build/gem serve --model /path/to/model.gguf --port 8080 # two-terminal debug
|
|
```
|
|
|
|
Model names resolve through `GEM_MODEL_DIR`, `<repo>/.state/models`, then the HuggingFace cache
|
|
(`HF_HOME`/`HF_HUB_CACHE`); explicit paths always win. A `.gguf` suffix is appended for names.
|
|
|
|
## Docs
|
|
|
|
- [Getting started (`docs/user_guide/`)](docs/user_guide/getting-started.md) — models, CLI,
|
|
server/OpenAI usage, hax, troubleshooting.
|
|
- [Build guide (`docs/build/`)](docs/build/build-guide.md) — prerequisites, CMake, variants,
|
|
tests, benchmark protocol.
|
|
- [Research (`docs/research/`)](docs/research/) — architecture, kernel roadmap, fused-split-KV
|
|
design, completion audit, and the engine research record (all kernel/benchmark evidence).
|