# gem

`gem` is a small, fast, **native C inference engine for dense Gemma 4 12B and 31B** on Apple
Silicon (M4 Max), built to be an offline OpenAI-compatible server for hax. It owns GGUF loading,
the fixed Gemma 4 graph, tokenizer, KV cache, sampling, Metal dispatch, and the minimal
HTTP/SSE chat/completions endpoint. It **never links libllama/ggml and never launches
`llama-server`**; the adjacent llama.cpp checkout is a reference and benchmark oracle only.

Highlights:

- strict mmap-backed GGUF parsing and 667/833-tensor manifests for 12B/31B;
- native Gemma 4 BPE, canonical turn/thinking/tool prompt formatting;
- full Metal decode/prefill for Q4_0 projections with Q8_0 tied embeddings;
- auto-routed long-context attention (1K SWA wrap → 4K BQ8/BK64 prefill → 8K split16 decode);
- exact-prefix KV reuse (2nd-turn ~70x faster), native sampler, speculative MTP sidecars;
- OpenAI-compatible `/v1/chat/completions` (SSE + non-stream) at 127.0.0.1, for hax (provider label
  `llamacpp` is only a protocol selector).

> **12B note:** the engine structurally supports 12B, but Q4_0 quantization of the 12B checkpoint
> provably flips next-token logits into a degenerate band (oracle-validated; see
> `artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md`). Correct 12B serving requires Q8_0 projection
> support (tracked engine feature). 12B at Q4_0 is caught by the `gem.12b_servability_gate` ctest and
> documented-unsupported; **31B Q4_0 is the validated serving path**.

## Quickstart

```sh
cmake -S . -B build && cmake --build build -j      # build
ctest --test-dir build --output-on-failure          # 70-test suite
build/gem generate --model gemma-4-31B-it-Q4_0 --prompt Hello     # name-resolved (GEM_MODEL_DIR / HF cache)
build/gem agent  --model gemma-4-31B-it-Q4_0 -- -p 'inspect this project'   # offline hax (preferred)
build/gem serve  --model /path/to/model.gguf --port 8080            # two-terminal debug
```

Model names resolve through `GEM_MODEL_DIR`, `<repo>/.state/models`, then the HuggingFace cache
(`HF_HOME`/`HF_HUB_CACHE`); explicit paths always win. A `.gguf` suffix is appended for names.

## Docs

- [Getting started (`docs/user_guide/`)](docs/user_guide/getting-started.md) — models, CLI,
  server/OpenAI usage, hax, troubleshooting.
- [Build guide (`docs/build/`)](docs/build/build-guide.md) — prerequisites, CMake, variants,
  tests, benchmark protocol.
- [Research (`docs/research/`)](docs/research/) — architecture, kernel roadmap, fused-split-KV
  design, completion audit, and the engine research record (all kernel/benchmark evidence).