trevon/gem
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gem/README.md

2.5 KiB
Raw Permalink Blame History

gem

gem is a small, fast, native C inference engine for dense Gemma 4 12B and 31B on Apple Silicon (M4 Max), built to be an offline OpenAI-compatible server for hax. It owns GGUF loading, the fixed Gemma 4 graph, tokenizer, KV cache, sampling, Metal dispatch, and the minimal HTTP/SSE chat/completions endpoint. It never links libllama/ggml and never launches llama-server; the adjacent llama.cpp checkout is a reference and benchmark oracle only.

Highlights:

  • strict mmap-backed GGUF parsing and 667/833-tensor manifests for 12B/31B;
  • native Gemma 4 BPE, canonical turn/thinking/tool prompt formatting;
  • full Metal decode/prefill for Q4_0 projections with Q8_0 tied embeddings;
  • auto-routed long-context attention (1K SWA wrap → 4K BQ8/BK64 prefill → 8K split16 decode);
  • exact-prefix KV reuse (2nd-turn ~70x faster), native sampler, speculative MTP sidecars;
  • OpenAI-compatible /v1/chat/completions (SSE + non-stream) at 127.0.0.1, for hax (provider label llamacpp is only a protocol selector).

12B note: the engine structurally supports 12B, but Q4_0 quantization of the 12B checkpoint provably flips next-token logits into a degenerate band (oracle-validated; see artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md). Correct 12B serving requires Q8_0 projection support (tracked engine feature). 12B at Q4_0 is caught by the gem.12b_servability_gate ctest and documented-unsupported; 31B Q4_0 is the validated serving path.

Quickstart

cmake -S . -B build && cmake --build build -j      # build
ctest --test-dir build --output-on-failure          # 70-test suite
build/gem generate --model gemma-4-31B-it-Q4_0 --prompt Hello     # name-resolved (GEM_MODEL_DIR / HF cache)
build/gem agent  --model gemma-4-31B-it-Q4_0 -- -p 'inspect this project'   # offline hax (preferred)
build/gem serve  --model /path/to/model.gguf --port 8080            # two-terminal debug

Model names resolve through GEM_MODEL_DIR, <repo>/.state/models, then the HuggingFace cache (HF_HOME/HF_HUB_CACHE); explicit paths always win. A .gguf suffix is appended for names.

Docs