trevon/gem
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gem/docs/user_guide/getting-started.md

4.8 KiB
Raw Permalink Blame History

Getting started

gem is a small, offline, spec-locked C inference engine for dense Gemma 4 12B and 31B on Apple Silicon. It owns GGUF loading, the fixed Gemma 4 graph, tokenizer, KV cache, sampling, Metal dispatch, and a minimal OpenAI-compatible HTTP/SSE server for hax. It never links libllama/ggml and never launches llama-server.

Models

Point gem at a GGUF by path or by name:

build/gem generate --model /path/to/gemma-4-31B-it-Q4_0.gguf --prompt Hello
build/gem generate --model gemma-4-31B-it-Q4_0 --prompt Hello      # name resolution

Name resolution order: an existing file path as given → GEM_MODEL_DIR (colon-separated directories) → the default <repo>/.state/models → the local HuggingFace cache (HF_HUB_CACHE, else $HF_HOME/hub, else ~/.cache/huggingface/hub). A .gguf suffix is appended automatically when the name lacks one. Environment: GEM_MODEL_DIR, HF_HOME, HF_HUB_CACHE.

Quantization requirement. The engine's strict manifest accepts Q4_0 projection weights with a Q8_0 tied embedding (norms/scales F32) — exactly the layout of llm.cpp-style single-GGUF conversions. Files in other quants are rejected at load with a clear message.

12B serving status (currently: documented-unsupported). The engine structurally supports 12B (48 layers, 667 tensors) and the metal path is correct, but Q4_0 quantization of the 12B checkpoint provably flips the next-token logits into a degenerate high-ID band (oracle-validated analysis in ../research/... and artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md). Correct 12B serving requires Q8_0 projection support, tracked as an engine feature. 12B at Q4_0 is caught by the gem.12b_servability_gate test and reported as unsupported. 31B Q4_0 is the validated serving path.

MTP (speculative) sidecar. With F32 KV, gem agent automatically discovers a colocated mtp-<model filename> assistant. Use --no-mtp to force direct decode.

Commands

build/gem print-config --model MODEL        # resolved engine/architecture JSON
build/gem generate  --model MODEL --prompt TEXT [--context N] [--max-tokens N]
                    [--temperature F] [--top-k N] [--top-p F] [--seed N]
                    [--command-span layer|token] [--kv-type f32|f16|f16-full]
build/gem serve     --model MODEL [--alias NAME] [--port N] [--mtp ASSISTANT] ...
build/gem agent     --model MODEL [--mtp|--no-mtp] [--hax PATH] [-- HAX_ARGS]
build/gem-inspect   [--gguf|--tensors|--bindings] MODEL   # manifest/metadata
build/gem-logits    --model MODEL --prompt TEXT --context N --output FILE.bin
build/gem-bench     --model MODEL --context N --prompt-tokens N --generation-tokens N \
                    --repetitions 3 --command-span token --kv-type f32

OpenAI server

gem serve (or gem agent, which owns the lifecycle and spawns hax) binds 127.0.0.1 (default port 8080) and serves:

Endpoint Purpose
GET /health liveness
GET /v1/models the loaded alias
GET /props engine properties
POST /v1/chat/completions OpenAI chat completions (SSE stream + non-stream)

Requests accept model, messages, tools, stream, max_tokens/max_completion_tokens, temperature, top_p. Responses include id, choices (with delta.tool_calls when streaming), finish_reason (stop/tool_calls), and usage.

There is no auth or CORS: it is a localhost service, one model per server. hax connects with HAX_LLAMACPP_BASE_URL=http://127.0.0.1:8080/v1 (llamacpp is hax's protocol-selector label; no llama.cpp participates).

Typical flows

build/gem agent --model gemma-4-31B-it-Q4_0 -- -p 'inspect this project'   # offline hax, preferred
build/gem serve  --model /path/to/model.gguf --alias gemma-4-31b            # two-terminal debug

Troubleshooting

  • "model not found" — no path matched: check GEM_MODEL_DIR and that the name resolves in .state/models or the HF cache.
  • "missing, type-incompatible tensor" — the file's quant/layout isn't the supported Q4_0-projection/Q8_0-embedding contract (e.g., a Q8_0 model passes inspect --tensors but fails load).
  • 12B output degenerates (e.g. repeated digits / template tokens) — documented-unsupported at Q4_0; see scripts/remediate-12b.sh --verify and the diagnosis above.
  • port already in use — pick --port or stop the old server.
  • large model footprint — weights are mmap'd no-copy; the HF cache holds the actual blobs (a gitignored cache, safe to let hf manage).
  • performance questions — follow docs/build/benchmark-protocol.md before drawing conclusions; the project's results live in artifacts/ and project memory (scripts/yams search --cwd --query ...).

Build details: docs/build/build-guide.md. Architecture and research: docs/research/.