72647bbaf7- CI warm: ccache dep, continue-on-error, 60min timeout, YAMS_DISABLE_FAISS - Faiss: YAMS_HAS_FAISS guard, Apple Clang disable, OpenMP detection - Metadata: retry constraint failures, deduplicate batch UPSERT entries - Docker: wget for Zig, gfortran, YAMS_DISABLE_FAISS=1 - CI config: free-disk-space v1.3.1, libomp-18-dev, experimental trigger - Weekly benchmarks: faiss disable, ccache, diagnostics capture - Simeon: bump to 7763a97, EntropyRouter, concept_count - CorpusStats: extraction readiness fields - Remove committed artifacts (.wrangler, .artifacts, dispatcher_external) - Fix .vscode/settings.json hardcoded path Signed-off-by: trvon <git@trevon.dev>
183 lines
7.4 KiB
Markdown
183 lines
7.4 KiB
Markdown
<p align="center">
|
|
<h1 align="center">YAMS — Yet Another Memory System</h1>
|
|
<h6 align="center">Persistent memory for LLMs and apps. Content-addressed storage with dedupe, compression, full-text and vector search.</h6>
|
|
</p>
|
|
<p align="center">
|
|
<img alt="license" src="https://img.shields.io/github/license/trvon/yams?style=flat-square">
|
|
<img alt="language" src="https://img.shields.io/github/languages/top/trvon/yams?style=flat-square">
|
|
<img alt="github builds" src="https://img.shields.io/github/actions/workflow/status/trvon/yams/release.yml">
|
|
<img alt="last commit" src="https://img.shields.io/github/last-commit/trvon/yams?style=flat-square">
|
|
</p>
|
|
|
|
> [!WARNING]
|
|
> **Experimental — not production ready.** Expect bugs and breaking changes until 1.0.
|
|
|
|
## Features
|
|
|
|
- SHA-256 content-addressed storage with block-level dedupe (Rabin chunking)
|
|
- Full-text search (SQLite FTS5) + semantic vector search (embeddings)
|
|
- Tree-sitter symbol extraction for 18 languages ([list](docs/user_guide/cli.md#symbol-extraction))
|
|
- Snapshot management with Merkle tree diffs and rename detection
|
|
- WAL-backed durability, high-throughput I/O, thread-safe
|
|
- CLI, MCP server, and C-ABI plugins (ONNX/GLiNER/ColBERT, S3 storage, PDF via ZYP)
|
|
- Interactive relevance tuning through CLI tuning and doctor workflows
|
|
|
|
## Documentation
|
|
|
|
| Topic | Link |
|
|
|----------------------|---------------------------------------------------------------------|
|
|
| Install | [docs/user_guide/installation.md](docs/user_guide/installation.md) |
|
|
| CLI reference | [docs/user_guide/cli.md](docs/user_guide/cli.md) |
|
|
| MCP server | [docs/user_guide/mcp.md](docs/user_guide/mcp.md) |
|
|
| Embeddings | [docs/user_guide/embeddings.md](docs/user_guide/embeddings.md) |
|
|
| Plugins | [docs/PLUGINS.md](docs/PLUGINS.md) |
|
|
| Build from source | [docs/BUILD.md](docs/BUILD.md) |
|
|
| Architecture | [docs/architecture/](docs/architecture/) |
|
|
| Benchmarks | [docs/benchmarks/README.md](docs/benchmarks/README.md) |
|
|
| Changelog | [docs/changelogs/](docs/changelogs/) |
|
|
| Roadmap | [docs/roadmap.md](docs/roadmap.md) |
|
|
|
|
## Links
|
|
|
|
- SourceHut: https://sr.ht/~trvon/yams/
|
|
- GitHub mirror: https://github.com/trvon/yams
|
|
- Docs site: https://yamsmemory.ai
|
|
- Discord: https://discord.gg/rTBmRHdTEc
|
|
- License: GPL-3.0-or-later
|
|
|
|
## Install
|
|
|
|
Supported: Linux x86_64/ARM64, macOS x86_64/ARM64, Windows x86_64.
|
|
|
|
```bash
|
|
# macOS
|
|
brew install trvon/yams/yams
|
|
|
|
# Docker
|
|
docker pull ghcr.io/trvon/yams:latest
|
|
|
|
# Debian/Ubuntu, Fedora/RHEL, Windows: see installation guide
|
|
```
|
|
|
|
Full install matrix and package repos: [docs/user_guide/installation.md](docs/user_guide/installation.md).
|
|
|
|
### Build from source
|
|
|
|
```bash
|
|
./setup.sh Release # Linux/macOS (auto-detects toolchain, runs Conan + Meson)
|
|
meson compile -C build/release
|
|
```
|
|
|
|
```pwsh
|
|
./setup.ps1 Release # Windows
|
|
meson compile -C build/release
|
|
```
|
|
|
|
Requires a C++20 toolchain (GCC 13+, Clang 16+, or MSVC 2022+ recommended), Meson, Ninja, CMake, pkg-config, and Conan. See [docs/BUILD.md](docs/BUILD.md).
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
yams init # interactive; use --auto for headless
|
|
yams add ./README.md --tags docs
|
|
yams add src/ --recursive --include="*.cpp,*.h" --tags code
|
|
|
|
yams search "config file" --limit 5
|
|
yams grep "TODO" --include="*.cpp"
|
|
|
|
yams list --limit 20
|
|
yams get <hash> -o ./output.bin
|
|
```
|
|
|
|
Shell completions: `yams completion bash|zsh|fish|powershell`. Install instructions: [docs/user_guide/cli.md#cmd-completion](docs/user_guide/cli.md#cmd-completion).
|
|
|
|
## MCP Server
|
|
|
|
YAMS ships an MCP server over stdio (JSON-RPC) for AI assistants.
|
|
|
|
```bash
|
|
yams serve
|
|
```
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"yams": { "command": "yams", "args": ["serve"] }
|
|
}
|
|
}
|
|
```
|
|
|
|
Tool reference and MCP client setup: [docs/user_guide/mcp.md](docs/user_guide/mcp.md).
|
|
|
|
## Plugins
|
|
|
|
```bash
|
|
yams plugin list # loaded plugins
|
|
yams plugin trust add ~/.local/lib/yams/plugins # trust a directory
|
|
yams plugin health # status
|
|
yams doctor plugin onnx # diagnose
|
|
```
|
|
|
|
Plugin architecture, trust model, and bundled plugins (ONNX, S3, ZYP, GLiNER, symbol extractor): [docs/PLUGINS.md](docs/PLUGINS.md).
|
|
|
|
### GPU acceleration (ONNX)
|
|
|
|
| Platform | Provider | Hardware |
|
|
|----------|------------|---------------------------------------------|
|
|
| macOS | CoreML | Apple Silicon Neural Engine + GPU |
|
|
| Linux | CUDA | NVIDIA GPUs |
|
|
| Linux | MIGraphX | AMD GPUs (ROCm) |
|
|
| Windows | DirectML | Any DirectX 12 GPU (NVIDIA, AMD, Intel) |
|
|
|
|
Auto-detected at build. Override with `YAMS_ONNX_GPU=auto|cuda|coreml|directml|migraphx|none`. Details: [plugins/onnx/README.md](plugins/onnx/README.md).
|
|
|
|
### Simeon backend
|
|
|
|
YAMS uses `simeon` by default for both dense vector embeddings and lexical (BM25) search — no model download required. Set `embeddings.backend = "simeon"` in your TOML config (or leave it unset; simeon is the default). Fine-grained knobs live under `[embeddings.simeon]`.
|
|
|
|
Simeon research now points to a practical ceiling-closing path for YAMS: pair
|
|
retrieval with **corpus adapters** that expose structure the generic vector
|
|
backend cannot infer alone — file paths, titles, sections, headings, issue IDs,
|
|
citations, or domain-specific anchors. English ArguAna experiments moved from
|
|
~0.32 nDCG@10 with BM25 to ~0.76 with a topic adapter, while schema diagnostics
|
|
showed the remaining headroom is structure recognition rather than more vector
|
|
similarity.
|
|
|
|
YAMS now exposes this as a first-class search component: implement
|
|
`yams::search::CorpusAdapter`, register it with `SearchEngine::addCorpusAdapter()`,
|
|
or rely on the built-in `YamsNativeCorpusAdapter` for path-fragment and
|
|
agent-memory metadata queries such as `pbi=PBI-043 task=list-json-refresh`.
|
|
The native adapter uses English-first query seeding: it breaks natural-language
|
|
requests into compact content/path fragments, keeps structured tokens such as
|
|
`docs/research` or `PBI-043`, and records `adapter_us`, `seed_count`, and
|
|
`path_seed_queries` in debug metadata for profiling. The component is enabled
|
|
by default with `SearchEngineConfig::enableCorpusAdapters` and fused as
|
|
`corpus_adapter` evidence.
|
|
|
|
PRs are welcome for qrel-backed English corpora, yams-specific corpus adapters,
|
|
and language/profile contributions that make this retrieval path testable beyond
|
|
the current research fixture.
|
|
|
|
Full config reference and dim tradeoffs: [docs/user_guide/embeddings.md](docs/user_guide/embeddings.md).
|
|
|
|
## Troubleshooting
|
|
|
|
```bash
|
|
yams doctor # full diagnostics
|
|
yams stats --verbose # storage statistics
|
|
yams repair --all # repair common issues
|
|
```
|
|
|
|
Build issues: [docs/BUILD.md](docs/BUILD.md). Empty `yams plugin list`? Add a trust path: `yams plugin trust add ~/.local/lib/yams/plugins`.
|
|
|
|
## Cite
|
|
|
|
```bibtex
|
|
@misc{yams,
|
|
author = {Trevon Williams},
|
|
title = {YAMS: Content-addressable storage with semantic search},
|
|
year = {2025},
|
|
publisher = {GitHub},
|
|
url = {https://github.com/trvon/yams}
|
|
}
|
|
```
|