7076e4b283* perf(daemon): coalesce metadata writes * refactor(metadata): split repository ops * fix(metadata): harden repository invariants * feat: test coverage and continued refactor work in metadata code * fix(graph): handle path-like explore queries - prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH * fix(daemon): quiesce shutdown background work - Interrupt pending DB acquires before worker shutdown - Wait for tracked background coroutines and timers to drain * fix(daemon): cancel repair work on shutdown Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting. * perf(metadata): reduce sqlite lock contention * perf(metadata): flatten batch content retries * fix(daemon): align shutdown wait budgets A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression. * perf(metadata): bound feedback write waits * perf(metadata): prep batch content before tx * fix(daemon): honor no-embeddings tasks Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing. * perf(daemon): batch fallback metadata writes * feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements * perf(metadata): flatten insert retries * fix(metadata): clear CI portability blockers Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path. * perf(metadata): batch writes and steady bench * fix(daemon): skip inactive post-ingest pollers * perf(metadata): tighten tag and path caches * perf(metadata): cache small batch upserts * fix(daemon): harden embedded host teardown * feat(docs): updating docs policies and language * perf(metadata): tune doc update retries * perf(metadata): bound download retries * perf(metadata): bound download metadata waits * bench(metadata): scope post-ingest content path * fix(daemon): export embedded test hooks * fix(daemon): harden test helper linkage * fix(build): add missing standard includes
172 lines
6.4 KiB
Markdown
172 lines
6.4 KiB
Markdown
<p align="center">
|
|
<h1 align="center">YAMS — Yet Another Memory System</h1>
|
|
<h6 align="center">Persistent memory for LLMs and apps. Content-addressed storage with dedupe, compression, full-text and vector search.</h6>
|
|
</p>
|
|
<p align="center">
|
|
<img alt="license" src="https://img.shields.io/github/license/trvon/yams?style=flat-square">
|
|
<img alt="language" src="https://img.shields.io/github/languages/top/trvon/yams?style=flat-square">
|
|
<img alt="github builds" src="https://img.shields.io/github/actions/workflow/status/trvon/yams/release.yml">
|
|
<img alt="last commit" src="https://img.shields.io/github/last-commit/trvon/yams?style=flat-square">
|
|
</p>
|
|
|
|
> [!WARNING]
|
|
> **Experimental — not production ready.** Expect bugs and breaking changes until 1.0.
|
|
|
|
## Features
|
|
|
|
- SHA-256 content-addressed storage with block-level dedupe (Rabin chunking)
|
|
- Full-text search (SQLite FTS5) + semantic vector search (embeddings)
|
|
- Tree-sitter symbol extraction for 18 languages ([list](docs/user_guide/cli.md#symbol-extraction))
|
|
- Snapshot management with Merkle tree diffs and rename detection
|
|
- WAL-backed durability, high-throughput I/O, thread-safe
|
|
- CLI, MCP server, and C-ABI plugins (ONNX/GLiNER/ColBERT, S3 storage, PDF via ZYP)
|
|
- Interactive relevance tuning through CLI tuning and doctor workflows
|
|
|
|
## Documentation
|
|
|
|
| Topic | Link |
|
|
|----------------------|---------------------------------------------------------------------|
|
|
| Install | [docs/user_guide/installation.md](docs/user_guide/installation.md) |
|
|
| CLI reference | [docs/user_guide/cli.md](docs/user_guide/cli.md) |
|
|
| MCP server | [docs/user_guide/mcp.md](docs/user_guide/mcp.md) |
|
|
| Embeddings | [docs/user_guide/embeddings.md](docs/user_guide/embeddings.md) |
|
|
| Plugins | [docs/PLUGINS.md](docs/PLUGINS.md) |
|
|
| Build from source | [docs/BUILD.md](docs/BUILD.md) |
|
|
| Architecture | [docs/architecture/](docs/architecture/) |
|
|
| Benchmarks | [docs/benchmarks/README.md](docs/benchmarks/README.md) |
|
|
| Changelog | [docs/changelogs/](docs/changelogs/) |
|
|
| Roadmap | [docs/roadmap.md](docs/roadmap.md) |
|
|
|
|
## Links
|
|
|
|
- SourceHut: https://sr.ht/~trvon/yams/
|
|
- GitHub mirror: https://github.com/trvon/yams
|
|
- Docs site: https://yamsmemory.ai
|
|
- Discord: https://discord.gg/rTBmRHdTEc
|
|
- License: GPL-3.0-or-later
|
|
|
|
## Community and contributing
|
|
|
|
Questions, bug reports, docs fixes, tests, and feature ideas are all welcome.
|
|
|
|
If you'd like to contribute, start with:
|
|
- `CONTRIBUTING.md` for workflow and expectations
|
|
- `CODE_OF_CONDUCT.md` for community standards
|
|
- `SECURITY.md` for responsible vulnerability reporting
|
|
- `.github/ISSUE_TEMPLATE/` for issue templates
|
|
|
|
If you're unsure whether something is worth opening, it usually is.
|
|
|
|
## Install
|
|
|
|
Supported: Linux x86_64/ARM64, macOS x86_64/ARM64, Windows x86_64.
|
|
|
|
```bash
|
|
# macOS
|
|
brew install trvon/yams/yams
|
|
|
|
# Docker
|
|
docker pull ghcr.io/trvon/yams:latest
|
|
|
|
# Debian/Ubuntu, Fedora/RHEL, Windows: see installation guide
|
|
```
|
|
|
|
Full install matrix and package repos: [docs/user_guide/installation.md](docs/user_guide/installation.md).
|
|
|
|
### Build from source
|
|
|
|
```bash
|
|
./setup.sh Release # Linux/macOS (auto-detects toolchain, runs Conan + Meson)
|
|
meson compile -C build/release
|
|
```
|
|
|
|
```pwsh
|
|
./setup.ps1 Release # Windows
|
|
meson compile -C build/release
|
|
```
|
|
|
|
Requires a C++20 toolchain (GCC 13+, Clang 16+, or MSVC 2022+ recommended), Meson, Ninja, CMake, pkg-config, and Conan. See [docs/BUILD.md](docs/BUILD.md).
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
yams init # interactive; use --auto for headless
|
|
yams add ./README.md --tags docs
|
|
yams add src/ --recursive --include="*.cpp,*.h" --tags code
|
|
|
|
yams search "config file" --limit 5
|
|
yams grep "TODO" --include="*.cpp"
|
|
|
|
yams list --limit 20
|
|
yams get <hash> -o ./output.bin
|
|
```
|
|
|
|
Shell completions: `yams completion bash|zsh|fish|powershell`. Install instructions: [docs/user_guide/cli.md#cmd-completion](docs/user_guide/cli.md#cmd-completion).
|
|
|
|
## MCP Server
|
|
|
|
YAMS ships an MCP server over stdio (JSON-RPC) for AI assistants.
|
|
|
|
```bash
|
|
yams serve
|
|
```
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"yams": { "command": "yams", "args": ["serve"] }
|
|
}
|
|
}
|
|
```
|
|
|
|
Tool reference and MCP client setup: [docs/user_guide/mcp.md](docs/user_guide/mcp.md).
|
|
|
|
## Plugins
|
|
|
|
```bash
|
|
yams plugin list # loaded plugins
|
|
yams plugin trust add ~/.local/lib/yams/plugins # trust a directory
|
|
yams plugin health # status
|
|
yams doctor plugin onnx # diagnose
|
|
```
|
|
|
|
Plugin architecture, trust model, and bundled plugins (ONNX, S3, ZYP, GLiNER, symbol extractor): [docs/PLUGINS.md](docs/PLUGINS.md).
|
|
|
|
### GPU acceleration (ONNX)
|
|
|
|
| Platform | Provider | Hardware |
|
|
|----------|------------|---------------------------------------------|
|
|
| macOS | CoreML | Apple Silicon Neural Engine + GPU |
|
|
| Linux | CUDA | NVIDIA GPUs |
|
|
| Linux | MIGraphX | AMD GPUs (ROCm) |
|
|
| Windows | DirectML | Any DirectX 12 GPU (NVIDIA, AMD, Intel) |
|
|
|
|
Auto-detected at build. Override with `YAMS_ONNX_GPU=auto|cuda|coreml|directml|migraphx|none`. Details: [plugins/onnx/README.md](plugins/onnx/README.md).
|
|
|
|
### Simeon backend
|
|
|
|
YAMS uses `simeon` by default for both dense vector embeddings and lexical (BM25) search — no model download required. Set `embeddings.backend = "simeon"` in your TOML config (or leave it unset; simeon is the default). Fine-grained knobs live under `[embeddings.simeon]`.
|
|
|
|
Full config reference and dim tradeoffs: [docs/user_guide/embeddings.md](docs/user_guide/embeddings.md).
|
|
|
|
## Troubleshooting
|
|
|
|
```bash
|
|
yams doctor # full diagnostics
|
|
yams stats --verbose # storage statistics
|
|
yams repair --all # repair common issues
|
|
```
|
|
|
|
Build issues: [docs/BUILD.md](docs/BUILD.md). Empty `yams plugin list`? Add a trust path: `yams plugin trust add ~/.local/lib/yams/plugins`.
|
|
|
|
## Cite
|
|
|
|
```bibtex
|
|
@misc{yams,
|
|
author = {Trevon Williams},
|
|
title = {YAMS: Content-addressable storage with semantic search},
|
|
year = {2025},
|
|
publisher = {GitHub},
|
|
url = {https://github.com/trvon/yams}
|
|
}
|
|
```
|