yams/docs/developer/build_system.md

Build System (Developer Reference)

Detailed reference for the YAMS build infrastructure, test organization, and CI flow.

For step-by-step build instructions, see Developer Setup. For compiler requirements and platform support, see BUILD.md.

1. Stack

Layer	Tool
Build	Meson ≥ 1.2
Bootstrap	CMake wrapper (CMakeLists.txt)
Dependencies	Conan 2.x or system/pkg-config/CMake prefixes
Generator	Ninja (preferred)
Optional	ccache, lld, clang-tidy

2. Directory Layout

Out-of-source builds under builddir/:

• Debug: builddir/ (default with Meson)
• Release: build-release/

3. ONNX / GenAI Paths

Options (Conan scope):

• enable_onnx (default True) — toggles embedding / GenAI integration
• use_conan_onnx (default False) — when True pull packaged ORT; when False allow system ORT

Meson options:

• enable-onnx — mirrors enable_onnx
• onnx-runtime-path — optional path to system ONNX Runtime

4. Configure & Build

# Using Meson (preferred)
meson setup builddir --buildtype=debugoptimized -Db_ndebug=true
ninja -C builddir

# Install
sudo meson install -C builddir

For integrator-style builds, prefer one of these entrypoints:

# Default bootstrap path (Conan + Meson)
./setup.sh Release

# Cache-only/offline Conan path
./setup.sh Release --offline

# System dependency path (no Conan, no network)
YAMS_USE_SYSTEM_DEPS=true \
YAMS_OFFLINE=true \
YAMS_PKG_CONFIG_PATH=/opt/yams-deps/lib/pkgconfig \
YAMS_CMAKE_PREFIX_PATH=/opt/yams-deps \
./setup.sh Release --system-deps --offline

5. Tests

Meson Test System (Primary)

YAMS uses Meson for test organization with hierarchical suite structure:

# Quick smoke tests (< 2min)
meson test -C builddir --suite unit:smoke

# Fast unit tests (sharded for parallel execution)
meson test -C builddir --suite unit

# Database-heavy tests
meson test -C builddir --suite unit:db

# Integration tests
meson test -C builddir --suite integration:smoke

# Performance benchmarks
meson test -C builddir --suite bench:perf

# List all available tests
meson test -C builddir --list

Test Categories

Category	Suite	Description	Typical Runtime
Unit	unit:fast	Fast, isolated unit tests	< 30s per shard
	unit:slow	Heavy unit tests (DB, lifecycle)	2-5 min
	unit:db	Database-intensive tests	1-3 min
	unit:vector	Vector database tests	1-2 min
	unit:smoke	Quick validation subset	< 2 min
	unit:stable	CI-safe gate tests	< 5 min
Integration	integration:smoke	Basic integration workflows	2-5 min
	integration:services	Service layer integration	5-10 min
	integration:daemon	Daemon lifecycle tests	5-10 min
	integration:cli	CLI command tests	3-5 min
Bench	bench:perf	Performance benchmarks	5-15 min
	bench:stress	Load/stress tests	10-30 min
	bench:isolated	Heavy single-process tests	5-10 min
Acceptance	acceptance	End-to-end validation	10-20 min

Common Test Workflows

# Developer iteration (fastest feedback)
meson test -C builddir --suite unit:smoke

# Pre-commit validation (stable subset)
meson test -C builddir --suite unit:stable

# Full unit test suite (sharded)
meson test -C builddir --suite unit

# Integration smoke (quick integration check)
meson test -C builddir --suite integration:smoke

# Run ALL benchmarks (unified suite)
meson test -C builddir --suite bench

# Run only isolated benchmarks
meson test -C builddir --suite bench:isolated

# Run only performance benchmarks (exclude isolated)
meson test -C builddir --suite bench:perf

# Specific test by name
meson test -C builddir repair_scheduling_adapter

# Verbose output
meson test -C builddir --suite unit:smoke -v

# Parallel execution with custom jobs
meson test -C builddir --suite unit -j 8

Test Naming Conventions

Tests follow the pattern: <component>_<aspect>_<variant>

Examples:

• content_store_basic - Basic content store operations
• repair_scheduling_adapter - Repair scheduling adapter coverage
• metadata_repository_cache - Metadata repository caching
• daemon_warm_latency - Daemon warm start latency benchmark

Note: Test organization and suite naming are being standardized; treat suite labels as best-effort until this work completes.

6. Key Build Options

Meson Option	Purpose
build-cli	Build command-line interface
build-mcp-server	Build MCP server
enable-tests / enable-benchmarks	Test/benchmark builds
enable-onnx	Enable ONNX vector/GenAI paths
plugin-symbols	Tree-sitter symbol extraction plugin
enable-sanitizers	Debug ASan/UBSan toggle

Configure options:

meson setup builddir -Denable-onnx=true -Dbuild-cli=true

7. Dependency Management Notes

Edit conanfile.py then re-run conan install for each configuration. Prefer pinned versions. For sandboxed or distro builds, skip Conan entirely and use --system-deps with explicit pkg-config/CMake prefixes.

8. Common Scenarios

Scenario	Commands
Fresh debug loop	Conan install (Debug) → meson setup → ninja → meson test
System-only (no Conan)	Provide all libs, --wrap-mode=nofallback, explicit pkg-config/CMake prefixes
Release packaging	Conan install (Release) → meson setup → ninja → package

9. CI Flow

conan profile detect --force
conan install . -of build/release -s build_type=Release --build=missing
meson setup build/release --native-file build/release/conan_meson_native.ini --buildtype=debugoptimized -Db_ndebug=true
ninja -C build/release
meson test -C build/release

Cache ~/.conan + optionally build/.

10. Troubleshooting Quick Table

Symptom	Fix
Missing native file	Re-run Conan install (regenerates native file)
ONNX disabled	Check enable-onnx option + provide system ORT path if needed
Link errors after dep change	Delete builddir and reconfigure from scratch
Tests undiscovered	Use meson test -C builddir --list; ensure enable-tests=true
Slow incremental builds	Enable ccache or check Unity build settings
Clang: "cannot find -lstdc++"	Install libstdc++ or use YAMS_COMPILER=gcc
Windows: Boost build failures	Install v143 toolset via VS Installer, clean Boost cache
Windows: Missing ONNX Runtime recipe	Export the local onnxruntime recipe (see Developer Setup)

11. Conventions

Out-of-source only. Keep portable optimized builds on debugoptimized + b_ndebug=true; reserve more aggressive tuning for explicit local benchmarking. Avoid committing generated build artifacts.

12. Test Organization (Active Improvement)

Current State: Test names and suites have some inconsistencies (e.g., unit_isolated_* tests in bench suite, milestone numbers in names).

In Progress: test organization standardization is standardizing test organization with:

• Semantic naming: <component>_<aspect>_<variant>
• Hierarchical suites: <category>[+<tag>]
• Clear categorization: unit/integration/bench/acceptance
• Removal of temporal coupling (no milestone numbers in test names)

See the repository issues/PRs for the current plan and status.

Current Test Listing

# View all tests with current names
meson test -C builddir --list

# Example output shows current organization:
# yams:unit / unit_shard0
# yams:unit / unit_shard1
# yams:unit+smoke / unit_smoke
# yams:integration+smoke / integration_smoke

Migration in Progress: Test names will be gradually updated to follow the new conventions while maintaining backward compatibility.

---

Revision notes:

• Migrated from CMake to Meson as primary build system
• Added comprehensive test organization documentation
• Linked to ongoing test standardization
• Simplified ONNX configuration