yams/meson.build
Trevon a3d065dd8d
feat(search): make topology-assisted retrieval the default (#57)
* fix(release): repair package repo publish

Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.

* fix(daemon): stop shutdown log storm

Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.

* refactor(cli): delegate serve to MCP binary

Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.

* refactor(cli): move daemon path helpers

Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.

* refactor(daemon): split embedded client

* perf(search): add lean-guided benchmark gates

Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.

Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.

* refactor(config): move daemon policies

* refactor(daemon): extract plugin runtime

* refactor(daemon): untangle service coordinators

Untangle CLI/MCP from daemon service coordinators and the daemon static lib.

- MetadataRepository uses a daemon-neutral tree-diff callback. The
  GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
  owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
  static lib. Drop ResourceGovernor coupling inside the writer loop; batch
  sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
  its own enqueuePostIngest; app services no longer call ServiceManager
  directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
  concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
  lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
  app services link cleanly without daemon.

Validated:
  meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
  yams-cli --version, yams-mcp-server init, yams serve --help smoke tests

Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.

* fix(cli): keep stdin add local fallback

No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.

Validated:
  meson compile -C build/debug -j4 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
  meson compile -C build/release -j4 yams-cli yams-mcp-server
  meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output

* feat(bench): add xplan multi-arm experiment harness

Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.

* feat(formal): expand topology/search Lean contracts

Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.

* feat(search,topology): construction purity + graph expansion path

Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.

* feat(topology): add KMeans engine and SGC identity contracts

Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.

* feat(daemon): topology readiness gates and selectable reranker

Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.

* chore(search): drop meta_path tuner knobs

Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].

* feat(bench): add topology_purity_validate plan

Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.

* chore: format add_command and config include order

Mechanical clang-format / include reorder only; no behavior change.

* feat(bench): decision-grade xplan loop for search engine

Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.

* feat(graph): denser product semantic_neighbor topK

Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.

* feat(search): extract topology assist and cross-rerank seams

Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.

* feat(bench): product-path search engine ablation plans

Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.

* fix(ci): repair pre-push fallbacks

* feat(bench): lock-free mid-run builds + multi-corpus gates

xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.

* fix(daemon): keep SearchEngine alive via shared_ptr cache

getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.

Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.

* feat(search): route vectors through cached topology

Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.

Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.

* feat(bench): consolidate search decision plans

Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.

Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.

* feat(search): refine topology vector routing

* fix(search): propagate daemon readiness

* feat(bench): measure generalized memory health

* fix(cli): honor in-process fallbacks

* ci: harden smolvm debug builds

* perf(search): route topology through filtered ANN

* feat(bench): measure read-write pressure

* perf(metadata): index symbol lookups on read pool

* fix(daemon): isolate startup reads and maintenance

* feat(bench): replay isolated generalized corpora

* feat(search): add evidence routing pipeline

Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.

* refactor(search): unify topology evidence path

Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.

* refactor(bench): prune superseded search lanes

Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.

* docs(search): retire stale experiment notes

Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.

* refactor(search): prune dead search surfaces

Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.

* fix(cli): clear build warnings

* fix(xplan): reject stale run identity

* refactor(search)!: remove legacy query stack

Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.

BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.

* refactor(search): type value stage traces

Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.

* refactor(search): isolate benchmark env overlays

Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.

* refactor(search): unify topology routing options

Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.

* fix(search): unify tuner config ownership

Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.

* fix(search): snapshot tuner state per query

Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.

* fix(search): enforce topology routing contract

Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.

* fix(daemon): let SQLite reclaim its file

Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.

* fix(metadata): tolerate missing symbol FTS

Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.

* build(fuzz): drop removed query parser target

The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.

* fix(bench): include writes in drain oracle

A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.

* fix(daemon): vacuum padded WAL databases safely
2026-07-13 20:26:43 -04:00

935 lines
34 KiB
Meson

project('yams', ['cpp', 'c'],
version: '0.18.1', # x-release-please-version
default_options: [
'buildtype=debugoptimized',
'b_ndebug=if-release',
'warning_level=3',
'cpp_std=c++20',
'c_std=c11',
],
)
fs = import('fs')
cpp = meson.get_compiler('cpp')
wasi_enabled = get_option('wasi')
if wasi_enabled
add_project_arguments('-DYAMS_WASI=1', language: 'cpp')
endif
_yams_include_dir = join_paths(meson.current_source_dir(), 'include')
if cpp.get_id() == 'msvc' or cpp.get_id() == 'clang-cl'
add_project_arguments('/I' + _yams_include_dir, language: 'cpp')
else
add_project_arguments('-I' + _yams_include_dir, language: 'cpp')
endif
if host_machine.system() == 'darwin'
add_global_link_arguments('-Wl,-no_warn_duplicate_libraries', language: 'cpp')
endif
# std::stacktrace is optional diagnostic sugar for assertion failures. Some
# libstdc++ releases expose the header/feature macro but keep the implementation
# in an extra experimental library; that compiles but fails final executable
# links. Probe compile+link, not just header availability.
_stacktrace_probe = '''
#include <stacktrace>
int main() {
auto trace = std::stacktrace::current();
return static_cast<int>(trace.size());
}
'''
if cpp.links(_stacktrace_probe, name: 'std::stacktrace compile+link')
add_project_arguments('-DYAMS_CORE_HAS_STACKTRACE=1', language: 'cpp')
message('std::stacktrace diagnostics enabled')
else
add_project_arguments('-DYAMS_CORE_HAS_STACKTRACE=0', language: 'cpp')
message('std::stacktrace diagnostics disabled: stacktrace did not link')
endif
if host_machine.system() == 'windows'
add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'cpp')
add_project_arguments('-D_CRT_SECURE_NO_WARNINGS', language: 'cpp')
add_project_arguments('-D_SILENCE_CXX20_OLD_SHARED_PTR_ATOMIC_SUPPORT_DEPRECATION_WARNING', language: 'cpp')
add_project_arguments('-DNOMINMAX', language: 'cpp')
add_project_arguments('-DWIN32_LEAN_AND_MEAN', language: 'cpp')
# Large source files like mcp_server.cpp exceed MSVC's default section limit
add_project_arguments('/bigobj', language: 'cpp')
endif
# --- libc++ hardening mode ---
# Apply hardening defines when using libc++ (Darwin always uses libc++, Linux when explicitly set)
libcxx_hardening = get_option('libcxx-hardening')
if libcxx_hardening != 'none'
# Check if we're actually using libc++
using_libcxx = false
if host_machine.system() == 'darwin'
# macOS always uses libc++
using_libcxx = true
elif cpp.get_id() == 'clang' or cpp.get_id() == 'clang-cl'
# On non-Darwin, check if -stdlib=libc++ is in use
# This is a heuristic; proper detection would check link flags
has_libcxx = cpp.has_header('__config', prefix: '#include <ciso646>')
if has_libcxx
using_libcxx = true
endif
endif
if using_libcxx
hardening_mode_map = {
'fast': '_LIBCPP_HARDENING_MODE_FAST',
'extensive': '_LIBCPP_HARDENING_MODE_EXTENSIVE',
'debug': '_LIBCPP_HARDENING_MODE_DEBUG',
}
hardening_define = hardening_mode_map.get(libcxx_hardening)
if hardening_define != ''
add_project_arguments('-D' + hardening_define, language: 'cpp')
message('Enabling libc++ hardening mode: ' + libcxx_hardening + ' (' + hardening_define + ')')
endif
else
warning('libc++ hardening requested but not using libc++ (detected: ' + cpp.get_id() + '). Hardening mode ignored.')
endif
endif
# --- ThreadSanitizer for race detection ---
enable_tsan = get_option('enable-tsan')
enable_asan = get_option('enable-asan')
enable_ubsan = get_option('enable-ubsan')
enable_dcheck = get_option('enable-dcheck')
if enable_tsan and enable_asan
error('TSAN and ASAN cannot be enabled in the same build. Configure separate build directories.')
endif
if (enable_tsan and enable_ubsan) or (enable_asan and enable_ubsan)
error('UBSan cannot be combined with TSan or ASan. Configure a separate build directory.')
endif
if enable_tsan
buildtype = get_option('buildtype')
# TSan is most useful in debug builds; warn if enabled in release
if buildtype == 'release' or buildtype == 'plain'
warning('ThreadSanitizer enabled in non-debug build. This will significantly impact performance.')
endif
# Check if compiler supports TSan
if cpp.has_argument('-fsanitize=thread')
# Use additional flags for better stack traces (sanitizer is set via b_sanitize option)
tsan_extra_flags = ['-fno-omit-frame-pointer']
add_project_arguments(tsan_extra_flags, language: 'cpp')
add_project_arguments(tsan_extra_flags, language: 'c')
message('ThreadSanitizer enabled - expect significant performance impact')
message('Note: TSan requires all code (including dependencies) to be compiled with -fsanitize=thread')
else
error('ThreadSanitizer requested but compiler does not support -fsanitize=thread')
endif
endif
if enable_asan
buildtype = get_option('buildtype')
# ASan is most useful in debug builds; warn if enabled in release
if buildtype == 'release' or buildtype == 'plain'
warning('AddressSanitizer enabled in non-debug build. This will significantly impact performance.')
endif
# Check if compiler supports ASan
if cpp.has_argument('-fsanitize=address')
# Use additional flags for better stack traces (sanitizer is set via b_sanitize option)
asan_extra_flags = ['-fno-omit-frame-pointer']
add_project_arguments(asan_extra_flags, language: 'cpp')
add_project_arguments(asan_extra_flags, language: 'c')
message('AddressSanitizer enabled - memory errors will fail fast')
else
error('AddressSanitizer requested but compiler does not support -fsanitize=address')
endif
endif
if enable_ubsan
buildtype = get_option('buildtype')
# UBSan is most useful in debug builds; warn if enabled in release
if buildtype == 'release' or buildtype == 'plain'
warning('UndefinedBehaviorSanitizer enabled in non-debug build. This will significantly impact performance.')
endif
# Check if compiler supports UBSan
if cpp.has_argument('-fsanitize=undefined')
# -fno-sanitize-recover=all: abort on first UB hit (turns UBSan into a hard gate, not just logging)
ubsan_extra_flags = ['-fno-omit-frame-pointer', '-fno-sanitize-recover=all']
add_project_arguments(ubsan_extra_flags, language: 'cpp')
add_project_arguments(ubsan_extra_flags, language: 'c')
message('UndefinedBehaviorSanitizer enabled - undefined behavior will abort')
else
error('UBSan requested but compiler does not support -fsanitize=undefined')
endif
endif
if enable_dcheck
add_project_arguments('-DYAMS_ENABLE_DCHECK=1', language: 'cpp')
message('YAMS_DCHECK enabled in this build (overrides NDEBUG for debug-only checks)')
endif
split_debug = get_option('split-debug')
if split_debug
if cpp.has_argument('-gsplit-dwarf')
add_project_arguments('-gsplit-dwarf', language: 'cpp')
add_project_arguments('-gsplit-dwarf', language: 'c')
message('split-dwarf enabled: debug info emitted to .dwo files (reduced .o size)')
else
warning('split-debug requested but compiler does not support -gsplit-dwarf')
endif
endif
# --- General Hardening Flags ---
hardening_cpp_args = []
hardening_c_args = []
hardening_link_args = []
if cpp.get_id() == 'msvc' or cpp.get_id() == 'clang-cl'
# MSVC security checks: /GS (Stack Buffer Security Check) and /sdl (Additional Security Checks)
hardening_cpp_args += ['/GS', '/sdl']
hardening_c_args += ['/GS', '/sdl']
# Control Flow Guard
if cpp.has_argument('/guard:cf')
hardening_cpp_args += '/guard:cf'
hardening_c_args += '/guard:cf'
hardening_link_args += '/guard:cf'
endif
else
# GCC/Clang security checks
# Bounds checking for libstdc++
hardening_cpp_args += '-D_GLIBCXX_ASSERTIONS'
# Stack smashing protection
if cpp.has_argument('-fstack-protector-strong')
hardening_cpp_args += '-fstack-protector-strong'
hardening_c_args += '-fstack-protector-strong'
endif
# Fortify source (only for optimized builds)
if get_option('optimization') != '0'
if cpp.has_argument('-D_FORTIFY_SOURCE=2')
hardening_cpp_args += '-D_FORTIFY_SOURCE=2'
hardening_c_args += '-D_FORTIFY_SOURCE=2'
endif
endif
endif
if host_machine.system() == 'linux'
# Read-only relocations and immediate binding
if cpp.has_link_argument('-Wl,-z,relro')
hardening_link_args += '-Wl,-z,relro'
endif
if cpp.has_link_argument('-Wl,-z,now')
hardening_link_args += '-Wl,-z,now'
endif
endif
add_project_arguments(hardening_cpp_args, language: 'cpp')
add_project_arguments(hardening_c_args, language: 'c')
add_project_link_arguments(hardening_link_args, language: ['cpp', 'c'])
if host_machine.system() == 'darwin' and cpp.get_id() == 'clang'
# Apple clang + external fmt/spdlog can fail when fmt enables
# consteval format-string checking inside spdlog headers.
add_project_arguments('-DFMT_CONSTEVAL=', language: 'cpp')
add_project_arguments('-Wno-macro-redefined', language: 'cpp')
endif
# --- C++20 Modules Support (Experimental) ---
# Detect and configure C++20 module compilation when enabled
yams_modules_enabled = get_option('enable-modules')
yams_modules_supported = false
yams_module_args = []
yams_module_flags = []
yams_use_modules_def = []
yams_module_cpp_args = []
if yams_modules_enabled
# Include module detection and configuration
subdir('meson')
endif
# --- Install prefix for plugin auto-trust ---
# Pass the install prefix to C++ code so the daemon knows where system plugins are installed
install_prefix = get_option('prefix')
# Escape backslashes for Windows paths to avoid compiler warnings about unrecognized escape sequences
install_prefix_escaped = install_prefix.replace('\\', '\\\\')
add_project_arguments('-DYAMS_INSTALL_PREFIX="' + install_prefix_escaped + '"', language: 'cpp')
message('YAMS install prefix: ' + install_prefix)
# --- Shared Conan/CMake discovery paths ---
conan_candidate_dirs = []
conan_gen_opt = get_option('conan-generators-dir')
if conan_gen_opt != ''
conan_candidate_dirs += [join_paths(meson.project_source_root(), conan_gen_opt)]
endif
base_build = meson.project_build_root()
default_candidates = [
base_build,
join_paths(base_build, 'conan'),
join_paths(base_build, 'conan', 'generators'),
join_paths(base_build, 'generators'),
join_paths(base_build, 'build-release', 'conan', 'generators'),
join_paths(base_build, 'build-debug', 'conan', 'generators'),
]
foreach variant : ['build-debug', 'build-release', 'debug', 'release']
variant_path = join_paths(base_build, variant)
if fs.exists(variant_path)
default_candidates += [variant_path]
default_candidates += [join_paths(variant_path, 'conan')]
default_candidates += [join_paths(variant_path, 'conan', 'generators')]
default_candidates += [join_paths(variant_path, 'generators')]
endif
endforeach
foreach cand : default_candidates
if fs.exists(cand)
conan_candidate_dirs += [cand]
endif
endforeach
cli11_cmake_paths = []
foreach base : conan_candidate_dirs
if fs.exists(base)
cli11_cmake_paths += base
foreach sub : ['CLI11', 'cli11']
cand = join_paths(base, sub)
if fs.exists(cand)
cli11_cmake_paths += cand
endif
endforeach
endif
endforeach
cli11_opt = get_option('cli11-cmake-path')
if cli11_opt != ''
cli11_cmake_paths += cli11_opt
endif
onnx_cmake_paths = []
foreach base : conan_candidate_dirs
if fs.exists(base)
onnx_cmake_paths += base
foreach sub : ['onnxruntime', 'onnxruntime/Release', 'onnxruntime/Debug']
cand = join_paths(base, sub)
if fs.exists(cand)
onnx_cmake_paths += cand
endif
endforeach
endif
endforeach
onnx_opt_path = get_option('onnxruntime-cmake-path')
if onnx_opt_path != ''
onnx_cmake_paths += onnx_opt_path
endif
# --- Shared dependency resolution ---
cli11_resolved_dep = dependency('cli11', required: false, method: 'pkg-config')
if not cli11_resolved_dep.found()
cli11_resolved_dep = dependency('CLI11', required: false, method: 'pkg-config')
endif
if not cli11_resolved_dep.found() and cli11_cmake_paths.length() > 0
cli11_resolved_dep = dependency('cli11', method: 'cmake', cmake_module_path: cli11_cmake_paths, required: false)
if not cli11_resolved_dep.found()
cli11_resolved_dep = dependency('CLI11', method: 'cmake', cmake_module_path: cli11_cmake_paths, required: false)
endif
endif
if not cli11_resolved_dep.found()
cli11_resolved_dep = dependency('CLI11', method: 'cmake', required: false)
endif
onnxruntime_resolved_dep = dependency('onnxruntime', method: 'pkg-config', required: false)
if not onnxruntime_resolved_dep.found()
onnxruntime_resolved_dep = dependency('onnxruntime', method: 'cmake', cmake_module_path: onnx_cmake_paths, required: false)
endif
if not onnxruntime_resolved_dep.found()
onnxruntime_resolved_dep = dependency('onnxruntime', required: false)
endif
if not onnxruntime_resolved_dep.found()
onnxruntime_resolved_dep = dependency('libonnxruntime', required: false)
endif
# Normalize ONNX Runtime linkage to the current library location instead of any stale
# Homebrew copies that may remain under /opt/homebrew/lib after upgrades.
if onnxruntime_resolved_dep.found()
onnxruntime_link_dirs = []
onnxruntime_pkg_libdir = onnxruntime_resolved_dep.get_variable(pkgconfig: 'libdir', default_value: '')
if onnxruntime_pkg_libdir != ''
onnxruntime_link_dirs += [onnxruntime_pkg_libdir]
endif
onnxruntime_cmake_prefix = onnxruntime_resolved_dep.get_variable(cmake: 'PACKAGE_PREFIX_DIR', default_value: '')
if onnxruntime_cmake_prefix != ''
onnxruntime_link_dirs += [join_paths(onnxruntime_cmake_prefix, 'lib')]
endif
if host_machine.system() == 'darwin'
foreach brew_onnx_prefix : ['/opt/homebrew/opt/onnxruntime/lib', '/usr/local/opt/onnxruntime/lib']
if fs.exists(brew_onnx_prefix)
onnxruntime_link_dirs = [brew_onnx_prefix] + onnxruntime_link_dirs
endif
endforeach
endif
onnxruntime_compile_dep = onnxruntime_resolved_dep.partial_dependency(compile_args: true, includes: true)
onnxruntime_cpp = meson.get_compiler('cpp')
onnxruntime_link_dep = onnxruntime_cpp.find_library('onnxruntime', dirs: onnxruntime_link_dirs, required: false)
if not onnxruntime_link_dep.found()
onnxruntime_link_dep = onnxruntime_cpp.find_library('onnxruntime', required: false)
endif
if onnxruntime_link_dep.found()
onnxruntime_resolved_dep = declare_dependency(
dependencies: [onnxruntime_compile_dep, onnxruntime_link_dep],
)
message('ONNX Runtime linkage normalized via current library search path')
endif
endif
tbb_resolved_dep = dependency('tbb', required: false, method: 'pkg-config')
if not tbb_resolved_dep.found()
tbb_resolved_dep = dependency('onetbb', required: false, method: 'pkg-config')
endif
if not tbb_resolved_dep.found()
tbb_resolved_dep = dependency('tbb', required: false)
endif
if not tbb_resolved_dep.found()
tbb_resolved_dep = dependency('onetbb', required: false)
endif
if host_machine.system() == 'darwin' and tbb_resolved_dep.found()
tbb_normalized = false
tbb_cpp = meson.get_compiler('cpp')
foreach brew_tbb_prefix : ['/opt/homebrew/opt/tbb', '/usr/local/opt/tbb']
if not tbb_normalized
brew_tbb_incdir = brew_tbb_prefix / 'include'
brew_tbb_libdir = brew_tbb_prefix / 'lib'
if fs.exists(brew_tbb_incdir) and fs.exists(brew_tbb_libdir)
tbb_link_dep = tbb_cpp.find_library('tbb', dirs: [brew_tbb_libdir], required: false)
if tbb_link_dep.found()
tbb_resolved_dep = declare_dependency(
include_directories: include_directories(brew_tbb_incdir),
dependencies: [tbb_link_dep],
)
tbb_normalized = true
message('TBB linkage normalized via Homebrew opt prefix')
endif
endif
endif
endforeach
endif
# --- Windows DLL deployment ---
# On Windows, DLLs must be in the same directory as the executable or in PATH.
# We extract the DLL paths from the dependency's pkg-config `bindir` variable,
# then use fs.copyfile() to deploy them to the build directory.
#
# This is the Meson-idiomatic approach that doesn't rely on Conan's runtime_deploy.
windows_runtime_dlls = [] # List of [dll_name, dll_full_path] tuples
if host_machine.system() == 'windows'
# Get TBB dependency and extract bindir
tbb_dep = tbb_resolved_dep
if tbb_dep.found()
tbb_prefix = tbb_dep.get_variable(pkgconfig: 'prefix', default_value: '')
tbb_bindir = tbb_dep.get_variable(pkgconfig: 'bindir', default_value: '')
if tbb_bindir != ''
# Resolve ${prefix} placeholder if present
tbb_bindir = tbb_bindir.replace('${prefix}', tbb_prefix)
foreach dll_name : ['tbb12.dll', 'tbbbind_2_5.dll', 'tbbmalloc.dll', 'tbbmalloc_proxy.dll']
dll_path = tbb_bindir / dll_name
if fs.exists(dll_path)
windows_runtime_dlls += [[dll_name, dll_path]]
endif
endforeach
endif
endif
# Get ONNX Runtime dependency and extract bindir (prefer Conan CMake config)
onnx_dep_check = onnxruntime_resolved_dep
onnx_bindir_candidates = []
if onnx_dep_check.found()
onnx_prefix = onnx_dep_check.get_variable(pkgconfig: 'prefix', default_value: '')
onnx_bindir = onnx_dep_check.get_variable(pkgconfig: 'bindir', default_value: '')
if onnx_bindir != ''
onnx_bindir = onnx_bindir.replace('${prefix}', onnx_prefix)
onnx_bindir_candidates += [onnx_bindir]
endif
# CMake configs from Conan often expose PACKAGE_PREFIX_DIR; fall back to <prefix>/bin
cmake_prefix = onnx_dep_check.get_variable(cmake: 'PACKAGE_PREFIX_DIR', default_value: '')
if cmake_prefix != ''
onnx_bindir_candidates += [join_paths(cmake_prefix, 'bin')]
elif onnx_prefix != ''
onnx_bindir_candidates += [join_paths(onnx_prefix, 'bin')]
endif
endif
# As a last resort, probe the Conan cache directly (~/.conan2/p/**/p/bin) via a tiny Python helper
py_mod = import('python')
if host_machine.system() == 'windows'
python3 = py_mod.find_installation('python', required: false)
if not python3.found()
python3 = py_mod.find_installation('python3', required: false)
endif
else
python3 = py_mod.find_installation('python3', required: false)
if not python3.found()
python3 = py_mod.find_installation('python', required: false)
endif
endif
if python3.found()
conan_probe = run_command(python3, ['-c', '''import pathlib, os
root = pathlib.Path(os.path.expanduser("~/.conan2/p"))
dirs = []
if root.exists():
for pattern in ("*/p/bin", "b/*/p/bin"):
for bin_dir in root.glob(pattern):
dll = bin_dir / "onnxruntime.dll"
if dll.exists():
dirs.append(str(bin_dir))
print(";".join(dirs))'''], check: false)
if conan_probe.returncode() == 0
conan_dirs = conan_probe.stdout().strip().split(';')
foreach d : conan_dirs
if d != ''
onnx_bindir_candidates += [d]
endif
endforeach
endif
else
message('Python not found (python3/python); skipping Conan cache probe for ONNX Runtime DLLs')
endif
foreach bindir : onnx_bindir_candidates
foreach dll_name : ['onnxruntime.dll', 'onnxruntime_providers_shared.dll']
dll_path = bindir / dll_name
if fs.exists(dll_path)
windows_runtime_dlls += [[dll_name, dll_path]]
endif
endforeach
endforeach
# Deduplicate DLL entries by name to avoid duplicate targets
dedup_map = {}
deduped = []
foreach dll_info : windows_runtime_dlls
name = dll_info[0]
if not dedup_map.has_key(name)
dedup_map += {name: true}
deduped += [dll_info]
endif
endforeach
windows_runtime_dlls = deduped
# Log discovered DLLs
foreach dll_info : windows_runtime_dlls
message('Found runtime DLL: ' + dll_info[0] + ' at ' + dll_info[1])
endforeach
endif
if not wasi_enabled
# Database backend selection (libsql or sqlite)
db_backend = get_option('database-backend')
require_fts5 = get_option('require-fts5')
# Try to find the selected database backend
using_libsql = false
if db_backend == 'libsql'
# Try to find libsql (provides sqlite3-compatible API)
# 1. First try system-installed libsql via pkg-config
libsql_dep = dependency('libsql', required: false)
if not libsql_dep.found()
libsql_dep = dependency('libsql-sqlite3', required: false)
endif
# 2. If not found, try building from subproject (requires Rust toolchain)
if not libsql_dep.found()
cargo_prog = find_program('cargo', required: false)
if cargo_prog.found()
message('libSQL not found on system, building from subproject (requires Rust)...')
libsql_subproj = subproject('libsql', required: false)
if libsql_subproj.found()
libsql_dep = libsql_subproj.get_variable('libsql_dep')
endif
endif
endif
if libsql_dep.found()
sqlite_dep = libsql_dep
using_libsql = true
message('Using libSQL database backend (MVCC concurrent writes enabled)')
else
warning('libSQL not found and subproject build failed. Falling back to standard SQLite3.')
warning('To use libSQL: install Rust toolchain (rustup) or system libsql package.')
sqlite_dep = dependency('sqlite3', required: true)
endif
else
sqlite_dep = dependency('sqlite3', required: true)
message('Using standard SQLite3 database backend')
endif
# Add compile definition for backend detection
if using_libsql
add_project_arguments('-DYAMS_LIBSQL_BACKEND=1', language: 'cpp')
else
add_project_arguments('-DYAMS_LIBSQL_BACKEND=0', language: 'cpp')
endif
# Skip runtime test when cross-compiling or using subproject sqlite (internal dep can't be used with cpp.run in newer meson)
if meson.is_cross_build()
warning('Cross-compiling: skipping SQLite FTS5 runtime check. Ensure FTS5 is enabled in your target SQLite build.')
elif sqlite_dep.type_name() == 'internal'
message('Using subproject sqlite3: skipping FTS5 runtime check. Subproject sqlite3 has FTS5 enabled by default.')
elif using_libsql
message('Using libSQL: skipping FTS5 runtime check. libSQL includes FTS5 support.')
else
fts5_prog = cpp.run(files('meson/fts5_check.cpp'), dependencies: sqlite_dep, name: 'sqlite3 FTS5 check')
if fts5_prog.returncode() != 0
msg = 'SQLite3 built without FTS5 support. Enable FTS5 in your package manager or Conan (e.g., -o "sqlite3/*:fts5=True").'
if require_fts5
error(msg)
else
warning(msg)
endif
endif
endif
endif
# Boost is a core requirement (e.g., Boost.Asio for daemon communication).
# Resolution order:
# 1. Per-library pkg-config probes against Conan's emitted boost_*.pc.
# Required path on Windows: meson 1.4's built-in Boost finder rejects
# Conan's MSVC layout (libs are named libboost_<comp>.lib, not the
# auto-link form boost_<comp>-vc14x-mt-x64-1_85.lib that meson scans
# for). pkg-config bypasses meson's Boost special-case entirely.
# 2. System Boost auto-detect — Linux/macOS fallback when no .pc files
# are on PKG_CONFIG_PATH (apt/brew Boost ships no pkg-config files).
# NOTE: Conan PkgConfigDeps emits per-component .pc files using HYPHEN
# separators (boost-system.pc, boost-thread.pc), not underscores. pkg-config
# resolves dependency names against the .pc filename, so we must match.
boost_system_dep = dependency('boost-system', method: 'pkg-config',
required: false)
boost_thread_dep = dependency('boost-thread', method: 'pkg-config',
required: false)
if boost_system_dep.found() and boost_thread_dep.found()
boost_dep = declare_dependency(
dependencies: [boost_system_dep, boost_thread_dep])
else
boost_dep = dependency('boost', modules: ['system', 'thread'],
required: true)
endif
meson.override_dependency('boost', boost_dep)
# --- RE2 regex engine (optional, for high-performance grep) ---
re2_opt = get_option('enable-re2')
re2_dep = dependency('re2', required: re2_opt, include_type: 'system')
if re2_dep.found()
add_project_arguments('-DYAMS_HAS_RE2=1', language: 'cpp')
message('RE2 regex engine enabled (grep will use RE2 with std::regex fallback)')
else
add_project_arguments('-DYAMS_HAS_RE2=0', language: 'cpp')
message('RE2 not found; grep will use std::regex only')
endif
# --- Version header generation ---
_override_version = get_option('yams-version')
# Detect version from git tags if not overridden
_git_version = ''
_git_cmd = find_program('git', required: false)
if _git_cmd.found() and _override_version == ''
# Try to get the most recent semver tag (v*)
_r = run_command(_git_cmd, ['describe', '--tags', '--match', 'v*', '--abbrev=0'], check: false)
if _r.returncode() == 0
_tag = _r.stdout().strip()
# Strip leading 'v' if present
if _tag.startswith('v')
_git_version = _tag.substring(1)
else
_git_version = _tag
endif
endif
endif
# Priority: command-line override > git tag > meson.project_version()
if _override_version != ''
_effective_version = _override_version
elif _git_version != ''
_effective_version = _git_version
else
_effective_version = meson.project_version()
endif
if host_machine.system() == 'windows'
python_exe = find_program('python', 'python3')
else
python_exe = find_program('python3', 'python')
endif
_git_cmd = find_program('git', required: false)
_git_exe = _git_cmd.found() ? _git_cmd.full_path() : 'git'
version_header_template = files('include/yams/version_generated.h.in')[0]
version_header_script = files('scripts/generate_version_header.py')[0]
# Generate version header at build time so installed binaries pick up the
# current commit even when source changes occur after the last Meson reconfigure.
version_header = custom_target(
'version_generated_header',
input: [version_header_template, version_header_script],
output: 'version_generated.h',
command: [
python_exe,
version_header_script,
'@INPUT0@',
'@OUTPUT@',
meson.project_source_root(),
meson.project_version(),
_override_version,
_git_exe,
],
build_always_stale: true,
build_by_default: true,
install: true,
install_dir: get_option('includedir') / 'yams',
)
# Make builddir available as include path for generated headers
# CLI will include as "version_generated.h" directly
generated_inc = include_directories('.')
version_generated_dep = declare_dependency(
include_directories: generated_inc,
sources: version_header,
)
yams_build_cli = get_option('build-cli') and not wasi_enabled
yams_build_mcp = get_option('build-mcp-server')
# Install public headers under the compiler's include prefix without duplicating
# the top-level 'include' directory (was producing prefix/include/include/...).
install_subdir('include', install_dir: get_option('includedir'), strip_directory: true)
# --- tl-expected global resolution ---
# tl-expected is a header-only library. Homebrew installs it without pkgconfig.
# Resolve it once here and override for all subdirs.
_tl_expected_dep = dependency('tl-expected', method: 'cmake', required: false)
if not _tl_expected_dep.found()
_tl_expected_dep = dependency('tl-expected', method: 'pkg-config', required: false)
endif
if not _tl_expected_dep.found()
# Manual fallback for homebrew installation. Probe both brew prefixes
# (Apple Silicon + Intel) guarded by the header actually existing.
foreach _brew_prefix : ['/opt/homebrew', '/usr/local']
if not _tl_expected_dep.found() and fs.exists(_brew_prefix / 'include' / 'tl' / 'expected.hpp')
_tl_expected_inc = include_directories(_brew_prefix / 'include', is_system: true)
_tl_expected_dep = declare_dependency(include_directories: _tl_expected_inc)
endif
endforeach
endif
meson.override_dependency('tl-expected', _tl_expected_dep)
# --- tinyfsm global resolution ---
# tinyfsm is a single-header FSM library vendored under third_party/.
# Added as a project-wide include because daemon FSM headers are included
# transitively by many targets via ServiceManager.h.
_tinyfsm_inc = include_directories('third_party/tinyfsm/include', is_system: true)
add_project_arguments(
'-isystem', meson.project_source_root() / 'third_party' / 'tinyfsm' / 'include',
language: 'cpp',
)
_tinyfsm_dep = declare_dependency(include_directories: _tinyfsm_inc)
meson.override_dependency('tinyfsm', _tinyfsm_dep)
# --- libarchive global resolution ---
# Homebrew on macOS installs libarchive as keg-only; pkg-config won't find it
# without the Conan-generated pkg_config_path. Resolve once and override only
# in the fallback path — overriding an already-resolved dependency is an error.
_libarchive_dep = dependency('libarchive', required: false)
if not _libarchive_dep.found() and host_machine.system() == 'darwin'
foreach _brew_prefix : ['/opt/homebrew', '/usr/local']
if not _libarchive_dep.found()
_brew_libarchive = _brew_prefix / 'opt' / 'libarchive'
if fs.exists(_brew_libarchive / 'include' / 'archive.h')
_libarchive_lib = cpp.find_library('archive',
dirs: [_brew_libarchive / 'lib'],
required: false)
if _libarchive_lib.found()
_libarchive_dep = declare_dependency(
dependencies: _libarchive_lib,
include_directories: include_directories(_brew_libarchive / 'include', is_system: true),
)
meson.override_dependency('libarchive', _libarchive_dep)
endif
endif
endif
endforeach
endif
if wasi_enabled
# WASI profile: compile a minimal subset (no daemon IPC, no plugins, no downloads).
# Keep this intentionally small so cross-compilers don't need system libs like
# OpenSSL/libcurl/sqlite/onnx.
subdir('src/core')
if get_option('build-mcp-server')
subdir('src/mcp')
endif
else
# Core libraries (ordered by dependency)
subdir('src/core')
subdir('src/crypto')
subdir('src/chunking')
subdir('src/compression')
subdir('src/storage')
subdir('src/wal')
subdir('src/manifest')
subdir('src/metadata')
subdir('src/integrity')
subdir('src/config')
subdir('src/extraction')
subdir('src/detection')
subdir('src/content')
subdir('src/downloader')
if get_option('build-plugins')
subdir('src/plugins')
endif
subdir('src/daemon/client')
subdir('src/embedding_simeon')
subdir('src/vector')
subdir('src/topology')
# Third-party libraries (must come before src/search which depends on symspell)
subdir('third_party/symspell')
subdir('src/search')
subdir('src/api')
subdir('src/ingest')
subdir('src/repair')
subdir('src/app/services')
subdir('src/daemon')
if get_option('enable-mobile-bindings')
subdir('src/mobile')
endif
if get_option('build-benchmarks')
subdir('src/benchmarks')
endif
if get_option('build-mcp-server')
subdir('src/mcp')
endif
endif
if get_option('build-plugins') and not wasi_enabled
subdir('plugins')
endif
if get_option('build-cli') and not wasi_enabled
subdir('src/cli')
subdir('tools/yams-cli')
if get_option('build-mcp-server')
subdir('tools/yams-mcp')
endif
endif
# Tests (enabled by default in Debug builds)
_tests_opt = get_option('build-tests')
_buildtype = get_option('buildtype')
_tests_enabled = (not wasi_enabled) and (_tests_opt or (_buildtype == 'debug' and not get_option('build-fuzzers')))
if _tests_enabled
subdir('tests')
endif
# Fuzzing harnesses (AFL++/libFuzzer)
if get_option('build-fuzzers') and not wasi_enabled
subdir('tools/fuzzing')
endif
sql_install_dir = join_paths(get_option('datadir'), 'yams', 'sql')
install_data('sql/reference_schema.sql', install_dir: sql_install_dir)
if not wasi_enabled
yams_pc_config = configuration_data()
yams_pc_config.set('PREFIX', get_option('prefix'))
yams_pc_config.set('LIBDIR', get_option('libdir'))
yams_pc_config.set('INCLUDEDIR', get_option('includedir'))
yams_pc_config.set('VERSION', _effective_version)
configure_file(
input: files('packaging/yams.pc.in'),
output: 'yams.pc',
configuration: yams_pc_config,
install: true,
install_dir: join_paths(get_option('libdir'), 'pkgconfig'),
)
endif
summary({
'project': meson.project_name(),
'version': meson.project_version(),
'cpp_std': get_option('cpp_std'),
'build_cli': yams_build_cli,
'build_mcp_server': yams_build_mcp,
}, section: 'YAMS Meson configuration', bool_yn: true)
if get_option('b_coverage')
gcovr = find_program('gcovr', required: false)
if gcovr.found()
_cpp_compiler = meson.get_compiler('cpp')
_gcov_executable = 'gcov'
if _cpp_compiler.get_id() == 'clang'
_llvm_cov = find_program('llvm-cov', required: false)
if _llvm_cov.found()
_gcov_executable = 'llvm-cov gcov -p'
endif
endif
coverage_report_target = custom_target('coverage-report',
output: 'coverage.html',
command: [
gcovr,
'--root', meson.project_source_root(),
'--gcov-executable', _gcov_executable,
'--object-directory', meson.project_build_root(),
'--merge-mode-functions', 'merge-use-line-min',
'--filter', join_paths(meson.project_source_root(), 'src'),
'--filter', join_paths(meson.project_source_root(), 'include'),
'--exclude', join_paths(meson.project_source_root(), 'src', 'benchmarks'),
'--html', '--html-details',
'--output', '@OUTPUT@',
meson.project_build_root(),
],
build_by_default: false,
build_always_stale: true,
)
coverage_txt_target = custom_target('coverage-summary',
output: 'coverage.txt',
command: [
gcovr,
'--root', meson.project_source_root(),
'--gcov-executable', _gcov_executable,
'--object-directory', meson.project_build_root(),
'--merge-mode-functions', 'merge-use-line-min',
'--filter', join_paths(meson.project_source_root(), 'src'),
'--filter', join_paths(meson.project_source_root(), 'include'),
'--exclude', join_paths(meson.project_source_root(), 'src', 'benchmarks'),
'--print-summary',
'--output', '@OUTPUT@',
meson.project_build_root(),
],
build_by_default: false,
build_always_stale: true,
)
message('Coverage reporting configured. Run tests then: ninja -C builddir coverage.txt && ninja -C builddir coverage.html')
else
warning('gcovr not found. Install gcovr to generate coverage reports.')
endif
endif
# Note: Runtime DLLs are now deployed via Conan's runtime_deploy deployer during conan install
# No custom meson install script needed