* fix(core): add assertions and harden daemon lifecycle * fix(daemon): tighten shutdown lifecycle invariants * fix(test): add PressureLimitedPoller capability-sleep shutdown test * refactor(vector): harden backend contracts Split the vector backend surface into narrower contracts and move entity read paths to Result<> so callers can distinguish backend errors from empty results. Also fixes the TurboQuant lazy-init race, makes VectorDatabase error state thread-safe, wires eager index preparation, and adds regression plus memory-scaling benchmark coverage. * fix(vector): address audit findings plus hardening Fix the five confirmed bugs: coroutine lifetime hazard in embedding_generator (await_with_timeout outliving captured state), VectorSystemManager init retry latch poisoning, incremental HNSW persistence race, vec0 registry key collisions, and simeon PMI output_dim/config() drift. Add invariant hardening: PostIngestQueue stage validation + numChunks assertion, VectorSystemManager success postcondition, VectorDatabase TurboQuant non-null assertions on all quantize/ dequantize paths, and sqlite_vec_backend Simeon PQ range and divisibility preconditions. Also adds regression coverage for init-retry and lock-busy paths and a locked node snapshot helper for safe incremental HNSW persistence. * fix(test): add PressureLimitedPoller wake-timer shutdown tests * chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec Replace unprotected std::getenv calls with locked getenvCopy helpers, add debug-level logging to previously-silent catch blocks in PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap PostIngestQueue timer-detach allocations in try-catch, and fix unsigned-widening / optional-deref / unused-include hygiene. Verified with LSP diagnostics and rebuilds of catch2_vector_submodule and catch2_daemon_component_submodule. * fix(daemon): harden PressureLimitedPoller inFlightCounter null check * fix(daemon): harden PostIngestQueue wait_until deadline checks * fix(daemon): clean up recovery sentinels regardless of salvage count Previously the .recovered-* sentinel and .corrupt-* file cleanup was gated on documentsSalvaged > 0, leaving quarantine artifacts to accumulate on repeated crash/recovery cycles where the corrupt DB had 0 salvageable docs. Move cleanup outside the salvage-count guard so stale quarantine artifacts are always removed after examination. * feat(vec0): restore persisted ANN index on daemon restart After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables and load a previously-persisted HNSW index. When found, the ANN index is set directly on the vec0 table, skipping the O(n log n) rebuild that was previously paid on every daemon restart. Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor of inline restoration in rebuildVec0DimUnlocked(), which has the vec0 table already created and ready for ann_index assignment. * fix: enable DCHECK by default and assert metadata counter consistency * fix(services): assert document deletion ordering content-before-metadata * fix(test): add fork-based assertion validation tests * fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE * fix(ci): enable -Denable-dcheck=true in all test CI lanes * fix(test): address assertion review findings * refactor(daemon): isolate poller config assertion checks * refactor(services): reduce deletion and poller complexity * style(ci): normalize test workflow spacing * refactor(services): extract list query helpers * refactor(services): extract list hydration helpers * refactor(daemon): extract metrics snapshot helpers * refactor(daemon): extract metrics counter snapshot * refactor(daemon): extract metrics resource snapshot * feat(graph): add daemon-backed explore Wire graph explore through daemon IPC, keep CLI rendering stable, and fall back locally when daemon transport or protocol skew fails. Also add graph context service/tests, improve graph follow-up hints, and keep release builds working with the current JSON library. * fix(services): prevent prune delete metadata orphans * fix(cli): honor data-dir in graph explore - bypass daemon explore when the CLI pins a data dir so\n temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n context * fix(daemon): make lifecycle stop completion safe * fix(daemon): gate db-open startup against shutdown * fix(metadata): serialize FTS indexed counter updates * test(daemon): simulate vector lock contention cross-process * test(daemon): isolate post-ingest identity batch * style(daemon): normalize test include order * refactor(graph): split cli graph helpers Keep graph command paths smaller and easier to evolve. Lock graph explore/topology contracts into focused presenter, dispatcher, and support tests so follow-up refactors keep the current IPC and hot-path behavior intact. * fix(config): harden expand_tilde and fix Windows XDG state path doubling - expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range on substr(2) for single-char input - getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment * fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax * fix(static-analysis): strip trailing text from nosemgrep comments * fix(doctor): add logging to daemon_check empty catch blocks * fix(db_repair): add logging to empty catch blocks * fix(chunking): add precondition checks for rabin polynomial and windowSize * fix(safety): add null precondition checks for Fold proved null-deref sites * fix(daemon): widen init-failure dispatch, add null/dbz preconditions InitializationFailedEvent races with shutdown — async init fails after teardown begins. Widen canDispatch to accept Ready/ShuttingDown. Add static_assert on SnapshotRing N>0. Add YAMS_PRECONDITION null guards on hex/stages/data/bytes. * ci(ubsan): use restricted build targets to avoid runner timeout * fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV * fix(config): cast parseTomlU32 to size_t for batch fields * fix(ci): use static helper for Windows timer leak, restrict ubsan tests
896 lines
32 KiB
Meson
896 lines
32 KiB
Meson
project('yams', ['cpp', 'c'],
|
|
version: '0.16.2', # x-release-please-version
|
|
default_options: [
|
|
'buildtype=debugoptimized',
|
|
'b_ndebug=if-release',
|
|
'warning_level=3',
|
|
'cpp_std=c++20',
|
|
'c_std=c11',
|
|
],
|
|
)
|
|
|
|
fs = import('fs')
|
|
|
|
cpp = meson.get_compiler('cpp')
|
|
|
|
wasi_enabled = get_option('wasi')
|
|
if wasi_enabled
|
|
add_project_arguments('-DYAMS_WASI=1', language: 'cpp')
|
|
endif
|
|
|
|
_yams_include_dir = join_paths(meson.current_source_dir(), 'include')
|
|
if cpp.get_id() == 'msvc' or cpp.get_id() == 'clang-cl'
|
|
add_project_arguments('/I' + _yams_include_dir, language: 'cpp')
|
|
else
|
|
add_project_arguments('-I' + _yams_include_dir, language: 'cpp')
|
|
endif
|
|
|
|
if host_machine.system() == 'darwin'
|
|
add_global_link_arguments('-Wl,-no_warn_duplicate_libraries', language: 'cpp')
|
|
endif
|
|
|
|
if host_machine.system() == 'windows'
|
|
add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'cpp')
|
|
add_project_arguments('-D_CRT_SECURE_NO_WARNINGS', language: 'cpp')
|
|
add_project_arguments('-D_SILENCE_CXX20_OLD_SHARED_PTR_ATOMIC_SUPPORT_DEPRECATION_WARNING', language: 'cpp')
|
|
add_project_arguments('-DNOMINMAX', language: 'cpp')
|
|
add_project_arguments('-DWIN32_LEAN_AND_MEAN', language: 'cpp')
|
|
# Large source files like mcp_server.cpp exceed MSVC's default section limit
|
|
add_project_arguments('/bigobj', language: 'cpp')
|
|
endif
|
|
|
|
# --- libc++ hardening mode ---
|
|
# Apply hardening defines when using libc++ (Darwin always uses libc++, Linux when explicitly set)
|
|
libcxx_hardening = get_option('libcxx-hardening')
|
|
|
|
if libcxx_hardening != 'none'
|
|
# Check if we're actually using libc++
|
|
using_libcxx = false
|
|
if host_machine.system() == 'darwin'
|
|
# macOS always uses libc++
|
|
using_libcxx = true
|
|
elif cpp.get_id() == 'clang' or cpp.get_id() == 'clang-cl'
|
|
# On non-Darwin, check if -stdlib=libc++ is in use
|
|
# This is a heuristic; proper detection would check link flags
|
|
has_libcxx = cpp.has_header('__config', prefix: '#include <ciso646>')
|
|
if has_libcxx
|
|
using_libcxx = true
|
|
endif
|
|
endif
|
|
|
|
if using_libcxx
|
|
hardening_mode_map = {
|
|
'fast': '_LIBCPP_HARDENING_MODE_FAST',
|
|
'extensive': '_LIBCPP_HARDENING_MODE_EXTENSIVE',
|
|
'debug': '_LIBCPP_HARDENING_MODE_DEBUG',
|
|
}
|
|
hardening_define = hardening_mode_map.get(libcxx_hardening)
|
|
if hardening_define != ''
|
|
add_project_arguments('-D' + hardening_define, language: 'cpp')
|
|
message('Enabling libc++ hardening mode: ' + libcxx_hardening + ' (' + hardening_define + ')')
|
|
endif
|
|
else
|
|
warning('libc++ hardening requested but not using libc++ (detected: ' + cpp.get_id() + '). Hardening mode ignored.')
|
|
endif
|
|
endif
|
|
|
|
# --- ThreadSanitizer for race detection ---
|
|
enable_tsan = get_option('enable-tsan')
|
|
enable_asan = get_option('enable-asan')
|
|
enable_ubsan = get_option('enable-ubsan')
|
|
enable_dcheck = get_option('enable-dcheck')
|
|
|
|
if enable_tsan and enable_asan
|
|
error('TSAN and ASAN cannot be enabled in the same build. Configure separate build directories.')
|
|
endif
|
|
if (enable_tsan and enable_ubsan) or (enable_asan and enable_ubsan)
|
|
error('UBSan cannot be combined with TSan or ASan. Configure a separate build directory.')
|
|
endif
|
|
|
|
if enable_tsan
|
|
buildtype = get_option('buildtype')
|
|
# TSan is most useful in debug builds; warn if enabled in release
|
|
if buildtype == 'release' or buildtype == 'plain'
|
|
warning('ThreadSanitizer enabled in non-debug build. This will significantly impact performance.')
|
|
endif
|
|
|
|
# Check if compiler supports TSan
|
|
if cpp.has_argument('-fsanitize=thread')
|
|
# Use additional flags for better stack traces (sanitizer is set via b_sanitize option)
|
|
tsan_extra_flags = ['-fno-omit-frame-pointer']
|
|
add_project_arguments(tsan_extra_flags, language: 'cpp')
|
|
add_project_arguments(tsan_extra_flags, language: 'c')
|
|
message('ThreadSanitizer enabled - expect significant performance impact')
|
|
message('Note: TSan requires all code (including dependencies) to be compiled with -fsanitize=thread')
|
|
else
|
|
error('ThreadSanitizer requested but compiler does not support -fsanitize=thread')
|
|
endif
|
|
endif
|
|
|
|
if enable_asan
|
|
buildtype = get_option('buildtype')
|
|
# ASan is most useful in debug builds; warn if enabled in release
|
|
if buildtype == 'release' or buildtype == 'plain'
|
|
warning('AddressSanitizer enabled in non-debug build. This will significantly impact performance.')
|
|
endif
|
|
|
|
# Check if compiler supports ASan
|
|
if cpp.has_argument('-fsanitize=address')
|
|
# Use additional flags for better stack traces (sanitizer is set via b_sanitize option)
|
|
asan_extra_flags = ['-fno-omit-frame-pointer']
|
|
add_project_arguments(asan_extra_flags, language: 'cpp')
|
|
add_project_arguments(asan_extra_flags, language: 'c')
|
|
message('AddressSanitizer enabled - memory errors will fail fast')
|
|
else
|
|
error('AddressSanitizer requested but compiler does not support -fsanitize=address')
|
|
endif
|
|
endif
|
|
|
|
if enable_ubsan
|
|
buildtype = get_option('buildtype')
|
|
# UBSan is most useful in debug builds; warn if enabled in release
|
|
if buildtype == 'release' or buildtype == 'plain'
|
|
warning('UndefinedBehaviorSanitizer enabled in non-debug build. This will significantly impact performance.')
|
|
endif
|
|
|
|
# Check if compiler supports UBSan
|
|
if cpp.has_argument('-fsanitize=undefined')
|
|
# -fno-sanitize-recover=all: abort on first UB hit (turns UBSan into a hard gate, not just logging)
|
|
ubsan_extra_flags = ['-fno-omit-frame-pointer', '-fno-sanitize-recover=all']
|
|
add_project_arguments(ubsan_extra_flags, language: 'cpp')
|
|
add_project_arguments(ubsan_extra_flags, language: 'c')
|
|
message('UndefinedBehaviorSanitizer enabled - undefined behavior will abort')
|
|
else
|
|
error('UBSan requested but compiler does not support -fsanitize=undefined')
|
|
endif
|
|
endif
|
|
|
|
if enable_dcheck
|
|
add_project_arguments('-DYAMS_ENABLE_DCHECK=1', language: 'cpp')
|
|
message('YAMS_DCHECK enabled in this build (overrides NDEBUG for debug-only checks)')
|
|
endif
|
|
|
|
|
|
# --- General Hardening Flags ---
|
|
hardening_cpp_args = []
|
|
hardening_c_args = []
|
|
hardening_link_args = []
|
|
|
|
if cpp.get_id() == 'msvc' or cpp.get_id() == 'clang-cl'
|
|
# MSVC security checks: /GS (Stack Buffer Security Check) and /sdl (Additional Security Checks)
|
|
hardening_cpp_args += ['/GS', '/sdl']
|
|
hardening_c_args += ['/GS', '/sdl']
|
|
# Control Flow Guard
|
|
if cpp.has_argument('/guard:cf')
|
|
hardening_cpp_args += '/guard:cf'
|
|
hardening_c_args += '/guard:cf'
|
|
hardening_link_args += '/guard:cf'
|
|
endif
|
|
else
|
|
# GCC/Clang security checks
|
|
|
|
# Bounds checking for libstdc++
|
|
hardening_cpp_args += '-D_GLIBCXX_ASSERTIONS'
|
|
|
|
# Stack smashing protection
|
|
if cpp.has_argument('-fstack-protector-strong')
|
|
hardening_cpp_args += '-fstack-protector-strong'
|
|
hardening_c_args += '-fstack-protector-strong'
|
|
endif
|
|
|
|
# Fortify source (only for optimized builds)
|
|
if get_option('optimization') != '0'
|
|
if cpp.has_argument('-D_FORTIFY_SOURCE=2')
|
|
hardening_cpp_args += '-D_FORTIFY_SOURCE=2'
|
|
hardening_c_args += '-D_FORTIFY_SOURCE=2'
|
|
endif
|
|
endif
|
|
endif
|
|
|
|
if host_machine.system() == 'linux'
|
|
# Read-only relocations and immediate binding
|
|
if cpp.has_link_argument('-Wl,-z,relro')
|
|
hardening_link_args += '-Wl,-z,relro'
|
|
endif
|
|
if cpp.has_link_argument('-Wl,-z,now')
|
|
hardening_link_args += '-Wl,-z,now'
|
|
endif
|
|
endif
|
|
|
|
add_project_arguments(hardening_cpp_args, language: 'cpp')
|
|
add_project_arguments(hardening_c_args, language: 'c')
|
|
add_project_link_arguments(hardening_link_args, language: ['cpp', 'c'])
|
|
|
|
if host_machine.system() == 'darwin' and cpp.get_id() == 'clang'
|
|
# Apple clang + external fmt/spdlog can fail when fmt enables
|
|
# consteval format-string checking inside spdlog headers.
|
|
add_project_arguments('-DFMT_CONSTEVAL=', language: 'cpp')
|
|
add_project_arguments('-Wno-macro-redefined', language: 'cpp')
|
|
endif
|
|
|
|
# --- C++20 Modules Support (Experimental) ---
|
|
# Detect and configure C++20 module compilation when enabled
|
|
yams_modules_enabled = get_option('enable-modules')
|
|
yams_modules_supported = false
|
|
yams_module_args = []
|
|
yams_module_flags = []
|
|
yams_use_modules_def = []
|
|
yams_module_cpp_args = []
|
|
|
|
if yams_modules_enabled
|
|
# Include module detection and configuration
|
|
subdir('meson')
|
|
endif
|
|
|
|
# --- Install prefix for plugin auto-trust ---
|
|
# Pass the install prefix to C++ code so the daemon knows where system plugins are installed
|
|
install_prefix = get_option('prefix')
|
|
# Escape backslashes for Windows paths to avoid compiler warnings about unrecognized escape sequences
|
|
install_prefix_escaped = install_prefix.replace('\\', '\\\\')
|
|
add_project_arguments('-DYAMS_INSTALL_PREFIX="' + install_prefix_escaped + '"', language: 'cpp')
|
|
message('YAMS install prefix: ' + install_prefix)
|
|
|
|
# --- Shared Conan/CMake discovery paths ---
|
|
conan_candidate_dirs = []
|
|
conan_gen_opt = get_option('conan-generators-dir')
|
|
if conan_gen_opt != ''
|
|
conan_candidate_dirs += [join_paths(meson.project_source_root(), conan_gen_opt)]
|
|
endif
|
|
|
|
base_build = meson.project_build_root()
|
|
default_candidates = [
|
|
base_build,
|
|
join_paths(base_build, 'conan'),
|
|
join_paths(base_build, 'conan', 'generators'),
|
|
join_paths(base_build, 'generators'),
|
|
join_paths(base_build, 'build-release', 'conan', 'generators'),
|
|
join_paths(base_build, 'build-debug', 'conan', 'generators'),
|
|
]
|
|
|
|
foreach variant : ['build-debug', 'build-release', 'debug', 'release']
|
|
variant_path = join_paths(base_build, variant)
|
|
if fs.exists(variant_path)
|
|
default_candidates += [variant_path]
|
|
default_candidates += [join_paths(variant_path, 'conan')]
|
|
default_candidates += [join_paths(variant_path, 'conan', 'generators')]
|
|
default_candidates += [join_paths(variant_path, 'generators')]
|
|
endif
|
|
endforeach
|
|
|
|
foreach cand : default_candidates
|
|
if fs.exists(cand)
|
|
conan_candidate_dirs += [cand]
|
|
endif
|
|
endforeach
|
|
|
|
cli11_cmake_paths = []
|
|
foreach base : conan_candidate_dirs
|
|
if fs.exists(base)
|
|
cli11_cmake_paths += base
|
|
foreach sub : ['CLI11', 'cli11']
|
|
cand = join_paths(base, sub)
|
|
if fs.exists(cand)
|
|
cli11_cmake_paths += cand
|
|
endif
|
|
endforeach
|
|
endif
|
|
endforeach
|
|
|
|
cli11_opt = get_option('cli11-cmake-path')
|
|
if cli11_opt != ''
|
|
cli11_cmake_paths += cli11_opt
|
|
endif
|
|
|
|
onnx_cmake_paths = []
|
|
foreach base : conan_candidate_dirs
|
|
if fs.exists(base)
|
|
onnx_cmake_paths += base
|
|
foreach sub : ['onnxruntime', 'onnxruntime/Release', 'onnxruntime/Debug']
|
|
cand = join_paths(base, sub)
|
|
if fs.exists(cand)
|
|
onnx_cmake_paths += cand
|
|
endif
|
|
endforeach
|
|
endif
|
|
endforeach
|
|
|
|
onnx_opt_path = get_option('onnxruntime-cmake-path')
|
|
if onnx_opt_path != ''
|
|
onnx_cmake_paths += onnx_opt_path
|
|
endif
|
|
|
|
# --- Shared dependency resolution ---
|
|
cli11_resolved_dep = dependency('cli11', required: false, method: 'pkg-config')
|
|
if not cli11_resolved_dep.found()
|
|
cli11_resolved_dep = dependency('CLI11', required: false, method: 'pkg-config')
|
|
endif
|
|
if not cli11_resolved_dep.found() and cli11_cmake_paths.length() > 0
|
|
cli11_resolved_dep = dependency('cli11', method: 'cmake', cmake_module_path: cli11_cmake_paths, required: false)
|
|
if not cli11_resolved_dep.found()
|
|
cli11_resolved_dep = dependency('CLI11', method: 'cmake', cmake_module_path: cli11_cmake_paths, required: false)
|
|
endif
|
|
endif
|
|
if not cli11_resolved_dep.found()
|
|
cli11_resolved_dep = dependency('CLI11', method: 'cmake', required: false)
|
|
endif
|
|
|
|
onnxruntime_resolved_dep = dependency('onnxruntime', method: 'pkg-config', required: false)
|
|
if not onnxruntime_resolved_dep.found()
|
|
onnxruntime_resolved_dep = dependency('onnxruntime', method: 'cmake', cmake_module_path: onnx_cmake_paths, required: false)
|
|
endif
|
|
if not onnxruntime_resolved_dep.found()
|
|
onnxruntime_resolved_dep = dependency('onnxruntime', required: false)
|
|
endif
|
|
if not onnxruntime_resolved_dep.found()
|
|
onnxruntime_resolved_dep = dependency('libonnxruntime', required: false)
|
|
endif
|
|
|
|
# Normalize ONNX Runtime linkage to the current library location instead of any stale
|
|
# Homebrew copies that may remain under /opt/homebrew/lib after upgrades.
|
|
if onnxruntime_resolved_dep.found()
|
|
onnxruntime_link_dirs = []
|
|
onnxruntime_pkg_libdir = onnxruntime_resolved_dep.get_variable(pkgconfig: 'libdir', default_value: '')
|
|
if onnxruntime_pkg_libdir != ''
|
|
onnxruntime_link_dirs += [onnxruntime_pkg_libdir]
|
|
endif
|
|
|
|
onnxruntime_cmake_prefix = onnxruntime_resolved_dep.get_variable(cmake: 'PACKAGE_PREFIX_DIR', default_value: '')
|
|
if onnxruntime_cmake_prefix != ''
|
|
onnxruntime_link_dirs += [join_paths(onnxruntime_cmake_prefix, 'lib')]
|
|
endif
|
|
|
|
if host_machine.system() == 'darwin'
|
|
foreach brew_onnx_prefix : ['/opt/homebrew/opt/onnxruntime/lib', '/usr/local/opt/onnxruntime/lib']
|
|
if fs.exists(brew_onnx_prefix)
|
|
onnxruntime_link_dirs = [brew_onnx_prefix] + onnxruntime_link_dirs
|
|
endif
|
|
endforeach
|
|
endif
|
|
|
|
onnxruntime_compile_dep = onnxruntime_resolved_dep.partial_dependency(compile_args: true, includes: true)
|
|
onnxruntime_cpp = meson.get_compiler('cpp')
|
|
onnxruntime_link_dep = onnxruntime_cpp.find_library('onnxruntime', dirs: onnxruntime_link_dirs, required: false)
|
|
if not onnxruntime_link_dep.found()
|
|
onnxruntime_link_dep = onnxruntime_cpp.find_library('onnxruntime', required: false)
|
|
endif
|
|
|
|
if onnxruntime_link_dep.found()
|
|
onnxruntime_resolved_dep = declare_dependency(
|
|
dependencies: [onnxruntime_compile_dep, onnxruntime_link_dep],
|
|
)
|
|
message('ONNX Runtime linkage normalized via current library search path')
|
|
endif
|
|
endif
|
|
|
|
tbb_resolved_dep = dependency('tbb', required: false, method: 'pkg-config')
|
|
if not tbb_resolved_dep.found()
|
|
tbb_resolved_dep = dependency('onetbb', required: false, method: 'pkg-config')
|
|
endif
|
|
if not tbb_resolved_dep.found()
|
|
tbb_resolved_dep = dependency('tbb', required: false)
|
|
endif
|
|
if not tbb_resolved_dep.found()
|
|
tbb_resolved_dep = dependency('onetbb', required: false)
|
|
endif
|
|
if host_machine.system() == 'darwin' and tbb_resolved_dep.found()
|
|
tbb_normalized = false
|
|
tbb_cpp = meson.get_compiler('cpp')
|
|
foreach brew_tbb_prefix : ['/opt/homebrew/opt/tbb', '/usr/local/opt/tbb']
|
|
if not tbb_normalized
|
|
brew_tbb_incdir = brew_tbb_prefix / 'include'
|
|
brew_tbb_libdir = brew_tbb_prefix / 'lib'
|
|
if fs.exists(brew_tbb_incdir) and fs.exists(brew_tbb_libdir)
|
|
tbb_link_dep = tbb_cpp.find_library('tbb', dirs: [brew_tbb_libdir], required: false)
|
|
if tbb_link_dep.found()
|
|
tbb_resolved_dep = declare_dependency(
|
|
include_directories: include_directories(brew_tbb_incdir),
|
|
dependencies: [tbb_link_dep],
|
|
)
|
|
tbb_normalized = true
|
|
message('TBB linkage normalized via Homebrew opt prefix')
|
|
endif
|
|
endif
|
|
endif
|
|
endforeach
|
|
endif
|
|
|
|
# --- Windows DLL deployment ---
|
|
# On Windows, DLLs must be in the same directory as the executable or in PATH.
|
|
# We extract the DLL paths from the dependency's pkg-config `bindir` variable,
|
|
# then use fs.copyfile() to deploy them to the build directory.
|
|
#
|
|
# This is the Meson-idiomatic approach that doesn't rely on Conan's runtime_deploy.
|
|
windows_runtime_dlls = [] # List of [dll_name, dll_full_path] tuples
|
|
if host_machine.system() == 'windows'
|
|
# Get TBB dependency and extract bindir
|
|
tbb_dep = tbb_resolved_dep
|
|
if tbb_dep.found()
|
|
tbb_prefix = tbb_dep.get_variable(pkgconfig: 'prefix', default_value: '')
|
|
tbb_bindir = tbb_dep.get_variable(pkgconfig: 'bindir', default_value: '')
|
|
if tbb_bindir != ''
|
|
# Resolve ${prefix} placeholder if present
|
|
tbb_bindir = tbb_bindir.replace('${prefix}', tbb_prefix)
|
|
foreach dll_name : ['tbb12.dll', 'tbbbind_2_5.dll', 'tbbmalloc.dll', 'tbbmalloc_proxy.dll']
|
|
dll_path = tbb_bindir / dll_name
|
|
if fs.exists(dll_path)
|
|
windows_runtime_dlls += [[dll_name, dll_path]]
|
|
endif
|
|
endforeach
|
|
endif
|
|
endif
|
|
|
|
# Get ONNX Runtime dependency and extract bindir (prefer Conan CMake config)
|
|
onnx_dep_check = onnxruntime_resolved_dep
|
|
|
|
onnx_bindir_candidates = []
|
|
if onnx_dep_check.found()
|
|
onnx_prefix = onnx_dep_check.get_variable(pkgconfig: 'prefix', default_value: '')
|
|
onnx_bindir = onnx_dep_check.get_variable(pkgconfig: 'bindir', default_value: '')
|
|
if onnx_bindir != ''
|
|
onnx_bindir = onnx_bindir.replace('${prefix}', onnx_prefix)
|
|
onnx_bindir_candidates += [onnx_bindir]
|
|
endif
|
|
|
|
# CMake configs from Conan often expose PACKAGE_PREFIX_DIR; fall back to <prefix>/bin
|
|
cmake_prefix = onnx_dep_check.get_variable(cmake: 'PACKAGE_PREFIX_DIR', default_value: '')
|
|
if cmake_prefix != ''
|
|
onnx_bindir_candidates += [join_paths(cmake_prefix, 'bin')]
|
|
elif onnx_prefix != ''
|
|
onnx_bindir_candidates += [join_paths(onnx_prefix, 'bin')]
|
|
endif
|
|
endif
|
|
|
|
# As a last resort, probe the Conan cache directly (~/.conan2/p/**/p/bin) via a tiny Python helper
|
|
py_mod = import('python')
|
|
python3 = py_mod.find_installation('python3', required: false)
|
|
if not python3.found()
|
|
python3 = py_mod.find_installation('python', required: false)
|
|
endif
|
|
|
|
if python3.found()
|
|
conan_probe = run_command(python3, ['-c', '''import pathlib, os
|
|
root = pathlib.Path(os.path.expanduser("~/.conan2/p"))
|
|
dirs = []
|
|
if root.exists():
|
|
for pattern in ("*/p/bin", "b/*/p/bin"):
|
|
for bin_dir in root.glob(pattern):
|
|
dll = bin_dir / "onnxruntime.dll"
|
|
if dll.exists():
|
|
dirs.append(str(bin_dir))
|
|
print(";".join(dirs))'''], check: false)
|
|
if conan_probe.returncode() == 0
|
|
conan_dirs = conan_probe.stdout().strip().split(';')
|
|
foreach d : conan_dirs
|
|
if d != ''
|
|
onnx_bindir_candidates += [d]
|
|
endif
|
|
endforeach
|
|
endif
|
|
else
|
|
message('Python not found (python3/python); skipping Conan cache probe for ONNX Runtime DLLs')
|
|
endif
|
|
|
|
foreach bindir : onnx_bindir_candidates
|
|
foreach dll_name : ['onnxruntime.dll', 'onnxruntime_providers_shared.dll']
|
|
dll_path = bindir / dll_name
|
|
if fs.exists(dll_path)
|
|
windows_runtime_dlls += [[dll_name, dll_path]]
|
|
endif
|
|
endforeach
|
|
endforeach
|
|
|
|
# Deduplicate DLL entries by name to avoid duplicate targets
|
|
dedup_map = {}
|
|
deduped = []
|
|
foreach dll_info : windows_runtime_dlls
|
|
name = dll_info[0]
|
|
if not dedup_map.has_key(name)
|
|
dedup_map += {name: true}
|
|
deduped += [dll_info]
|
|
endif
|
|
endforeach
|
|
windows_runtime_dlls = deduped
|
|
|
|
# Log discovered DLLs
|
|
foreach dll_info : windows_runtime_dlls
|
|
message('Found runtime DLL: ' + dll_info[0] + ' at ' + dll_info[1])
|
|
endforeach
|
|
endif
|
|
|
|
if not wasi_enabled
|
|
# Database backend selection (libsql or sqlite)
|
|
db_backend = get_option('database-backend')
|
|
require_fts5 = get_option('require-fts5')
|
|
|
|
# Try to find the selected database backend
|
|
using_libsql = false
|
|
if db_backend == 'libsql'
|
|
# Try to find libsql (provides sqlite3-compatible API)
|
|
# 1. First try system-installed libsql via pkg-config
|
|
libsql_dep = dependency('libsql', required: false)
|
|
if not libsql_dep.found()
|
|
libsql_dep = dependency('libsql-sqlite3', required: false)
|
|
endif
|
|
|
|
# 2. If not found, try building from subproject (requires Rust toolchain)
|
|
if not libsql_dep.found()
|
|
cargo_prog = find_program('cargo', required: false)
|
|
if cargo_prog.found()
|
|
message('libSQL not found on system, building from subproject (requires Rust)...')
|
|
libsql_subproj = subproject('libsql', required: false)
|
|
if libsql_subproj.found()
|
|
libsql_dep = libsql_subproj.get_variable('libsql_dep')
|
|
endif
|
|
endif
|
|
endif
|
|
|
|
if libsql_dep.found()
|
|
sqlite_dep = libsql_dep
|
|
using_libsql = true
|
|
message('Using libSQL database backend (MVCC concurrent writes enabled)')
|
|
else
|
|
warning('libSQL not found and subproject build failed. Falling back to standard SQLite3.')
|
|
warning('To use libSQL: install Rust toolchain (rustup) or system libsql package.')
|
|
sqlite_dep = dependency('sqlite3', required: true)
|
|
endif
|
|
else
|
|
sqlite_dep = dependency('sqlite3', required: true)
|
|
message('Using standard SQLite3 database backend')
|
|
endif
|
|
|
|
# Add compile definition for backend detection
|
|
if using_libsql
|
|
add_project_arguments('-DYAMS_LIBSQL_BACKEND=1', language: 'cpp')
|
|
else
|
|
add_project_arguments('-DYAMS_LIBSQL_BACKEND=0', language: 'cpp')
|
|
endif
|
|
|
|
# Skip runtime test when cross-compiling or using subproject sqlite (internal dep can't be used with cpp.run in newer meson)
|
|
if meson.is_cross_build()
|
|
warning('Cross-compiling: skipping SQLite FTS5 runtime check. Ensure FTS5 is enabled in your target SQLite build.')
|
|
elif sqlite_dep.type_name() == 'internal'
|
|
message('Using subproject sqlite3: skipping FTS5 runtime check. Subproject sqlite3 has FTS5 enabled by default.')
|
|
elif using_libsql
|
|
message('Using libSQL: skipping FTS5 runtime check. libSQL includes FTS5 support.')
|
|
else
|
|
fts5_prog = cpp.run(files('meson/fts5_check.cpp'), dependencies: sqlite_dep, name: 'sqlite3 FTS5 check')
|
|
if fts5_prog.returncode() != 0
|
|
msg = 'SQLite3 built without FTS5 support. Enable FTS5 in your package manager or Conan (e.g., -o "sqlite3/*:fts5=True").'
|
|
if require_fts5
|
|
error(msg)
|
|
else
|
|
warning(msg)
|
|
endif
|
|
endif
|
|
endif
|
|
endif
|
|
|
|
# Boost is a core requirement (e.g., Boost.Asio for daemon communication).
|
|
# Resolution order:
|
|
# 1. Per-library pkg-config probes against Conan's emitted boost_*.pc.
|
|
# Required path on Windows: meson 1.4's built-in Boost finder rejects
|
|
# Conan's MSVC layout (libs are named libboost_<comp>.lib, not the
|
|
# auto-link form boost_<comp>-vc14x-mt-x64-1_85.lib that meson scans
|
|
# for). pkg-config bypasses meson's Boost special-case entirely.
|
|
# 2. System Boost auto-detect — Linux/macOS fallback when no .pc files
|
|
# are on PKG_CONFIG_PATH (apt/brew Boost ships no pkg-config files).
|
|
# NOTE: Conan PkgConfigDeps emits per-component .pc files using HYPHEN
|
|
# separators (boost-system.pc, boost-thread.pc), not underscores. pkg-config
|
|
# resolves dependency names against the .pc filename, so we must match.
|
|
boost_system_dep = dependency('boost-system', method: 'pkg-config',
|
|
required: false)
|
|
boost_thread_dep = dependency('boost-thread', method: 'pkg-config',
|
|
required: false)
|
|
if boost_system_dep.found() and boost_thread_dep.found()
|
|
boost_dep = declare_dependency(
|
|
dependencies: [boost_system_dep, boost_thread_dep])
|
|
else
|
|
boost_dep = dependency('boost', modules: ['system', 'thread'],
|
|
required: true)
|
|
endif
|
|
meson.override_dependency('boost', boost_dep)
|
|
|
|
# --- RE2 regex engine (optional, for high-performance grep) ---
|
|
re2_opt = get_option('enable-re2')
|
|
re2_dep = dependency('re2', required: re2_opt, include_type: 'system')
|
|
if re2_dep.found()
|
|
add_project_arguments('-DYAMS_HAS_RE2=1', language: 'cpp')
|
|
message('RE2 regex engine enabled (grep will use RE2 with std::regex fallback)')
|
|
else
|
|
add_project_arguments('-DYAMS_HAS_RE2=0', language: 'cpp')
|
|
message('RE2 not found; grep will use std::regex only')
|
|
endif
|
|
|
|
# --- Version header generation ---
|
|
_override_version = get_option('yams-version')
|
|
|
|
# Detect version from git tags if not overridden
|
|
_git_version = ''
|
|
_git_cmd = find_program('git', required: false)
|
|
if _git_cmd.found() and _override_version == ''
|
|
# Try to get the most recent semver tag (v*)
|
|
_r = run_command(_git_cmd, ['describe', '--tags', '--match', 'v*', '--abbrev=0'], check: false)
|
|
if _r.returncode() == 0
|
|
_tag = _r.stdout().strip()
|
|
# Strip leading 'v' if present
|
|
if _tag.startswith('v')
|
|
_git_version = _tag.substring(1)
|
|
else
|
|
_git_version = _tag
|
|
endif
|
|
endif
|
|
endif
|
|
|
|
# Priority: command-line override > git tag > meson.project_version()
|
|
if _override_version != ''
|
|
_effective_version = _override_version
|
|
elif _git_version != ''
|
|
_effective_version = _git_version
|
|
else
|
|
_effective_version = meson.project_version()
|
|
endif
|
|
|
|
python_exe = find_program('python3', 'python')
|
|
_git_cmd = find_program('git', required: false)
|
|
_git_exe = _git_cmd.found() ? _git_cmd.full_path() : 'git'
|
|
version_header_template = files('include/yams/version_generated.h.in')[0]
|
|
version_header_script = files('scripts/generate_version_header.py')[0]
|
|
|
|
# Generate version header at build time so installed binaries pick up the
|
|
# current commit even when source changes occur after the last Meson reconfigure.
|
|
version_header = custom_target(
|
|
'version_generated_header',
|
|
input: [version_header_template, version_header_script],
|
|
output: 'version_generated.h',
|
|
command: [
|
|
python_exe,
|
|
version_header_script,
|
|
'@INPUT0@',
|
|
'@OUTPUT@',
|
|
meson.project_source_root(),
|
|
meson.project_version(),
|
|
_override_version,
|
|
_git_exe,
|
|
],
|
|
build_always_stale: true,
|
|
build_by_default: true,
|
|
install: true,
|
|
install_dir: get_option('includedir') / 'yams',
|
|
)
|
|
|
|
# Make builddir available as include path for generated headers
|
|
# CLI will include as "version_generated.h" directly
|
|
generated_inc = include_directories('.')
|
|
version_generated_dep = declare_dependency(
|
|
include_directories: generated_inc,
|
|
sources: version_header,
|
|
)
|
|
|
|
yams_build_cli = get_option('build-cli') and not wasi_enabled
|
|
yams_build_mcp = get_option('build-mcp-server')
|
|
|
|
# Install public headers under the compiler's include prefix without duplicating
|
|
# the top-level 'include' directory (was producing prefix/include/include/...).
|
|
install_subdir('include', install_dir: get_option('includedir'), strip_directory: true)
|
|
|
|
# --- tl-expected global resolution ---
|
|
# tl-expected is a header-only library. Homebrew installs it without pkgconfig.
|
|
# Resolve it once here and override for all subdirs.
|
|
_tl_expected_dep = dependency('tl-expected', method: 'cmake', required: false)
|
|
if not _tl_expected_dep.found()
|
|
_tl_expected_dep = dependency('tl-expected', method: 'pkg-config', required: false)
|
|
endif
|
|
if not _tl_expected_dep.found()
|
|
# Manual fallback for homebrew installation. Probe both brew prefixes
|
|
# (Apple Silicon + Intel) guarded by the header actually existing.
|
|
foreach _brew_prefix : ['/opt/homebrew', '/usr/local']
|
|
if not _tl_expected_dep.found() and fs.exists(_brew_prefix / 'include' / 'tl' / 'expected.hpp')
|
|
_tl_expected_inc = include_directories(_brew_prefix / 'include', is_system: true)
|
|
_tl_expected_dep = declare_dependency(include_directories: _tl_expected_inc)
|
|
endif
|
|
endforeach
|
|
endif
|
|
meson.override_dependency('tl-expected', _tl_expected_dep)
|
|
|
|
# --- tinyfsm global resolution ---
|
|
# tinyfsm is a single-header FSM library vendored under third_party/.
|
|
# Added as a project-wide include because daemon FSM headers are included
|
|
# transitively by many targets via ServiceManager.h.
|
|
_tinyfsm_inc = include_directories('third_party/tinyfsm/include', is_system: true)
|
|
add_project_arguments(
|
|
'-isystem', meson.project_source_root() / 'third_party' / 'tinyfsm' / 'include',
|
|
language: 'cpp',
|
|
)
|
|
_tinyfsm_dep = declare_dependency(include_directories: _tinyfsm_inc)
|
|
meson.override_dependency('tinyfsm', _tinyfsm_dep)
|
|
|
|
# --- libarchive global resolution ---
|
|
# Homebrew on macOS installs libarchive as keg-only; pkg-config won't find it
|
|
# without the Conan-generated pkg_config_path. Resolve once and override only
|
|
# in the fallback path — overriding an already-resolved dependency is an error.
|
|
_libarchive_dep = dependency('libarchive', required: false)
|
|
if not _libarchive_dep.found() and host_machine.system() == 'darwin'
|
|
foreach _brew_prefix : ['/opt/homebrew', '/usr/local']
|
|
if not _libarchive_dep.found()
|
|
_brew_libarchive = _brew_prefix / 'opt' / 'libarchive'
|
|
if fs.exists(_brew_libarchive / 'include' / 'archive.h')
|
|
_libarchive_lib = cpp.find_library('archive',
|
|
dirs: [_brew_libarchive / 'lib'],
|
|
required: false)
|
|
if _libarchive_lib.found()
|
|
_libarchive_dep = declare_dependency(
|
|
dependencies: _libarchive_lib,
|
|
include_directories: include_directories(_brew_libarchive / 'include', is_system: true),
|
|
)
|
|
meson.override_dependency('libarchive', _libarchive_dep)
|
|
endif
|
|
endif
|
|
endif
|
|
endforeach
|
|
endif
|
|
|
|
if wasi_enabled
|
|
# WASI profile: compile a minimal subset (no daemon IPC, no plugins, no downloads).
|
|
# Keep this intentionally small so cross-compilers don't need system libs like
|
|
# OpenSSL/libcurl/sqlite/onnx.
|
|
subdir('src/core')
|
|
|
|
if get_option('build-mcp-server')
|
|
subdir('src/mcp')
|
|
endif
|
|
else
|
|
# Core libraries (ordered by dependency)
|
|
subdir('src/core')
|
|
subdir('src/crypto')
|
|
subdir('src/chunking')
|
|
subdir('src/compression')
|
|
subdir('src/storage')
|
|
subdir('src/wal')
|
|
subdir('src/manifest')
|
|
subdir('src/metadata')
|
|
subdir('third_party/hdbscan-cpp')
|
|
meson.override_dependency('yams_hdbscan_cpp', hdbscan_dep)
|
|
subdir('src/integrity')
|
|
subdir('src/config')
|
|
subdir('src/extraction')
|
|
subdir('src/detection')
|
|
subdir('src/content')
|
|
subdir('src/downloader')
|
|
if get_option('build-plugins')
|
|
subdir('src/plugins')
|
|
endif
|
|
subdir('src/daemon/client')
|
|
subdir('src/embedding_simeon')
|
|
subdir('src/vector')
|
|
subdir('src/topology')
|
|
|
|
# Third-party libraries (must come before src/search which depends on symspell)
|
|
subdir('third_party/symspell')
|
|
|
|
subdir('src/search')
|
|
subdir('src/api')
|
|
subdir('src/ingest')
|
|
subdir('src/repair')
|
|
subdir('src/app/services')
|
|
subdir('src/daemon')
|
|
if get_option('enable-mobile-bindings')
|
|
subdir('src/mobile')
|
|
endif
|
|
if get_option('build-benchmarks')
|
|
subdir('src/benchmarks')
|
|
endif
|
|
|
|
if get_option('build-mcp-server')
|
|
subdir('src/mcp')
|
|
endif
|
|
endif
|
|
|
|
if get_option('build-plugins') and not wasi_enabled
|
|
subdir('plugins')
|
|
endif
|
|
|
|
if get_option('build-cli') and not wasi_enabled
|
|
subdir('src/cli')
|
|
subdir('tools/yams-cli')
|
|
if get_option('build-mcp-server')
|
|
subdir('tools/yams-mcp')
|
|
endif
|
|
endif
|
|
|
|
# Tests (enabled by default in Debug builds)
|
|
_tests_opt = get_option('build-tests')
|
|
_buildtype = get_option('buildtype')
|
|
_tests_enabled = (not wasi_enabled) and (_tests_opt or (_buildtype == 'debug' and not get_option('build-fuzzers')))
|
|
if _tests_enabled
|
|
subdir('tests')
|
|
endif
|
|
|
|
# Fuzzing harnesses (AFL++/libFuzzer)
|
|
if get_option('build-fuzzers') and not wasi_enabled
|
|
subdir('tools/fuzzing')
|
|
endif
|
|
|
|
sql_install_dir = join_paths(get_option('datadir'), 'yams', 'sql')
|
|
install_data('sql/reference_schema.sql', install_dir: sql_install_dir)
|
|
|
|
if not wasi_enabled
|
|
yams_pc_config = configuration_data()
|
|
yams_pc_config.set('PREFIX', get_option('prefix'))
|
|
yams_pc_config.set('LIBDIR', get_option('libdir'))
|
|
yams_pc_config.set('INCLUDEDIR', get_option('includedir'))
|
|
yams_pc_config.set('VERSION', _effective_version)
|
|
configure_file(
|
|
input: files('packaging/yams.pc.in'),
|
|
output: 'yams.pc',
|
|
configuration: yams_pc_config,
|
|
install: true,
|
|
install_dir: join_paths(get_option('libdir'), 'pkgconfig'),
|
|
)
|
|
endif
|
|
|
|
summary({
|
|
'project': meson.project_name(),
|
|
'version': meson.project_version(),
|
|
'cpp_std': get_option('cpp_std'),
|
|
'build_cli': yams_build_cli,
|
|
'build_mcp_server': yams_build_mcp,
|
|
}, section: 'YAMS Meson configuration', bool_yn: true)
|
|
|
|
if get_option('b_coverage')
|
|
gcovr = find_program('gcovr', required: false)
|
|
if gcovr.found()
|
|
_cpp_compiler = meson.get_compiler('cpp')
|
|
_gcov_executable = 'gcov'
|
|
if _cpp_compiler.get_id() == 'clang'
|
|
_llvm_cov = find_program('llvm-cov', required: false)
|
|
if _llvm_cov.found()
|
|
_gcov_executable = 'llvm-cov gcov -p'
|
|
endif
|
|
endif
|
|
|
|
coverage_report_target = custom_target('coverage-report',
|
|
output: 'coverage.html',
|
|
command: [
|
|
gcovr,
|
|
'--root', meson.project_source_root(),
|
|
'--gcov-executable', _gcov_executable,
|
|
'--object-directory', meson.project_build_root(),
|
|
'--merge-mode-functions', 'merge-use-line-min',
|
|
'--filter', join_paths(meson.project_source_root(), 'src'),
|
|
'--filter', join_paths(meson.project_source_root(), 'include'),
|
|
'--exclude', join_paths(meson.project_source_root(), 'src', 'benchmarks'),
|
|
'--html', '--html-details',
|
|
'--output', '@OUTPUT@',
|
|
meson.project_build_root(),
|
|
],
|
|
build_by_default: false,
|
|
build_always_stale: true,
|
|
)
|
|
|
|
coverage_txt_target = custom_target('coverage-summary',
|
|
output: 'coverage.txt',
|
|
command: [
|
|
gcovr,
|
|
'--root', meson.project_source_root(),
|
|
'--gcov-executable', _gcov_executable,
|
|
'--object-directory', meson.project_build_root(),
|
|
'--merge-mode-functions', 'merge-use-line-min',
|
|
'--filter', join_paths(meson.project_source_root(), 'src'),
|
|
'--filter', join_paths(meson.project_source_root(), 'include'),
|
|
'--exclude', join_paths(meson.project_source_root(), 'src', 'benchmarks'),
|
|
'--print-summary',
|
|
'--output', '@OUTPUT@',
|
|
meson.project_build_root(),
|
|
],
|
|
build_by_default: false,
|
|
build_always_stale: true,
|
|
)
|
|
|
|
message('Coverage reporting configured. Run tests then: ninja -C builddir coverage.txt && ninja -C builddir coverage.html')
|
|
else
|
|
warning('gcovr not found. Install gcovr to generate coverage reports.')
|
|
endif
|
|
endif
|
|
|
|
# Note: Runtime DLLs are now deployed via Conan's runtime_deploy deployer during conan install
|
|
# No custom meson install script needed
|