yams/tests/unit/metadata/search_query_helpers_catch2_test.cpp
Trevon 3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00

136 lines
4.8 KiB
C++

#include <catch2/catch_test_macros.hpp>
#include <string>
#include <vector>
#include "../../common/test_helpers_catch2.h"
#include "src/metadata/repository/search_query_helpers.hpp"
namespace yams::metadata {
#ifdef YAMS_TESTING
void testingResetSearchQueryHelperCaches();
#endif
} // namespace yams::metadata
namespace {
void resetSearchQueryHelperCaches() {
#ifdef YAMS_TESTING
yams::metadata::testingResetSearchQueryHelperCaches();
#endif
}
} // namespace
TEST_CASE("Search query helpers: parseFts5ModeEnv honors supported values",
"[unit][metadata][fts5][search-query-helpers]") {
using yams::metadata::Fts5QueryMode;
using yams::metadata::parseFts5ModeEnv;
yams::test::ScopedEnvVar env("YAMS_FTS_MODE");
SECTION("unset defaults to smart") {
env.unset();
CHECK((parseFts5ModeEnv() == Fts5QueryMode::Smart));
}
SECTION("simple is case-insensitive") {
env.set("SIMPLE");
CHECK((parseFts5ModeEnv() == Fts5QueryMode::Simple));
}
SECTION("nl shorthand selects natural mode") {
env.set("nl");
CHECK((parseFts5ModeEnv() == Fts5QueryMode::Natural));
}
SECTION("natural selects natural mode") {
env.set("natural");
CHECK((parseFts5ModeEnv() == Fts5QueryMode::Natural));
}
SECTION("unknown values fall back to smart") {
env.set("surprising-value");
CHECK((parseFts5ModeEnv() == Fts5QueryMode::Smart));
}
}
TEST_CASE("Search query helpers: includeSearchSnippets caches env-derived behavior",
"[unit][metadata][fts5][search-query-helpers]") {
using yams::metadata::includeSearchSnippets;
yams::test::ScopedEnvVar env("YAMS_SEARCH_INCLUDE_SNIPPET");
SECTION("unset defaults to disabled") {
env.unset();
resetSearchQueryHelperCaches();
CHECK_FALSE((includeSearchSnippets()));
}
SECTION("recognized falsey values disable snippets") {
for (const char* value : {"0", "false", "off", "no"}) {
env.set(value);
resetSearchQueryHelperCaches();
INFO(value);
CHECK_FALSE((includeSearchSnippets()));
}
}
SECTION("non-empty truthy values enable snippets") {
for (const char* value : {"1", "true", "yes", "debug"}) {
env.set(value);
resetSearchQueryHelperCaches();
INFO(value);
CHECK((includeSearchSnippets()));
}
}
}
TEST_CASE("Search query helpers: buildDiagnosticAltOrQuery strips punctuation and limits terms",
"[unit][metadata][fts5][search-query-helpers]") {
using yams::metadata::buildDiagnosticAltOrQuery;
SECTION("empty input yields empty query") {
CHECK((buildDiagnosticAltOrQuery({}).empty()));
}
SECTION("filters short terms and quotes normalized tokens") {
const std::vector<std::string> tokens = {"a", "the", "alpha,", "(beta)", "gamma.",
"delta", "epsilon", "zeta", "eta"};
CHECK((buildDiagnosticAltOrQuery(tokens) ==
"\"alpha\" OR \"beta\" OR \"gamma\" OR \"delta\" OR \"epsilon\""));
}
}
TEST_CASE("Search query helpers: hasAdvancedFts5Operators distinguishes power-user syntax",
"[unit][metadata][fts5][search-query-helpers]") {
using yams::metadata::hasAdvancedFts5Operators;
CHECK((hasAdvancedFts5Operators("title:hello")));
CHECK((hasAdvancedFts5Operators("hello NEAR/3 world")));
CHECK((hasAdvancedFts5Operators("\"exact phrase\"")));
CHECK((hasAdvancedFts5Operators("(foo OR bar) NOT baz")));
CHECK_FALSE((hasAdvancedFts5Operators("dendritic cells (DCs)")));
CHECK_FALSE((hasAdvancedFts5Operators("path:/tmp/file.txt")));
CHECK_FALSE((hasAdvancedFts5Operators("hello and world")));
CHECK_FALSE((hasAdvancedFts5Operators("plain terms only")));
// C++ scope qualifiers are literal content, not FTS field selectors.
CHECK_FALSE((hasAdvancedFts5Operators("SqliteVecBackend::bruteForceSearch")));
CHECK_FALSE(
(hasAdvancedFts5Operators("yams::vector::SqliteVecBackend::Impl::bruteForceSearch")));
}
TEST_CASE("Search query helpers: sanitizeFts5UserQuery quotes C++ qualified names",
"[unit][metadata][fts5][search-query-helpers]") {
using yams::metadata::sanitizeFts5UserQuery;
// A C++ qualified name must reach SQLite as a quoted literal phrase; otherwise FTS5
// parses the leading qualifier as a column selector and fails with "no such column".
CHECK((sanitizeFts5UserQuery("SqliteVecBackend::bruteForceSearch") ==
"\"SqliteVecBackend::bruteForceSearch\""));
CHECK((sanitizeFts5UserQuery("yams::vector::SqliteVecBackend::Impl::bruteForceSearch") ==
"\"yams::vector::SqliteVecBackend::Impl::bruteForceSearch\""));
// Single-colon field syntax remains available for power users.
CHECK((sanitizeFts5UserQuery("title:hello") == "title:hello"));
}