yams/tests/unit/cli/doctor_command_catch2_test.cpp
Trevon 73d408a4a6
[05/14] Consolidate configuration authority (#145)
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays

LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays
gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as
test-only benchmark levers, but tests/scripts/report_config_key_usage.py
shows 45 of them are set by no test, no bench arm, no script, and no
doc: they were levers nobody pulled, each shadowing a SearchEngineConfig
field that benches and tests already set directly. The entire
YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene
note that graph-vector fusion is a parked lever.

Removed (CANDIDATE rows of the report, verbatim):
  YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE
  YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD
  YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION
  YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION
  YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST
  YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE
  YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS
  YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT
  YAMS_SEARCH_GRAPH_ENABLE_PATHS
  YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY
  YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS
  YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS
  YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS
  YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K
  YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE
  YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY
  YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL
  YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER
  YAMS_SEARCH_GRAPH_HOP_DECAY
  YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW
  YAMS_SEARCH_GRAPH_MAX_HOPS
  YAMS_SEARCH_GRAPH_MAX_NEIGHBORS
  YAMS_SEARCH_GRAPH_MAX_PATHS
  YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE
  YAMS_SEARCH_GRAPH_TEXT_WEIGHT
  YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS
  YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING
  YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION
  YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING
  YAMS_SEARCH_GRAPH_VECTOR_WEIGHT
  YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS
  YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY
  YAMS_SEARCH_RERANK_TOP_K
  YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE
  YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK
  YAMS_SEARCH_TIERED_MIN_CANDIDATES
  YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH
  YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS
  YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS
  YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY
  YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE
  YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK
  YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER
  YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS
  YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE

Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of
YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE
(different clamps, later block won when both were set); only the
current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by
two identical blocks and is now read once.

A test builds the environment over a recording lookup and asserts none
of the retired names is ever consulted, and that the surviving names
are consulted exactly once. production_environment_keys.txt shrinks by
45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale.

The search_latency_graph_rerank task in .vscode/tasks.json set two of the
retired overlays; those assignments are dropped so the task does not claim a
configuration the code no longer honours.

Signed-off-by: trvon <me@trevon.dev>

* refactor(config): drop unread embeddings.* keys and the no-op toggles

yams config embeddings enable/disable/tune wrote embeddings.auto_generate,
embeddings.generation_delay_ms, and embeddings.batch_size, and the
migrator seeded them (plus embeddings.auto_on_add) into every new
config.toml. No daemon code reads any of them: the post-ingest queue
always dispatches embed jobs when an embed queue exists, and the only
batch size the resolvers read is embeddings.runtime.batch_size. The
toggle was a switch wired to nothing, and yams status advised users to
flip it.

- embeddings enable and disable are removed. enable duplicated
  embeddings model <name> once its dead keys are gone; a kept no-op
  would be a shim. status no longer prints an auto-generation line and
  shows the runtime batch size when set. tune presets write
  embeddings.runtime.batch_size and drop the delay.
- config_migration no longer seeds auto_generate, auto_on_add,
  generation_delay_ms, or a top-level batch_size.
- ConfigResolver and the compatibility policy honour a legacy
  [embeddings] batch_size silently when the runtime key is absent, so
  configs in the wild get their value applied without a file rewrite (the
  migrator exits early for v3 configs, so a rewrite would never run).
- status and model hints point at embeddings model <name>.

Declared behavior change: `yams config embeddings enable|disable` are
unknown subcommands now.

Declared behavior change: the legacy [embeddings] batch_size was written by
`yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16)
but no resolver ever read it. Honouring it as a fallback applies those numbers
for the first time; the one consumer whose result moves is the doctor embedding
repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32
on a stock config. The embedding service already defaulted to 32.

Signed-off-by: trvon <me@trevon.dev>

* feat(config): add [tuning] keys for the setters only env could reach

Seventeen TuneAdvisor setters were exercised by tests but had no TOML
key, so production could reach them only through a deprecated YAMS_*
overlay: connection slot sizing, the CPU-high threshold, the ONNX
concurrency and reservation knobs, model-eviction thresholds, ingest
worker cap, store/embed channel capacities, work-coordinator threads,
and connection lifetime. AGENTS.md says tuning lives in typed config;
these were half-migrated.

Each gets a [tuning] key resolved through ConfigResolver alongside the
existing ones, with provenance recorded, and each is included in
resetConfiguredOverrides() so a reload that drops a key reverts it
(the three with dedicated reset helpers call those). The environment
names keep working as compatibility inputs; nothing new is added to the
YAMS_* surface.

A table-driven test sets all 17 keys, checks every getter, then reloads
with an empty [tuning] section and checks every value reverted.
docs/architecture/environment-overrides.md lists the keys.

The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds
in (0, 1), connection_lifetime_s at most 86400) keep their default when
handed a value outside the range. The resolver checks the same ranges first
and warns, so an ignored value is never recorded as config provenance; test
covers all three shapes.

Signed-off-by: trvon <me@trevon.dev>

* fix(config): report 0 for [tuning] integer keys instead of ignoring it

For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below
every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back
as "unset", and silently fall through to the default while the resolver
still recorded config provenance for it. applyUint32 now warns and skips
the value: nothing changes for the operator except that the mistake is
visible.

Two keys mean something by 0 and are exempt: post_ingest_threads (0 =
auto, which the config migrator seeds into every config and
`yams config` displays as "(auto)") and connection_lifetime_s (0 = no
recycling; its setter takes a signed sentinel and accepts 0, and the
environment spelling always did). Both keep their provenance and stay
silent, so a stock config produces no warning.

Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and
record no provenance, pool_ipc_max = 48 still applies, and the two
zero-valid keys apply with provenance.

Signed-off-by: trvon <me@trevon.dev>

* refactor(config): one embedding-runtime resolver, one tuning baseline

Two resolvers read the same five environment keys and three TOML keys
for the embedding runtime with different error semantics:
ConfigResolver::resolveEmbeddingConfig used the strict typed readers
(warn and keep the default), while resolveEmbeddingRuntimePolicy in
config_resolver_policies.cpp used std::stoull inside a bare catch, so
YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on
the other. The compatibility entry point is now a projection of the
typed resolution onto the EmbeddingRuntimePolicy shape that library
callers (vector, repair) still consume; it lives in ConfigResolver.cpp
because DaemonConfig is declared there. 91 lines of duplicate parsing
go away.

yams doctor tuning carried its own [tuning] baseline table and it had
drifted: worker_poll_ms 75 against the daemon default of 150,
pool_io_max sized from hardware against a default of 32, so
`doctor tuning --apply` wrote defaults as divergent overrides. The
baseline now reads TuneAdvisor's effective values, which is what
"baseline" should mean.

Tests: the compatibility policy rejects 12abc and agrees with the typed
resolver; the doctor proposal equals TuneAdvisor for every key.

Signed-off-by: trvon <me@trevon.dev>

* refactor(config): stop writing the unread search.reranker_model_path

yams init wrote both search.reranker_model (the name the daemon resolves
through ConfigResolver::resolveRerankerModel) and
search.reranker_model_path, a derived on-disk path nothing in the
daemon, search, or vector code reads; yams config search reranker
status then echoed it back as if it mattered. init no longer writes the
path key and config no longer displays it. The branch that erases a
stale reranker_model_path line from an existing config is kept so old
files converge.

Of the three "aliased key clusters" flagged by the audit, this was the
only real drift: the three embedding-dimension keys are already one
centralized alias set in config_helpers.cpp (read any, write all), and
msl_stack_log_warn_bytes is a metrics counter name, not a config key.

portability_allowlist.txt: three init_command.cpp anchors reflowed; no
new environment reads.

The model command's post-download hint still told operators to set
search.reranker_model_path; that line goes too (the two lines above it
already name the commands that configure the reranker).

Signed-off-by: trvon <me@trevon.dev>

* fix(config): validate tuning ranges and pairs

Reject unsupported scalar and inverted paired limits before changing
runtime overrides or provenance. Preserve explicit zero ONNX reservations
with a distinct unset sentinel and avoid false default-backend conflicts.

Cover all 27 integer keys, inherited limits, and reservation reset.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-09-12 10:09:19 -04:00

349 lines
12 KiB
C++

// CLI Doctor Command tests
// Catch2 migration from GTest (yams-3s4 / yams-cli)
//
// Tests for doctor command subcommands.
#include <catch2/catch_test_macros.hpp>
#include <sqlite3.h>
#include <yams/cli/doctor/doctor_context.h>
#include <yams/cli/yams_cli.h>
#include <yams/daemon/components/TuneAdvisor.h>
#include <algorithm>
#include <filesystem>
#include <iostream>
#include <map>
#include <optional>
#include <sstream>
#include <string>
#include <vector>
// Include test helpers for ScopedEnvVar
#include "../../common/test_helpers_catch2.h"
namespace fs = std::filesystem;
using namespace yams::cli;
namespace {
/**
* RAII helper for CLI tests with proper isolation and cleanup.
*/
struct CliTestHelper {
fs::path tempDir;
std::optional<yams::test::ScopedEnvVar> configEnv;
std::optional<yams::test::ScopedEnvVar> dataEnv;
std::optional<yams::test::ScopedEnvVar> nonInteractiveEnv;
std::optional<yams::test::ScopedEnvVar> disableDaemonEnv;
CliTestHelper() {
tempDir = yams::test::make_temp_dir("yams_cli_catch2_test_");
fs::create_directories(tempDir);
configEnv.emplace("YAMS_CONFIG", (tempDir / "config.toml").string());
dataEnv.emplace("YAMS_DATA_DIR", (tempDir / "data").string());
nonInteractiveEnv.emplace(std::string("YAMS_NON_INTERACTIVE"),
std::optional<std::string>("1"));
// Disable daemon autostart to prevent cleanup crashes
disableDaemonEnv.emplace(std::string("YAMS_CLI_DISABLE_DAEMON_AUTOSTART"),
std::optional<std::string>("1"));
}
~CliTestHelper() {
configEnv.reset();
dataEnv.reset();
nonInteractiveEnv.reset();
disableDaemonEnv.reset();
std::error_code ec;
fs::remove_all(tempDir, ec);
}
std::unique_ptr<YamsCLI> makeCli() const { return std::make_unique<YamsCLI>(); }
int runCommand(const std::vector<std::string>& args) {
auto cli = makeCli();
std::vector<char*> argv;
argv.reserve(args.size());
for (const auto& arg : args) {
argv.push_back(const_cast<char*>(arg.c_str()));
}
return cli->run(static_cast<int>(argv.size()), argv.data());
}
};
/**
* RAII helper to capture and restore cout.
*/
class CaptureStdout {
public:
CaptureStdout() : oldCout_(std::cout.rdbuf(buffer_.rdbuf())) {}
~CaptureStdout() { std::cout.rdbuf(oldCout_); }
std::string str() const { return buffer_.str(); }
CaptureStdout(const CaptureStdout&) = delete;
CaptureStdout& operator=(const CaptureStdout&) = delete;
private:
std::ostringstream buffer_;
std::streambuf* oldCout_;
};
} // namespace
TEST_CASE("DoctorCommand - prune help does not run doctor diagnostics", "[cli][doctor][catch2]") {
CliTestHelper helper;
// When running `yams doctor prune` without arguments (help mode),
// it should NOT execute the default doctor diagnostics (runAll)
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "prune"};
cli->run(3, const_cast<char**>(argv));
std::string output = capture.str();
// Should show prune help text
CHECK(output.find("YAMS Doctor Prune") != std::string::npos);
CHECK(output.find("Composite Categories") != std::string::npos);
// Should NOT show doctor diagnostics output
CHECK(output.find("Collecting system information") == std::string::npos);
CHECK(output.find("Daemon Status") == std::string::npos);
}
TEST_CASE("DoctorCommand - prune with category does not run doctor diagnostics",
"[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
auto cli = helper.makeCli();
// Parse args: yams doctor prune --category build-artifacts
const char* argv[] = {"yams", "doctor", "prune", "--category", "build-artifacts"};
// Note: This will fail if daemon not running, but that's OK for the test
// We're just checking it doesn't run doctor diagnostics
cli->run(5, const_cast<char**>(argv));
std::string output = capture.str();
// Should NOT show doctor diagnostics output
CHECK(output.find("Collecting system information") == std::string::npos);
}
// Note: We don't test bare "yams doctor" because it actually runs diagnostics
// which may hang or take a long time. The key fix is ensuring subcommands
// like "prune" don't trigger diagnostics.
TEST_CASE("DoctorCommand - prune help displays without errors", "[cli][doctor][catch2]") {
CliTestHelper helper;
// Test that the UI frames are built correctly
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "prune"};
// Should not throw
REQUIRE_NOTHROW(cli->run(3, const_cast<char**>(argv)));
std::string output = capture.str();
// Check for proper formatting
CHECK(output.find("YAMS Doctor Prune") != std::string::npos);
CHECK(output.find("build-artifacts") != std::string::npos);
CHECK(output.find("Usage Examples") != std::string::npos);
// Should not contain error messages (case-insensitive check)
std::string lowerOutput = output;
std::transform(lowerOutput.begin(), lowerOutput.end(), lowerOutput.begin(), ::tolower);
// Allow "error" in words like "stderr" but not standalone
bool hasError = lowerOutput.find(" error") != std::string::npos ||
lowerOutput.find("error ") != std::string::npos ||
lowerOutput.find("failed") != std::string::npos;
CHECK_FALSE(hasError);
}
TEST_CASE("DoctorCommand - dedupe help displays correctly", "[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "dedupe", "--help"};
(void)cli->run(4, const_cast<char**>(argv));
std::string output = capture.str();
// Should show dedupe options
CHECK(output.find("--mode") != std::string::npos);
CHECK(output.find("--strategy") != std::string::npos);
CHECK(output.find("--apply") != std::string::npos);
CHECK(output.find("semantic") != std::string::npos);
CHECK(output.find("--semantic-threshold") != std::string::npos);
}
TEST_CASE("DoctorCommand - repair subcommand help displays correctly", "[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "repair", "--help"};
(void)cli->run(4, const_cast<char**>(argv));
std::string output = capture.str();
// Should show repair options
CHECK(output.find("--embeddings") != std::string::npos);
CHECK(output.find("--fts5") != std::string::npos);
}
TEST_CASE("DoctorCommand - repair with no flags does not fall through to full diagnostics",
"[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
const int rc = helper.runCommand({"yams", "doctor", "repair"});
const std::string output = capture.str();
CHECK(rc == 0);
CHECK(output.find("Nothing to repair.") != std::string::npos);
CHECK(output.find("Collecting system information") == std::string::npos);
}
TEST_CASE("DoctorCommand - validate help displays correctly", "[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "validate", "--help"};
(void)cli->run(4, const_cast<char**>(argv));
std::string output = capture.str();
// Should show validate options
CHECK(output.find("--graph") != std::string::npos);
}
TEST_CASE("DoctorCommand - validate with no flags does not run full doctor",
"[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
const int rc = helper.runCommand({"yams", "doctor", "validate"});
const std::string output = capture.str();
CHECK(rc == 0);
CHECK(output.find("Nothing to validate. Use --graph.") != std::string::npos);
CHECK(output.find("Collecting system information") == std::string::npos);
}
TEST_CASE("DoctorCommand - tuning help displays correctly", "[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "tuning", "--help"};
(void)cli->run(4, const_cast<char**>(argv));
std::string output = capture.str();
// Should show tuning options
CHECK(output.find("--apply") != std::string::npos);
}
TEST_CASE("DoctorCommand - top-level help displays all subcommands", "[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
auto cli = helper.makeCli();
const char* argv[] = {"yams", "doctor", "--help"};
(void)cli->run(3, const_cast<char**>(argv));
std::string output = capture.str();
// Should list available subcommands
CHECK(output.find("prune") != std::string::npos);
CHECK(output.find("dedupe") != std::string::npos);
CHECK(output.find("repair") != std::string::npos);
}
TEST_CASE("DoctorCommand - plugin missing target fails cleanly without full doctor fallback",
"[cli][doctor][catch2]") {
CliTestHelper helper;
CaptureStdout capture;
const int rc = helper.runCommand({"yams", "doctor", "plugin", "definitely-not-a-real-plugin"});
const std::string output = capture.str();
CHECK(rc == 0);
CHECK(output.find("Plugin Doctor: definitely-not-a-real-plugin") != std::string::npos);
CHECK(output.find("Not found as path or name in default dirs") != std::string::npos);
CHECK(output.find("Collecting system information") == std::string::npos);
}
TEST_CASE("DoctorCommand - reports non-vec0 vector schema with fix hint",
"[cli][doctor][vec0][catch2]") {
CliTestHelper helper;
const fs::path dataDir = helper.tempDir / "data";
fs::create_directories(dataDir);
const fs::path dbPath = dataDir / "vectors.db";
sqlite3* db = nullptr;
REQUIRE(sqlite3_open(dbPath.string().c_str(), &db) == SQLITE_OK);
char* err = nullptr;
REQUIRE(sqlite3_exec(db,
"CREATE TABLE doc_embeddings(rowid INTEGER PRIMARY KEY, embedding BLOB)",
nullptr, nullptr, &err) == SQLITE_OK);
if (err) {
sqlite3_free(err);
}
sqlite3_close(db);
CaptureStdout capture;
const int rc = helper.runCommand({"yams", "doctor"});
const std::string output = capture.str();
CHECK(rc == 0);
CHECK(output.find("doc_embeddings table not using vec0 virtual table") != std::string::npos);
CHECK(output.find("yams doctor --recreate-vectors --stop-daemon") != std::string::npos);
}
TEST_CASE("DoctorCommand - tuning baseline proposes the daemon's own defaults",
"[cli][doctor][tuning][catch2]") {
using yams::daemon::TuneAdvisor;
std::ostringstream os;
yams::cli::doctor::DoctorContext::applyTuningBaseline(os, false);
std::map<std::string, std::string> proposed;
std::istringstream lines(os.str());
std::string line;
while (std::getline(lines, line)) {
const auto eq = line.find(" = ");
if (line.rfind(" tuning.", 0) == 0 && eq != std::string::npos) {
proposed[line.substr(2, eq - 2)] = line.substr(eq + 3);
}
}
REQUIRE_FALSE(proposed.empty());
// A doctor baseline that disagrees with TuneAdvisor rewrites defaults as overrides.
CHECK(proposed.at("tuning.worker_poll_ms") == std::to_string(TuneAdvisor::workerPollMs()));
CHECK(proposed.at("tuning.pool_cooldown_ms") == std::to_string(TuneAdvisor::poolCooldownMs()));
CHECK(proposed.at("tuning.pool_ipc_min") == std::to_string(TuneAdvisor::poolMinSizeIpc()));
CHECK(proposed.at("tuning.pool_ipc_max") == std::to_string(TuneAdvisor::poolMaxSizeIpc()));
CHECK(proposed.at("tuning.pool_io_min") == std::to_string(TuneAdvisor::poolMinSizeIpcIo()));
CHECK(proposed.at("tuning.pool_io_max") == std::to_string(TuneAdvisor::poolMaxSizeIpcIo()));
CHECK(proposed.at("tuning.idle_shrink_hold_ms") ==
std::to_string(TuneAdvisor::idleShrinkHoldMs()));
CHECK(proposed.at("tuning.backpressure_read_pause_ms") ==
std::to_string(TuneAdvisor::backpressureReadPauseMs()));
}