trevon/yams
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
yams/tests/unit/daemon/tuning_allocation_catch2_test.cpp
Trevon 73d408a4a6
 [05/14] Consolidate configuration authority (#145) 
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays

LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays
gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as
test-only benchmark levers, but tests/scripts/report_config_key_usage.py
shows 45 of them are set by no test, no bench arm, no script, and no
doc: they were levers nobody pulled, each shadowing a SearchEngineConfig
field that benches and tests already set directly. The entire
YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene
note that graph-vector fusion is a parked lever.

Removed (CANDIDATE rows of the report, verbatim):
  YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE
  YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD
  YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION
  YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION
  YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST
  YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE
  YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS
  YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT
  YAMS_SEARCH_GRAPH_ENABLE_PATHS
  YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY
  YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS
  YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS
  YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS
  YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K
  YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE
  YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY
  YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL
  YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER
  YAMS_SEARCH_GRAPH_HOP_DECAY
  YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW
  YAMS_SEARCH_GRAPH_MAX_HOPS
  YAMS_SEARCH_GRAPH_MAX_NEIGHBORS
  YAMS_SEARCH_GRAPH_MAX_PATHS
  YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE
  YAMS_SEARCH_GRAPH_TEXT_WEIGHT
  YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS
  YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING
  YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION
  YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING
  YAMS_SEARCH_GRAPH_VECTOR_WEIGHT
  YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS
  YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY
  YAMS_SEARCH_RERANK_TOP_K
  YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE
  YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK
  YAMS_SEARCH_TIERED_MIN_CANDIDATES
  YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH
  YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS
  YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS
  YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY
  YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE
  YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK
  YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER
  YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS
  YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE

Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of
YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE
(different clamps, later block won when both were set); only the
current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by
two identical blocks and is now read once.

A test builds the environment over a recording lookup and asserts none
of the retired names is ever consulted, and that the surviving names
are consulted exactly once. production_environment_keys.txt shrinks by
45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale.

The search_latency_graph_rerank task in .vscode/tasks.json set two of the
retired overlays; those assignments are dropped so the task does not claim a
configuration the code no longer honours.

Signed-off-by: trvon <me@trevon.dev>

* refactor(config): drop unread embeddings.* keys and the no-op toggles

yams config embeddings enable/disable/tune wrote embeddings.auto_generate,
embeddings.generation_delay_ms, and embeddings.batch_size, and the
migrator seeded them (plus embeddings.auto_on_add) into every new
config.toml. No daemon code reads any of them: the post-ingest queue
always dispatches embed jobs when an embed queue exists, and the only
batch size the resolvers read is embeddings.runtime.batch_size. The
toggle was a switch wired to nothing, and yams status advised users to
flip it.

- embeddings enable and disable are removed. enable duplicated
  embeddings model <name> once its dead keys are gone; a kept no-op
  would be a shim. status no longer prints an auto-generation line and
  shows the runtime batch size when set. tune presets write
  embeddings.runtime.batch_size and drop the delay.
- config_migration no longer seeds auto_generate, auto_on_add,
  generation_delay_ms, or a top-level batch_size.
- ConfigResolver and the compatibility policy honour a legacy
  [embeddings] batch_size silently when the runtime key is absent, so
  configs in the wild get their value applied without a file rewrite (the
  migrator exits early for v3 configs, so a rewrite would never run).
- status and model hints point at embeddings model <name>.

Declared behavior change: `yams config embeddings enable|disable` are
unknown subcommands now.

Declared behavior change: the legacy [embeddings] batch_size was written by
`yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16)
but no resolver ever read it. Honouring it as a fallback applies those numbers
for the first time; the one consumer whose result moves is the doctor embedding
repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32
on a stock config. The embedding service already defaulted to 32.

Signed-off-by: trvon <me@trevon.dev>

* feat(config): add [tuning] keys for the setters only env could reach

Seventeen TuneAdvisor setters were exercised by tests but had no TOML
key, so production could reach them only through a deprecated YAMS_*
overlay: connection slot sizing, the CPU-high threshold, the ONNX
concurrency and reservation knobs, model-eviction thresholds, ingest
worker cap, store/embed channel capacities, work-coordinator threads,
and connection lifetime. AGENTS.md says tuning lives in typed config;
these were half-migrated.

Each gets a [tuning] key resolved through ConfigResolver alongside the
existing ones, with provenance recorded, and each is included in
resetConfiguredOverrides() so a reload that drops a key reverts it
(the three with dedicated reset helpers call those). The environment
names keep working as compatibility inputs; nothing new is added to the
YAMS_* surface.

A table-driven test sets all 17 keys, checks every getter, then reloads
with an empty [tuning] section and checks every value reverted.
docs/architecture/environment-overrides.md lists the keys.

The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds
in (0, 1), connection_lifetime_s at most 86400) keep their default when
handed a value outside the range. The resolver checks the same ranges first
and warns, so an ignored value is never recorded as config provenance; test
covers all three shapes.

Signed-off-by: trvon <me@trevon.dev>

* fix(config): report 0 for [tuning] integer keys instead of ignoring it

For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below
every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back
as "unset", and silently fall through to the default while the resolver
still recorded config provenance for it. applyUint32 now warns and skips
the value: nothing changes for the operator except that the mistake is
visible.

Two keys mean something by 0 and are exempt: post_ingest_threads (0 =
auto, which the config migrator seeds into every config and
`yams config` displays as "(auto)") and connection_lifetime_s (0 = no
recycling; its setter takes a signed sentinel and accepts 0, and the
environment spelling always did). Both keep their provenance and stay
silent, so a stock config produces no warning.

Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and
record no provenance, pool_ipc_max = 48 still applies, and the two
zero-valid keys apply with provenance.

Signed-off-by: trvon <me@trevon.dev>

* refactor(config): one embedding-runtime resolver, one tuning baseline

Two resolvers read the same five environment keys and three TOML keys
for the embedding runtime with different error semantics:
ConfigResolver::resolveEmbeddingConfig used the strict typed readers
(warn and keep the default), while resolveEmbeddingRuntimePolicy in
config_resolver_policies.cpp used std::stoull inside a bare catch, so
YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on
the other. The compatibility entry point is now a projection of the
typed resolution onto the EmbeddingRuntimePolicy shape that library
callers (vector, repair) still consume; it lives in ConfigResolver.cpp
because DaemonConfig is declared there. 91 lines of duplicate parsing
go away.

yams doctor tuning carried its own [tuning] baseline table and it had
drifted: worker_poll_ms 75 against the daemon default of 150,
pool_io_max sized from hardware against a default of 32, so
`doctor tuning --apply` wrote defaults as divergent overrides. The
baseline now reads TuneAdvisor's effective values, which is what
"baseline" should mean.

Tests: the compatibility policy rejects 12abc and agrees with the typed
resolver; the doctor proposal equals TuneAdvisor for every key.

Signed-off-by: trvon <me@trevon.dev>

* refactor(config): stop writing the unread search.reranker_model_path

yams init wrote both search.reranker_model (the name the daemon resolves
through ConfigResolver::resolveRerankerModel) and
search.reranker_model_path, a derived on-disk path nothing in the
daemon, search, or vector code reads; yams config search reranker
status then echoed it back as if it mattered. init no longer writes the
path key and config no longer displays it. The branch that erases a
stale reranker_model_path line from an existing config is kept so old
files converge.

Of the three "aliased key clusters" flagged by the audit, this was the
only real drift: the three embedding-dimension keys are already one
centralized alias set in config_helpers.cpp (read any, write all), and
msl_stack_log_warn_bytes is a metrics counter name, not a config key.

portability_allowlist.txt: three init_command.cpp anchors reflowed; no
new environment reads.

The model command's post-download hint still told operators to set
search.reranker_model_path; that line goes too (the two lines above it
already name the commands that configure the reranker).

Signed-off-by: trvon <me@trevon.dev>

* fix(config): validate tuning ranges and pairs

Reject unsupported scalar and inverted paired limits before changing
runtime overrides or provenance. Preserve explicit zero ONNX reservations
with a distinct unset sentinel and avoid false default-backend conflicts.

Cover all 27 integer keys, inherited limits, and reservation reset.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-09-12 10:09:19 -04:00

529 lines
24 KiB
C++
Raw Permalink Blame History

// TuneAdvisor / ResourceGovernor allocation bug tests (Catch2)
// Tests 4 bugs: integer truncation, governor cap inversion, ONNX zero shared capacity,
// stage starvation. These tests should FAIL against the current buggy code (TDD red phase).
#include <climits>
#include <cstdlib>
#include <string>
#include "../../common/env_compat.h"
#include "../../common/test_helpers_catch2.h"
#include <catch2/catch_test_macros.hpp>
#include <yams/compat/unistd.h>
#include <yams/daemon/components/ResourceGovernor.h>
#include <yams/daemon/components/TuneAdvisor.h>
using namespace yams::daemon;
namespace {
class ProfileGuard {
TuneAdvisor::Profile prev_;
public:
explicit ProfileGuard(TuneAdvisor::Profile profile) : prev_(TuneAdvisor::tuningProfile()) {
TuneAdvisor::setTuningProfile(profile);
}
~ProfileGuard() { TuneAdvisor::setTuningProfile(prev_); }
ProfileGuard(const ProfileGuard&) = delete;
ProfileGuard& operator=(const ProfileGuard&) = delete;
};
using yams::test::ScopedEnvVar;
class HwGuard {
unsigned prev_;
public:
explicit HwGuard(unsigned hw) : prev_(TuneAdvisor::hardwareConcurrency()) {
TuneAdvisor::setHardwareConcurrencyForTests(hw);
}
~HwGuard() { TuneAdvisor::setHardwareConcurrencyForTests(prev_); }
HwGuard(const HwGuard&) = delete;
HwGuard& operator=(const HwGuard&) = delete;
};
/// Reset all per-stage overrides to 0 (no override) and dynamic caps to UINT32_MAX (unset)
void resetPostIngestOverrides() {
TuneAdvisor::setPostExtractionConcurrent(0);
TuneAdvisor::setPostKgConcurrent(0);
TuneAdvisor::setPostSymbolConcurrent(0);
TuneAdvisor::setPostEntityConcurrent(0);
TuneAdvisor::setPostTitleConcurrent(0);
TuneAdvisor::setPostEmbedConcurrent(0);
TuneAdvisor::setPostExtractionConcurrentDynamicCap(UINT32_MAX);
TuneAdvisor::setPostKgConcurrentDynamicCap(UINT32_MAX);
TuneAdvisor::setPostSymbolConcurrentDynamicCap(UINT32_MAX);
TuneAdvisor::setPostEntityConcurrentDynamicCap(UINT32_MAX);
TuneAdvisor::setPostTitleConcurrentDynamicCap(UINT32_MAX);
TuneAdvisor::setPostEmbedConcurrentDynamicCap(UINT32_MAX);
TuneAdvisor::setPostIngestTotalConcurrent(0);
}
void setAllPostIngestStagesActive() {
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::Extraction, true);
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::KnowledgeGraph, true);
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::Symbol, true);
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::Entity, true);
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::Title, true);
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::Embed, true);
}
} // namespace
// =============================================================================
// Test Group A: postIngestTotalConcurrent scaling across core counts
// =============================================================================
TEST_CASE("postIngestTotalConcurrent scaling across core counts",
"[daemon][tune][allocation][catch2]") {
SECTION("Budget is monotonically non-decreasing in hardware concurrency") {
resetPostIngestOverrides();
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
for (auto profile : {TuneAdvisor::Profile::Efficient, TuneAdvisor::Profile::Balanced,
TuneAdvisor::Profile::Aggressive}) {
ProfileGuard profileGuard(profile);
uint32_t prev = 0;
for (unsigned hw : {2u, 4u, 8u, 16u}) {
HwGuard hwGuard(hw);
uint32_t budget = TuneAdvisor::postIngestTotalConcurrent();
INFO("profile=" << static_cast<int>(profile) << " hw=" << hw << " budget=" << budget
<< " prev=" << prev);
if (prev != 0) {
CHECK(budget >= prev);
}
prev = budget;
}
}
}
SECTION("Budget is monotonically non-decreasing in profile aggressiveness") {
resetPostIngestOverrides();
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
for (unsigned hw : {2u, 4u, 8u}) {
HwGuard hwGuard(hw);
uint32_t eff, bal, agg;
{
ProfileGuard g(TuneAdvisor::Profile::Efficient);
eff = TuneAdvisor::postIngestTotalConcurrent();
}
{
ProfileGuard g(TuneAdvisor::Profile::Balanced);
bal = TuneAdvisor::postIngestTotalConcurrent();
}
{
ProfileGuard g(TuneAdvisor::Profile::Aggressive);
agg = TuneAdvisor::postIngestTotalConcurrent();
}
INFO("hw=" << hw << " eff=" << eff << " bal=" << bal << " agg=" << agg);
CHECK(eff <= bal);
CHECK(bal <= agg);
}
}
SECTION("8-core Balanced budget exceeds minimum floor (BUG 1 regression)") {
resetPostIngestOverrides();
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
HwGuard hwGuard(8);
ProfileGuard profileGuard(TuneAdvisor::Profile::Balanced);
setAllPostIngestStagesActive();
const uint32_t budget = TuneAdvisor::postIngestTotalConcurrent();
INFO("hw=8 profile=Balanced budget=" << budget);
// The active-stage floor must leave one slot per configured stage after reserving host
// threads. Direct TuneAdvisor tests publish their stage assumptions explicitly.
CHECK(budget >= 6);
}
SECTION("16-core Aggressive budget is substantially larger than 2-core Efficient") {
resetPostIngestOverrides();
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
uint32_t budget2Eff = 0;
{
HwGuard hwGuard(2);
ProfileGuard profileGuard(TuneAdvisor::Profile::Efficient);
budget2Eff = TuneAdvisor::postIngestTotalConcurrent();
}
uint32_t budget16Agg = 0;
{
HwGuard hwGuard(16);
ProfileGuard profileGuard(TuneAdvisor::Profile::Aggressive);
budget16Agg = TuneAdvisor::postIngestTotalConcurrent();
}
INFO("budget2Eff=" << budget2Eff << " budget16Agg=" << budget16Agg);
CHECK(budget16Agg >= budget2Eff + 2);
}
}
// =============================================================================
// Test Group B: postIngestBudgetedConcurrency stage starvation
// =============================================================================
TEST_CASE("postIngestBudgetedConcurrency stage starvation",
"[daemon][tune][allocation][starvation][catch2]") {
SECTION("New lifecycle allocates a two-slot budget only to published stages") {
resetPostIngestOverrides();
setAllPostIngestStagesActive();
TuneAdvisor::setPostIngestTotalConcurrent(2);
TuneAdvisor::ConfiguredOverrideLifecycleLease lifecycle;
REQUIRE(lifecycle.ownsInitialization());
lifecycle.commitInitialization();
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::Extraction, true);
TuneAdvisor::setPostIngestStageActive(TuneAdvisor::PostIngestStage::KnowledgeGraph, true);
const auto budget = TuneAdvisor::testing_postIngestBudget(false);
CHECK(budget.extraction == 1);
CHECK(budget.kg == 1);
CHECK(budget.symbol == 0);
CHECK(budget.entity == 0);
CHECK(budget.title == 0);
CHECK(budget.embed == 0);
resetPostIngestOverrides();
}
SECTION("Concurrent stage owners keep activity published until the last release") {
resetPostIngestOverrides();
setAllPostIngestStagesActive();
TuneAdvisor::ConfiguredOverrideLifecycleLease lifecycle;
REQUIRE(lifecycle.ownsInitialization());
lifecycle.commitInitialization();
const auto firstOwner =
TuneAdvisor::acquirePostIngestStageActivity(TuneAdvisor::PostIngestStage::Extraction);
const auto secondOwner =
TuneAdvisor::acquirePostIngestStageActivity(TuneAdvisor::PostIngestStage::Extraction);
CHECK((TuneAdvisor::postIngestStageActiveMask() & 1u) != 0u);
TuneAdvisor::releasePostIngestStageActivity(TuneAdvisor::PostIngestStage::Extraction,
firstOwner);
CHECK((TuneAdvisor::postIngestStageActiveMask() & 1u) != 0u);
TuneAdvisor::releasePostIngestStageActivity(TuneAdvisor::PostIngestStage::Extraction,
firstOwner);
CHECK((TuneAdvisor::postIngestStageActiveMask() & 1u) != 0u);
TuneAdvisor::releasePostIngestStageActivity(TuneAdvisor::PostIngestStage::Extraction,
secondOwner);
CHECK((TuneAdvisor::postIngestStageActiveMask() & 1u) == 0u);
}
SECTION("Stale owners cannot clear activity from a newer lifecycle") {
resetPostIngestOverrides();
TuneAdvisor::PostIngestStageActivityToken staleOwner = 0;
{
TuneAdvisor::ConfiguredOverrideLifecycleLease firstLifecycle;
REQUIRE(firstLifecycle.ownsInitialization());
firstLifecycle.commitInitialization();
staleOwner = TuneAdvisor::acquirePostIngestStageActivity(
TuneAdvisor::PostIngestStage::KnowledgeGraph);
}
TuneAdvisor::ConfiguredOverrideLifecycleLease nextLifecycle;
REQUIRE(nextLifecycle.ownsInitialization());
nextLifecycle.commitInitialization();
const auto currentOwner = TuneAdvisor::acquirePostIngestStageActivity(
TuneAdvisor::PostIngestStage::KnowledgeGraph);
TuneAdvisor::releasePostIngestStageActivity(TuneAdvisor::PostIngestStage::KnowledgeGraph,
staleOwner);
CHECK((TuneAdvisor::postIngestStageActiveMask() & (1u << 1u)) != 0u);
TuneAdvisor::releasePostIngestStageActivity(TuneAdvisor::PostIngestStage::KnowledgeGraph,
currentOwner);
CHECK((TuneAdvisor::postIngestStageActiveMask() & (1u << 1u)) == 0u);
}
SECTION("All 6 stages get at least 1 slot on 8-core Balanced (BUG 4 regression)") {
resetPostIngestOverrides();
setAllPostIngestStagesActive();
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
ScopedEnvVar envExtraction("YAMS_POST_EXTRACTION_CONCURRENT", "0");
ScopedEnvVar envKg("YAMS_POST_KG_CONCURRENT", "0");
ScopedEnvVar envSymbol("YAMS_POST_SYMBOL_CONCURRENT", "0");
ScopedEnvVar envEntity("YAMS_POST_ENTITY_CONCURRENT", "0");
ScopedEnvVar envTitle("YAMS_POST_TITLE_CONCURRENT", "0");
ScopedEnvVar envEmbed("YAMS_POST_EMBED_CONCURRENT", "0");
HwGuard hwGuard(8);
ProfileGuard profileGuard(TuneAdvisor::Profile::Balanced);
uint32_t totalBudget = TuneAdvisor::postIngestTotalConcurrent();
uint32_t extraction = TuneAdvisor::postExtractionConcurrent();
uint32_t kg = TuneAdvisor::postKgConcurrent();
uint32_t symbol = TuneAdvisor::postSymbolConcurrent();
uint32_t entity = TuneAdvisor::postEntityConcurrent();
uint32_t title = TuneAdvisor::postTitleConcurrent();
uint32_t embed = TuneAdvisor::postEmbedConcurrent();
INFO("hw=8 profile=Balanced totalBudget=" << totalBudget);
INFO("alloc extraction=" << extraction << " kg=" << kg << " symbol=" << symbol << " entity="
<< entity << " title=" << title << " embed=" << embed);
// Currently totalBudget=2, only extraction=1 embed=1, rest=0. All should be >=1.
CHECK(extraction >= 1);
CHECK(kg >= 1);
CHECK(symbol >= 1);
CHECK(entity >= 1);
CHECK(title >= 1);
CHECK(embed >= 1);
}
SECTION("extraction and embed always get at least 1 when budget >= 2 (invariant)") {
resetPostIngestOverrides();
setAllPostIngestStagesActive();
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
ScopedEnvVar envExtraction("YAMS_POST_EXTRACTION_CONCURRENT", "0");
ScopedEnvVar envKg("YAMS_POST_KG_CONCURRENT", "0");
ScopedEnvVar envSymbol("YAMS_POST_SYMBOL_CONCURRENT", "0");
ScopedEnvVar envEntity("YAMS_POST_ENTITY_CONCURRENT", "0");
ScopedEnvVar envTitle("YAMS_POST_TITLE_CONCURRENT", "0");
ScopedEnvVar envEmbed("YAMS_POST_EMBED_CONCURRENT", "0");
HwGuard hwGuard(2);
ProfileGuard profileGuard(TuneAdvisor::Profile::Efficient);
uint32_t totalBudget = TuneAdvisor::postIngestTotalConcurrent();
INFO("hw=2 profile=Efficient totalBudget=" << totalBudget);
CHECK(totalBudget >= 2);
CHECK(TuneAdvisor::postExtractionConcurrent() >= 1);
CHECK(TuneAdvisor::postEmbedConcurrent() >= 1);
}
}
// =============================================================================
// Test Group C: ResourceGovernor scaling cap monotonicity
// =============================================================================
TEST_CASE("ResourceGovernor scaling cap monotonicity",
"[daemon][tune][allocation][governor][catch2]") {
SECTION("Warning caps never exceed Normal caps at 2-core Balanced (BUG 2 regression)") {
resetPostIngestOverrides();
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
HwGuard hwGuard(2);
ProfileGuard profileGuard(TuneAdvisor::Profile::Balanced);
auto& governor = ResourceGovernor::instance();
governor.testing_updateScalingCaps(ResourcePressureLevel::Normal);
auto normalCaps = governor.getScalingCaps();
governor.testing_updateScalingCaps(ResourcePressureLevel::Warning);
auto warningCaps = governor.getScalingCaps();
INFO("normalCaps ex=" << normalCaps.extractionConcurrency
<< " kg=" << normalCaps.kgConcurrency
<< " embed=" << normalCaps.embedConcurrency
<< " ingest=" << normalCaps.ingestWorkers
<< " search=" << normalCaps.searchConcurrency);
INFO("warningCaps ex=" << warningCaps.extractionConcurrency
<< " kg=" << warningCaps.kgConcurrency
<< " embed=" << warningCaps.embedConcurrency
<< " ingest=" << warningCaps.ingestWorkers
<< " search=" << warningCaps.searchConcurrency);
// Currently buggy: max(2, 1/2) = 2 > 1 for extraction/kg at small defaults
CHECK(warningCaps.ingestWorkers <= normalCaps.ingestWorkers);
CHECK(warningCaps.searchConcurrency <= normalCaps.searchConcurrency);
CHECK(warningCaps.extractionConcurrency <= normalCaps.extractionConcurrency);
CHECK(warningCaps.kgConcurrency <= normalCaps.kgConcurrency);
CHECK(warningCaps.embedConcurrency <= normalCaps.embedConcurrency);
// Restore Normal
governor.testing_updateScalingCaps(ResourcePressureLevel::Normal);
}
SECTION("Caps are monotonically non-increasing across all pressure levels") {
resetPostIngestOverrides();
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
HwGuard hwGuard(4);
ProfileGuard profileGuard(TuneAdvisor::Profile::Balanced);
auto& governor = ResourceGovernor::instance();
governor.testing_updateScalingCaps(ResourcePressureLevel::Normal);
auto normalCaps = governor.getScalingCaps();
governor.testing_updateScalingCaps(ResourcePressureLevel::Warning);
auto warningCaps = governor.getScalingCaps();
governor.testing_updateScalingCaps(ResourcePressureLevel::Critical);
auto criticalCaps = governor.getScalingCaps();
governor.testing_updateScalingCaps(ResourcePressureLevel::Emergency);
auto emergencyCaps = governor.getScalingCaps();
INFO("Normal: ingest=" << normalCaps.ingestWorkers
<< " search=" << normalCaps.searchConcurrency
<< " ex=" << normalCaps.extractionConcurrency
<< " kg=" << normalCaps.kgConcurrency
<< " embed=" << normalCaps.embedConcurrency);
INFO("Warning: ingest=" << warningCaps.ingestWorkers
<< " search=" << warningCaps.searchConcurrency
<< " ex=" << warningCaps.extractionConcurrency
<< " kg=" << warningCaps.kgConcurrency
<< " embed=" << warningCaps.embedConcurrency);
INFO("Critical: ingest=" << criticalCaps.ingestWorkers
<< " search=" << criticalCaps.searchConcurrency
<< " ex=" << criticalCaps.extractionConcurrency
<< " kg=" << criticalCaps.kgConcurrency
<< " embed=" << criticalCaps.embedConcurrency);
INFO("Emergency: ingest=" << emergencyCaps.ingestWorkers
<< " search=" << emergencyCaps.searchConcurrency
<< " ex=" << emergencyCaps.extractionConcurrency
<< " kg=" << emergencyCaps.kgConcurrency
<< " embed=" << emergencyCaps.embedConcurrency);
// Normal >= Warning
CHECK(normalCaps.ingestWorkers >= warningCaps.ingestWorkers);
CHECK(normalCaps.searchConcurrency >= warningCaps.searchConcurrency);
CHECK(normalCaps.extractionConcurrency >= warningCaps.extractionConcurrency);
CHECK(normalCaps.kgConcurrency >= warningCaps.kgConcurrency);
CHECK(normalCaps.embedConcurrency >= warningCaps.embedConcurrency);
// Warning >= Critical
CHECK(warningCaps.ingestWorkers >= criticalCaps.ingestWorkers);
CHECK(warningCaps.searchConcurrency >= criticalCaps.searchConcurrency);
CHECK(warningCaps.extractionConcurrency >= criticalCaps.extractionConcurrency);
CHECK(warningCaps.kgConcurrency >= criticalCaps.kgConcurrency);
CHECK(warningCaps.embedConcurrency >= criticalCaps.embedConcurrency);
// Critical >= Emergency
CHECK(criticalCaps.ingestWorkers >= emergencyCaps.ingestWorkers);
CHECK(criticalCaps.searchConcurrency >= emergencyCaps.searchConcurrency);
CHECK(criticalCaps.extractionConcurrency >= emergencyCaps.extractionConcurrency);
CHECK(criticalCaps.kgConcurrency >= emergencyCaps.kgConcurrency);
CHECK(criticalCaps.embedConcurrency >= emergencyCaps.embedConcurrency);
// Restore Normal
governor.testing_updateScalingCaps(ResourcePressureLevel::Normal);
}
SECTION("Warning caps never exceed Normal caps at 4-core Aggressive") {
resetPostIngestOverrides();
ScopedEnvVar envMaxThreads("YAMS_MAX_THREADS", "0");
ScopedEnvVar envPostIngest("YAMS_POST_INGEST_TOTAL_CONCURRENT", "0");
HwGuard hwGuard(4);
ProfileGuard profileGuard(TuneAdvisor::Profile::Aggressive);
auto& governor = ResourceGovernor::instance();
governor.testing_updateScalingCaps(ResourcePressureLevel::Normal);
auto normalCaps = governor.getScalingCaps();
governor.testing_updateScalingCaps(ResourcePressureLevel::Warning);
auto warningCaps = governor.getScalingCaps();
INFO("normalCaps ex=" << normalCaps.extractionConcurrency
<< " kg=" << normalCaps.kgConcurrency
<< " embed=" << normalCaps.embedConcurrency);
INFO("warningCaps ex=" << warningCaps.extractionConcurrency
<< " kg=" << warningCaps.kgConcurrency
<< " embed=" << warningCaps.embedConcurrency);
CHECK(warningCaps.ingestWorkers <= normalCaps.ingestWorkers);
CHECK(warningCaps.searchConcurrency <= normalCaps.searchConcurrency);
CHECK(warningCaps.extractionConcurrency <= normalCaps.extractionConcurrency);
CHECK(warningCaps.kgConcurrency <= normalCaps.kgConcurrency);
CHECK(warningCaps.embedConcurrency <= normalCaps.embedConcurrency);
// Restore Normal
governor.testing_updateScalingCaps(ResourcePressureLevel::Normal);
}
}
// =============================================================================
// Test Group D: ONNX shared capacity
// =============================================================================
TEST_CASE("ONNX shared capacity", "[daemon][tune][allocation][onnx][catch2]") {
SECTION("ONNX max concurrent exceeds total reserved (BUG 3 regression)") {
resetPostIngestOverrides();
// Clear programmatic ONNX overrides so defaults are used.
TuneAdvisor::setOnnxMaxConcurrent(0);
TuneAdvisor::setOnnxGlinerReserved(UINT32_MAX);
TuneAdvisor::setOnnxEmbedReserved(UINT32_MAX);
TuneAdvisor::setOnnxRerankerReserved(UINT32_MAX);
for (auto profile : {TuneAdvisor::Profile::Efficient, TuneAdvisor::Profile::Balanced,
TuneAdvisor::Profile::Aggressive}) {
ProfileGuard profileGuard(profile);
for (unsigned hw : {2u, 4u}) {
HwGuard hwGuard(hw);
uint32_t maxConc = TuneAdvisor::onnxMaxConcurrent();
uint32_t reserved = TuneAdvisor::onnxGlinerReserved() +
TuneAdvisor::onnxEmbedReserved() +
TuneAdvisor::onnxRerankerReserved();
INFO("profile=" << static_cast<int>(profile) << " hw=" << hw
<< " maxConc=" << maxConc << " reserved=" << reserved);
// Must leave at least 1 shared slot beyond total reserved
CHECK(maxConc > reserved);
}
}
}
SECTION("ONNX scaled max in TuningManager logic exceeds total reserved") {
resetPostIngestOverrides();
TuneAdvisor::setOnnxMaxConcurrent(0);
TuneAdvisor::setOnnxGlinerReserved(UINT32_MAX);
TuneAdvisor::setOnnxEmbedReserved(UINT32_MAX);
TuneAdvisor::setOnnxRerankerReserved(UINT32_MAX);
for (auto profile : {TuneAdvisor::Profile::Efficient, TuneAdvisor::Profile::Balanced,
TuneAdvisor::Profile::Aggressive}) {
ProfileGuard profileGuard(profile);
for (unsigned hw : {2u, 4u}) {
HwGuard hwGuard(hw);
uint32_t maxConc = TuneAdvisor::onnxMaxConcurrent();
uint32_t reserved = TuneAdvisor::onnxGlinerReserved() +
TuneAdvisor::onnxEmbedReserved() +
TuneAdvisor::onnxRerankerReserved();
double scale = TuneAdvisor::profileScale();
// Mirrors TuningManager::configureOnnxConcurrencyRegistry logic
// (post-fix: reserved + 1 to guarantee shared capacity):
uint32_t scaledMax = static_cast<uint32_t>(maxConc * scale);
scaledMax = std::max(scaledMax, 2u);
scaledMax = std::max(scaledMax, reserved + 1);
INFO("profile=" << static_cast<int>(profile) << " hw=" << hw << " scale=" << scale
<< " maxConc=" << maxConc << " reserved=" << reserved
<< " scaledMax=" << scaledMax);
// Must have at least 1 shared slot beyond reserved
CHECK(scaledMax > reserved);
}
}
}
}
Reference in a new issue View git blame Copy permalink