* test(memory-sync): stop the sync service off the apply lock in stage hooks
Two integration tests stop the sync service from a stage observer to
simulate a stop during apply or backfill. The observer runs under
ServiceManager's apply lock, and the blocking stop() then takes the
service's callback lock, the reverse of the worker's order (callback
lock, then apply lock in applyMemorySyncWinners). TSan on a gcc Debug
build reports a lock-order inversion in daemon_memory_sync_apply_stop and
daemon_memory_sync_backfill_stop. stop() also joins the worker, which can
be blocked on that apply lock, so the test can hang.
Production shutdown calls stop() without the apply lock, so this is a
test hazard only. StopFromStage runs stop() on a helper thread, returns
once stopRequested() is visible, and joins after the stage call returns.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep the memory-sync status request alive until the call runs (#353)
fetchMemorySyncStatus passed a temporary MemorySyncRequest to
DaemonClient::call(), which is a coroutine taking the request by
reference, and returned the awaitable from the provider lambda.
runDaemonClient awaits it only after the lambda returns, so the call
read a destroyed request.
Under TSan on a gcc Debug build, `yams daemon status -d` aborted while
copying the request into the Request variant: "requested allocation size
0x7ffed4d77a51" (a stack address read as the key length). That failed
integration_smoke. ASan does not flag it without stack-use-after-return
detection.
The provider is now a coroutine lambda that owns the request. The other
providers that return call() awaitables capture their request by value
and keep the closure alive while awaiting (awaitResultSync), so they are
not affected.
Signed-off-by: trvon <me@trevon.dev>
* fix(extraction): return plugin output as a copy taken under its lock (#354)
PluginProcess::read_stdout() and read_stderr() returned a span into the
output buffer after releasing the buffer mutex. The reader thread keeps
appending to that buffer, and a reallocation frees the storage the span
points to. JsonRpcClient::read_next_response() scans that span for a
newline, so a plugin that writes while a response is being parsed reads
freed memory.
TSan on a gcc Debug build reports the race in plugin_process
(read_stdout_loop's insert against the test's unlocked read). Both
accessors now return std::vector<std::byte>, copied under the lock.
Callers use auto or pass the result as a span, so they are unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(client): stop the write loop once close() has reset the socket (#355)
close() resets AsioConnection::socket on the strand. A write loop that
still had frames queued, or a new async_write_frame() after close, went
on to async_write(*socket, ...) through the null unique_ptr. Debug builds
with _GLIBCXX_ASSERTIONS abort there; other builds bind a reference to
null.
Seen on a gcc Debug TSan build: client_timeout_recovery aborted at
asio_connection.cpp:233, and Catch2's fatal-signal handler then
allocated inside the SIGABRT handler (TSan "signal-unsafe call") and
hung until the 60 s test timeout.
The write loop now ends with NetworkError when the socket is gone or
closed, mirroring the read loop, and the timeout path checks the socket
before closing it.
Signed-off-by: trvon <me@trevon.dev>
* test(tsan): apply the suppressions file to every test (#356)
tsan.suppressions reached only the CLI, smoke, search and services
suites, the ones whose test() env sets TSAN_OPTIONS. Every other test
ran unsuppressed. A default 'tsan' test setup now sets TSAN_OPTIONS for
all tests in a TSan build. Tests that already set it use the same value.
Two narrow rules cover false positives that a gcc TSan build reports
and clang does not:
- boost::asio::detail::ref_count_down: fetch_sub(release) then a
standalone acquire fence, which gcc TSan does not model
(PostIngestQueue, VectorIndexCoordinator, WorkCoordinator reports).
- walIndexTryHdr / walIndexWriteHdr: SQLite's lock-free WAL-index header
protocol. SQLITE_NO_TSAN is clang-only (metadata commit, symspell
commit, KG path node reports).
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): run ServiceManager teardown even when the shared FSM is stopped (#357)
ServiceManagerFsm keeps its state in tinyfsm's process-wide storage.
ServiceManager::shutdown() returned early when that state was already
ShuttingDown or Stopped, so another manager's shutdown in the same
process cancelled this one's teardown. The common case: a second daemon
that loses the instance lock destroys its never-started ServiceManager,
whose destructor runs shutdown() and drives the shared FSM to Stopped.
The running daemon's stop() then skipped every shutdown phase. Its
WorkCoordinator threads and background coroutines kept running and kept
the ServiceManager alive.
gcc TSan reported the result in daemon_catch2's concurrent-start test: a
worker still inside the FTS5 consumer read TuningSnapshotRegistry after
exit() had destroyed it (heap-use-after-free).
shutdownInvoked_ is the per-instance guard. The shared state now only
decides whether to dispatch ShutdownEvent, and completeShutdown()
already accepts a Stopped state. A new test starts a daemon, loses a
second start against it, stops the first, and checks that its
ServiceManager is released.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): leave the repair hysteresis timestamps to the run loop (#358)
stop() cleared repairBusySince_ and repairReadySince_ from the caller's
thread while the runLoop thread updates them in updateRepairHysteresis().
TSan on a gcc Debug build reports the race in
lifecycle_single_instance_release and grep_service_expectations
(daemon.cpp:1141/1142 against runLoop at 914).
Only runLoop uses these fields, and start() resets them before the loop
thread is created, so stop() no longer touches them.
Signed-off-by: trvon <me@trevon.dev>
* test(sync-indexing): scale the indexing waits with the test timeout scale (#359)
sync_indexing_catch2 waits a fixed 3 s for a freshly added document to
show up in grep, with a 1.5 s per-call timeout and a 2 s timing check.
Under gcc TSan everything runs about 10x slower, so "Sync add then
immediate grep" failed REQUIRE(found) at 3 s.
The test now gets _test_timeout_cpp_arg like the smoke tests and
multiplies those three bounds by YAMS_TEST_TIMEOUT_SCALE (1 in normal
builds, 10 under TSan).
Signed-off-by: trvon <me@trevon.dev>
* fix(build): keep the S3 plugin mapped in Linux TSan builds (#360)
storage_submodule passed every test (3167 assertions) under gcc TSan and
then crashed in exit(). The S3 plugin links its own static OpenSSL,
whose init registers atexit(OPENSSL_cleanup) from inside the plugin.
glibc ties that handler to the DSO and runs it at dlclose, but TSan's
atexit interceptor keeps it until process exit and calls into the
unmapped plugin (gdb: at_exit_callback_installed_at -> unmapped pc in
OPENSSL_cleanup of libyams_object_storage_s3.so).
Linux TSan builds now link the plugin with -z nodelete. Other builds are
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test(cli): scale the serve_delegation per-call timeout with the build (#361)
serve_delegation copies the yams CLI into a temp dir and runs it with a
fixed 10 s timeout. In an ASan build the binary is over 500 MB, and on
macOS the first exec of a freshly copied executable waits for the system
policy scan, so under load a call can exceed 10 s. The pre-push ASan
lane failed this way twice with no product change.
The script now takes --timeout-scale, and meson passes the build's test
timeout scale: 1 by default, 3 in the ASan lane, 10 under TSan. With
scale 3 the suite passes in 50 s at load average 24.
Signed-off-by: trvon <me@trevon.dev>
* test(daemon): scale the socket probe budgets in sanitizer builds (#362)
"unix_socket_io_permitted: real probe completes within budget" requires
the first probe in under 100 ms and the cached call in under 50 us. On a
gcc ASan build running tests in parallel, the first probe missed 100 ms
while the code under test was unchanged.
The budgets are now 10x under ASan or TSan, detected at compile time,
and unchanged in normal builds.
Signed-off-by: trvon <me@trevon.dev>
* build(tsan): use atomic coverage counters in TSan + coverage builds (#363)
gcov counters are plain increments. A gcc Debug build with
-Db_sanitize=thread -Db_coverage=true reported every counter that two
threads touch as a data race (`Location is global '__gcov0...'`), 32
reports in the first 43 tests and none in YAMS code.
When b_coverage is on in a TSan build, the project now adds
-fprofile-update=atomic for C and C++. Other builds are unchanged.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
5eb01f4048