5eb01f4048* test(memory-sync): stop the sync service off the apply lock in stage hooks Two integration tests stop the sync service from a stage observer to simulate a stop during apply or backfill. The observer runs under ServiceManager's apply lock, and the blocking stop() then takes the service's callback lock, the reverse of the worker's order (callback lock, then apply lock in applyMemorySyncWinners). TSan on a gcc Debug build reports a lock-order inversion in daemon_memory_sync_apply_stop and daemon_memory_sync_backfill_stop. stop() also joins the worker, which can be blocked on that apply lock, so the test can hang. Production shutdown calls stop() without the apply lock, so this is a test hazard only. StopFromStage runs stop() on a helper thread, returns once stopRequested() is visible, and joins after the stage call returns. Signed-off-by: trvon <me@trevon.dev> * fix(cli): keep the memory-sync status request alive until the call runs (#353) fetchMemorySyncStatus passed a temporary MemorySyncRequest to DaemonClient::call(), which is a coroutine taking the request by reference, and returned the awaitable from the provider lambda. runDaemonClient awaits it only after the lambda returns, so the call read a destroyed request. Under TSan on a gcc Debug build, `yams daemon status -d` aborted while copying the request into the Request variant: "requested allocation size 0x7ffed4d77a51" (a stack address read as the key length). That failed integration_smoke. ASan does not flag it without stack-use-after-return detection. The provider is now a coroutine lambda that owns the request. The other providers that return call() awaitables capture their request by value and keep the closure alive while awaiting (awaitResultSync), so they are not affected. Signed-off-by: trvon <me@trevon.dev> * fix(extraction): return plugin output as a copy taken under its lock (#354) PluginProcess::read_stdout() and read_stderr() returned a span into the output buffer after releasing the buffer mutex. The reader thread keeps appending to that buffer, and a reallocation frees the storage the span points to. JsonRpcClient::read_next_response() scans that span for a newline, so a plugin that writes while a response is being parsed reads freed memory. TSan on a gcc Debug build reports the race in plugin_process (read_stdout_loop's insert against the test's unlocked read). Both accessors now return std::vector<std::byte>, copied under the lock. Callers use auto or pass the result as a span, so they are unchanged. Signed-off-by: trvon <me@trevon.dev> * fix(client): stop the write loop once close() has reset the socket (#355) close() resets AsioConnection::socket on the strand. A write loop that still had frames queued, or a new async_write_frame() after close, went on to async_write(*socket, ...) through the null unique_ptr. Debug builds with _GLIBCXX_ASSERTIONS abort there; other builds bind a reference to null. Seen on a gcc Debug TSan build: client_timeout_recovery aborted at asio_connection.cpp:233, and Catch2's fatal-signal handler then allocated inside the SIGABRT handler (TSan "signal-unsafe call") and hung until the 60 s test timeout. The write loop now ends with NetworkError when the socket is gone or closed, mirroring the read loop, and the timeout path checks the socket before closing it. Signed-off-by: trvon <me@trevon.dev> * test(tsan): apply the suppressions file to every test (#356) tsan.suppressions reached only the CLI, smoke, search and services suites, the ones whose test() env sets TSAN_OPTIONS. Every other test ran unsuppressed. A default 'tsan' test setup now sets TSAN_OPTIONS for all tests in a TSan build. Tests that already set it use the same value. Two narrow rules cover false positives that a gcc TSan build reports and clang does not: - boost::asio::detail::ref_count_down: fetch_sub(release) then a standalone acquire fence, which gcc TSan does not model (PostIngestQueue, VectorIndexCoordinator, WorkCoordinator reports). - walIndexTryHdr / walIndexWriteHdr: SQLite's lock-free WAL-index header protocol. SQLITE_NO_TSAN is clang-only (metadata commit, symspell commit, KG path node reports). Signed-off-by: trvon <me@trevon.dev> * fix(daemon): run ServiceManager teardown even when the shared FSM is stopped (#357) ServiceManagerFsm keeps its state in tinyfsm's process-wide storage. ServiceManager::shutdown() returned early when that state was already ShuttingDown or Stopped, so another manager's shutdown in the same process cancelled this one's teardown. The common case: a second daemon that loses the instance lock destroys its never-started ServiceManager, whose destructor runs shutdown() and drives the shared FSM to Stopped. The running daemon's stop() then skipped every shutdown phase. Its WorkCoordinator threads and background coroutines kept running and kept the ServiceManager alive. gcc TSan reported the result in daemon_catch2's concurrent-start test: a worker still inside the FTS5 consumer read TuningSnapshotRegistry after exit() had destroyed it (heap-use-after-free). shutdownInvoked_ is the per-instance guard. The shared state now only decides whether to dispatch ShutdownEvent, and completeShutdown() already accepts a Stopped state. A new test starts a daemon, loses a second start against it, stops the first, and checks that its ServiceManager is released. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): leave the repair hysteresis timestamps to the run loop (#358) stop() cleared repairBusySince_ and repairReadySince_ from the caller's thread while the runLoop thread updates them in updateRepairHysteresis(). TSan on a gcc Debug build reports the race in lifecycle_single_instance_release and grep_service_expectations (daemon.cpp:1141/1142 against runLoop at 914). Only runLoop uses these fields, and start() resets them before the loop thread is created, so stop() no longer touches them. Signed-off-by: trvon <me@trevon.dev> * test(sync-indexing): scale the indexing waits with the test timeout scale (#359) sync_indexing_catch2 waits a fixed 3 s for a freshly added document to show up in grep, with a 1.5 s per-call timeout and a 2 s timing check. Under gcc TSan everything runs about 10x slower, so "Sync add then immediate grep" failed REQUIRE(found) at 3 s. The test now gets _test_timeout_cpp_arg like the smoke tests and multiplies those three bounds by YAMS_TEST_TIMEOUT_SCALE (1 in normal builds, 10 under TSan). Signed-off-by: trvon <me@trevon.dev> * fix(build): keep the S3 plugin mapped in Linux TSan builds (#360) storage_submodule passed every test (3167 assertions) under gcc TSan and then crashed in exit(). The S3 plugin links its own static OpenSSL, whose init registers atexit(OPENSSL_cleanup) from inside the plugin. glibc ties that handler to the DSO and runs it at dlclose, but TSan's atexit interceptor keeps it until process exit and calls into the unmapped plugin (gdb: at_exit_callback_installed_at -> unmapped pc in OPENSSL_cleanup of libyams_object_storage_s3.so). Linux TSan builds now link the plugin with -z nodelete. Other builds are unchanged. Signed-off-by: trvon <me@trevon.dev> * test(cli): scale the serve_delegation per-call timeout with the build (#361) serve_delegation copies the yams CLI into a temp dir and runs it with a fixed 10 s timeout. In an ASan build the binary is over 500 MB, and on macOS the first exec of a freshly copied executable waits for the system policy scan, so under load a call can exceed 10 s. The pre-push ASan lane failed this way twice with no product change. The script now takes --timeout-scale, and meson passes the build's test timeout scale: 1 by default, 3 in the ASan lane, 10 under TSan. With scale 3 the suite passes in 50 s at load average 24. Signed-off-by: trvon <me@trevon.dev> * test(daemon): scale the socket probe budgets in sanitizer builds (#362) "unix_socket_io_permitted: real probe completes within budget" requires the first probe in under 100 ms and the cached call in under 50 us. On a gcc ASan build running tests in parallel, the first probe missed 100 ms while the code under test was unchanged. The budgets are now 10x under ASan or TSan, detected at compile time, and unchanged in normal builds. Signed-off-by: trvon <me@trevon.dev> * build(tsan): use atomic coverage counters in TSan + coverage builds (#363) gcov counters are plain increments. A gcc Debug build with -Db_sanitize=thread -Db_coverage=true reported every counter that two threads touch as a data race (`Location is global '__gcov0...'`), 32 reports in the first 43 tests and none in YAMS code. When b_coverage is on in a TSan build, the project now adds -fprofile-update=atomic for C and C++. Other builds are unchanged. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|---|---|---|
| .. | ||
| meson.build | ||
| README.md | ||
| s3_plugin.cpp | ||
S3 Object Storage Plugin (Experimental)
Overview
- Provides an S3/R2-compatible object storage backend via the YAMS plugin system.
- Implements basic object operations (PUT/GET/HEAD/DELETE) with AWS Signature V4.
- Adds list() using S3 ListObjectsV2.
- Includes a first-cut multipart upload (initiate/upload-part/complete/abort).
Status
- Experimental: APIs and behavior may change. Not all S3 features are covered.
- Multipart: first-cut implementation with optional checksum and SSE‑KMS headers.
- list(): implements ListObjectsV2 with prefix and delimiter parsing; bounded pagination.
Provider validation status
- Cloudflare R2: tested in this repository (smoke + storage benchmark path).
- AWS S3: supported by the same S3-compatible plugin/configuration path, but not currently validated by automated tests in this repository.
Configuration
- BackendConfig.type: "s3"
- BackendConfig.url: "s3:///"
- BackendConfig.region: AWS region (e.g., "us-east-1"), or "auto" for R2.
- BackendConfig.usePathStyle: true to force path-style addressing.
- Credentials (config.credentials map or environment):
- access_key / secret_key / session_token
- endpoint: custom endpoint host (e.g., ".r2.cloudflarestorage.com")
Environment Variables
- AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN
- AWS_REGION (used by the smoke test; region is otherwise taken from BackendConfig)
- S3_TEST_BUCKET, S3_TEST_ENDPOINT, S3_TEST_USE_PATH_STYLE=1 (smoke test)
Build Notes
- Requires libcurl and OpenSSL development libraries present at build time.
- Target name: yams_object_storage_s3
Testing
- Unit coverage:
meson test -C build/debug storage_submodule --print-errorlogs(includes signer and storage submodule tests). - Optional network smoke test:
meson test -C build/debug s3_plugin_smoke --print-errorlogs. - The smoke test requires the environment variables above and performs a PUT/GET/HEAD/LIST/DELETE round-trip.
- Helper script:
scripts/dev/run_s3_plugin_smoke.sh [build/debug].
Example (R2)
export AWS_ACCESS_KEY_ID="<r2-access-key>"
export AWS_SECRET_ACCESS_KEY="<r2-secret-key>"
export AWS_REGION="auto"
export S3_TEST_BUCKET="<bucket>"
export S3_TEST_ENDPOINT="<accountid>.r2.cloudflarestorage.com"
export S3_TEST_USE_PATH_STYLE=0
meson test -C build/debug s3_plugin_smoke --print-errorlogs
Example (AWS S3, supported path)
export AWS_ACCESS_KEY_ID="<aws-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<aws-secret-access-key>"
export AWS_REGION="us-east-1"
export S3_TEST_BUCKET="<bucket>"
export S3_TEST_ENDPOINT="s3.us-east-1.amazonaws.com"
export S3_TEST_USE_PATH_STYLE=0
meson test -C build/debug s3_plugin_smoke --print-errorlogs
Note: set S3_TEST_ENDPOINT to host only. Do not include https:// or /<bucket>.
Note: Cloudflare API bearer tokens are not S3 credentials; use an R2 S3 Access Key ID + Secret
Access Key pair from R2 -> Manage R2 API tokens.
If you only have a Cloudflare API token, create temporary S3 credentials via
POST /accounts/{account_id}/r2/temp-access-credentials and export returned
accessKeyId, secretAccessKey, and sessionToken as AWS env vars.
Security
- Credentials are taken from config and/or standard AWS environment variables.
- Signing uses AWS SigV4; verify environment and endpoint correctness for non-AWS providers.
Enhancements in this slice
- Stronger URI encoding for keys and canonicalization.
- Prefix-aware listing with delimiter support using XML parsing of ListObjectsV2.
- Optional checksum/SSE headers for PUT and multipart flows (signed):
checksumAlgorithm = "sha256"addsx-amz-checksum-sha256.sseKmsKeyIdaddsx-amz-server-side-encryption: aws:kmsand key id.storageClasssetsx-amz-storage-class.
Prefix-aware listing and prefixes
- Core API (
IStorageBackend::list): returns a flat list of keys for a given prefix. - Plugin API (
object_storage_v1via adapter): acceptsdelimiter,pageToken,maxKeysinopts_jsonand returns JSON:{ "items": ["key1", ...], "nextPageToken": "...", "prefixes": ["dir1/", ...] }.- Not all backends populate
prefixes; clients can infer prefixes from keys when unset.
Signer header coverage
- The SigV4 signer includes optional headers in the signed set when provided (SSE‑KMS, storage class, checksum).
Limitations / Future Work
- Canonical query/signing consolidation across signer and callers.
- SSE‑C, CRC32C, compose, versioning, presign URLs, and robust pagination APIs.
- DR presence and replication hooks aligned with
object_storage_v1interface.