The system-service entries were superseded by the per-user unit in the same stack, one ci fix was listed twice, and the promotion merge is not a change.
Signed-off-by: trvon <me@trevon.dev>
Switching the main test user's embedding backend to onnxruntime (no
model installed) and back left state behind that the later phases then
tripped over. Probe the runtime with a third user whose own config
enables plugin autoload with the auto backend (config_version = 3 so
the daemon does not migrate it to the simeon default), and keep the main
user on yams init's configuration throughout.
Signed-off-by: trvon <me@trevon.dev>
A CompressedStorageEngine wrapping an engine without a base path used a
fixed /tmp/yams-compressed-wrapper as its own base, and the StorageEngine
base creates objects/ and temp/ there. With the daemon now running as a
systemd user unit for each user (UMask=0077), whichever user's daemon
started first owned that directory, and every other user's daemon with
compression enabled (the yams init default) failed at startup with
"Failed to create storage directory: Permission denied" and never became
ready. Found by the package validation: two lingering users, reboot, the
second daemon stuck initializing.
Suffix the scratch directory with the real UID on POSIX.
Signed-off-by: trvon <me@trevon.dev>
First runs of the user-unit validation on all four lanes found harness
gaps, fixed here:
- Arch's `su -` does not run pam_systemd, so the login shell had no
XDG_RUNTIME_DIR and the CLI looked for /tmp/yams-daemon-<uid>.sock;
the checks now provide the runtime dir a real login session gets and
report whether su set it.
- Ubuntu 24.04's systemctl --user needs the user D-Bus bus: the
Debian/Ubuntu lane image gains dbus-user-session (standard on real
installs) and the checks pass the bus address when present.
- CLI commands fall back to an in-process daemon instead of spawning
one, so the lifecycle check uses `yams daemon start` (which starts the
unit) rather than expecting an ordinary command to start it.
- `yams init`'s default embedder is the built-in simeon backend, which
skips the onnx model provider by design. The onnx phase now checks
that init's config autoloads plugins, then switches [embeddings]
backend to onnxruntime to prove the bundled runtime loads (from the
user journal), and restores init's config so the later phases run the
user's real configuration.
docs/install.md no longer claims ordinary CLI commands start the unit:
`yams daemon start` and on-demand starters do.
Signed-off-by: trvon <me@trevon.dev>
The package validation now exercises the per-user service:
- service: the user unit and user preset are installed, the unit is
enabled for all users (systemctl --global), no system unit remains,
and the unit does not hide the user's home or /tmp.
- user: a normal user with lingering gets a user manager; the unit is
enabled and active without a manual start and `systemd-analyze --user
verify` is clean; the socket is in $XDG_RUNTIME_DIR and owned by the
user; through a login shell with no YAMS_* variables the CLI reports
status, adds from $HOME and /tmp, searches, stores in
~/.local/share/yams and logs to ~/.local/state/yams, and the only
daemon is the unit's MainPID; `yams daemon stop`, CLI auto-start and
`yams daemon restart` go through systemd; `yams daemon install --user`
enables the packaged unit without writing an override; a second user
cannot reach the first one's daemon.
- onnx: after `yams init` the restarted user daemon logs "Using bundled
ONNX Runtime" and `yams plugin health` lists the onnx plugin.
- persist: with lingering the user's daemon is back after a reboot with
the data intact.
- upgrade from 0.20.3: the old system unit is stopped, removed and
leaves no enablement links, /var/lib/yams is left alone, the user unit
is enabled for all users and the user's daemon and CLI work.
- remove: running user daemons are stopped, global enablement is gone,
no daemon process is left, user corpora survive (deb: also purge).
- minimal: bare base images still install with declared dependencies
only; the account checks are gone with the yams account.
The Fedora lane image replaces the systemd-user PAM stack, which cannot
run in a container (unix_chkpwd, pam_selinux, pam_loginuid), with a
minimal one that still runs pam_systemd, so user@.service starts.
Signed-off-by: trvon <me@trevon.dev>
The package scripts only addressed user managers that had a D-Bus user
bus socket (/run/user/<uid>/bus). Debian and Ubuntu without
dbus-user-session have none, so upgrades did not reload or restart running
user daemons and removal left them running on deleted binaries. Detect a
manager by its private socket (/run/user/<uid>/systemd/private), which
systemctl --user uses directly, and pass only XDG_RUNTIME_DIR.
Signed-off-by: trvon <me@trevon.dev>
The generated config.toml (yams init) set daemon.socket_path =
"/tmp/yams-daemon.sock" and pid_file = "/tmp/yams-daemon.pid". Both are
the same for every user on a host, so with the packaged systemd user unit
two users' daemons fought for one socket, and an initialized user's
daemon left $XDG_RUNTIME_DIR for a world-visible /tmp path. Write them
empty so the CLI and the daemon fall back to the per-user defaults
($XDG_RUNTIME_DIR/yams-daemon.sock, /tmp/yams-daemon-<uid>.sock without a
runtime dir). Configs written before keep their explicit values.
Signed-off-by: trvon <me@trevon.dev>
With the daemon shipped as a systemd user unit, the CLI must not fight
systemd for it:
- Auto-start (DaemonClient::startDaemon): when the unit is installed but
not running and would run exactly the daemon the client expects (same
socket, data dir and config as the default resolution, default
binary), start it with `systemctl --user start` instead of forking a
copy that a later login would race for the socket. Flag or
environment overrides (--socket, --data-dir, YAMS_DAEMON_SOCKET,
YAMS_DATA_DIR, YAMS_CONFIG, YAMS_DAEMON_BIN) keep the fork path.
- `yams daemon start|stop|restart` use `systemctl --user` for an
installed/active unit (start/restart without custom path options,
stop/restart without --socket), falling back to IPC if systemctl
fails.
- `yams daemon install --user` with no options enables the packaged unit
(`systemctl --user enable --now`) instead of writing a copy into
~/.config/systemd/user that would shadow it; with options it still
writes that override and says so. `uninstall --user` reloads the
manager after removing an override and explains how to mask the
globally enabled packaged unit for one account.
The decision is a pure function (userUnitMatchesPaths) with Catch2
coverage; systemctl is only run with whitelisted verbs and only when a
user manager is reachable ($XDG_RUNTIME_DIR set, Linux).
Signed-off-by: trvon <me@trevon.dev>
The service is now user-bound: each user runs their own daemon with
their own config ($XDG_CONFIG_HOME/yams/config.toml), corpus
($XDG_DATA_HOME/yams) and socket ($XDG_RUNTIME_DIR/yams-daemon.sock),
which is exactly what the CLI already resolves. The system service, its
yams account/group, the group socket and the /etc/yams/config.toml
configuration go away.
- Unit: /usr/lib/systemd/user/yams-daemon.service runs
`yams-daemon --foreground` with default path resolution,
Restart=on-failure, WantedBy=default.target, and light hardening
(NoNewPrivileges, LockPersonality, RestrictRealtime, RestrictSUIDSGID,
native syscalls only). No ProtectHome/PrivateTmp: the daemon reads the
files its user adds.
- Enablement: /usr/lib/systemd/user-preset/80-yams.preset plus
`systemctl --global preset` on first install (and on upgrade from a
release with the system unit), so the unit starts with every user's
next session; administrators override with a user preset or
`systemctl --global disable`, and later upgrades keep that choice.
Package scripts never start per-user instances: they reload running
user managers, try-restart running user daemons on upgrade and stop
them on removal (`systemctl --user` via runuser per /run/user/<uid>),
then `systemctl --global disable` on removal.
- Migration from 0.20.x and the system-service design: deb preinst, rpm
%pre and Arch pre_upgrade stop and disable the old system unit while
its file is still installed (deb also purges deb-systemd-helper
state). /var/lib/yams, the old system corpus that the CLI never
reached, is left alone and documented; user corpora are never
touched.
- The sysusers entry, account creation and the deb's passwd and
init-system-helpers dependencies are dropped; the deb purge no longer
deletes /var/lib/yams.
- Install paths move to lib/systemd/user and lib/systemd/user-preset in
meson, the Arch stage/PKGBUILD and the install-surface policy.
- docs/install.md: the user service, enabling it now
(`systemctl --user enable --now`), lingering, plugins via the user's
config, opting out, upgrade and removal behaviour, and the 0.20.3
leftover.
Signed-off-by: trvon <me@trevon.dev>
The packaged service becomes a systemd user unit that runs each user's
own daemon on the CLI's default per-user socket, with that user's
config and data. There is no system daemon left to discover, and the
daemon always runs as the same user as the CLI, so both workarounds for
the system-service design go:
- Revert "feat(cli): find the packaged system daemon socket without
configuration" (client-side /run/yams/yams-daemon.sock discovery, the
systemd refusal in `daemon stop|restart`, the spawn refusal and the
status mismatch suppression, plus its tests).
- Revert "fix(cli): send file content to a daemon that runs as another
user" (peer-UID probe and content upload in `yams add`).
Per-user socket resolution is back to: explicit flag/env/config, then
$XDG_RUNTIME_DIR/yams-daemon.sock (or the /tmp fallback).
Signed-off-by: trvon <me@trevon.dev>
The packaged service had no configuration file it would read: with
HOME=/var/lib/yams it looked for /var/lib/yams/.config/yams/config.toml,
which nothing creates. Plugins load only with [daemon]
auto_load_plugins = true, so the onnx and glint plugins and the bundled
ONNX Runtime shipped in the packages were never used by the service
("No plugins loaded"), and an administrator had no obvious place to turn
them on.
The unit now sets YAMS_CONFIG=/etc/yams/config.toml (the package ships
no file; ProtectSystem=strict leaves /etc readable). With
auto_load_plugins = true there, the sandboxed daemon loads all three
plugins and logs "Using bundled ONNX Runtime 1.23.0 from
/usr/lib/yams/onnxruntime/libonnxruntime.so.1 (no system copy found)",
ready=true. docs/install.md describes the file and how to check which
runtime was loaded; the validation's onnx phase writes it, restarts the
unit and checks the bundled runtime and plugin health.
Signed-off-by: trvon <me@trevon.dev>
`meson install` of a release build with --with-onnx failed:
ERROR: Tried to install symlink to missing file
.../usr/lib/yams/onnxruntime/libonnxruntime.so.1
Meson installs symlinks in declaration order and refuses one whose target
is not installed yet. list-onnxruntime-libs.py printed them in name order,
so libonnxruntime.so -> .so.1 came before .so.1 -> .so.1.23.0. Emit each
chain from the real file outwards (by link depth). Found by building the
packages the release way (setup.sh Release --with-onnx, meson install).
Signed-off-by: trvon <me@trevon.dev>
The lane images boot systemd and carry the tools the checks use, so
systemd-sysusers always existed and the deb postinst's broken `adduser`
fallback was never exercised.
Each lane now also installs the package in the raw distro image
(debian:trixie-slim, ubuntu:24.04, fedora:42, the Arch base) with no
systemd running and nothing preinstalled, so only declared dependencies
are present. It checks that the install finishes (deb: dpkg status
"install ok installed", not half-configured), that the yams account and
group exist, and that `yams add` + `yams search` work without the
service.
A new onnx phase checks, on the systemd lane, that the package ships the
private runtime in <libdir>/yams/onnxruntime and the onnx plugin, that
the system daemon logs "Using bundled ONNX Runtime", and that
`yams plugin health` lists the onnx plugin.
Signed-off-by: trvon <me@trevon.dev>
The deb postinst created the account with systemd-sysusers and fell back
to `adduser`, which the package never depended on. In a minimal Debian or
Ubuntu container (no systemd, no adduser) installation stopped in
postinst with "adduser: not found" and left the package half-configured.
The postinst still prefers the shipped sysusers.d entry, then falls back
to groupadd/useradd from `passwd`, now a declared dependency (Priority:
required, so it costs nothing). If neither works it prints what to do,
skips enabling the service and exits 0: the CLI does not need the account,
so a missing account no longer breaks the install. The rpm %post gets the
same groupadd/useradd fallback when systemd-sysusers cannot run; Arch
already relies on pacman's sysusers hook plus an explicit
systemd-sysusers call.
Signed-off-by: trvon <me@trevon.dev>
The resolution tests used ELF names (libonnxruntime.so.1) everywhere,
but the policy only treats .dylib files as runtime libraries on macOS,
so the macOS test lane would have failed. Fixtures now use the host's
names (libonnxruntime.1.dylib / libonnxruntime.1.23.0.dylib on macOS)
and build paths with fs::path; the test is not registered on Windows,
where resolution is the DLL search for onnxruntime.dll.
grim (Linux, Debug): 7/7 cases, 34 assertions pass.
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin cached compiled CoreML models directly in the model
directory. With the plugin now able to run on either the system or the
bundled ONNX Runtime, a cache compiled by one version was reused by the
other: on macOS, after a run on Homebrew's 1.30 the bundled 1.23 failed
every embedding batch with 'Non-zero status code returned while running
..._CoreML_... node'. The default cache is now
<model dir>/coreml-ort-<runtime version>; YAMS_COREML_CACHE_DIR still
overrides it.
Verified on macOS arm64: a run on the system runtime creates
coreml-ort-1.30.0, a run with Homebrew's copy hidden uses the bundled
runtime, creates coreml-ort-1.23.0 and embeds new documents without
CoreML errors.
Signed-off-by: trvon <me@trevon.dev>
On macOS meson.build puts the Homebrew onnxruntime prefix first when
normalizing ONNX Runtime linkage, so the private bundle copied
/opt/homebrew/opt/onnxruntime/lib even in a Conan build. That dylib links
other Homebrew libraries (abseil, protobuf, re2) by absolute path, so the
bundled copy would not load on a Mac without them. Bundle only from the
dependency's own libdir (pkg-config/CMake), skipping Homebrew prefixes;
the Homebrew formula uses Homebrew's onnxruntime directly.
Verified on macOS arm64: the bundle source changes from
/opt/homebrew/opt/onnxruntime/lib to the Conan package libdir.
Signed-off-by: trvon <me@trevon.dev>
rpm's dependency generator advertised every ELF library in the package
as a system soname, including the bundled ONNX Runtime
(libonnxruntime.so.1()(64bit)), so the yams RPM could satisfy other
packages' ONNX Runtime dependencies, and it emitted requires on the
private libyams_onnx_resource/libzpdf it ships itself. Filter provides
from /usr/lib/yams and drop requires on those private sonames.
Signed-off-by: trvon <me@trevon.dev>
Packages now carry the private fallback ONNX Runtime in
/usr/lib/yams/onnxruntime/ (libonnxruntime.so -> .so.1 -> .so.1.x.y and
libonnxruntime_providers_shared.so). Without an allowlist entry the
layout check rejects them as unreviewed paths. The pattern admits only
those library names under that directory.
Signed-off-by: trvon <me@trevon.dev>
No package carried ONNX Runtime: tools/yams-cli tried to install_subdir()
the runtime's libdir, but read it with get_variable('libdir') from the
dependency meson.build had already normalized into a declare_dependency,
which has no variables, so it installed nothing. Packaged ONNX and Glint
plugins therefore only worked where the user had installed a compatible
runtime themselves.
meson.build now records the directories it resolved ONNX Runtime from,
and the runtime install bundles the shared library from the first that
holds one into <libdir>/yams/onnxruntime/ (install tag runtime). A small
configure-time helper (scripts/list-onnxruntime-libs.py) lists the real
files and the soname symlinks, so the chain libonnxruntime.so ->
.so.1 -> .so.1.23.0 is recreated instead of tripled, and only the core
library and providers_shared are taken (no static archives, CMake or
pkg-config files). The plugins use this copy only when no compatible
system copy exists. Windows is unchanged: onnxruntime.dll already ships
next to the executables in the MSI and is found by the loader search.
- scripts/split-debug-symbols.sh leaves the bundled runtime
byte-identical: it is an upstream release build, already stripped, and
re-stripping would void a macOS code signature.
- install_surface_policy: <libdir>/yams/onnxruntime/* is allowed and
exempt from first-party export/hardening checks; the Glint size budget
grows to 850,000 bytes (711,352 measured with real ONNX Runtime code).
- Homebrew formulas depend on Homebrew's onnxruntime, which the plugins
prefer, so they drop the archive's private copy (rm_rf
lib/yams/onnxruntime).
- Arch: optdepends now reads 'onnxruntime: use the system ONNX Runtime
instead of the bundled copy' (PKGBUILD and the CI .PKGINFO).
Measured on grim (Linux x86_64 release, --with-onnx, stripped staged
payload): 87,883,247 -> 110,236,706 bytes (+22.4 MB: the 22,207,288-byte
libonnxruntime.so.1.23.0, 14,632-byte providers_shared, and +111 KB of
real GLiNER code in yams_glint.so).
Signed-off-by: trvon <me@trevon.dev>
On Linux the Glint plugin always built in mock mode: meson found the ONNX
Runtime headers but its find_library('onnxruntime') looked in a libdir
taken from the already-normalized dependency, found nothing, and dropped
YAMS_USE_ONNX_RUNTIME, so GLiNER never ran (the v0.20.3 deb's glint has
no ONNX Runtime dependency either). On macOS it linked Homebrew's copy
directly.
Glint now uses the shared ONNX Runtime loader like the ONNX plugin: it
compiles against the headers only (ORT_API_MANUAL_INIT through
ort_cxx_api_wrapper.h), and before creating the session it resolves a
runtime with the same system-first, bundled-fallback policy. If none is
available the session reports "ONNX Runtime unavailable: ..." and entity
extraction is disabled instead of silently running the mock. Plugin
health gains an onnx_runtime object (available, source, version, path or
error), and [plugins.glint] runtime_library pins a copy.
Verified on grim (Linux, release, staged tree in a clean debian:testing
container): "Glint plugin: ONNX Runtime enabled (loaded at runtime)" at
configure, "[Glint] Loaded ONNX model with 6 inputs, 5 outputs" at run
time, and the daemon's title+NL stage extracts entities ("nl=5" for a
document naming people, companies, a place and a date).
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
On the Arch package build the executables link the system libmagic, whose
dependencies (libzstd, libz) are also linked statically from Conan. ld then
exported the private copies so libmagic would bind to them: 600+ dynamic
symbols per binary (ZSTD_*, deflate*, inflate*), interposing one library
copy into another, and scripts/ci/check_install_surface.py failed the
Arch stage (export budget 64, forbidden third-party exports). The Debian
build, without that system dependency, exported 43.
Link optimized Linux builds with --exclude-libs,ALL so symbols from static
archives never enter .dynsym. YAMS executables export nothing plugins rely
on (plugins resolve against their own dependencies and the C ABI), so this
only removes the leak.
Signed-off-by: trvon <me@trevon.dev>
The install-surface cleanup moved libyams_onnx_resource from /usr/lib to
/usr/lib/yams, matching scripts/ci/install_surface_policy.json. The
package validation allowlist only accepted libyams_* directly in the
library directory and would have failed every lane; accept both, and
nested paths under /usr/share/doc/yams like the policy does.
Signed-off-by: trvon <me@trevon.dev>
The systemd fix added lib/sysusers.d/yams.conf without an install tag, so
'meson install --tags runtime' dropped it, and the install-surface policy
did not allow the path, so the release gate would have failed on it.
Signed-off-by: trvon <me@trevon.dev>
rpm expands macros inside comments. The generated spec's comment named
%systemd_postun_with_restart, which systemd-rpm-macros defines with
required arguments, so rpmbuild on Fedora 42 (rpm 4.20 with the macros
installed) stopped with "The %systemd_postun_with_restart macro
requires some arguments". Debian and Ubuntu builders do not define the
macro, which is why release CI never saw it. Write the names as %% in
the comments, including the %postun mention in the %post scriptlet.
Checked with rpmbuild -bb of the generated spec in fedora:42 with
systemd-rpm-macros: the previous spec fails, this one builds and its
%post carries the systemd-sysusers call.
Signed-off-by: trvon <me@trevon.dev>
x86_64 runs on a loaded host showed three harness problems:
- The Debian/Ubuntu and Fedora lane images lacked pam_systemd, so
user@.service could not start and `yams daemon install --user` looked
broken. Install libpam-systemd / systemd-pam. Where a host still
cannot start a user manager in a container, the check reports INFO
instead of failing, since that says nothing about yams.
- `systemctl --user -M user@` needs machined; query the user manager as
that user instead. The "CLI uses its unit" check now also requires that
the only daemon running for the user is the unit's MainPID, so a
CLI-spawned second daemon no longer passes.
- A container reboot can sit in 'starting' past 60 s under load; wait up
to three minutes and print pending jobs if it still does not boot.
Signed-off-by: trvon <me@trevon.dev>
Running the gate against v0.20.3 and branch packages on Debian, Ubuntu,
Fedora and Arch showed harness gaps, fixed here:
- The driver ran only under bash 4+ (mapfile, empty arrays under set -u);
it now works with macOS's bash 3.2 so it runs on Docker Desktop too.
- Paths dropped by the image's dpkg path-exclude (docs) or rpm nodocs
no longer fail the permission check, and rpm's /usr/lib/.build-id links
and the package doc/license directories are on the allowlist.
- The upgrade check pins the system socket, so it measures whether the
corpus survived rather than whether an older CLI can discover it.
- Debian keeps enablement links until purge, so for deb the dangling-link
check runs after purge.
- New per-user service check: `yams daemon install --user` beside the
system unit starts a working user unit, and that user's CLI uses it.
The unit comment and install guide now describe the socket as group
read/write with no access for others (the daemon sets owner rwx, group
rw), rather than a literal 0660.
Signed-off-by: trvon <me@trevon.dev>
`yams daemon install --user` writes WorkingDirectory=<data dir> into the
unit. On an account that never ran yams, ~/.local/share/yams does not
exist yet, so systemd failed every start with 200/CHDIR and restarted
the unit forever while the command printed "[OK] ... enabled and
started". Create the directory before writing the unit and fail clearly
if that is impossible.
Found by the new per-user service check in package validation.
Signed-off-by: trvon <me@trevon.dev>
The old package-validate.sh only checked that the unit came up and that
`yams daemon status` answered with YAMS_DAEMON_SOCKET set by hand, so it
passed while nobody's CLI ever reached the system daemon, the socket was
world-writable and `yams add ~/file` failed.
package-validate.sh stays the single entry point (same --only/--deb/
--rpm/--arch flags the workflow uses) and now drives
package-validate-checks.sh inside Debian, Ubuntu, Fedora and Arch
containers booted with systemd as PID 1 (private cgroup namespace; the
host's cgroups are not shared). Per lane it checks: install with no
manual steps; the installed file list against a runtime allowlist with
no headers, static libs, pkg-config or CMake files; no world-writable,
setuid or non-root files; resolvable DT_NEEDED and $ORIGIN-or-/usr/lib
RPATH/RUNPATH; unit enabled and active as the yams account; socket group
yams and closed to others; `systemd-analyze verify` and a
`systemd-analyze security` exposure budget; root, a yams-group member
and a non-member using the CLI with no YAMS_* variables (add from $HOME
and /tmp, search, shared corpus, no private fallback store, clear
directory error, non-member denied, `daemon stop` deferring to systemd);
state after a container reboot; removal and deb purge. With
--upgrade-from-<fmt> it installs an older package, stores a document,
upgrades and checks the restart and the document. Every check reports
PASS/FAIL, a summary table follows, and any FAIL fails the run.
release.yml's existing validation steps (the gate before artifacts are
uploaded for publishing) pass the latest published release's packages as
the upgrade source. The unreferenced packaging/systemd/Dockerfile is
removed; the lane images take a BASE_IMAGE argument and add binutils
and util-linux for the checks.
Signed-off-by: trvon <me@trevon.dev>
The packaged yams-daemon.service ran under DynamicUser= and its
ExecStartPost chmod'ed /run/yams/yams-daemon.sock to 0666, so every
local account could read, add and delete documents in the system
corpus. It also used the deprecated PermissionsStartOnly= and a light
sandbox (ProtectSystem=full only).
The unit now runs as a static `yams` system account created from
/usr/lib/sysusers.d/yams.conf (deb postinst, rpm %post and the Arch
install script run systemd-sysusers before starting it). The daemon's
own socket mode (owner rwx, group rw) with Group=yams and UMask=0007
gives a group-only socket, and /run/yams, /var/lib/yams and
/var/log/yams are 0750, so access is granted by adding a user to the
`yams` group. The chmod and PermissionsStartOnly are gone.
Sandboxing: ProtectSystem=strict, ProtectHome, PrivateTmp,
PrivateDevices, empty CapabilityBoundingSet, kernel/clock/hostname/
cgroup protection, ProtectProc=invisible, RestrictAddressFamilies
(unix, inet, inet6, netlink), RestrictNamespaces, LockPersonality,
RemoveIPC and a @system-service syscall filter without @privileged and
@resources. MemoryDenyWriteExecute stays off for ONNX Runtime.
Package scripts: Arch pre_remove now disables as well as stops (it left
a dangling multi-user.target.wants link), and deb purge removes
/var/lib/yams and /var/log/yams, including the /var/{lib,log}/private
copies earlier DynamicUser= releases left. Upgrades keep the corpus:
systemd moves the DynamicUser= state back to /var/lib/yams on first
start under User=yams.
docs/install.md describes the group, the client socket order, and what
the sandboxed service can and cannot read.
Signed-off-by: trvon <me@trevon.dev>
`yams add <path>` passes the path to the daemon, which reads the file
itself. The packaged system daemon runs as its own account with
ProtectHome= and PrivateTmp=, so adding ~/notes.txt failed with
"Permission denied" and a /tmp file was accepted but never stored (the
daemon's /tmp is private).
When SO_PEERCRED/getpeereid shows the daemon's UID differs from ours,
the CLI now reads single files itself and sends their bytes (up to
15 MiB, under the 16 MiB IPC frame) named by their absolute path.
Directory adds stay daemon-side walks (include/exclude patterns and
ignore files), so for such a daemon they now fail up front with an
explanation instead of queueing a walk that finds nothing. Same-user
daemons keep the path-based flow.
Signed-off-by: trvon <me@trevon.dev>
The deb, rpm and Arch packages start yams-daemon.service on
/run/yams/yams-daemon.sock, but clients only looked at the per-user
default (XDG_RUNTIME_DIR, /var/run as root, /tmp/yams-daemon-<uid>.sock).
With no YAMS_DAEMON_SOCKET set, root and normal users silently fell back
to an in-process daemon and a private store in ~/.local/share/yams, so
the installed service was never used.
Client-side resolution now goes: an explicit socket (flag, env, config),
the per-user daemon when its socket exists or a personal data directory
was named (flag, env, core.data_dir from yams init), then the system
socket when it exists and is connectable (root or the yams group), and
finally the per-user default. Daemons keep binding what
resolve_runtime_paths() names; `yams daemon install --user` writes the
per-user socket, never the discovered one.
Because systemd owns the system unit, `yams daemon stop|restart` refuse
it with a systemctl hint and the client never tries to spawn a daemon on
that socket. `daemon status` no longer flags the system corpus as a data
directory mismatch. socket_utils also gains a peer-UID probe used by the
next change.
The two socket default tests now use the per-user resolver so they do
not depend on whether the build host runs the packaged service.
Signed-off-by: trvon <me@trevon.dev>
Meson rewrites the install name of installed dylibs to their absolute
install path and points consumers at it, so the staged macOS payload
referenced /opt/homebrew/lib/yams/libyams_onnx_resource.dylib: yams-cli
from an unzipped release archive (or a staging dir) died in dyld
("Library not loaded ... no such file"), and Homebrew depended on its
prefix symlinks to resolve it.
A darwin-only Meson install script (scripts/macos-rpath-install-names.sh,
tag runtime) restores @rpath/libyams_onnx_resource.dylib as the library
id and in yams-cli/yams-daemon and the plugins, which already carry
@executable_path/../lib/yams and @loader_path/.. RPATHs.
Verified on macOS arm64 (ONNX + Glint release build): otool -L shows
@rpath/libyams_onnx_resource.dylib; a `zip -ry` archive of the stage
unzipped to an unrelated directory runs `yams --version`, codesign -v
passes, and the Glint plugin dlopens from there.
Signed-off-by: trvon <me@trevon.dev>
On a macOS Release build staged and split with split-debug-symbols.sh,
check_install_surface reported every binary as shipping a symbol table.
The only stab left after strip -S -x is the 'OPT radr://5614542' entry
ld64 always writes; it carries no debug information. Unstripped binaries
still trip the check (135,012 stabs in an unsplit yams-cli).
Verified on macOS arm64 (ONNX + Glint build): RED with 8 false
'stripped' violations before, GREEN after.
Signed-off-by: trvon <me@trevon.dev>
meson_options.txt had 47 options. Several were read but never set by
any workflow, script or doc, several only fed test-lane timeouts, and
enable-{tsan,asan,ubsan} duplicated b_sanitize. 47 -> 33.
Removed (behavior kept at the previous default everywhere):
- require-fts5 (configure now always errors on a SQLite without FTS5,
the old default), conan-generators-dir, cli11-cmake-path,
onnxruntime-cmake-path (Conan/CMake discovery under the build dir is
unchanged), split-debug.
- unit-timeout, integration-timeout, integration-ui-timeout,
integration-smoke-timeout, integration-daemon-timeout and stress-iters:
replaced by the same constants at their call sites (smoke keeps its
test-timeout-scale multiplier). Only meson/profiles/ci-fast.ini and
nightly-soak.ini set them, and nothing used those profiles; both go.
- enable-integration-services/-smoke and enable-daemon-integration:
folded into enable-integration-tests. Debug builds still register every
integration suite; nothing set the sub-toggles.
Kept, because third_party/symspell's meson.build reads them:
bench-timeout, enable-tsan and enable-asan. YAMS itself now derives all
sanitizer handling (extra flags, TSan runtime links, test timeouts) from
b_sanitize; setup.sh still mirrors its choice into enable-tsan/asan for
the submodule, and the CI ASan lane keeps -Denable-asan=true for the
same reason. enable-ubsan is a deprecated no-op (a warning if it
disagrees with b_sanitize) so build dirs configured with it still
regenerate; the UBSan lane passes only -Db_sanitize=undefined and still
gets -fno-sanitize-recover=all (standalone UBSan aborts; ASan+UBSan stays
recoverable, as before).
Conan: the without_* Boost list was applied only to iOS/Android. YAMS
uses Boost.Asio (header-only) and links only boost_system/boost_thread
on every platform (asio/ssl uses OpenSSL directly; no spawn/context), so
the list now applies everywhere. Debug Boost package on grim: 45 static
libraries / 229 MiB -> 10 / 6.8 MiB (Release: 45 / 30 MiB before).
Verified on grim: ./setup.sh Debug --no-tsan --with-tests reconfigures
an existing build/debug and builds; an existing build/release (configured
with -Denable-tsan/-Denable-asan) regenerates.
Signed-off-by: trvon <me@trevon.dev>
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.
Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
yams_plugin_*, local *) and yams_plugin.exp (ld64
-exported_symbols_list) are applied to every plugin (s3, onnx, glint,
zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
marked YAMS_PLUGIN_API so it stays exported under hidden visibility.
libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
exported.
- It no longer links spdlog. Its messages reached the host log only
because the executable's spdlog interposed the library's copy; with the
copy hidden, the registry would have logged through a private default
logger to stdout. The registry now takes a host log sink
(OnnxConcurrencyRegistry::setLogSink); TuningManager and
AbiModelProviderAdapter install a spdlog forwarder from the header-only
onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.
Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
S3 and zyp plugins and asserts the entry points resolve while libcurl,
OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
not compile before; passes after). `meson test --suite plugins`: 5/5.
Measured on grim (release, stripped staged payload):
file bytes before after exports before after
libyams_object_storage_s3.so 7,854,064 6,240,112 11,147 10
yams_zyp.so 80,112 63,728 103 8
libyams_onnx_resource.so 477,472 30,840 895 20
bin/yams-cli 33,712,056 33,859,512 80 39
bin/yams-daemon 28,389,784 28,537,304 98 39
The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.
Signed-off-by: trvon <me@trevon.dev>
The audit found -fstack-protector-strong, RELRO/BIND_NOW, NX and
_FORTIFY_SOURCE=2 but no stack clash protection or control-flow
protection.
- -fstack-clash-protection on GCC/Clang where supported.
- Linux x86_64: -fcf-protection=full; Linux AArch64:
-mbranch-protection=standard (get_supported_arguments drops either
where the compiler rejects it).
- _FORTIFY_SOURCE=3 (with -U first so a toolchain default does not
produce redefinition warnings) for optimized builds; glibc < 2.34
treats it as level 2.
FORTIFY was not inert: the shipped v0.20.3 yams-cli imports
__memcpy_chk, and a grim (clang 21, glibc 2.43) build imports six
__*_chk functions. The count stays small by construction because most
libc calls in the binaries come from the Conan static archives (spdlog,
fmt, SQLite, libcurl, OpenSSL), which are compiled without FORTIFY;
level 3 did not change the count on this tree.
The x86 IBT/SHSTK property note is the AND of every linked object, so
the binaries still carry none: the Conan archives are not built with
-fcf-protection. check_install_surface keeps that as a warning; making
it a requirement needs CET-built dependencies.
Measured on grim (release -O3, stripped staged payload):
file before after delta
bin/yams-cli 33,261,496 33,712,056 +1.4%
bin/yams-daemon 28,033,432 28,389,784 +1.3%
bin/yams-mcp-server 16,573,864 16,741,800 +1.0%
libyams_object_storage_s3.so 7,837,680 7,854,064 +0.2%
yams_zyp.so 80,112 80,112 0.0%
Signed-off-by: trvon <me@trevon.dev>
- macOS zips were built with `zip -r`, which follows symlinks: bin/yams
(-> yams-cli) shipped as a second 34 MB copy and each versioned dylib
link became another full file. `zip -ry` stores the links.
- RPMs used rpm's gzip-class default payload (37.5 MB vs 28.6 MB for the
zstd .deb of the same tree); the spec now sets w19.zstdio (rpm >= 4.14).
The spec's License also said Apache-2.0.
- The Arch PKGBUILD and the CI .PKGINFO generator declared Apache-2.0
(the project is GPL-3.0-or-later) and depended on liburing, libarchive,
taglib, sqlite, openssl, curl and boost-libs, none of which the
installed binaries load: readelf shows they need only libstdc++,
libgcc_s, libc and libm, so depends is gcc-libs and glibc.
- packaging/appimage was never built by any workflow; it and the
*.AppImage globs in release.yml and the release manifest go away.
- packaging/homebrew/yams.rb was a stale CMake-era formula; release.yml
renders the tap formula from yams.rb.template, which stays.
Signed-off-by: trvon <me@trevon.dev>
Each executable links the whole static core, so most of it is unreachable
from any single binary, and every PIE carried ~0.9 MiB of RELA relative
relocations.
Optimized, non-coverage builds now compile with -ffunction-sections
-fdata-sections and link with --gc-sections and --build-id (the latter
keeps the debug split's .build-id lookup independent of the toolchain
default); lld gets --icf=safe when it is the linker. Linux links add
-z pack-relative-relocs (DT_RELR): it needs glibc >= 2.36 at runtime,
below the GLIBC_2.38 floor shipped binaries already require (v0.20.3
objdump -T), and the link probe drops it where the toolchain cannot emit
it. macOS uses -dead_strip. Debug, -O0 and coverage builds are unchanged.
Measured on grim (Linux x86_64, release -O3, stripped staged payload):
file before after delta
bin/yams-cli 36,733,304 33,261,496 -9.5%
bin/yams-daemon 30,685,792 28,033,432 -8.6%
bin/yams-mcp-server 19,865,472 16,573,864 -16.6%
libyams_object_storage_s3.so 8,239,016 7,837,680 -4.9%
libyams_onnx_resource.so 481,488 477,472 -0.8%
staged runtime total 92.0 MiB 82.7 MiB -10.1%
yams-daemon .rela.dyn: 0xd9158 (889,176 B) -> 0xb250 RELA + 0x2380 RELR.
The staged binaries init, add and search a document with a private HOME.
Signed-off-by: trvon <me@trevon.dev>
`./setup.sh Profiling` used Debug as its base build type, so Conan built
every dependency (spdlog, fmt, SQLite, Boost, protobuf, ...) unoptimized
while YAMS itself was -O2. Zone timings and allocation profiles from that
tree misattribute time to dependency code that is fast in shipped
binaries.
Profiling now uses the Release Conan profile and the Release Meson
configuration (-O3 -g, NDEBUG, PIE) plus Tracy, and keeps frame pointers
(-fno-omit-frame-pointer, -mno-omit-leaf-frame-pointer when supported)
for Tracy/perf/Instruments call stacks. TSan stays off for Profiling, as
before; --tsan still opts in.
Configure options, build/profiling on grim:
before: conan build_type=Debug; buildtype=debugoptimized (-O2 -g),
b_ndebug=true, enable-profiling=true, enable-tsan=false,
no frame-pointer flags
after: conan build_type=Release; buildtype=release, debug=true (-O3 -g),
b_ndebug=true, b_pie=true, enable-profiling=true,
enable-tsan=false, b_sanitize=none, -fno-omit-frame-pointer
Signed-off-by: trvon <me@trevon.dev>
Release used buildtype=debugoptimized (-O2 -g, strip=false) and relied on
scripts/prune-runtime-install.sh to strip the staged payload, discarding
the debug info. Crash reports from shipped binaries could not be
symbolicated and the build type label did not match what shipped.
- setup.sh/setup.ps1 Release and the Meson project default use
buildtype=release. setup.sh adds -Ddebug=true for Release so -O3 code
keeps DWARF for the split, and passes -Db_pie=true explicitly (also a
project default) so reconfigured build dirs pick it up. Windows keeps
release without -g so no PDBs land in the MSI stage. Debug/Fuzzing are
unchanged (-O0 -g, assertions on).
- scripts/split-debug-symbols.sh replaces prune-runtime-install.sh: ELF
files get objcopy --only-keep-debug (compressed) into <debug-dir>/
<path>.debug plus .build-id/xx/yyyy.debug links, strip
--strip-unneeded and --add-gnu-debuglink; Mach-O files get dsymutil
dSYMs and strip -S -x. Without a debug dir it only strips.
- The release job splits into $BUILD_DIR/debug-symbols and uploads it as
a debug-symbols-<suffix> workflow artifact (30 days) instead of
shipping or dropping it. build-deb.sh, the Dockerfile, PKGBUILD and
build-arch-pkg.sh call the split script in strip-only mode.
Measured on grim (Linux x86_64, clang 21, stripped staged payload):
file debugoptimized release -O3 delta
bin/yams-cli 35,528,968 36,733,304 +3.4%
bin/yams-daemon 29,731,312 30,685,792 +3.2%
bin/yams-mcp-server 19,504,928 19,865,472 +1.8%
libyams_onnx_resource.so 477,280 481,488 +0.9%
libyams_object_storage_s3.so 8,226,608 8,239,016 +0.2%
yams_zyp.so 79,936 80,040 +0.1%
Debug symbols: 309 MiB (compressed .debug files); gdb resolves
`info line main` on the stripped daemon through the .build-id tree.
check_install_surface: no debug sections or .symtab remain; the
remaining violations are the export leaks fixed later in this series.
Signed-off-by: trvon <me@trevon.dev>
