Commit graph

121 commits

Author SHA1 Message Date
Trevon
d398556ee5
fix(tests): replace frozen expectations and harden release gates (#323)
* test(search,daemon,storage): assert behaviour instead of frozen literals

Several suites encoded exact internal values that legitimate tuning and
ranking changes are allowed to move: SCIENTIFIC-era profile weights and
rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation
numbers. They failed on every platform after those changes landed, so the
real behaviour regressions they should catch were hidden behind noise.

Replace the frozen literals with contract assertions: weights are finite,
non-negative and normalize, state selection still holds, capability-gated
sources stay zeroed, and hybrid results keep matching canonical hash/path
sets. Deadlines are not widened and the daemon paths they cover are
unchanged.

Also add independent corpus and adapter oracles, deterministic collector
cancellation and late-result gates, and synchronize ServiceManager
shutdown observation, so the tests fail for the behaviour they name.

The portability allowlist keeps its two reviewed raw test-env reads; only
their line numbers moved.

Signed-off-by: trvon <git@trevon.dev>

* ci(tests): summarise Meson failures without flooding the log

A failing test run printed the whole Meson suite output, so the actual
failing test names and their assertion messages were buried and the
uploaded logs were dominated by passing-test chatter.

Add a bounded reporter that extracts the failing test names plus their
surrounding diagnostics, keeps an overall cap on emitted lines, and is
unit-tested against representative Meson output. Wire it into the Tests
workflow so failures stay readable.

Signed-off-by: trvon <git@trevon.dev>

* build(deb): validate candidate identity, mapping and upgrade ordering

The Debian packaging scripts took filenames and versions on trust, so a
misnamed candidate or a wrong package ordering could pass validation while
the real install path was never exercised.

Fail closed on candidate identity, version and architecture, verify the
package-to-source mapping, seed the legacy-only layout, wait on the actual
services, and assert install/upgrade/remove ordering for both Debian and
RPM in an offline container. Cases that cannot be proven on a host are
reported as skipped rather than silently passing.

Signed-off-by: trvon <git@trevon.dev>

---------

Signed-off-by: trvon <git@trevon.dev>
2026-10-19 19:16:57 -06:00
Trevon
111988c6e4
feat(topology): typed event time and session for topology inputs (#305)
* feat(topology): typed event time and session for topology inputs

Add `eventTime` (epoch seconds, UTC) and `sessionId` to
`TopologyDocumentInput`. The extractor fills them from two documented
metadata keys: `event_time` (integer epoch seconds, or ISO-8601
`YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive
datetimes are UTC) takes precedence over the document createdTime; an
unparsable value is skipped with a debug log, never an error, and a
zero createdTime leaves the field unset. `session_id` is copied
verbatim and empty when absent. The keys are only consulted when the
extraction loads metadata; otherwise createdTime is the only source.

`parseTopologyEventTime` is public for tests and future callers; the
calendar math is Hinnant's days_from_civil with range-checked fields.
Extraction stats count the event time source and invalid values.

Nothing downstream reads the fields yet. Artifacts, codecs and both
routing fingerprints are unchanged, which the new test pins.

Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp):
parser accept/reject tables, metadata precedence over createdTime,
invalid ignored, session_id, includeMetadata=false fallback, and
byte-identical snapshot payload plus equal construction and road
fingerprints with and without the typed fields.

Signed-off-by: trvon <me@trevon.dev>

* bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar

Every exported corpus document now carries `metadata.event_time`
(ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM`
haystack date, first-seen date per session) and `metadata.session_id`.
Document ids, titles, texts, queries and qrels are unchanged in the
default mode.

`--round-docs` adds one document per session round (`<sid>__r<n>`): a
user turn opens a round and the replies that follow join it; a leading
assistant turn is its own round. Round docs share the session's
`event_time`/`session_id` and add `round_index`; qrels stay at the
session level.

`--full-haystack` also writes `haystacks.jsonl`, one line per question
with `haystack_session_ids` and the normalized `haystack_dates`, so
per-question membership survives the global corpus deduplication
(the 2026-10-10 preflight lost it).

The converter is split into `convert` / `write_outputs`; `--self-test`
runs both on a three-session synthetic fixture without a download and
is the regression check for the default output.

Signed-off-by: trvon <me@trevon.dev>

* bench(retrieval): carry BEIR corpus metadata onto ingested documents

`retrieval_quality_bench` ingests the BEIR corpus as a directory, which
cannot carry per-file metadata, so `event_time`, `session_id` and
`round_index` from corpus.jsonl never reached the topology extractor.

`BEIRDocument::metadata` now holds the corpus line's `metadata` object
(strings as is, numbers and booleans stringified; both the generic and
the SciFact loader read it), and `applyBenchmarkDocumentMetadata`
copies it onto the stored documents matched by exact file path through
the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can
repeat it). The bench runs it after the ingest wait, for cold and
reused corpora alike, and logs how many documents were updated or not
found.

Tests: loader parses and coerces metadata; apply helper writes the
keys, skips documents without metadata, counts missing ones and is
idempotent against a real MetadataRepository.

The portability allowlist entries for the two touched bench files are
re-pointed at their shifted lines; no raw environment read was added.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-10-14 08:47:04 -06:00
Trevon
dc7a4fb499
fix: unblock the 0.20.5 release (aarch64 budget, MSVC linkage, mobile configure, Windows MSI) (#263)
* fix(ci): size the onnx resource budget for aarch64 page alignment

libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.

Signed-off-by: trvon <me@trevon.dev>

* fix(plugins): declare the S3 legacy factory with the plugin export macro

93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.

The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.

Signed-off-by: trvon <me@trevon.dev>

* fix(build): drop install_tag from the mobile install_headers call

324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).

Signed-off-by: trvon <me@trevon.dev>

* fix(packaging): install the Windows onnx resource DLL next to the executables

yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.

Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.

Signed-off-by: trvon <me@trevon.dev>

* ci(release): smoke-test the staged Windows CLI before building the MSI

The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 18:47:16 -06:00
trvon
c4f0ce4338
ci(packaging): probe the bundled ONNX Runtime with a separate user
Switching the main test user's embedding backend to onnxruntime (no
model installed) and back left state behind that the later phases then
tripped over. Probe the runtime with a third user whose own config
enables plugin autoload with the auto backend (config_version = 3 so
the daemon does not migrate it to the simeon default), and keep the main
user on yams init's configuration throughout.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 10:12:02 -06:00
trvon
c1483e4478
ci(packaging): make the user-unit checks match real sessions
First runs of the user-unit validation on all four lanes found harness
gaps, fixed here:

- Arch's `su -` does not run pam_systemd, so the login shell had no
  XDG_RUNTIME_DIR and the CLI looked for /tmp/yams-daemon-<uid>.sock;
  the checks now provide the runtime dir a real login session gets and
  report whether su set it.
- Ubuntu 24.04's systemctl --user needs the user D-Bus bus: the
  Debian/Ubuntu lane image gains dbus-user-session (standard on real
  installs) and the checks pass the bus address when present.
- CLI commands fall back to an in-process daemon instead of spawning
  one, so the lifecycle check uses `yams daemon start` (which starts the
  unit) rather than expecting an ordinary command to start it.
- `yams init`'s default embedder is the built-in simeon backend, which
  skips the onnx model provider by design. The onnx phase now checks
  that init's config autoloads plugins, then switches [embeddings]
  backend to onnxruntime to prove the bundled runtime loads (from the
  user journal), and restores init's config so the later phases run the
  user's real configuration.

docs/install.md no longer claims ordinary CLI commands start the unit:
`yams daemon start` and on-demand starters do.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 09:18:38 -06:00
trvon
e0b0b575e2
ci(packaging): validate the systemd user unit in every lane
The package validation now exercises the per-user service:

- service: the user unit and user preset are installed, the unit is
  enabled for all users (systemctl --global), no system unit remains,
  and the unit does not hide the user's home or /tmp.
- user: a normal user with lingering gets a user manager; the unit is
  enabled and active without a manual start and `systemd-analyze --user
  verify` is clean; the socket is in $XDG_RUNTIME_DIR and owned by the
  user; through a login shell with no YAMS_* variables the CLI reports
  status, adds from $HOME and /tmp, searches, stores in
  ~/.local/share/yams and logs to ~/.local/state/yams, and the only
  daemon is the unit's MainPID; `yams daemon stop`, CLI auto-start and
  `yams daemon restart` go through systemd; `yams daemon install --user`
  enables the packaged unit without writing an override; a second user
  cannot reach the first one's daemon.
- onnx: after `yams init` the restarted user daemon logs "Using bundled
  ONNX Runtime" and `yams plugin health` lists the onnx plugin.
- persist: with lingering the user's daemon is back after a reboot with
  the data intact.
- upgrade from 0.20.3: the old system unit is stopped, removed and
  leaves no enablement links, /var/lib/yams is left alone, the user unit
  is enabled for all users and the user's daemon and CLI work.
- remove: running user daemons are stopped, global enablement is gone,
  no daemon process is left, user corpora survive (deb: also purge).
- minimal: bare base images still install with declared dependencies
  only; the account checks are gone with the yams account.

The Fedora lane image replaces the systemd-user PAM stack, which cannot
run in a container (unix_chkpwd, pam_selinux, pam_loginuid), with a
minimal one that still runs pam_systemd, so user@.service starts.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 08:30:55 -06:00
trvon
999cafc64a
fix(packaging): reach running user managers without a D-Bus user bus
The package scripts only addressed user managers that had a D-Bus user
bus socket (/run/user/<uid>/bus). Debian and Ubuntu without
dbus-user-session have none, so upgrades did not reload or restart running
user daemons and removal left them running on deleted binaries. Detect a
manager by its private socket (/run/user/<uid>/systemd/private), which
systemctl --user uses directly, and pass only XDG_RUNTIME_DIR.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 08:30:39 -06:00
trvon
e389e384b1
feat(packaging): ship the daemon as a systemd user unit
The service is now user-bound: each user runs their own daemon with
their own config ($XDG_CONFIG_HOME/yams/config.toml), corpus
($XDG_DATA_HOME/yams) and socket ($XDG_RUNTIME_DIR/yams-daemon.sock),
which is exactly what the CLI already resolves. The system service, its
yams account/group, the group socket and the /etc/yams/config.toml
configuration go away.

- Unit: /usr/lib/systemd/user/yams-daemon.service runs
  `yams-daemon --foreground` with default path resolution,
  Restart=on-failure, WantedBy=default.target, and light hardening
  (NoNewPrivileges, LockPersonality, RestrictRealtime, RestrictSUIDSGID,
  native syscalls only). No ProtectHome/PrivateTmp: the daemon reads the
  files its user adds.
- Enablement: /usr/lib/systemd/user-preset/80-yams.preset plus
  `systemctl --global preset` on first install (and on upgrade from a
  release with the system unit), so the unit starts with every user's
  next session; administrators override with a user preset or
  `systemctl --global disable`, and later upgrades keep that choice.
  Package scripts never start per-user instances: they reload running
  user managers, try-restart running user daemons on upgrade and stop
  them on removal (`systemctl --user` via runuser per /run/user/<uid>),
  then `systemctl --global disable` on removal.
- Migration from 0.20.x and the system-service design: deb preinst, rpm
  %pre and Arch pre_upgrade stop and disable the old system unit while
  its file is still installed (deb also purges deb-systemd-helper
  state). /var/lib/yams, the old system corpus that the CLI never
  reached, is left alone and documented; user corpora are never
  touched.
- The sysusers entry, account creation and the deb's passwd and
  init-system-helpers dependencies are dropped; the deb purge no longer
  deletes /var/lib/yams.
- Install paths move to lib/systemd/user and lib/systemd/user-preset in
  meson, the Arch stage/PKGBUILD and the install-surface policy.
- docs/install.md: the user service, enabling it now
  (`systemctl --user enable --now`), lingering, plugins via the user's
  config, opting out, upgrade and removal behaviour, and the 0.20.3
  leftover.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 07:55:01 -06:00
trvon
ca1710fdcc
fix(packaging): give the system daemon an /etc configuration file
The packaged service had no configuration file it would read: with
HOME=/var/lib/yams it looked for /var/lib/yams/.config/yams/config.toml,
which nothing creates. Plugins load only with [daemon]
auto_load_plugins = true, so the onnx and glint plugins and the bundled
ONNX Runtime shipped in the packages were never used by the service
("No plugins loaded"), and an administrator had no obvious place to turn
them on.

The unit now sets YAMS_CONFIG=/etc/yams/config.toml (the package ships
no file; ProtectSystem=strict leaves /etc readable). With
auto_load_plugins = true there, the sandboxed daemon loads all three
plugins and logs "Using bundled ONNX Runtime 1.23.0 from
/usr/lib/yams/onnxruntime/libonnxruntime.so.1 (no system copy found)",
ready=true. docs/install.md describes the file and how to check which
runtime was loaded; the validation's onnx phase writes it, restarts the
unit and checks the bundled runtime and plugin health.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 00:57:34 -06:00
trvon
e9d19abd45
fix(build): install the bundled ONNX Runtime symlinks target-first
`meson install` of a release build with --with-onnx failed:

  ERROR: Tried to install symlink to missing file
         .../usr/lib/yams/onnxruntime/libonnxruntime.so.1

Meson installs symlinks in declaration order and refuses one whose target
is not installed yet. list-onnxruntime-libs.py printed them in name order,
so libonnxruntime.so -> .so.1 came before .so.1 -> .so.1.23.0. Emit each
chain from the real file outwards (by link depth). Found by building the
packages the release way (setup.sh Release --with-onnx, meson install).

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 00:18:42 -06:00
trvon
2cd2ff68e3
ci(packaging): validate on bare base images and check the bundled ONNX Runtime
The lane images boot systemd and carry the tools the checks use, so
systemd-sysusers always existed and the deb postinst's broken `adduser`
fallback was never exercised.

Each lane now also installs the package in the raw distro image
(debian:trixie-slim, ubuntu:24.04, fedora:42, the Arch base) with no
systemd running and nothing preinstalled, so only declared dependencies
are present. It checks that the install finishes (deb: dpkg status
"install ok installed", not half-configured), that the yams account and
group exist, and that `yams add` + `yams search` work without the
service.

A new onnx phase checks, on the systemd lane, that the package ships the
private runtime in <libdir>/yams/onnxruntime and the onnx plugin, that
the system daemon logs "Using bundled ONNX Runtime", and that
`yams plugin health` lists the onnx plugin.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 00:01:55 -06:00
trvon
a4a9dd4418
fix(packaging): create the yams account on systems without systemd or adduser
The deb postinst created the account with systemd-sysusers and fell back
to `adduser`, which the package never depended on. In a minimal Debian or
Ubuntu container (no systemd, no adduser) installation stopped in
postinst with "adduser: not found" and left the package half-configured.

The postinst still prefers the shipped sysusers.d entry, then falls back
to groupadd/useradd from `passwd`, now a declared dependency (Priority:
required, so it costs nothing). If neither works it prints what to do,
skips enabling the service and exits 0: the CLI does not need the account,
so a missing account no longer breaks the install. The rpm %post gets the
same groupadd/useradd fallback when systemd-sysusers cannot run; Arch
already relies on pacman's sysusers hook plus an explicit
systemd-sysusers call.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 00:01:01 -06:00
trvon
14175fc3e5
build(rpm): keep private /usr/lib/yams libraries out of rpm provides
rpm's dependency generator advertised every ELF library in the package
as a system soname, including the bundled ONNX Runtime
(libonnxruntime.so.1()(64bit)), so the yams RPM could satisfy other
packages' ONNX Runtime dependencies, and it emitted requires on the
private libyams_onnx_resource/libzpdf it ships itself. Filter provides
from /usr/lib/yams and drop requires on those private sonames.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 22:43:19 -06:00
trvon
6e25a8b90f
ci(packaging): allow the bundled ONNX Runtime in package validation
Packages now carry the private fallback ONNX Runtime in
/usr/lib/yams/onnxruntime/ (libonnxruntime.so -> .so.1 -> .so.1.x.y and
libonnxruntime_providers_shared.so). Without an allowlist entry the
layout check rejects them as unreviewed paths. The pattern admits only
those library names under that directory.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 22:42:44 -06:00
trvon
e895fe2f94
build(packaging): bundle ONNX Runtime privately in <libdir>/yams/onnxruntime
No package carried ONNX Runtime: tools/yams-cli tried to install_subdir()
the runtime's libdir, but read it with get_variable('libdir') from the
dependency meson.build had already normalized into a declare_dependency,
which has no variables, so it installed nothing. Packaged ONNX and Glint
plugins therefore only worked where the user had installed a compatible
runtime themselves.

meson.build now records the directories it resolved ONNX Runtime from,
and the runtime install bundles the shared library from the first that
holds one into <libdir>/yams/onnxruntime/ (install tag runtime). A small
configure-time helper (scripts/list-onnxruntime-libs.py) lists the real
files and the soname symlinks, so the chain libonnxruntime.so ->
.so.1 -> .so.1.23.0 is recreated instead of tripled, and only the core
library and providers_shared are taken (no static archives, CMake or
pkg-config files). The plugins use this copy only when no compatible
system copy exists. Windows is unchanged: onnxruntime.dll already ships
next to the executables in the MSI and is found by the loader search.

- scripts/split-debug-symbols.sh leaves the bundled runtime
  byte-identical: it is an upstream release build, already stripped, and
  re-stripping would void a macOS code signature.
- install_surface_policy: <libdir>/yams/onnxruntime/* is allowed and
  exempt from first-party export/hardening checks; the Glint size budget
  grows to 850,000 bytes (711,352 measured with real ONNX Runtime code).
- Homebrew formulas depend on Homebrew's onnxruntime, which the plugins
  prefer, so they drop the archive's private copy (rm_rf
  lib/yams/onnxruntime).
- Arch: optdepends now reads 'onnxruntime: use the system ONNX Runtime
  instead of the bundled copy' (PKGBUILD and the CI .PKGINFO).

Measured on grim (Linux x86_64 release, --with-onnx, stripped staged
payload): 87,883,247 -> 110,236,706 bytes (+22.4 MB: the 22,207,288-byte
libonnxruntime.so.1.23.0, 14,632-byte providers_shared, and +111 KB of
real GLiNER code in yams_glint.so).

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 22:42:15 -06:00
trvon
4563f92d55
ci(packaging): allow private libraries under lib/yams in package validation
The install-surface cleanup moved libyams_onnx_resource from /usr/lib to
/usr/lib/yams, matching scripts/ci/install_surface_policy.json. The
package validation allowlist only accepted libyams_* directly in the
library directory and would have failed every lane; accept both, and
nested paths under /usr/share/doc/yams like the policy does.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:16:29 -06:00
trvon
ad0c1e04cf
build: install the yams sysusers entry with the runtime set
The systemd fix added lib/sysusers.d/yams.conf without an install tag, so
'meson install --tags runtime' dropped it, and the install-surface policy
did not allow the path, so the release gate would have failed on it.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:14:10 -06:00
trvon
3c64e1fb8f
fix(packaging): escape rpm macro names in spec comments
rpm expands macros inside comments. The generated spec's comment named
%systemd_postun_with_restart, which systemd-rpm-macros defines with
required arguments, so rpmbuild on Fedora 42 (rpm 4.20 with the macros
installed) stopped with "The %systemd_postun_with_restart macro
requires some arguments". Debian and Ubuntu builders do not define the
macro, which is why release CI never saw it. Write the names as %% in
the comments, including the %postun mention in the %post scriptlet.

Checked with rpmbuild -bb of the generated spec in fedora:42 with
systemd-rpm-macros: the previous spec fails, this one builds and its
%post carries the systemd-sysusers call.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:45 -06:00
trvon
7ef6cc9366
ci(packaging): make the per-user service and reboot checks host-robust
x86_64 runs on a loaded host showed three harness problems:

- The Debian/Ubuntu and Fedora lane images lacked pam_systemd, so
  user@.service could not start and `yams daemon install --user` looked
  broken. Install libpam-systemd / systemd-pam. Where a host still
  cannot start a user manager in a container, the check reports INFO
  instead of failing, since that says nothing about yams.
- `systemctl --user -M user@` needs machined; query the user manager as
  that user instead. The "CLI uses its unit" check now also requires that
  the only daemon running for the user is the unit's MainPID, so a
  CLI-spawned second daemon no longer passes.
- A container reboot can sit in 'starting' past 60 s under load; wait up
  to three minutes and print pending jobs if it still does not boot.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:34 -06:00
trvon
25114dadc7
ci(packaging): harden package validation from its first real runs
Running the gate against v0.20.3 and branch packages on Debian, Ubuntu,
Fedora and Arch showed harness gaps, fixed here:

- The driver ran only under bash 4+ (mapfile, empty arrays under set -u);
  it now works with macOS's bash 3.2 so it runs on Docker Desktop too.
- Paths dropped by the image's dpkg path-exclude (docs) or rpm nodocs
  no longer fail the permission check, and rpm's /usr/lib/.build-id links
  and the package doc/license directories are on the allowlist.
- The upgrade check pins the system socket, so it measures whether the
  corpus survived rather than whether an older CLI can discover it.
- Debian keeps enablement links until purge, so for deb the dangling-link
  check runs after purge.
- New per-user service check: `yams daemon install --user` beside the
  system unit starts a working user unit, and that user's CLI uses it.

The unit comment and install guide now describe the socket as group
read/write with no access for others (the daemon sets owner rwx, group
rw), rather than a literal 0660.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:27 -06:00
trvon
53f0061d31
ci(release): gate packages on a real systemd install, CLI and upgrade run
The old package-validate.sh only checked that the unit came up and that
`yams daemon status` answered with YAMS_DAEMON_SOCKET set by hand, so it
passed while nobody's CLI ever reached the system daemon, the socket was
world-writable and `yams add ~/file` failed.

package-validate.sh stays the single entry point (same --only/--deb/
--rpm/--arch flags the workflow uses) and now drives
package-validate-checks.sh inside Debian, Ubuntu, Fedora and Arch
containers booted with systemd as PID 1 (private cgroup namespace; the
host's cgroups are not shared). Per lane it checks: install with no
manual steps; the installed file list against a runtime allowlist with
no headers, static libs, pkg-config or CMake files; no world-writable,
setuid or non-root files; resolvable DT_NEEDED and $ORIGIN-or-/usr/lib
RPATH/RUNPATH; unit enabled and active as the yams account; socket group
yams and closed to others; `systemd-analyze verify` and a
`systemd-analyze security` exposure budget; root, a yams-group member
and a non-member using the CLI with no YAMS_* variables (add from $HOME
and /tmp, search, shared corpus, no private fallback store, clear
directory error, non-member denied, `daemon stop` deferring to systemd);
state after a container reboot; removal and deb purge. With
--upgrade-from-<fmt> it installs an older package, stores a document,
upgrades and checks the restart and the document. Every check reports
PASS/FAIL, a summary table follows, and any FAIL fails the run.

release.yml's existing validation steps (the gate before artifacts are
uploaded for publishing) pass the latest published release's packages as
the upgrade source. The unreferenced packaging/systemd/Dockerfile is
removed; the lane images take a BASE_IMAGE argument and add binutils
and util-linux for the checks.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:15 -06:00
trvon
ef119cb5d5
fix(packaging): run the system daemon as a yams account behind a group socket
The packaged yams-daemon.service ran under DynamicUser= and its
ExecStartPost chmod'ed /run/yams/yams-daemon.sock to 0666, so every
local account could read, add and delete documents in the system
corpus. It also used the deprecated PermissionsStartOnly= and a light
sandbox (ProtectSystem=full only).

The unit now runs as a static `yams` system account created from
/usr/lib/sysusers.d/yams.conf (deb postinst, rpm %post and the Arch
install script run systemd-sysusers before starting it). The daemon's
own socket mode (owner rwx, group rw) with Group=yams and UMask=0007
gives a group-only socket, and /run/yams, /var/lib/yams and
/var/log/yams are 0750, so access is granted by adding a user to the
`yams` group. The chmod and PermissionsStartOnly are gone.

Sandboxing: ProtectSystem=strict, ProtectHome, PrivateTmp,
PrivateDevices, empty CapabilityBoundingSet, kernel/clock/hostname/
cgroup protection, ProtectProc=invisible, RestrictAddressFamilies
(unix, inet, inet6, netlink), RestrictNamespaces, LockPersonality,
RemoveIPC and a @system-service syscall filter without @privileged and
@resources. MemoryDenyWriteExecute stays off for ONNX Runtime.

Package scripts: Arch pre_remove now disables as well as stops (it left
a dangling multi-user.target.wants link), and deb purge removes
/var/lib/yams and /var/log/yams, including the /var/{lib,log}/private
copies earlier DynamicUser= releases left. Upgrades keep the corpus:
systemd moves the DynamicUser= state back to /var/lib/yams on first
start under User=yams.

docs/install.md describes the group, the client socket order, and what
the sandboxed service can and cannot read.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:08 -06:00
trvon
545f461ae1
build(macos): keep libyams_onnx_resource @rpath-relative after install
Meson rewrites the install name of installed dylibs to their absolute
install path and points consumers at it, so the staged macOS payload
referenced /opt/homebrew/lib/yams/libyams_onnx_resource.dylib: yams-cli
from an unzipped release archive (or a staging dir) died in dyld
("Library not loaded ... no such file"), and Homebrew depended on its
prefix symlinks to resolve it.

A darwin-only Meson install script (scripts/macos-rpath-install-names.sh,
tag runtime) restores @rpath/libyams_onnx_resource.dylib as the library
id and in yams-cli/yams-daemon and the plugins, which already carry
@executable_path/../lib/yams and @loader_path/.. RPATHs.

Verified on macOS arm64 (ONNX + Glint release build): otool -L shows
@rpath/libyams_onnx_resource.dylib; a `zip -ry` archive of the stage
unzipped to an unrelated directory runs `yams --version`, codesign -v
passes, and the Glint plugin dlopens from there.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 20:29:46 -06:00
trvon
454c95245b
ci: ignore ld64's radr://5614542 marker when checking Mach-O strip state
On a macOS Release build staged and split with split-debug-symbols.sh,
check_install_surface reported every binary as shipping a symbol table.
The only stab left after strip -S -x is the 'OPT radr://5614542' entry
ld64 always writes; it carries no debug information. Unstripped binaries
still trip the check (135,012 stabs in an unsplit yams-cli).

Verified on macOS arm64 (ONNX + Glint build): RED with 8 false
'stripped' violations before, GREEN after.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 19:44:42 -06:00
trvon
93f2439471
build(plugins): export only the plugin C ABI; hide onnx_resource internals
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.

Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
  yams_plugin_*, local *) and yams_plugin.exp (ld64
  -exported_symbols_list) are applied to every plugin (s3, onnx, glint,
  zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
  marked YAMS_PLUGIN_API so it stays exported under hidden visibility.

libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
  visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
  exported.
- It no longer links spdlog. Its messages reached the host log only
  because the executable's spdlog interposed the library's copy; with the
  copy hidden, the registry would have logged through a private default
  logger to stdout. The registry now takes a host log sink
  (OnnxConcurrencyRegistry::setLogSink); TuningManager and
  AbiModelProviderAdapter install a spdlog forwarder from the header-only
  onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.

Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
  S3 and zyp plugins and asserts the entry points resolve while libcurl,
  OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
  assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
  not compile before; passes after). `meson test --suite plugins`: 5/5.

Measured on grim (release, stripped staged payload):

  file                          bytes before  after       exports before  after
  libyams_object_storage_s3.so   7,854,064   6,240,112          11,147     10
  yams_zyp.so                       80,112      63,728             103      8
  libyams_onnx_resource.so         477,472      30,840             895     20
  bin/yams-cli                  33,712,056  33,859,512              80     39
  bin/yams-daemon               28,389,784  28,537,304              98     39

The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 18:51:34 -06:00
trvon
d534f090bd
build(packaging): store symlinks in zips, zstd rpm, correct Arch metadata
- macOS zips were built with `zip -r`, which follows symlinks: bin/yams
  (-> yams-cli) shipped as a second 34 MB copy and each versioned dylib
  link became another full file. `zip -ry` stores the links.
- RPMs used rpm's gzip-class default payload (37.5 MB vs 28.6 MB for the
  zstd .deb of the same tree); the spec now sets w19.zstdio (rpm >= 4.14).
  The spec's License also said Apache-2.0.
- The Arch PKGBUILD and the CI .PKGINFO generator declared Apache-2.0
  (the project is GPL-3.0-or-later) and depended on liburing, libarchive,
  taglib, sqlite, openssl, curl and boost-libs, none of which the
  installed binaries load: readelf shows they need only libstdc++,
  libgcc_s, libc and libm, so depends is gcc-libs and glibc.
- packaging/appimage was never built by any workflow; it and the
  *.AppImage globs in release.yml and the release manifest go away.
- packaging/homebrew/yams.rb was a stale CMake-era formula; release.yml
  renders the tap formula from yams.rb.template, which stays.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 17:15:46 -06:00
trvon
8250b061fc
build: release buildtype with split debug symbols
Release used buildtype=debugoptimized (-O2 -g, strip=false) and relied on
scripts/prune-runtime-install.sh to strip the staged payload, discarding
the debug info. Crash reports from shipped binaries could not be
symbolicated and the build type label did not match what shipped.

- setup.sh/setup.ps1 Release and the Meson project default use
  buildtype=release. setup.sh adds -Ddebug=true for Release so -O3 code
  keeps DWARF for the split, and passes -Db_pie=true explicitly (also a
  project default) so reconfigured build dirs pick it up. Windows keeps
  release without -g so no PDBs land in the MSI stage. Debug/Fuzzing are
  unchanged (-O0 -g, assertions on).
- scripts/split-debug-symbols.sh replaces prune-runtime-install.sh: ELF
  files get objcopy --only-keep-debug (compressed) into <debug-dir>/
  <path>.debug plus .build-id/xx/yyyy.debug links, strip
  --strip-unneeded and --add-gnu-debuglink; Mach-O files get dsymutil
  dSYMs and strip -S -x. Without a debug dir it only strips.
- The release job splits into $BUILD_DIR/debug-symbols and uploads it as
  a debug-symbols-<suffix> workflow artifact (30 days) instead of
  shipping or dropping it. build-deb.sh, the Dockerfile, PKGBUILD and
  build-arch-pkg.sh call the split script in strip-only mode.

Measured on grim (Linux x86_64, clang 21, stripped staged payload):

  file                         debugoptimized  release -O3   delta
  bin/yams-cli                     35,528,968   36,733,304   +3.4%
  bin/yams-daemon                  29,731,312   30,685,792   +3.2%
  bin/yams-mcp-server              19,504,928   19,865,472   +1.8%
  libyams_onnx_resource.so            477,280      481,488   +0.9%
  libyams_object_storage_s3.so      8,226,608    8,239,016   +0.2%
  yams_zyp.so                          79,936       80,040   +0.1%

Debug symbols: 309 MiB (compressed .debug files); gdb resolves
`info line main` on the stripped daemon through the .build-id tree.
check_install_surface: no debug sections or .symtab remain; the
remaining violations are the export leaks fixed later in this series.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 16:53:23 -06:00
trvon
324be33abb
build: install only the runtime set from meson install
`meson install` of a Release tree staged 2.4 GiB: all 28 internal static
libraries (1.3 GiB), the whole include/ tree (459 files), 22 pkg-config
files for libraries that were never meant to be linked from outside, and
the ONNX resource library in the system libdir. Packages hid this by
deleting include/, *.a and pkgconfig/ after the fact.

- Internal static libraries are `install: false` and their pkg.generate()
  calls are gone, as is the top-level yams.pc (it required yams-api, which
  is not installable). Benchmarks are no longer installed either.
- Headers are not installed: no installable C++ library exists to link
  them against. The one SDK surface, the mobile C ABI, installs
  mobile_bindings.h and its .pc under the 'devel' install tag.
- Runtime installs carry explicit install_tag: 'runtime' where Meson
  cannot infer one (data, systemd unit/preset, the yams alias symlink,
  Windows DLL copies, the libzpdf install script), so
  `meson install --tags runtime` stages the same set as a plain install.
- libyams_onnx_resource moves from <libdir> to the private <libdir>/yams
  on Linux/macOS, without a soname version since nothing outside this
  build links it. yams-cli/yams-daemon RUNPATH becomes $ORIGIN/../lib/yams
  (@executable_path on macOS) and the ONNX/Glint plugins use $ORIGIN/..
  so they resolve it and the bundled ONNX Runtime, which also moves to
  <libdir>/yams (the ONNX plugin's runtime loader already searches its
  parent directory). Windows keeps its existing DLL layout.
- Homebrew templates and the macOS install smoke follow the new layout.
- scripts/prune-runtime-install.sh only strips now.

Measured on grim (Linux x86_64 Release, prefix /usr/local):
  staged paths  522 -> 11, staged bytes 2.4 GiB -> 1.0 GiB (unstripped)
  check_install_surface path violations: 6 -> 0
  `--tags devel` stages nothing; ldd resolves libyams_onnx_resource from
  bin/../lib/yams and the staged `yams --version` runs.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 14:22:10 -06:00
trvon
3c619f1624
ci: add an install surface gate for the staged release payload
Packages have only looked small because scripts/prune-runtime-install.sh
deletes headers and static archives and strips after `meson install`. A
plain install of a Release tree stages 2.4 GiB: 28 internal static
libraries (1.3 GiB), 459 headers and 22 .pc files, unstripped binaries
(yams-cli 497 MiB), an S3 plugin exporting 11,154 symbols including
OpenSSL, libcurl, zlib, spdlog and nlohmann, and an ONNX resource library
in the system libdir.

scripts/ci/check_install_surface.py stages `meson install --destdir` for
a build dir (or reads a staged tree) and asserts it against
scripts/ci/install_surface_policy.json: an allowed runtime path set and a
forbidden devel set, required paths, per-binary size budgets, exported
symbol budgets and forbidden third-party export patterns, $ORIGIN-only
RUNPATH, ELF hardening (PIE, BIND_NOW, GNU_RELRO, NX; CET and FORTIFY
reported as warnings) and stripped binaries. ELF inspection uses
readelf/nm; Mach-O uses otool/nm. Standard library only.

The release packaging job runs it on the exact staged payload after the
existing runtime asset checks. On current experimental it is RED
(Linux x86_64 Release, grim):

  raw meson install:  43 violations (paths, debug info, exports, sizes)
  after prune script: 29 violations (exports, libdir placement)

Size budgets start at the measured stripped baseline plus ~10% and are
tightened by the follow-up build changes.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 14:16:09 -06:00
Trevon
577adf1e0f
build: standardize the Debug build dir on build/debug (#248)
* build: standardize the Debug build dir on build/debug

`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.

Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
  doc examples -> `build/debug`

Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.

* build: migrate remaining Debug consumers off builddir

Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.

- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
  `-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
  Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
  (and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
  READMEs/examples use `build/debug`.

`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
2026-10-01 08:45:20 -06:00
trvon
5eeb3b8239
fix(ci): drop --force from wrangler r2 object put
The #206 revert restored the pre-September upload code, which passed
--force, while the workflow still pins wrangler 4.69.0. That version's
'r2 object put' has no --force option (it overwrites without prompting)
and exits with 'Unknown argument: force', so the v0.20.3 release failed
its first R2 upload and no apt, yum or Arch repository was published.
Immutable objects are still checked with 'get' before any write.

Signed-off-by: trvon <me@trevon.dev>
2026-09-30 14:48:44 -06:00
trvon
09e4064a07
ci(mobile): add host mobile SDK lane and ABI check
libyams_mobile stopped compiling without anyone noticing because no CI
lane built it: every lane forced YAMS_ENABLE_MOBILE_BINDINGS=false, and
the "Check mobile ABI surface" step was commented out and pointed at
scripts/ci/check_mobile_abi.sh, which was never committed.

Add a linux-x86_64-mobile matrix entry that configures with
enable-mobile-bindings and enable-mobile-tests, compiles only the
yams_mobile and catch2_mobile_abi targets, runs the ABI check, and runs
the mobile meson suite. Other lanes keep mobile bindings off.

Add scripts/ci/check_mobile_abi.sh. It compiles the public header as C11
and C++20 and takes the address of every YAMS_MOBILE_API function, so a
declaration hidden from C (the extern "C" bug fixed in the previous
commit) fails. It then checks that the library exports exactly the
declared yams_mobile_* functions with unmangled names.

This lane proves the Linux x86_64 host build only. It does not show that
an Android .so or iOS XCFramework loads on a device.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:49 -06:00
Trevon
468e1f9982
Revert "ci: make immutable uploads atomic" (#206)
This reverts commit 296e7320.

296e7320 moved R2 publication from wrangler (authenticated by
CLOUDFLARE_API_TOKEN) to boto3 over R2's S3 API, which needs
R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (and EXPERIMENTAL_* pairs). Those
secrets were never configured, so the first stable release built after
the change (v0.20.2, run 36084509789) failed at 'Publish channel package
repositories to R2' with 'Missing channel-specific Cloudflare R2 S3
authority', and the APT/RPM/Arch repositories were not updated.

Restore the wrangler path so releases publish with the existing token.
2026-09-25 08:44:43 -06:00
trvon
e167498be4
fix(ci): build aarch64 Arch packages on an Arch Linux ARM base image
archlinux/archlinux only publishes linux/amd64, so the native aarch64 Arch
lane failed with 'exec format error' before provisioning. Both Arch
Dockerfiles now take ARCH_BASE_IMAGE; the aarch64 matrix entry pins
ghcr.io/menci/archlinuxarm by digest and x86_64 keeps the official image.
package-validate.sh forwards ARCH_BASE_IMAGE to the systemd substrate.
2026-09-24 15:00:32 -06:00
trvon
89bf7165b7
fix(ci): stop package-validate from exiting silently on a missing build dir
The release Arch job passes --arch explicitly, but the script still ran
deb/rpm discovery against the default build/release dir. find fails on the
missing dir, pipefail + set -e turn that into a silent exit 1. Only discover
packages for selected lanes and tolerate a missing build dir.
2026-09-24 14:52:18 -06:00
Trevon
bb0be0d587
Merge pull request #192 from trvon/test/fuzz-parsers
fix: harden search, UTF-8, extraction and compression parsers; libFuzzer lane on Linux
2026-09-23 20:25:20 -06:00
trvon
15f3e3db73
refactor(compression): remove the never-built LZMA compressor
lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but
the build only finds xz/liblzma, which does not ship those headers, so
every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it
out. No build could read or write LZMA blocks, and its decoder never
bounded output by the expected size.

The policy still chose LZMA for cold/archival and large inactive data;
compression then failed ('Compressor not available') and batch stores
kept those blocks uncompressed. Select Zstandard, which already uses the
archive level for old data.

Remove the compressor, its test, the enable-lzma option and the build
flags that passed it. The LZMA algorithm id stays in the header format,
and compression.lzma_level remains accepted in config (now unused).
2026-09-23 19:31:37 -06:00
trvon
0288f02715
fix(release): build Arch packages with the ONNX plugin
scripts/build-arch-pkg.sh still ran a plain ./setup.sh Release, which no
longer enables ONNX, so both Arch release targets would have shipped
without the ONNX embedding provider and Glint. Pass --with-onnx like the
other release lanes and fail packaging if the plugin is missing from the
stage, since the Arch lane skips the ONNX smoke test.
2026-09-23 17:26:16 -06:00
trvon
e6de1978c2
Merge branch 'refactor/remove-symbol-abi' into release/v0.20-integration 2026-09-22 13:55:37 -06:00
trvon
c39ed84bf7
refactor(plugins)!: remove the symbol_extractor_v1 plugin ABI
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.

Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.

Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.

BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
2026-09-22 13:30:35 -06:00
trvon
438f8a8a7b
ci(release): refuse stable releases from non-version refs
A stable workflow_dispatch took the release tag from GITHUB_REF_NAME
without checking it, so dispatching from main published a release
tagged main. Require refs/tags/v* or refs/tags/yams-v*.

Fetch main and experimental with explicit refspecs in the candidate
preflight so a tag named main cannot shadow the branch, and drop the
workflow_run trust path from draft reconciliation now that the workflow
is dispatch-only.
2026-09-22 12:18:20 -06:00
trvon
2bfc5334eb
ci(release): decouple draft promotion and enforce non-squash merge policy
- Decouple draft-promotion.yml from automated Tests workflow_run triggers,
  making promotion draft creation manual via workflow_dispatch to prevent
  GitHub from auto-chaining active experimental feature PRs into a stack
  targeting main.
- Enforce and document the mandatory Promotion Merge Policy across
  reconcile_promotion_pr.py, validate_release_candidate.py, and draft-promotion.yml
  step summary: promotions must use a merge commit (--no-ff) or fast-forward,
  never squash-merge into main, preserving conventional commit history and
  unbroken git ancestry.
- Unhide refactor commits in release-please-config.json so architectural
  refactorings are tracked under Changed in CHANGELOG.md.
- Update unit tests in test_reconcile_promotion_pr.py and
  test_validate_release_candidate.py for the new policy invariants.

Signed-off-by: trvon <me@trevon.dev>
2026-09-22 07:58:31 -06:00
trvon
ee51abcb1a
refactor(symbols): remove in-tree tree-sitter symbol extractor plugin and tests
Remove legacy Tree-sitter symbol extractor plugin (plugins/symbol_extractor_treesitter),
the unused plugin-symbols build option, and associated tests and benchmark targets:
- Drop plugins/symbol_extractor_treesitter/ (grammar_loader, symbol_extractor, plugin_api).
- Remove plugin-symbols Meson option and packaging references.
- Remove dedicated symbol plugin tests and benchmark targets (symbol_extraction_bench,
  symbol_extractor_all_langs_catch2_test, symbol_extractor_edge_cases_catch2_test,
  symbol_extractor_solidity_catch2_test, symbol_extractor_query_test,
  symbol_extractor_validation_test).
- Clean up dead symbol extraction plugin helper and test from hnsw_entity_search_bench.
- Prune obsolete portability allowlist entries.

Signed-off-by: trvon <me@trevon.dev>
2026-09-22 07:37:30 -06:00
trvon
9c7f780d4c
refactor(symbols): remove symbol extraction subsystem and add migration 40
Drop legacy Tree-sitter symbol extraction subsystem, associated database
tables, triggers, and AST edge types:
- Migration 40 drops symbol_metadata, document_symbol_extraction_state,
  symbol_metadata_fts, associated triggers, and prunes historical bloated
  topology snapshots while preserving active snapshot topology,
  semantic_neighbor edges, and non-symbol observed_as provenance edges.
  Exclude nl_entity:% nodes from kg_nodes / kg_edges pruning to preserve
  canonicalized method-type technology entities.
  Provides complete downSQL restoring symbol tables, indexes, FTS virtual table,
  and sync triggers for clean v39 rollbacks.
- In EntityGraphService, ensure locally resolved documentDbId propagates to
  WriteBatch for knowledgeGraphDocumentId to ensure accurate enrichment
  token completion, and only populate batch->documentIdToDelete for snapshot
  documents with a resolved document database ID.
- In SymbolEnricher and SearchService, restore one-directional symbol-contains-query
  matching with non-empty checks to prevent short substrings matching broad queries.
- In tests/unit/metadata/symbol_metadata_schema_test.cpp, migrate test DB up to
  version 39 so tests pass against their target version prior to v40 table drops.
- Remove Tree-sitter grammar download and compile workflows from CLI init
- Default plugin-symbols to false.
- Fix macOS daemon client and CLI executable resolution using
  _NSGetExecutablePath with candidate fallbacks.
- Add migration validation tests and symbol enrichment unit tests.

Signed-off-by: trvon <me@trevon.dev>
2026-09-22 07:37:30 -06:00
Trevon
e4b32e98bd
fix: fence enrichment completion and reconcile promotion prerequisites (#157)
Some checks failed
Secret scan / gitleaks (push) Failing after 8s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
* test(ci): stabilize main matrix (#112)

Signed-off-by: trvon <me@trevon.dev>

* ci(release): preview experimental promotion (#111)

Signed-off-by: trvon <me@trevon.dev>

* ci(release): reconcile draft promotion safely (#113)

* ci(release): reconcile draft promotion

Create promotion visibility independently of release readiness.
Leave existing PRs untouched to preserve concurrent maintainer edits.
Validate against current main without masking candidate rejection.

Signed-off-by: trvon <me@trevon.dev>

* ci: isolate hook Git environments

Prevent temporary Git fixtures from modifying the calling worktree.
Clear repository-local hook variables before the unchanged build gates,
and isolate validator and fixture Git subprocesses from caller overrides.
Exercise linked-worktree config, ref and index preservation offline.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>

* fix(ci): reject invalid candidate evidence

Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values.

Signed-off-by: trvon <me@trevon.dev>

* fix: preserve completion and retrieval contracts

Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures.

Signed-off-by: trvon <me@trevon.dev>

* fix: fence asynchronous enrichment completion

Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage.

Signed-off-by: trvon <me@trevon.dev>

* fix: bound semantic graph output reservation

Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward.

Signed-off-by: trvon <me@trevon.dev>

* fix: fence standalone embedding repair completion

Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass.

Signed-off-by: trvon <me@trevon.dev>

* fix: preserve enrichment failure reporting

Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged.

Signed-off-by: trvon <me@trevon.dev>

* test: align explicit hash prefix boundaries

Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun.

Signed-off-by: trvon <me@trevon.dev>

* test: align UI hash validation boundary

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-09-12 18:53:12 -06:00
trvon
296e732064 ci: make immutable uploads atomic
Signed-off-by: trvon <me@trevon.dev>
2026-09-04 17:14:43 -06:00
trvon
618bd5e0c0 ci: isolate experimental package repositories
Signed-off-by: trvon <me@trevon.dev>
2026-09-04 17:14:43 -06:00
trvon
0ab39df476
ci: attest experimental release artifacts
Signed-off-by: trvon <me@trevon.dev>
2026-09-03 21:39:17 -06:00
0xkite
000863e664
docs: replace SourceHut with Forgejo mirror (#91)
Signed-off-by: trvon <me@trevon.dev>
Co-authored-by: trvon <me@trevon.dev>
2026-09-03 17:16:57 -06:00
Trevon
6c1846ba80
refactor(daemon): decompose ServiceManager responsibilities (#79)
* fix(ci): enable OpenGrep custom ignore support

Signed-off-by: trvon <me@trevon.dev>

* fix(p2p): centralize backoff cap and bound test polling

Signed-off-by: trvon <me@trevon.dev>

* refactor(daemon): split ServiceManager responsibilities

Signed-off-by: trvon <me@trevon.dev>

* fix(daemon): narrow ServiceManager test-hook declarations

Signed-off-by: trvon <me@trevon.dev>

* chore(architecture): ignore local Brick artifacts

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-08-30 20:08:00 -04:00