Skip to content

pg push service#547

Open
mxfeinberg wants to merge 2 commits into
kenn-io:mainfrom
mxfeinberg:pg-push-service
Open

pg push service#547
mxfeinberg wants to merge 2 commits into
kenn-io:mainfrom
mxfeinberg:pg-push-service

Conversation

@mxfeinberg
Copy link
Copy Markdown

@mxfeinberg mxfeinberg commented May 26, 2026

Problem

I track token usage across AI tools with agentsview, and lately I've been
running more remote agent tools on machines other than my primary dev box.
Each of those machines records its own sessions locally. The Postgres support
(pg push / pg serve) already lets me review everything together, but
keeping the shared database current meant either remembering to run pg push
by hand on each machine or wiring up a bespoke cron job per host. I wanted a
touchless way to sync data from multiple machines into one place so remote
agent token usage shows up in my primary dashboard automatically, without
maintaining custom cron entries everywhere.

I also made some updates to hermes specific code because although tokens
were being passed through, the associated costs were not being aggregated
properly.

pg push service

A long-running auto-push daemon and a first-class way to install it as an OS
service, so a recorder machine keeps the shared Postgres current on its own.

agentsview pg push --watch start a foreground daemon that pushes to Postgres
shortly after sessions change, with a periodic floor as a safety net:

  • A file watcher (reusing the same machinery as serve) coalesces change
    events over a debounce window (--debounce, default 30s) and triggers a
    local sync + incremental push.
  • A periodic floor (--interval, default 15m) pushes regardless of file
    events and covers any directories the watch budget couldn't cover.
  • Pushes are serialized by a single goroutine and connect to Postgres lazily,
    reconnecting on error. A transiently unreachable database is logged and
    retried on the next trigger rather than crashing the daemon.
  • A single-instance lock prevents two watchers from racing on the push
    watermarks. On shutdown (SIGINT/SIGTERM) it performs one bounded final flush.
  • It reads the same [pg] config and project filters as pg push, and
    honors result_content_blocked_categories so the push path no longer
    diverges from serve.

agentsview pg service install|uninstall|status|start|stop|logs
installs and manages the daemon as a per-user OS service:

  • launchd LaunchAgent on macOS, systemd --user unit on Linux, behind a small
    platform abstraction with pure (golden-tested) unit-file rendering.
  • install validates that the Postgres DSN is resolvable before creating the
    service, and surfaces the systemd linger requirement so the service keeps
    running on headless boxes after logout.
  • status reports the manager state plus the last successful push time;
    logs -f tails the daemon log and survives log rotation.

The daemon reads the DSN from the existing config file (no credentials are
copied into unit files), so protecting config.toml is sufficient.

Notes

  • Platform support targets macOS (launchd) and Linux (systemd --user); the
    daemon command itself is cross-platform Go.
  • The generated unit always pins AGENTSVIEW_DATA_DIR to the install-time data
    dir, so the service resolves the same config regardless of the environment it
    starts in.
  • The watch daemon's initial resync is push-oriented and skips the serve-side
    Vacuum/BackfillSignals steps, since signal recomputation happens on the
    serve side.
  • A windows service was not added because I don't currently have access to a
    windows machine that I could use for testing.

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (6a35195)

No Medium, High, or Critical issues found.

All review outputs are clean; no actionable findings to include.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (5101e30)

High confidence issues remain; PR should not merge until the import break is fixed.

High

  • cmd/agentsview/pg_service.go:14
    New files import go.kenn.io/agentsview/internal/..., but the module is github.com/wesm/agentsview, with no module-path change in the diff. These packages will not resolve.
    Fix: Use the repository’s existing import path, e.g. github.com/wesm/agentsview/internal/config, in all new files.

Medium

  • cmd/agentsview/pg_service_manager.go:26
    buildServiceSpec validates pg.url using the installer’s current environment, but the rendered launchd/systemd units only preserve AGENTSVIEW_DATA_DIR. A config like pg.url = "${PG_URL}" installs successfully, then the service fails at runtime because PG_URL is absent.
    Fix: Either reject env-dependent PG URLs during service install or render the required PG environment variables into the service unit.

  • internal/pidlock/pidlock.go:26
    Acquire is not atomic: two concurrent processes can both see no live lock, write the shared .tmp file, rename over the lock path, and both return success.
    Fix: Acquire with an atomic primitive such as os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, ...), removing stale locks and retrying as needed.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (93270cc)

Medium-risk issues remain around single-instance locking and service environment propagation.

Medium

  • internal/pidlock/pidlock.go:41
    Acquire is not atomic: after checking/removing a stale lock, it writes a temp file and renames it over the target. Two near-simultaneous starts can both acquire the “single-instance” lock, allowing multiple pg push --watch daemons and potentially concurrent resyncs against the same temp DB path.
    Fix: Create the lock with an atomic primitive such as os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, ...) or an OS file lock, and avoid releasing a lock file that no longer belongs to the current process.

  • cmd/agentsview/pg_service_manager.go:27
    Service install validates the resolved PG URL using the current environment, but the generated launchd/systemd units only preserve AGENTSVIEW_DATA_DIR. Configurations that use AGENTSVIEW_PG_URL or ${VAR} expansion can pass install and then fail when the background service starts without those variables.
    Fix: Persist the required resolved PG settings into the service environment/unit, support an environment file, or reject env-dependent PG URLs during service install with a clear error.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (9b511a2)

High severity issue found.

High

  • cmd/agentsview/pg_service_systemd.go:40

    systemdManager.render interpolates spec.BinPath and spec.DataDir directly into a systemd unit using quoted strings. systemd unit files are not shell scripts, but newlines and quote characters inside these values can break out of the intended directive and inject additional unit directives.

    Since spec.DataDir comes from configuration, a crafted data dir containing quote/newline-injected directives could cause agentsview pg service install to write, enable, and start a user service that runs attacker-chosen commands as the current user.

    Suggested remediation: escape or quote values using systemd-safe syntax, or reject unsafe path characters such as \n, \r, ", and control bytes before writing the unit. Cover both ExecStart and Environment with tests.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (1929c73)

High: PR has compile-breaking import path issues and a service environment persistence bug that can make installed services run with different config than validated.

High

  • cmd/agentsview/pg_watch.go:13
    The new command files import internal packages via go.kenn.io/agentsview/..., but this repository’s module path is github.com/wesm/agentsview. Because these are internal packages, the mismatched import path will break compilation for the added watch/service code.
    Fix: Change the new imports in pg_watch.go, pg_service.go, pg_service_manager.go, and their tests to github.com/wesm/agentsview/internal/....

Medium

  • cmd/agentsview/pg_service_manager.go:20
    Service installation only rejects env-dependent pg.url, but the installed service environment only preserves AGENTSVIEW_DATA_DIR. If the install-time config depends on other env vars such as AGENTSVIEW_PG_SCHEMA, AGENTSVIEW_PG_MACHINE, or agent session directory env vars, foreground install validation can succeed while the background service later runs with different schema, machine, or source directories.
    Fix: Either reject all env-derived config that affects service runtime, or render the required resolved environment values into both launchd and systemd units.

Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (b559140)

No Medium, High, or Critical findings were reported.

All completed reviews found no actionable issues; one review was skipped due to quota exhaustion.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

Note: gemini review skipped (agent quota exhausted)

@mxfeinberg mxfeinberg marked this pull request as ready for review May 26, 2026 21:57
@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (bfd3d7d)

No Medium, High, or Critical findings were reported.

All completed reviews found no issues; one review was skipped due to quota exhaustion.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

Note: gemini review skipped (agent quota exhausted)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 27, 2026

roborev: Combined Review (84da010)

High: PR has a compile-blocking module import issue; also includes Medium concerns around Linux service logs and a flaky stdin test.

High

  • cmd/agentsview/pg_service.go:13
    New files import go.kenn.io/agentsview/internal/..., but this repository’s module path is github.com/wesm/agentsview. Go will treat those as external internal-package imports and fail to compile.

    Fix: Change the new imports to the repository module path, e.g. github.com/wesm/agentsview/internal/config, consistently across the added files and tests.

Medium

  • cmd/agentsview/pg_service_systemd.go:33
    The systemd unit ignores spec.LogPath, while pg service logs tails pg-watch.log. stderr/stdout fatal messages from the service go to journald instead of that file, so Linux users can get an empty or incomplete service log.

    Fix: Either configure StandardOutput/StandardError to append to spec.LogPath, or make the systemd log command read from journald.

  • cmd/agentsview/pg_service_test.go in TestPromptYesNo
    The test temporarily overrides the global os.Stdin variable. This introduces a race condition and can cause flaky failures if other tests or goroutines access os.Stdin concurrently.

    Fix: Refactor promptYesNo in cmd/agentsview/pg_service.go to accept an io.Reader, passing os.Stdin in production code and a pipe/mock reader in tests.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 27, 2026

roborev: Combined Review (0084777)

Medium: tailFile can miss new log data after truncation or get stuck after rotation.

Medium

  • Location: cmd/agentsview/pg_service.go:696-724
    Problem: tailFile only checks the already-open file descriptor. If the log is truncated and rewritten past the current offset, the prefix of the new log can be skipped. If the log is rotated, the open descriptor can keep pointing at the old unlinked inode and remain at EOF indefinitely.
    Fix: Periodically stat the path itself and reopen when the path’s file identity changes or when truncation is detected.

Low-severity findings were omitted per instructions.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 28, 2026

roborev: Combined Review (ad32c91)

No Medium, High, or Critical findings were reported across the reviews.

All reported findings were below the requested severity threshold and have been omitted.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@mxfeinberg
Copy link
Copy Markdown
Author

@wesm , I just pushed the last couple of tweaks to this work. As mentioned in my top comment, I added a system service (for MacOS and Linux) intended to push data from remote machines to a central pg instance without the need for a dedicated cron or manual pushes. It was motivated by a deployment of hermes that I just started using. I also added some other tweaks to the hermes parser because none of its tokens were being priced properly via my deployment.

Let me know if there's any interest in merging these changes or changes like this! No worries if not, but I've been a fan of using agentsview and roborev per a recommendation from Hugh, and I thought I could contribute back.

Squashes the pg-push-service branch. Included changes:

- chore: gitignore graphify-out analysis directory
- feat: add pidlock package for single-instance locking
- refactor: harden pidlock (EPERM-as-alive, idempotent Release, more tests)
- refactor: make startFileWatcher accept an onChange callback
- refactor: add setupLogFileNamed helper
- refactor: extract resolvePushProjects; add watch fields to PGPushConfig
- test: cover config-conflict branches in resolvePushProjects
- feat: add coalescing push control loop for pg watch
- feat: bound pushLoop shutdown flush with a timeout
- feat: add lazy-reconnecting pgPusher and resolveWatchTargets
- test: cover pgPusher reconnect after EnsureSchema error
- feat: add pg push --watch auto-push daemon
- refactor: log unwatched roots and document classifier wiring in pg watch
- feat: add service spec and manager interface for pg service
- feat: add launchd service manager for pg service
- fix: stop launchd service via bootout so KeepAlive does not restart it
- feat: add systemd --user service manager and platform selector
- fix: quote paths in systemd unit and add start/stop/uninstall tests
- feat: add pg service command group (install/uninstall/status/start/stop/logs)
- fix: recover from log truncation in pg service logs -f; test tailFile and promptYesNo
- docs: document pg push --watch and pg service
- fix: respect result_content_blocked_categories on pg push and pg push --watch
- feat: show last-push in pg service status; warn on watch-only flags; note KeepAlive tradeoff
- fix session aggregate token columns, add more models
- address lock issues, reject env dependent URLs
- fix: harden pg service rendering and dedupe env-URL check
- feat: warn at pg service install about uninherited runtime env vars
- fix: clear lint findings and make pg-service path tests cross-platform
- fix: capture systemd stdout/stderr in log and inject promptYesNo reader
- fix(pg): sync usage_events for zero-message sessions on push
- fix(hermes): catalog-price "included" usage with no cost source
- chore(db): bump dataVersion to 30 for hermes cost re-parse

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wesm wesm force-pushed the pg-push-service branch from ad32c91 to b4493cb Compare May 28, 2026 19:35
@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 28, 2026

roborev: Combined Review (b4493cb)

No Medium, High, or Critical findings were reported.

The only finding was Low severity and has been omitted per instructions.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 28, 2026

roborev: Combined Review (3345c2b)

Summary verdict: One medium issue needs attention before merge.

Medium

  • internal/pricing/fallback.go:5: FallbackPricing() adds new gpt-5.5 and openrouter/owl-alpha fallback rows, but FallbackVersion was not bumped. Existing databases that already stored the current fallback seed version will skip re-seeding, so these new rows will not be added and Hermes gpt-5.5 usage can remain unpriced.

    Fix: Bump FallbackVersion whenever changing the fallback pricing list, and consider adding a test that fails when fallback rows change without a version change.


Synthesized from 2 reviews (agents: codex | types: default, security)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant