Skip to content

feat(vfs): thread-safe config registry for per-database VFS configuration#1157

Open
corylanou wants to merge 6 commits intomainfrom
issue-1150-feat-vfs-support-thread-safe-configuration-via-uri-parameters-and-pragmas-no-setenv-required
Open

feat(vfs): thread-safe config registry for per-database VFS configuration#1157
corylanou wants to merge 6 commits intomainfrom
issue-1150-feat-vfs-support-thread-safe-configuration-via-uri-parameters-and-pragmas-no-setenv-required

Conversation

@corylanou
Copy link
Copy Markdown
Collaborator

@corylanou corylanou commented Feb 23, 2026

Merge Blocker

This PR currently depends directly on github.com/corylanou/sqlite3vfs v0.0.0-20260421183553-1656bf6ea7f7, from branch litestream-uri-filename-support in the fork. This is intentional for review/CI and avoids any replace directive.

Do not merge this PR until psanford/sqlite3vfs#18 is merged upstream and this dependency/import path is switched back to github.com/psanford/sqlite3vfs.

Description

Add thread-safe VFS configuration for per-database runtime settings and URI parameters. This removes the need for loadable-extension users to rely on process-wide setenv() after startup, and lets callers configure options such as replica_url and poll_interval directly in the SQLite database URI.

Key changes:

  • Config registry (vfs_config.go): SetVFSConfig/GetVFSConfig/DeleteVFSConfig keyed by database name, protected by sync.RWMutex with defensive copies on set/get to prevent concurrent mutation.
  • URI configuration: supported VFS settings can be parsed from SQLite URI parameters and merged over registry configuration per connection.
  • Temporary direct fork dependency: imports github.com/corylanou/sqlite3vfs directly from branch litestream-uri-filename-support until add URI parameter support via optional FilenameOpener interface psanford/sqlite3vfs#18 is available upstream. There is intentionally no replace directive.
  • No internal sqlite3vfs fork: removed the copied internal/sqlite3vfs package from this PR.
  • Unified parser: GoLitestreamConfigure() and URI config now share the same VFSConfig.Set() validation path.
  • Per-connection clients: openMainDB() creates a per-connection ReplicaClient when config specifies replica_url; clients are cleaned up on VFSFile.Close() and on Open() failure.
  • Optional env vars: LitestreamVFSRegister() no longer requires LITESTREAM_REPLICA_URL, enabling per-database config via registry or URI parameters.
  • No PRAGMA support: PRAGMA-based configuration was removed per review feedback.

Motivation and Context

setenv() is not thread-safe on Linux. Library users who don't know VFS config values at process startup cannot safely call os.Setenv later. This PR adds registry and URI-based configuration so callers can configure each database connection without mutating process-wide environment variables.

Fixes #1150
Fixes #1231

How Has This Been Tested?

  • go test -tags vfs -count=1 .
  • go test -tags vfs -count=1 ./cmd/litestream-vfs
  • go test github.com/corylanou/sqlite3vfs
  • go vet -tags vfs . ./cmd/litestream-vfs
  • go build -tags "SQLITE3VFS_LOADABLE_EXT vfs" ./cmd/litestream-vfs
  • Commit hooks: go-imports-repo, go-vet-repo-mod, go-staticcheck-repo-mod

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist

  • My code follows the code style of this project (go fmt, go vet)
  • I have tested my changes
  • I have updated the documentation accordingly (if needed)

Copy link
Copy Markdown
Owner

@benbjohnson benbjohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with setting parameters in the URL but I think the PRAGMA part is overkill. These should all be set at initialization time and PRAGMAs can be used anytime.

@corylanou corylanou changed the title feat(vfs): thread-safe config registry + extended PRAGMAs feat(vfs): thread-safe config registry for per-database VFS configuration Feb 25, 2026
@corylanou corylanou force-pushed the issue-1150-feat-vfs-support-thread-safe-configuration-via-uri-parameters-and-pragmas-no-setenv-required branch from 661885e to 6929314 Compare March 3, 2026 20:18
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 3, 2026

PR Build Metrics

⚠️ Attention needed — vulnerabilities found, Go toolchain outdated (1.25.8 → 1.25.9)

Check Status Summary
Binary size 35.96 MB (+56.0 KB / +0.15%)
Dependencies ℹ️ 3 added, 2 removed
Vulnerabilities ⚠️ Issues found — expand details below
Go toolchain ⚠️ 1.25.8 → 1.25.9 available
Module graph 1206 edges (+2)

Binary Size

Size Change
Base (bd2c6e6) 35.90 MB
PR (2283bf6) 35.96 MB +56.0 KB (+0.15%)

Dependency Changes

Added:

  • github.com/lmittmann/tint v1.1.3
  • github.com/mattn/go-isatty v0.0.20
  • github.com/psanford/sqlite3vfs v0.0.0-20260421161058-a9ece85b5678

Removed:

  • github.com/mattn/go-isatty v0.0.20 // indirect
  • github.com/psanford/sqlite3vfs v0.0.0-20251127171934-4e34e03a991a // direct

govulncheck Output

=== Symbol Results ===

Vulnerability #1: GO-2026-4947
    Unexpected work during chain building in crypto/x509
  More info: https://pkg.go.dev/vuln/GO-2026-4947
  Standard library
    Found in: crypto/x509@go1.25.8
    Fixed in: crypto/x509@go1.25.9
    Example traces found:
      #1: internal/resumable_reader.go:76:22: internal.ResumableReader.Read calls http.readWriteCloserBody.Read, which eventually calls x509.Certificate.Verify

Vulnerability #2: GO-2026-4946
    Inefficient policy validation in crypto/x509
  More info: https://pkg.go.dev/vuln/GO-2026-4946
  Standard library
    Found in: crypto/x509@go1.25.8
    Fixed in: crypto/x509@go1.25.9
    Example traces found:
      #1: internal/resumable_reader.go:76:22: internal.ResumableReader.Read calls http.readWriteCloserBody.Read, which eventually calls x509.Certificate.Verify

Vulnerability #3: GO-2026-4870
    Unauthenticated TLS 1.3 KeyUpdate record can cause persistent connection
    retention and DoS in crypto/tls
  More info: https://pkg.go.dev/vuln/GO-2026-4870
  Standard library
    Found in: crypto/tls@go1.25.8
    Fixed in: crypto/tls@go1.25.9
    Example traces found:
      #1: nats/replica_client.go:195:25: nats.ReplicaClient.connect calls nats.Connect, which eventually calls tls.Conn.Handshake
      #2: server.go:135:31: litestream.Start calls http.Server.Serve, which eventually calls tls.Conn.HandshakeContext
      #3: internal/resumable_reader.go:76:22: internal.ResumableReader.Read calls http.readWriteCloserBody.Read, which calls tls.Conn.Read
      #4: internal/hexdump.go:30:14: internal.Hexdump calls fmt.Fprintf, which calls tls.Conn.Write
      #5: heartbeat.go:55:30: litestream.HeartbeatClient.Ping calls http.Client.Do, which eventually calls tls.Dialer.DialContext

Vulnerability #4: GO-2026-4865
    JsBraceDepth Context Tracking Bugs (XSS) in html/template
  More info: https://pkg.go.dev/vuln/GO-2026-4865
  Standard library
    Found in: html/template@go1.25.8
    Fixed in: html/template@go1.25.9
    Example traces found:
      #1: nats/replica_client.go:460:32: nats.ReplicaClient.DeleteAll calls errors.joinError.Error, which calls template.Error.Error
      #2: server.go:135:31: litestream.Start calls http.Server.Serve, which eventually calls template.Template.Execute
      #3: server.go:135:31: litestream.Start calls http.Server.Serve, which eventually calls template.Template.ExecuteTemplate
      #4: replica.go:712:21: litestream.Replica.Restore calls http.http2requestBody.Close, which eventually calls template.Template.Funcs
      #5: replica.go:712:21: litestream.Replica.Restore calls http.http2requestBody.Close, which eventually calls template.Template.Parse
      #6: replica.go:1637:27: litestream.WriteTXIDFile calls fmt.Fprintln, which eventually calls template.context.String

Your code is affected by 4 vulnerabilities from the Go standard library.
This scan also found 2 vulnerabilities in packages you import and 1
vulnerability in modules you require, but your code doesn't appear to call these
vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric Value
Build time 53s
Go version go1.25.8
Commit 2283bf6

History (4 previous)

Commit Updated Status Summary
9f96d07 2026-04-21 17:06 UTC 35.96 MB (+56.0 KB / +0.15%)
ca975b3 2026-04-21 12:50 UTC 35.96 MB (+56.0 KB / +0.15%)
85f55ab 2026-04-20 23:52 UTC 35.90 MB (0.0 KB / 0.00%)
e9ad8b7 2026-03-11 01:02 UTC 35.82 MB (0.0 KB / 0.00%)

🤖 Updated on each push.

Replace the requirement for setenv() (not thread-safe on Linux) with a
Go config registry that allows per-database VFS configuration at
runtime. Library users who don't know config values at process startup
can now safely configure each database connection without calling
os.Setenv.

Key changes:
- Add VFSConfig registry (SetVFSConfig/GetVFSConfig/DeleteVFSConfig)
  keyed by database name with RWMutex protection
- openMainDB() checks registry and creates per-connection ReplicaClient
  when config specifies a replica_url
- Per-connection clients are cleaned up on VFSFile.Close()
- LitestreamVFSRegister() no longer requires LITESTREAM_REPLICA_URL env
  var, enabling per-database config via registry
- Add GoLitestreamConfigure C export for Python/C callers
- Add PRAGMAs: litestream_poll_interval (R/W), litestream_cache_size
  (R/W), litestream_hydration_enabled (R/O), litestream_log_level (R/W),
  litestream_replica_url (R/O)
- Protect PollInterval/CacheSize PRAGMA access with mutex

Fixes #1150
Address Ben's review feedback: PRAGMAs are overkill for config that
should be set at initialization time. Also fix issues found during
code review.

Changes:
- Remove all new PRAGMAs (poll_interval, cache_size, hydration_enabled,
  log_level, replica_url)
- Return defensive copies from SetVFSConfig/GetVFSConfig to prevent
  concurrent mutation of shared config
- Add nil client check in openMainDB to fail early with clear error
- Close per-connection client on f.Open() failure to prevent leaks
- Revert unnecessary mutex addition in monitorReplicaClient
- Add TestVFSConfig_CopyOnSetAndGet and TestVFS_NilClientReturnsError
@corylanou corylanou force-pushed the issue-1150-feat-vfs-support-thread-safe-configuration-via-uri-parameters-and-pragmas-no-setenv-required branch from 6929314 to 5f029df Compare March 11, 2026 01:01
Expose SQLite URI parameters to the Litestream VFS via an internal sqlite3vfs fork, then parse supported VFS settings from database URIs. This lets loadable-extension users set options such as poll_interval without PRAGMA support.
@github-actions github-actions Bot added metrics: vulns-found govulncheck found vulnerabilities metrics: go-update Go toolchain has a newer patch release labels Apr 20, 2026
@corylanou
Copy link
Copy Markdown
Collaborator Author

Updated for re-review:

  • Removed PRAGMA-based configuration per review feedback.
  • Added URI parameter support so options like poll_interval can be set from the SQLite database URI.
  • Added an internal sqlite3vfs fork solely to expose SQLite filename/URI metadata to the Go VFS layer.
  • Updated the PR body to note that this also fixes Ability to set the poll interval for VFS when loaded as an extension #1231.

Latest commit is d63169d; CI is green.

@corylanou corylanou requested a review from benbjohnson April 21, 2026 00:07
Close per-connection replica clients if Init fails so URI-based connections do not leak backend resources.
Replace the internal sqlite3vfs fork with the upstream module import and a temporary replace to the URI-support branch from psanford/sqlite3vfs#18.

Add an end-to-end URI replica_url test and keep test fault injection active for FilenameOpener-based opens.
Remove the temporary replace directive and import the corylanou/sqlite3vfs fork directly until upstream URI filename support is merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

metrics: go-update Go toolchain has a newer patch release metrics: vulns-found govulncheck found vulnerabilities ready for review

Projects

None yet

2 participants