feat: Add KV store API for automation state persistence

[jpshackelford]

Summary

This PR implements the KV store API for automation state persistence, including both the design document and the service-side implementation.

Problem

One of the use cases for automations is implementing integrations. Some integrations are stateless—webhook responders that receive an event, do work, and complete. But other jobs require small amounts of data storage to work effectively.

For example, an automation that summarizes Slack messages might store the last processed timestamp, then on the next run fetch only items since that date. But where should it store this data?

A GitHub repo is not the right tool. External services (JSONBin, Redis Cloud, etc.) work but require users to provision and manage infrastructure. If external systems are required for such a common use case, that erodes the simplicity of a batteries-included platform.

Solution

A built-in key-value store API scoped per-automation:

• Easy — simple GET/SET, familiar Redis-like semantics
• Flexible — JSON values, counters, lists, nested paths
• Secure — application-level encryption, strict isolation between automations
• Concurrent — batch operations with optimistic concurrency control

We do not need high performance—just simplicity and security for occasional state persistence.

Changes

Design Document

• docs/kv-store-design.md - Full design specification for the KV store feature

New Files

• automation/kv_router.py - KV API endpoints including batch operations
• automation/kv_schemas.py - Request/response schemas including batch operation types
• automation/kv_helpers.py - Validation and encryption helpers
• automation/utils/kv.py - JWT and encryption utilities
• migrations/versions/005_add_kv_store.py - Database migration
• tests/test_kv_router.py - Integration test coverage
• tests/test_kv_batch.py - Unit tests for batch operations
• tests/test_kv_helpers.py - Unit tests for helpers

Modified Files

• automation/models.py - Added AutomationKV model and enable_kv_store flag on Automation
• automation/schemas.py - Added enable_kv_store to request/response schemas
• automation/router.py - Include enable_kv_store in automation creation
• automation/dispatcher.py - Generate AUTOMATION_KV_TOKEN for runs with KV enabled
• automation/config.py - Added kv_secret setting
• automation/app.py - Mount KV router
• pyproject.toml - Added pyjwt and jwcrypto dependencies

API Endpoints

Endpoint	Method	Description
/v1/kv	GET	List all keys
/v1/kv/batch	POST	Atomic batch operations with optimistic concurrency
/v1/kv/{key}	GET	Get value (supports ?path= and ?meta=true for version)
/v1/kv/{key}	PUT	Set value (supports ?nx, ?xx, ?if_version)
/v1/kv/{key}	PATCH	Update nested path (supports ?if_version)
/v1/kv/{key}	DELETE	Delete key (supports ?if_version)
/v1/kv/{key}/incr	POST	Atomic increment
/v1/kv/{key}/decr	POST	Atomic decrement
/v1/kv/{key}/lpush	POST	Push to list front
/v1/kv/{key}/rpush	POST	Push to list back
/v1/kv/{key}/lpop	POST	Pop from list front
/v1/kv/{key}/rpop	POST	Pop from list back
/v1/kv/{key}/len	GET	Get list length

Concurrency Controls

Single-Document Model

All state for an automation is stored in a single encrypted JSON document. This eliminates multi-key deadlock scenarios—all operations serialize through a single row lock.

Lock Timeout (Deadlock Prevention)

• 5-second lock timeout prevents stuck transactions from cascading
• Returns HTTP 409 Conflict so clients can retry with backoff

Optimistic Concurrency ($version)

State includes a $version counter that auto-increments on every write. Use if_version for safe read-modify-write patterns.

On individual endpoints (PUT, PATCH, DELETE):

PUT /v1/kv/state?if_version=4
→ 200 OK (version was 4, now 5)
→ 409 Conflict {"error": "version_mismatch", "expected_version": 4, "actual_version": 6}

On batch endpoint:

POST /v1/kv/batch
{
  "if_version": 4,
  "operations": [
    {"op": "set", "key": "state", "value": {...}}
  ]
}
// → 200: {"version": 5, "results": [...]}
// → 409: {"error": "version_mismatch", "expected_version": 4, "actual_version": 6}

Get current version with ?meta=true:

GET /v1/kv/state?meta=true
// → {"key": "state", "value": {...}, "version": 5, "created_at": "...", "updated_at": "..."}

Batch Operations

Execute multiple operations atomically in a single transaction:

POST /v1/kv/batch
{
  "operations": [
    {"op": "incr", "key": "counter"},
    {"op": "rpush", "key": "log", "value": {"event": "sync"}}
  ]
}
// → {"version": 5, "results": [...]}

Benefits:

• Single decrypt → N modifications → single encrypt → commit
• Reduces crypto overhead for multiple updates
• All operations succeed or none do (atomic)

Reserved Keys

Keys starting with $ are reserved for system use (e.g., $version). Client attempts to set/delete $-prefixed keys are rejected with HTTP 400.

When to Use Version Checks

Operation	Version Check Needed?	Why
PUT (set)	Yes	Read-modify-write pattern
PATCH	Yes	Read-modify-write pattern
DELETE	Optional	Can be useful for safe cleanup
incr/decr	No	Atomic, conflict-free
lpush/rpush	No	Atomic, conflict-free
lpop/rpop	No	Atomic, conflict-free

Deployment

Configuration

The KV store requires the AUTOMATION_KV_SECRET environment variable for JWT signing and value encryption.

Chart PR: OpenHands/OpenHands-Cloud#576 wires this up by reusing the existing jwt-secret — no new secrets needed!

Out of Scope

As noted in the design doc comment, admin debugging tooling for KV data is out of scope for this PR. The encrypted data will be secure at rest; admin tools can be added in a follow-up issue once the permission model is clarified with the platform team.

Related

• Closes Feature: Key-Value Store for Automation State Persistence #67
• Related: Implement hierarchical concurrency limits for automation runs #72 (concurrency limits), Implement event queue management with backpressure strategies #74 (backpressure strategies)
• Chart changes: feat(automation): Add KV store secret configuration OpenHands-Cloud#576

---

This PR was updated by an AI agent (OpenHands) on behalf of @jpshackelford.

[github-actions]

🚀 Deploy Preview PR Created/Updated

A deploy preview has been created/updated for this PR.

Deploy PR: https://github.com/OpenHands/deploy/pull/3920
Automation SHA: 4fc3f05e245b7c3fc24f749c53cee39d0676e5d6
Last updated: May 19, 2026, 12:39:19 PM ET

Once the deploy PR's CI passes, the automation service will be deployed to the feature environment.

[github-actions]

[jpshackelford]

🔗 Chart PR Linked for Preview Deploy

The deploy preview PR has been updated to use the preview chart from OpenHands-Cloud PR #576, which adds the AUTOMATION_KV_SECRET environment variable configuration.

Deploy PR: https://github.com/OpenHands/deploy/pull/3920
Chart Version: 0.4.13-alpha.576

Once the deploy PR's CI passes, the feature environment will include both:

• The KV store API code changes from this PR
• The chart changes that wire up the KV secret

---

This comment was created by an AI agent (OpenHands) on behalf of the user.

[jpshackelford]

Out of Scope: Admin Debugging Tooling

We are calling out of scope for this implementation:

• An automation-specific role/permission that would allow an engineer to obtain a key to list, retrieve, and manipulate KV data stored for an automation
• Admin CLI or API endpoints for debugging encrypted KV store contents

Context: Automations are linked to organizations (org_id), but the automation service delegates role/permission management to the main OpenHands platform. Adding admin debugging capabilities would require coordination with the platform team to define appropriate permissions (e.g., automation:kv:admin).

For now: All KV data is encrypted at rest. Admin debugging tooling can be added in a follow-up issue once the permission model is clarified.

---

This comment was migrated from PR #68 (design document PR) which has been merged into this PR.

[jpshackelford]

✅ KV Store E2E Tests Passed (Thorough Mode)

Tested against jps01.r9.all-hands.dev with PR-69 image:

============================================================
KV STORE E2E TEST (THOROUGH MODE)
Running 26 tests
============================================================

[TEST] SET and GET
  PUT /test_key: 201
  GET /test_key: 200
  PASS

[TEST] DELETE
  DELETE /to_delete: 200
  GET after delete: 404
  PASS

[TEST] INCR and DECR
  INCR by 5: 200, value=15
  DECR by 3: 200, value=12
  PASS

[TEST] List operations
  RPUSH 'a': 200, length=1
  LPUSH 'z': 200, length=4
  LPOP: 200, value=z
  RPOP: 200, value=c
  LEN: 200, length=2
  PASS

[TEST] Nested path operations
  PATCH database.port=5433: 200
  GET with path: 200, value=5433
  PASS

[TEST] Conditional SET (nx)
  PUT with nx=true (new): 201
  PUT with nx=true (exists): 409
  PASS

[TEST] List keys
  GET /kv: 200, count=7
  PASS

[TEST] GET with metadata
  GET with meta=true: 200
  created_at: 2026-04-25T07:03:03.895583+00:00
  PASS

[TEST] GET non-existent key
  GET /nonexistent: 404
  PASS

[TEST] GET non-existent nested path
  GET with invalid path: 404
  PASS

[TEST] DELETE non-existent key
  DELETE /nonexistent: 200, deleted=False
  PASS

[TEST] Conditional SET (xx)
  PUT with xx=true (missing): 409
  PUT with xx=true (exists): 200
  PASS

[TEST] PATCH non-existent key
  PATCH /nonexistent: 404
  PASS

[TEST] INCR new key
  INCR new key: 200, value=1
  PASS

[TEST] DECR new key
  DECR new key: 200, value=-1
  PASS

[TEST] INCR non-numeric
  INCR string value: 400
  PASS

[TEST] LPOP empty list
  LPOP empty: 200, value=None
  PASS

[TEST] LPOP non-existent key
  LPOP nonexistent: 200, value=None
  PASS

[TEST] RPUSH to non-list
  RPUSH to dict: 400
  PASS

[TEST] LEN non-existent key
  LEN nonexistent: 404
  PASS

[TEST] Special characters in key
  PUT /test-key_123: 201
  PASS

[TEST] Store null value
  PUT null: 422
  PASS

[TEST] Various JSON types
  string_type: OK
  number_int: OK
  number_float: OK
  boolean_true: OK
  boolean_false: OK
  array_type: OK
  nested_obj: OK
  PASS

[TEST] Auth - missing token
  GET without token: 422
  PASS

[TEST] Auth - invalid token
  GET with invalid token: 401
  PASS

[TEST] Invalid JSON body
  PUT invalid JSON: 422
  PASS

============================================================
RESULTS (THOROUGH): 26 passed, 0 failed
============================================================

KV_STORE_ALL_TESTS_PASSED

All 26 test cases verified:

• ✅ Basic SET/GET with proper status codes (201 create, 200 read)
• ✅ DELETE with 404 on subsequent GET
• ✅ INCR/DECR numeric operations
• ✅ List operations (RPUSH, LPUSH, LPOP, RPOP, LEN)
• ✅ Nested path PATCH and GET
• ✅ Conditional SET with nx=true and xx=true
• ✅ List all keys
• ✅ Metadata retrieval with meta=true
• ✅ Error handling (404 for missing keys, 400 for type errors, 409 for conflicts)
• ✅ Auth validation (401 invalid token, 422 missing token)
• ✅ Input validation (422 for null body, invalid JSON)
• ✅ Various JSON types (string, int, float, bool, array, object)

---

This comment was created by an AI agent (OpenHands) on behalf of @jpshackelford.

[jpshackelford]

Still doing some clean-up in response to OpenHands review feedback.

[tofarr]

• There is a lot of code related to wheterh kv is enabled or whether it is enabled for a automation - I think it should just always be enabled - something that each automation gets by default
• Do we need to update the skill for automations to let the agent know that this facility is available?
• We have a lot of code related to encryption. Could we use the Cipher class from the sdk to simplify / remove some of this?

[tofarr]

Picking this up on John's behalf since it's unblocking some work on my end. Addressed @tofarr's review in 4fc3f05:

1. "a lot of code related to whether kv is enabled / per-automation" — Removed entirely. There is no longer an enable_kv_store field on Automation, no enable_kv_store knob in the schemas / router / preset router, and no per-automation lock-timeout override. The dispatcher injects AUTOMATION_KV_TOKEN for every run whenever the service has AUTOMATION_KV_SECRET configured, and the row-lock timeout is now a single service-wide config value (AUTOMATION_KV_LOCK_TIMEOUT_MS, default 5000ms).

2. "update the skill for automations" — Deferring to a follow-up PR; tracked in the design doc's implementation checklist. Wanted to keep this PR scoped to the service-side cleanup so it's easier to review.

3. "could we use the Cipher class from the SDK" — Done. utils/kv.py now uses openhands.sdk.utils.cipher.Cipher for value encryption (Fernet under the hood). Dropped the cryptography>=42 direct dependency and the custom AES-256-GCM JWE wrapper code. Since Fernet emits URL-safe base64 text rather than raw bytes, automation_kv.state_encrypted is now TEXT (was BYTEA/LargeBinary); the migration was updated in place and the old 007_add_kv_lock_timeout migration is gone (its only purpose was the per-automation timeout column).

Also:

• Skipped the new COMMENT ON TABLE statements when running migrations on SQLite so the existing test_db.py::TestSqliteMigrations tests still pass.
• Updated docs/kv-store-design.md, docs/kv-store-client-guide.md, and docs/kv-store-test-plan.md to match (removed all references to enable_kv_store, per-automation kv_lock_timeout_ms, AUTOMATION_JWT_SECRET, jwcrypto, JWE/A256GCM, and LargeBinary).

KV tests (test_kv_helpers, test_kv_batch, test_kv_concurrency) and test_db.py all pass locally. Remaining test ERRORs are testcontainers/Docker-related and reproduce on main too, so they're not from this change.

This comment was written by an AI agent (OpenHands) on behalf of @jpshackelford.

[tofarr]

Dropping by as a near-term consumer — I'm building slack-channel-listener (a skill + plugin that lets users wire any Slack channel up as an OpenHands trigger), and this PR is exactly the primitive I need. Before I saw it I was about to invent a SLACK_STATE_DIR=/automation/storage/state convention with a bundled SQLite file, which would have been the wrong layer to solve it at.

I've gone ahead and built against the API as specced in docs/kv-store-client-guide.md — state.py in extensions#245 has a pluggable Store interface with two backends, KVApiStore (preferred, activates when AUTOMATION_KV_TOKEN is injected) and SQLiteStore (fallback, keeps the skill usable on runtimes that don't ship this PR yet). The KV implementation uses the single-document model with ?if_version=N and jittered exponential backoff on 409 per the client guide.

A few pieces of consumer feedback while it's fresh:

Strong points

• The single-document model + row-lock + $version story is the right call for this workload. Concurrent contention is rare (cron is single-instance by default) and when it does happen, optimistic retry is dead simple to implement client-side.
• Per-automation isolation + encryption-at-rest means I don't have to think about secrets management for state — this would have been the most annoying part of rolling my own.
• AUTOMATION_KV_TOKEN + AUTOMATION_ENABLE_KV_STORE env-var injection in dispatcher.py makes the client-side detection trivial: I just check for the token's presence in get_store().
• Batch operations are over-spec for my use case but I can see them being load-bearing for skills that need to update several counters atomically (e.g., "increment per-user mention count AND push to recent-mentions list").

Things I bumped into while implementing

1. No TTL primitive. For my use case I need to drop entries older than 2 × poll_lookback on every run to stay under the 64 KB cap. I implemented it client-side as a prune_older_than(cutoff_iso) that does an RMW and filters the dict — fine, but a server-side ?ttl=seconds on PUT (or a periodic-prune query) would generalise nicely for any skill that accumulates per-key timestamps. Not a blocker, but file under "obvious follow-up".
2. 64 KB ceiling. For my {<channel>:<ts>: {...}} shape that's ~600–1000 entries. Plenty with pruning, but 413 with no per-key delete fallback would be a bad surprise for callers who haven't read the guide carefully. Right now I handle 413 by aggressively pruning to last-24h and retrying once, but it's not a great UX — the script either has to know its own retention policy or guess. Two things would help:
   • The 413 response body could include current_size_bytes so clients can decide how much to prune.
   • Document somewhere prominent that 64 KB is JSON-encoded after encryption-overhead, not raw value bytes, so callers can size accurately.
3. AUTOMATION_API_URL is implicit. The client guide uses it freely but it's an existing env var, not introduced by this PR. My factory ends up checking both AUTOMATION_API_URL and (fallback) OPENHANDS_CLOUD_API_URL for resilience. Worth being explicit in the client guide that this env var pre-dates the KV store and is contributed by the dispatcher in general, not by enable_kv_store: true.
4. Python helper would be high-leverage. I ended up writing ~60 lines of urllib-based RMW-with-retry logic that essentially recapitulates the safe_update() example in the client guide. Every skill that uses KV will rewrite some version of this. Worth considering whether a tiny openhands_automation_kv PyPI package (or, simpler, an automation-kv plugin in OpenHands/extensions) should land alongside this PR with claim() / txn() / incr() helpers and built-in backoff. I'm happy to take the first cut at that if you want — extensions#245 has working code I can extract.
5. reactions:read is not the right concurrency primitive for everything Slack. Just confirming explicitly that this PR removes the temptation to use it — my earlier draft of the slack-channel-listener was using Slack reactions as a state machine (claim by adding 👀, finish with ✅), and it's clever but it leaks implementation detail into the user-facing channel and is fragile to scope changes. The KV store is unambiguously the right layer. I've now demoted reactions in extensions#245 to opt-in visual UX only.

Question I couldn't answer from the docs

• What's the lifetime of AUTOMATION_KV_TOKEN? For sub-10-minute cron runs it doesn't matter, but I'd like to know what "long" means before recommending the KV path for batch-mode automations.

Looking forward to this landing. Happy to be the first dogfood consumer once it does — I'll flip extensions#245 from draft to ready-for-review the moment a Cloud build with this PR is reachable.

This comment was posted by an AI agent (OpenHands) on behalf of @tofarr.

[jpshackelford]

@tofarr Awesome feedback thank you. Will incorporate your feedback or you are of course welcome to do so.

Beyond that, things I was thinking of doing still include:

• Building the client library and a CLI to make it easy to build with this and to allow developers to debug their work given the that DB values are encrypted. My preferred way to do that would be to make automation repo a mono repo so that it cloud include the client library and cli packages. I recognize that that could be disruptive and happy to fallback to a separate repo if needed.

• I had a few more tests I wanted to add to test suite.

• Commits against this PR by @tofarr or other OpenHands/automation maintainers welcome!