Add deadlock detection middleware for VirtualObjects

[yshaul]

Summary

• Adds an invocation middleware infrastructure to the SDK, allowing middleware to wrap handler execution (use) and outbound service calls (use_outbound) at the Restate protocol level (distinct from Rack HTTP middleware)
• Ships a built-in DeadlockDetection middleware that catches re-entrant VirtualObject calls that would otherwise block forever — raises a DeadlockError (409) immediately instead of silently hanging
• Includes 15 unit tests, a complete example (examples/middleware/deadlock_detection.rb), and README documentation

How it works

Restate VirtualObjects serialize exclusive handler access per key. If handler A on key "x" calls handler B on the same VO key "x", the second call waits for the first to finish — which never happens. The middleware tracks held locks via a header (x-restate-held-locks) propagated through the call chain:

1. Inbound — reads the held-locks header; if the current exclusive handler targets an already-held VO+key, raises DeadlockError; otherwise appends the lock
2. Outbound — injects the header into every downstream call; detects same-service deadlocks on the outbound side

Test plan

• All 84 tests pass (15 new + 69 existing)
• Deadlock detected on same VO+key for exclusive handlers
• Shared handlers on same key pass through (no deadlock)
• Lock propagation through call chain
• Lock cleanup on handler completion and errors
• Outbound header injection and same-service detection
• Basic services skip deadlock detection

Made with Cursor