Add configurable delay for sandbox cleanup after automation runs complete

[jpshackelford]

Summary

This PR implements configurable sandbox cleanup delay as requested in issue #31. Sandboxes can now remain available for inspection after automation runs complete, making it easier to debug automation failures.

Changes

Configuration

• Added AUTOMATION_SANDBOX_CLEANUP_DELAY_MINS environment variable (default: 60 minutes)
• When set to 0: immediate cleanup (legacy behavior)
• When > 0: sandboxes remain available for that many minutes after run completion

Database

• Added cleanup_at column to automation_runs table (nullable DateTime with index)
• New migration: 003_add_cleanup_at.py

Code Changes

automation/config.py

• Added sandbox_cleanup_delay_mins setting

automation/models.py

• Added cleanup_at field to AutomationRun model

automation/router.py

• Modified complete_run to set cleanup_at instead of immediate cleanup when delay > 0

automation/watchdog.py

• Added _compute_cleanup_at helper function
• Modified _verify_and_mark_run to set cleanup_at instead of immediate cleanup
• Added cleanup_pending_sandboxes function that processes runs past their cleanup deadline
• Updated watchdog_loop to call cleanup scanner each interval

automation/schemas.py

• Added cleanup_at field to AutomationRunResponse

Tests

• Added tests for _compute_cleanup_at
• Added tests for delayed cleanup behavior in watchdog
• Added tests for cleanup_pending_sandboxes function
• Added tests for complete_run endpoint cleanup behavior
• Added tests for the new config setting

SDK Behavior Note

Regarding the question in the issue comments:

if the user provides custom SDK code using OpenHandsCloudWorkspace(keep_alive=False) will the sandbox be shutdown by the SDK script at the end of its execution lifecycle regardless of our scheduler?

Answer: The SDK's OpenHandsCloudWorkspace does not directly delete the sandbox. When the context manager exits, it calls the automation service's callback endpoint (/runs/{run_id}/complete), and the automation service decides whether to cleanup based on:

1. The run's keep_alive flag (set on the AutomationRun model)
2. The sandbox_cleanup_delay_mins setting

So the SDK's keep_alive parameter in OpenHandsCloudWorkspace is not directly related to sandbox deletion control - that's handled entirely by the automation service.

Example Usage

# Keep sandboxes available for 2 hours after completion
export AUTOMATION_SANDBOX_CLEANUP_DELAY_MINS=120

# Immediate cleanup (legacy behavior)
export AUTOMATION_SANDBOX_CLEANUP_DELAY_MINS=0

Testing

The unit tests pass with the new behavior. Integration tests require Docker and were not run in this environment.

Closes #31

---

This PR was created by an AI assistant (OpenHands) on behalf of a user.

@jpshackelford can click here to continue refining the PR

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	1	0	100%
app.py	85	39	54%	31, 34, 37, 43, 45, 48, 51–54, 57–58, 61, 68–69, 72–73, 77, 85–86, 89, 96–97, 99, 102–103, 106, 111–119, 121–123
auth.py	77	6	92%	59, 92, 149, 200, 208–209
config.py	38	0	100%
constants.py	12	0	100%
db.py	44	26	40%	37–39, 48–49, 51–52, 54, 62, 69, 79, 82–83, 87–88, 96, 104, 109, 114, 117–123
dispatcher.py	135	34	74%	53, 73, 85, 87–88, 157, 161, 184, 203–205, 233–235, 238–240, 275–276, 300–307, 328–329, 339–340, 347–348, 350
exceptions.py	4	0	100%
execution.py	222	127	42%	45–47, 56, 89–92, 100–102, 110, 115–119, 133, 135, 137–140, 142–147, 149, 151–158, 160–161, 163, 178, 183, 194, 200–202, 213, 219–221, 268, 276, 280, 282–283, 288–289, 294, 342, 344, 362, 365–368, 390–393, 395, 403–404, 407, 413, 476–482, 485–486, 488, 490–492, 495, 498, 501–504, 506, 509–510, 513–515, 519, 522, 526–529, 531, 539–540, 544–546, 548–554, 558, 560, 569–571, 573–575
logger.py	53	18	66%	25–26, 36, 46–47, 49–55, 68, 88, 90–93
models.py	66	0	100%
preset_router.py	84	25	70%	169–170, 175–182, 187, 190, 192–193, 203–206, 208–212, 217, 226
router.py	110	60	45%	72–73, 93, 95, 98, 100, 114, 127, 129–130, 132–133, 136–138, 149–151, 169–172, 191, 194, 197, 204, 206, 244–246, 249–251, 255–256, 261, 265–268, 271–275, 277, 285, 287–288, 293–294, 297, 302, 304–305, 314, 335–337, 341
scheduler.py	57	9	84%	124–125, 162–163, 178–179, 189–190, 192
schemas.py	145	11	92%	67–69, 71, 129, 153–154, 157, 162, 167, 173
uploads.py	107	59	44%	138–141, 149–151, 157–158, 161, 170–171, 174–175, 183–184, 186–189, 192–195, 197, 199–201, 203–206, 208–209, 211, 226, 232–233, 236, 239, 242, 245, 247, 260–261, 275, 278–280, 282–283, 285, 291–292, 305, 313–315, 319
watchdog.py	157	58	63%	87–88, 100–101, 106–107, 112–113, 118–120, 128, 130, 265, 301–302, 304, 306, 315, 317–319, 321, 328–330, 332–334, 336–337, 339, 413, 418–419, 443, 445, 451–454, 456, 458–460, 463–467, 469–475, 477
presets
__init__.py	0	0	100%
storage
__init__.py	5	0	100%
factory.py	11	0	100%
file_store.py	18	5	72%	11, 20, 25, 30, 54
google_cloud.py	75	10	86%	103–108, 142–143, 196, 198
s3.py	115	14	87%	100, 102–103, 107, 109, 190, 213–215, 269–270, 275, 337–338
utils
__init__.py	4	0	100%
api_key.py	32	24	25%	40–41, 46–48, 50, 55, 60, 62–65, 67–68, 70–71, 73, 79, 81–82, 89, 91–92, 98
cron.py	45	6	86%	39, 45, 74, 80, 123, 140
run.py	75	12	84%	74–76, 172–174, 179–181, 228, 234–235
sandbox.py	108	63	41%	51–52, 57–60, 62–64, 66–72, 84, 86, 96–97, 99–101, 103–104, 107–108, 114, 120–122, 138–141, 174, 178–180, 217–218, 220, 222–225, 230–231, 234, 236–237, 243–245, 250–252, 257–258, 266–268, 270
tarball_validation.py	48	0	100%
time.py	3	0	100%
TOTAL	1936	606	68%

[github-actions]

🚀 Deploy Preview PR Created/Updated

A deploy preview has been created/updated for this PR.

Deploy PR: https://github.com/OpenHands/deploy/pull/3638
Automation SHA: eadea664d51fe762d703c5e6a31e12c9af390d1a
Last updated: Apr 01, 2026, 07:29:44 PM ET

Once the deploy PR's CI passes, the automation service will be deployed to the feature environment.