fix(trt): explicit TRT context/engine teardown + VRAM pre-check to prevent CUDA OOM on reload

[livepeer-tessa]

Summary

Fixes #723 — streamdiffusion-sdxl workers enter ERROR state after pipeline param updates due to CUDA OOM.

Root Causes

Two bugs conspired to leave ~22 GB of GPU memory pinned after cleanup_gpu_memory():

1. Incorrect teardown order in Engine.__del__ (utilities.py)**

TensorRT requires the execution context to be destroyed before the engine. The old code did del self.engine / del self.context (Python reference deletion, no ordering guarantee). Setting self.context = None first forces the C++ IExecutionContext destructor to run before ICudaEngine is released.

2. Manual __del__() call is unreliable (wrapper.py)**

cleanup_gpu_memory() called unet_engine.engine.__del__() explicitly. Python's destructor protocol doesn't guarantee immediate native teardown when called this way — the object can still be alive in the GC graph and TRT CUDA memory stays pinned.

Changes

src/streamdiffusion/acceleration/tensorrt/utilities.py

• Engine.__del__: set self.context = None then self.engine = None before the del statements to ensure the C++ destructors fire in the correct order.

src/streamdiffusion/wrapper.py

• New static helper _destroy_trt_engine: explicitly nullifies context → engine → frees buffers on any Engine wrapper. Replaces the fragile manual __del__() call.
• cleanup_gpu_memory rewrite: uses _destroy_trt_engine on UNet, VAE encoder/decoder, and ControlNet engines; calls gc.collect() between stages; logs non-PyTorch residual VRAM so operators can spot incomplete teardown.
• VRAM pre-flight check in _load_model: after cleanup, checks torch.cuda.mem_get_info(). If free VRAM < 2 GB, raises RuntimeError with an actionable message instead of letting the process OOM mid-load and exhaust the 3-restart budget.

Testing

Cannot be tested locally without a 24 GB GPU + TRT engine build. Tested via log inspection and code review against the crash trace in #723.

Checklist:

• Both changed files parse cleanly (ast.parse)
• Teardown order matches TRT documentation (context before engine)
• No silent except: pass that would hide new failures
• VRAM check raises with an actionable message rather than logging and continuing