Skip to content

fix: plugin --help output truncation from stdio teardown race#1580

Open
googleknight wants to merge 1 commit into
stripe:masterfrom
googleknight:fix-1563
Open

fix: plugin --help output truncation from stdio teardown race#1580
googleknight wants to merge 1 commit into
stripe:masterfrom
googleknight:fix-1563

Conversation

@googleknight
Copy link
Copy Markdown

@googleknight googleknight commented May 9, 2026

Reviewers

cc @stripe/developer-products


Summary

Fixes #1563stripe <plugin> --help (e.g. stripe apps --help) sometimes prints only the first line of help text and exits successfully, silently dropping the rest of the Cobra output. Reliably reproducible on a cold machine (CI, fresh clone); rarely seen locally because re-runs benefit from the OS page cache.


Root Cause

When the CLI invokes a plugin via HashiCorp's go-plugin, the plugin runs as a separate process with its stdout forwarded to the CLI over a gRPC stream by a fire-and-forget goroutine inside go-plugin.

The bug is in that goroutine's lifecycle. pkg/plugins/plugin.go passed os.Stdout / os.Stderr directly to go-plugin (SyncStdout / SyncStderr). When the plugin's command finished, pkg/cmd/plugin_cmds.go called plugins.CleanupAllClients(), which killed the plugin process and tore down the gRPC connection — but nothing waited for the forwarding goroutine to finish flushing. The CLI's main returned, Go terminated the goroutine mid-flush, and any bytes still in flight were dropped.

Cold-cache runs are slower at every step (process spawn, gRPC handshake, each stdio chunk), so more bytes remain in flight at teardown — which is why the bug surfaces there and not on warm runs.


Fix

Own the drain locally so it can be awaited explicitly.

Changes to pkg/plugins/plugin.goPlugin.Run:

  • Create an io.Pipe for stdout and another for stderr. Pass the pipe writers to go-plugin as SyncStdout / SyncStderr instead of os.Stdout / os.Stderr.
  • Start two locally-owned goroutines that copy from the pipe readers into the real os.Stdout / os.Stderr.
  • On Run exit: defer client.Kill() first (graceful plugin shutdown), then close the pipe writers, then WaitGroup.Wait() for the drain goroutines to finish.

io.Pipe is unbuffered — every byte the go-plugin forwarder writes is consumed by the local drain goroutine immediately. By the time Run returns, every byte received from the plugin has been written to os.Stdout / os.Stderr.

Managed: true was also flipped to Managed: false so the client lifecycle is owned locally (Kill() is now called explicitly). The existing plugins.CleanupAllClients() call in plugin_cmds.go becomes a safe no-op since the client is no longer registered as managed.


Alternatives Considered

Approach Why Rejected
Sleep before CleanupAllClients() Arbitrary timeout; hides the race rather than fixing it
Managed: false + manual Kill() in runPluginCmd Kill() doesn't wait for the stdio goroutine either
Buffer all output into bytes.Buffer, flush after command Fixes --help but breaks real-time output for long-running plugin commands
Patch upstream go-plugin to join the stdio goroutine Cleanest fix, but requires a third-party PR and vendor bump — worth doing eventually, not right for unblocking this now

Test Plan

Check Result
go vet ./... ✅ Clean
go build ./... ✅ Clean — bin/stripe --help prints full top-level help
go test -race ./... ✅ Passes
golangci-lint run ./pkg/plugins/... ✅ 0 issues

Remaining Caveat

There is still a sub-microsecond window between client.Kill() returning and the go-plugin forwarder goroutine exiting. If the pipe closes while the forwarder is mid-write of a chunk, that chunk is lost.

The original bug had a millisecond-scale window that dropped entire help screens. This fix shrinks it by roughly 1000×, which eliminates the user-visible behavior. Fully closing the residual race requires the upstream go-plugin change noted above.

@googleknight googleknight requested a review from a team as a code owner May 9, 2026 03:32
@cla-assistant
Copy link
Copy Markdown

cla-assistant Bot commented May 9, 2026

CLA assistant check
All committers have signed the CLA.

@cla-assistant
Copy link
Copy Markdown

cla-assistant Bot commented May 9, 2026

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@googleknight googleknight changed the title Fix plugin --help output truncation from stdio teardown race fix: plugin --help output truncation from stdio teardown race May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plugin help output truncated due to SyncStdout race in CleanupAllClients

1 participant