Skip to content

Revert "Tee output so we can see it in journald logs" for dataproc#354

Merged
june-hua merged 1 commit intomasterfrom
june/dataproc-fix
Mar 31, 2026
Merged

Revert "Tee output so we can see it in journald logs" for dataproc#354
june-hua merged 1 commit intomasterfrom
june/dataproc-fix

Conversation

@june-hua
Copy link
Copy Markdown
Contributor

@june-hua june-hua commented Mar 31, 2026

This partially reverts commit 9bd46a1.

Symptoms:

  • google-startup-scripts.service stuck in "activating" for hours
  • google-dataproc-agent.service never starts (ConditionResult=no)

Suspected root Cause:

  1. The exec > >(tee -a "${POST_STARTUP_OUTPUT_FILE}") creates a tee subprocess that remains alive
  2. The background process that waits for Jupyter inherits file descriptors from tee
  3. When the main startup script exits, tee subprocess stays alive because the background process still has the pipes open
  4. google-startup-scripts.service remains stuck in "activating" state indefinitely, waiting for all child processes
  5. google-dataproc-agent.service has After=google-startup-scripts.service, so it never starts
  6. This creates a race condition where the background Jupyter configuration runs while Dataproc is still writing its config

PHP-140163

…proc

This partially reverts commit 9bd46a1.

Root Cause:
1. The exec > >(tee -a "${POST_STARTUP_OUTPUT_FILE}") creates a tee
subprocess that remains alive
2. The background process that waits for Jupyter inherits file
descriptors from tee
3. When the main startup script exits, tee subprocess stays alive
because the background process still has the pipes open
4. google-startup-scripts.service remains stuck in "activating" state
indefinitely, waiting for all child processes
5. google-dataproc-agent.service has
After=google-startup-scripts.service, so it never starts
6. This creates a race condition where the background Jupyter
configuration runs while Dataproc is still writing its config

Symptoms:
- google-startup-scripts.service stuck in "activating" for hours
  - google-dataproc-agent.service never starts (ConditionResult=no)
@june-hua june-hua self-assigned this Mar 31, 2026
@june-hua june-hua requested review from a team as code owners March 31, 2026 00:59
@june-hua june-hua merged commit 72c9814 into master Mar 31, 2026
13 of 15 checks passed
@june-hua june-hua deleted the june/dataproc-fix branch March 31, 2026 12:43
@june-hua june-hua changed the title Revert "Tee output so we can see it in journald logs (#346)" for dataproc Revert "Tee output so we can see it in journald logs" for dataproc Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants