[Bug]: Applications are reporting their error states after terminate

[PawelPlesniak]

Description

In a production environment, the error state of messages is frequently reported after the applications are terminated.

Steps to reproduce

Take a HW run as np04daq, anything using the TPC
Boot the apps, take a run, then end the run
Once terminate is ran, there are several CRITICAL log records that appear after terminate has finished.

Version and environment

fddaq-v5.6.0-rc4

Impact

Small/Isolated

Anything else?

No response

[PawelPlesniak]

Screenshots being uploaded shortly

[PawelPlesniak]

This is likely addressed with #878, testing required

[PawelPlesniak]

Possibly related

[PawelPlesniak]

Main issue

Which has a variable number of applications reporting, even if running the same session

[PawelPlesniak]

Here is an example log, many processes are failing to terminate after SIGQUIT

[2026/04/30 13:10:16 UTC] INFO       shell_utils.py:112                       drunc.unified_shell.shell_utils                    Running command: 'terminate'
[2026/04/30 13:10:16 UTC] INFO       ssh_process_manager.py:203               drunc.process_manager.SSH_SHELL_process_manager    Terminating
[2026/04/30 13:10:16 UTC] INFO       ssh_process_manager.py:206               drunc.process_manager.SSH_SHELL_process_manager    Killing all the known processes before exiting
[2026/04/30 13:10:16 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'unknown' from provided UUIDs ---
[2026/04/30 13:10:16 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 6 process(es) with role 'unknown' from 18 candidates
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process eee1bd37-03be-4857-a3c4-663f402fe81f (PID 279143) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process c0fa5c5b-8566-4930-bee6-5ee96add6ecc (PID 279142) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 5c1fa090-b495-411b-ae42-5a1bac330c5d (PID 1351026) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 3091111f-ce0f-40c3-9cb1-b59312e9fed4 (PID 527345) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 54630d55-0d6d-406d-a175-f78d002fe432 (PID 527346) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 8d851fba-d546-4428-bb07-756efad6cf52 (PID 517061) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'crp5-wiec' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 54630d55-0d6d-406d-a175-f78d002fe432 (PID 527346) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'crp4-wiec' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:17 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'crp23-crates-np04srv031' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:17 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'runp02srv003eth1' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:17 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'runp02srv003eth0' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 3091111f-ce0f-40c3-9cb1-b59312e9fed4 (PID 527345) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:17 UTC] ERROR      ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 5c1fa090-b495-411b-ae42-5a1bac330c5d (PID 1351026) still did not terminate after SIGKILL signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process eee1bd37-03be-4857-a3c4-663f402fe81f (PID 279143) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process c0fa5c5b-8566-4930-bee6-5ee96add6ecc (PID 279142) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 8d851fba-d546-4428-bb07-756efad6cf52 (PID 517061) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:17 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'runp04srv031tde' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:18 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'unknown' complete ---
[2026/04/30 13:10:18 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'application' from provided UUIDs ---
[2026/04/30 13:10:18 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 8 process(es) with role 'application' from 18 candidates
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process f570b64e-c009-49c7-9137-5241bedab2e0 (PID 1270127) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process e72fa14d-e464-46be-bf79-9765134430d1 (PID 745235) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 5f589b55-dfbe-4ebd-8cdc-e652d56fc07d (PID 494258) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 25543637-3935-4acb-8781-dbf96e6c783b (PID 745450) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 17228a33-3b46-4170-819a-2f9d2616c31b (PID 1270293) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process cd95bb13-e154-4313-a6ff-759345245ef7 (PID 2936508) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 5ff01ff1-1952-4941-b55b-af21e737c4fa (PID 2936722) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 860e775b-fc9a-4acd-9e81-0d811beb6b3d (PID 2936901) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'dfo-01' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'mlt' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 25543637-3935-4acb-8781-dbf96e6c783b (PID 745450) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 5ff01ff1-1952-4941-b55b-af21e737c4fa (PID 2936722) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'tc-maker-tpc' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 17228a33-3b46-4170-819a-2f9d2616c31b (PID 1270293) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 860e775b-fc9a-4acd-9e81-0d811beb6b3d (PID 2936901) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-s04-d0' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'tp-stream-writer' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'crp4-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'crp5-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'crp23-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process e72fa14d-e464-46be-bf79-9765134430d1 (PID 745235) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:19 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 5f589b55-dfbe-4ebd-8cdc-e652d56fc07d (PID 494258) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:20 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process f570b64e-c009-49c7-9137-5241bedab2e0 (PID 1270127) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:20 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process cd95bb13-e154-4313-a6ff-759345245ef7 (PID 2936508) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:20 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'application' complete ---
[2026/04/30 13:10:20 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'segment-controller' from provided UUIDs ---
[2026/04/30 13:10:20 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 3 process(es) with role 'segment-controller' from 18 candidates
[2026/04/30 13:10:21 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process def3091e-bc55-4eac-8ac3-fe42003c0552 (PID 2935985) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:21 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process d39824d0-4080-401b-a835-16ec8134da44 (PID 2936422) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:21 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 616f0a3e-182b-4d98-b64e-4c06f13ffb4a (PID 2936218) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:21 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:10:21 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'trg-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:10:21 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'tpc-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:22 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process def3091e-bc55-4eac-8ac3-fe42003c0552 (PID 2935985) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:22 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 616f0a3e-182b-4d98-b64e-4c06f13ffb4a (PID 2936218) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:22 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process d39824d0-4080-401b-a835-16ec8134da44 (PID 2936422) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:22 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'segment-controller' complete ---
[2026/04/30 13:10:22 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'root-controller' from provided UUIDs ---
[2026/04/30 13:10:22 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 1 process(es) with role 'root-controller' from 18 candidates
[2026/04/30 13:10:23 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 7f1ed30b-8613-4c4c-92b1-fe648d10c711 (PID 2935911) did not terminate after SIGQUIT signal.
[2026/04/30 13:10:24 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'root-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:10:24 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 7f1ed30b-8613-4c4c-92b1-fe648d10c711 (PID 2935911) terminated forcibly following SIGKILL signal.
[2026/04/30 13:10:24 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'root-controller' complete ---
[2026/04/30 13:10:24 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'infrastructure-applications' from provided UUIDs ---
                                                              Terminated process
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ session                 ┃ friendly name                 ┃ user    ┃ host         ┃ uuid                                 ┃ alive ┃ exit-code ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ np02-release-testing-pm │ root-controller               │ np04daq │ np04-srv-024 │ 7f1ed30b-8613-4c4c-92b1-fe648d10c711 │ False │ 137       │
│ np02-release-testing-pm │   df-controller               │ np04daq │ np04-srv-024 │ def3091e-bc55-4eac-8ac3-fe42003c0552 │ False │ 0         │
│ np02-release-testing-pm │     df-s04-d0                 │ np04daq │ np04-srv-004 │ 17228a33-3b46-4170-819a-2f9d2616c31b │ False │ 0         │
│ np02-release-testing-pm │     dfo-01                    │ np04daq │ np04-srv-011 │ 5f589b55-dfbe-4ebd-8cdc-e652d56fc07d │ False │ 137       │
│ np02-release-testing-pm │     tp-stream-writer          │ np04daq │ np04-srv-001 │ f570b64e-c009-49c7-9137-5241bedab2e0 │ False │ 137       │
│ np02-release-testing-pm │   tpc-controller              │ np04daq │ np04-srv-024 │ d39824d0-4080-401b-a835-16ec8134da44 │ False │ 137       │
│ np02-release-testing-pm │     crp23-controller          │ np04daq │ np04-srv-024 │ 860e775b-fc9a-4acd-9e81-0d811beb6b3d │ False │ 0         │
│ np02-release-testing-pm │       crp23-crates-np04srv031 │ np04daq │ np04-srv-013 │ 8d851fba-d546-4428-bb07-756efad6cf52 │ False │ 137       │
│ np02-release-testing-pm │       runp04srv031tde         │ np04daq │ np04-srv-031 │ 5c1fa090-b495-411b-ae42-5a1bac330c5d │ False │ 0         │
│ np02-release-testing-pm │     crp4-controller           │ np04daq │ np04-srv-024 │ cd95bb13-e154-4313-a6ff-759345245ef7 │ False │ 137       │
│ np02-release-testing-pm │       crp4-wiec               │ np04daq │ np04-srv-012 │ 54630d55-0d6d-406d-a175-f78d002fe432 │ False │ 0         │
│ np02-release-testing-pm │       runp02srv003eth0        │ np04daq │ np02-srv-003 │ c0fa5c5b-8566-4930-bee6-5ee96add6ecc │ False │ 137       │
│ np02-release-testing-pm │     crp5-controller           │ np04daq │ np04-srv-024 │ 5ff01ff1-1952-4941-b55b-af21e737c4fa │ False │ 0         │
│ np02-release-testing-pm │       crp5-wiec               │ np04daq │ np04-srv-012 │ 3091111f-ce0f-40c3-9cb1-b59312e9fed4 │ False │ 137       │
│ np02-release-testing-pm │       runp02srv003eth1        │ np04daq │ np02-srv-003 │ eee1bd37-03be-4857-a3c4-663f402fe81f │ False │ 137       │
│ np02-release-testing-pm │     tc-maker-tpc              │ np04daq │ np04-srv-018 │ 25543637-3935-4acb-8781-dbf96e6c783b │ False │ 0         │
│ np02-release-testing-pm │   trg-controller              │ np04daq │ np04-srv-024 │ 616f0a3e-182b-4d98-b64e-4c06f13ffb4a │ False │ 0         │
│ np02-release-testing-pm │     mlt                       │ np04daq │ np04-srv-018 │ e72fa14d-e464-46be-bf79-9765134430d1 │ False │ 137       │
└─────────────────────────┴───────────────────────────────┴─────────┴──────────────┴──────────────────────────────────────┴───────┴───────────┘
[2026/04/30 13:10:25 UTC] INFO       shell_utils.py:135                       drunc.utils.ShellContext                           You will not be able to issue commands to the controller anymore.
[2026/04/30 13:10:25 UTC] INFO       shell_utils.py:137                       drunc.utils.ShellContext                           Controller driver has been deleted.
drunc-unified-shell > [2026/04/30 13:10:25 UTC] CRITICAL   process_manager.py:246                   drunc.process_manager.SSH_SHELL_process_manager    Process runp04srv031tde has died with a return code 0

[PawelPlesniak]

Note - running an integration test as the superuser in a tmux setup for production has had mostly 0 exitt codes

[2026/04/30 13:16:36 UTC] INFO       ssh_process_manager.py:203               drunc.process_manager.SSH_SHELL_process_manager    Terminating
[2026/04/30 13:16:36 UTC] INFO       ssh_process_manager.py:206               drunc.process_manager.SSH_SHELL_process_manager    Killing all the known processes before exiting
[2026/04/30 13:16:36 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'unknown' from provided UUIDs ---
[2026/04/30 13:16:36 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'application' from provided UUIDs ---
[2026/04/30 13:16:36 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 8 process(es) with role 'application' from 13 candidates
[2026/04/30 13:16:37 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process e3b2f4df-6c12-43f7-b2df-512d5cb8a1d8 (PID 2977857) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:37 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'hsi-fake-01' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:37 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'mlt' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:37 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 8f76ee42-fcc6-4f4e-972e-d9e45dcf9ce6 (PID 2976628) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:37 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'tp-stream-writer' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:37 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process f7fdc959-1a8d-4792-b842-669f903086a2 (PID 2977571) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:39 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'hsi-fake-to-tc-app' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:39 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 2092183a-795a-4c20-97a8-10df53fb00f1 (PID 2977441) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:39 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'tc-maker-1' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:39 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process d9392cca-bd6f-4871-89d3-18dda2abe4a2 (PID 2977991) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:41 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'ru-01' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:41 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 97db59ff-4210-40a5-8a4f-838be6906f52 (PID 2976316) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:46 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process e98a2a5a-8a11-4eef-b4c6-f8004b4ff70e (PID 2976798) did not terminate after SIGQUIT signal.
[2026/04/30 13:16:46 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 98381642-d1a2-403a-897d-21f25f1c8917 (PID 2976958) did not terminate after SIGQUIT signal.
[2026/04/30 13:16:47 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-01' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:16:47 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'dfo-01' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 137
[2026/04/30 13:16:47 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process e98a2a5a-8a11-4eef-b4c6-f8004b4ff70e (PID 2976798) terminated forcibly following SIGKILL signal.
[2026/04/30 13:16:47 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 98381642-d1a2-403a-897d-21f25f1c8917 (PID 2976958) terminated forcibly following SIGKILL signal.
[2026/04/30 13:16:48 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'application' complete ---
[2026/04/30 13:16:48 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'segment-controller' from provided UUIDs ---
[2026/04/30 13:16:48 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 4 process(es) with role 'segment-controller' from 13 candidates
[2026/04/30 13:16:48 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'hsi-fake-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:49 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process f0615818-0e76-4903-8748-7d882fba4c4f (PID 2977719) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:49 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'ru-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:50 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:50 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 335057d6-50a8-433c-8ed5-aa2e52a39a72 (PID 2976111) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:50 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 9c3d2428-dcdb-46c1-a0e6-049975669b33 (PID 2976411) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:50 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'trg-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:51 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process fa25ef67-e399-433d-b982-d58d6806490d (PID 2977149) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:51 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'segment-controller' complete ---
[2026/04/30 13:16:51 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'root-controller' from provided UUIDs ---
[2026/04/30 13:16:51 UTC] INFO       ssh_process_lifetime_manager_shell.py:51 drunc.drunc.processes.ssh_process_lifetime_manager Killing 1 process(es) with role 'root-controller' from 13 candidates
[2026/04/30 13:16:52 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'root-controller' (session: 'np02-release-testing-pm', user: 'np04daq') process exited with exit code 0
[2026/04/30 13:16:52 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process b5b5ad00-da9f-4e57-90a6-566a9324d87d (PID 2976038) terminated gracefully following SIGQUIT signal.
[2026/04/30 13:16:52 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'root-controller' complete ---
[2026/04/30 13:16:52 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'infrastructure-applications' from provided UUIDs ---
                                                         Terminated process
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ session                 ┃ friendly name          ┃ user    ┃ host      ┃ uuid                                 ┃ alive ┃ exit-code ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ np02-release-testing-pm │ root-controller        │ np04daq │ localhost │ b5b5ad00-da9f-4e57-90a6-566a9324d87d │ False │ 0         │
│ np02-release-testing-pm │   df-controller        │ np04daq │ localhost │ 9c3d2428-dcdb-46c1-a0e6-049975669b33 │ False │ 0         │
│ np02-release-testing-pm │     df-01              │ np04daq │ localhost │ 98381642-d1a2-403a-897d-21f25f1c8917 │ False │ 137       │
│ np02-release-testing-pm │     dfo-01             │ np04daq │ localhost │ e98a2a5a-8a11-4eef-b4c6-f8004b4ff70e │ False │ 137       │
│ np02-release-testing-pm │     tp-stream-writer   │ np04daq │ localhost │ 8f76ee42-fcc6-4f4e-972e-d9e45dcf9ce6 │ False │ 0         │
│ np02-release-testing-pm │   hsi-fake-controller  │ np04daq │ localhost │ f0615818-0e76-4903-8748-7d882fba4c4f │ False │ 0         │
│ np02-release-testing-pm │     hsi-fake-01        │ np04daq │ localhost │ e3b2f4df-6c12-43f7-b2df-512d5cb8a1d8 │ False │ 0         │
│ np02-release-testing-pm │     hsi-fake-to-tc-app │ np04daq │ localhost │ d9392cca-bd6f-4871-89d3-18dda2abe4a2 │ False │ 0         │
│ np02-release-testing-pm │   ru-controller        │ np04daq │ localhost │ 335057d6-50a8-433c-8ed5-aa2e52a39a72 │ False │ 0         │
│ np02-release-testing-pm │     ru-01              │ np04daq │ localhost │ 97db59ff-4210-40a5-8a4f-838be6906f52 │ False │ 0         │
│ np02-release-testing-pm │   trg-controller       │ np04daq │ localhost │ fa25ef67-e399-433d-b982-d58d6806490d │ False │ 0         │
│ np02-release-testing-pm │     mlt                │ np04daq │ localhost │ f7fdc959-1a8d-4792-b842-669f903086a2 │ False │ 0         │
│ np02-release-testing-pm │     tc-maker-1         │ np04daq │ localhost │ 2092183a-795a-4c20-97a8-10df53fb00f1 │ False │ 0         │
└─────────────────────────┴────────────────────────┴─────────┴───────────┴──────────────────────────────────────┴───────┴───────────┘
[2026/04/30 13:16:52 UTC] INFO       shell_utils.py:135                       drunc.utils.ShellContext                           You will not be able to issue commands to the controller anymore.
[2026/04/30 13:16:52 UTC] INFO       shell_utils.py:137                       drunc.utils.ShellContext                           Controller driver has been deleted.

[Aurashk]

Thanks for the details @PawelPlesniak

Think it's really down to this code segment

It's a bit of a tricky situation, our methods to detect whether the process is dead are still reporting the process as alive - so we don't really want to record the process as dead and clean up. But we also used our last resort method of SIGKILL so we have no other tools to kill it. Ideally you would never reach this code path in normal operations but apparently it's quite common. As a quick workaround I can change the error to a warning so it's flagged for investigation, and do the normal cleanup so you don't end up with that critical log. In essence with this patch the process manager would be saying: 'I can't shut down the process with SIGKILL and I don't know why, so I'm just going to emit a warning and forget it exists'. Ideally we want to understand what's actually happening here causing the SIGKILL to fail, but robust handling of that would require more investigation. What do you think?

Another note about the 137 codes, maybe we need to try extending the timeout a bit and see if gets more consistent zero's? I'm going to make the logging clearer on this - it's SIGQUIT failing within the timeout we've chosen, so we resorted to SIGKILL that causes the 137 codes

[Aurashk]

Possibly related

This should be partially addressed with #896

[PawelPlesniak]

Based on the feedback from the CT meeting, the release will be held back by this, so I think we can go straight to robust implementation. If we mark this as a warning, the integration tests will all fail, which while will cause others to panic. If there are any logs or tests that I can run to make this easier, let me know

[PawelPlesniak]

As for the image, I cannot see it across multiple browsers, not sure what the cause for that is but feel free to ignore this comment

[Aurashk]

Based on the feedback from the CT meeting, the release will be held back by this, so I think we can go straight to robust implementation. If we mark this as a warning, the integration tests will all fail, which while will cause others to panic. If there are any logs or tests that I can run to make this easier, let me know

Would be really useful to see what's going on with the process that emits the error, if you use ps --long-format you should be able to get the remote pid, then please let me know the output of cat /proc/<pid>/status and ps -p <pid> -o pid,ppid,stat,comm,args,pcpu,pmem before and after terminate

Having said that though, this might just be a timeout configs issue if we're not giving SIGKILL enough time #900 (comment) so would check that first

[PawelPlesniak]

Using a TPC run, I get the following

                                                                               Processes running
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ session                 ┃ friendly name                 ┃ user    ┃ host         ┃ uuid                                 ┃ alive                    ┃ exit-code ┃ executable ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ np02-release-testing-pm │ root-controller               │ np04daq │ np04-srv-024 │ 846ab1da-9e6b-4cc4-9f44-a866c3606e78 │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │   df-controller               │ np04daq │ np04-srv-024 │ 117038dc-0457-47c8-93e0-7ba40c327522 │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │     df-s04-d0                 │ np04daq │ np04-srv-004 │ 553d4206-37d6-4295-85fb-59be59362403 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │     dfo-01                    │ np04daq │ np04-srv-011 │ fa3a8050-dfe6-444c-8945-d084f2c6d760 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │     tp-stream-writer          │ np04daq │ np04-srv-001 │ a2321131-07c1-45e5-b830-ab13bd7ed1ec │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │   tpc-controller              │ np04daq │ np04-srv-024 │ 350c4c48-bf87-4ea5-9321-638e1d5656c4 │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │     crp23-controller          │ np04daq │ np04-srv-024 │ 0468f378-0ff1-44e1-8887-5ac910969953 │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │       crp23-crates-np04srv031 │ np04daq │ np04-srv-013 │ 5f74c1bd-9c22-4595-8755-cc554e35c5d3 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │       runp04srv031tde         │ np04daq │ np04-srv-031 │ 669098d7-6937-4257-92b4-97684d6816df │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │     crp4-controller           │ np04daq │ np04-srv-024 │ bde0e777-e6e6-4de0-9c3d-0f790586d3a7 │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │       crp4-wiec               │ np04daq │ np04-srv-012 │ 1aa698db-8d2c-4167-ac36-d9075820a575 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │       runp02srv003eth0        │ np04daq │ np02-srv-003 │ e1e3c777-297c-4fbd-8687-52b6939d459b │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │     crp5-controller           │ np04daq │ np04-srv-024 │ 9c72c34f-5e80-4159-957b-9b76cf15a92e │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │       crp5-wiec               │ np04daq │ np04-srv-012 │ 681e09ce-3357-4698-800a-d41d1bd342c4 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │       runp02srv003eth1        │ np04daq │ np02-srv-003 │ 0c350558-f7d6-4286-a654-6912cad91d54 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │     tc-maker-tpc              │ np04daq │ np04-srv-018 │ 52d41663-ee28-444b-87ec-69ca3e17fa40 │ source; daq_application  │ True      │ 0          │
│ np02-release-testing-pm │   trg-controller              │ np04daq │ np04-srv-024 │ 58569948-75f2-4636-8e95-8a89c7e9a49e │ source; drunc-controller │ True      │ 0          │
│ np02-release-testing-pm │     mlt                       │ np04daq │ np04-srv-018 │ eb31591a-bd39-4b07-85d6-d88425d58d31 │ source; daq_application  │ True      │ 0          │
└─────────────────────────┴───────────────────────────────┴─────────┴──────────────┴──────────────────────────────────────┴──────────────────────────┴───────────┴────────────┘

While running:

 ps aux | grep runp04srv031tde
np04daq  3195806  0.0  0.0  12416  8192 ?        Ss   17:41   0:00 /usr/bin/ssh np04daq@np04-srv-031 -o StrictHostKeyChecking=no -tt echo "SSHPM: Starting process $$ on host $HOSTNAME as user $USER";export SPACK_RELEASES_DIR="/cvmfs/dunedaq-development.opensciencegrid.org/candidates";export DUNEDAQ_ERS_VERBOSITY_LEVEL="1";export DUNEDAQ_ERS_FATAL="erstrace,lstdout,protobufstream(monkafka.cern.ch:30092)";export CONNECTION_PORT="30005";export DUNEDAQ_ERS_STREAM_LIBS="erskafka";export DUNEDAQ_ELISA_LOGBOOK_APPARATUS="pddp";export IPM_ZMQ_IO_THREADS="2";export DUNEDAQ_SESSION="np02-release-testing-pm";export DUNEDAQ_ERS_ERROR="erstrace,throttle,lstdout,protobufstream(monkafka.cern.ch:30092)";export DUNEDAQ_DB_PATH="/nfs/sw/dunedaq/dunedaq-fddaq-v5.6.0-rc4-dev-k8s/ehn1-daqconfigs:/nfs/sw/dunedaq/dunedaq-fddaq-v5.6.0-rc4-dev-k8s/sourcecode/dpdklibs:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/fddaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/snbmodules-v2.1.0-ifehbgiwvlldonziswudhl2lzphpgfxd/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/fddaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/daqsystemtest-v4.8.1-lophjfhusbud5yvjdt2dpcpgqelireft/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/restcmd-v1.4.2-uccvea2l2cl545w6mozv3kvoddu2nf5p/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/oksconflibs-v1.2.0-sz5wkleyocqs3syp27exsn7qjfez3rtd/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/oksutils-v1.0.0-tbg5yw5k4ai5zpa4zu3j3utsq45konbl/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/listrev-v6.2.0-feqnmi3idmbshqxzd3olqiiq72j6ny65/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dfmodules-v5.3.0-r7nnmw62zb6kfga673oafpfp7ybng53g/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/hsilibs-v4.10.0-4fzsvxnvswytit3e7q7o7mu3o2ehtltw/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/timinglibs-v4.7.0-ojnze3wqfhtlxpftklsrocq2iqd2tohl/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/datahandlinglibs-v3.2.3-w6gkiw72lovkddmlthghmk4cr6nba2qf/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/appfwk-v6.1.0-iklxh3l2hu6bj4oohsf6ribcfzaj32me/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/iomanager-v3.2.0-wz7fsj7jc3uaa6llsize34j5rubcxsci/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/appmodel-v5.1.0-yqllmu5r3tise3dqpehags2hnmrfx3ph/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/confmodel-v2.7.0-w7gugb2g7b5dtjxonm5dapfzgu2qrldj/share:/cvmfs/dunedaq-development.opensciencegrid.org/candidates/coredaq-v5.6.0-rc4-a9/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/conffwk-v2.1.1-dbbmhsee77z2rbb7bkx76i6hrspal66h/share";export DUNEDAQ_ERS_DEBUG_LEVEL="-1";export DUNE_DAQ_BASE_RELEASE="fddaq-v5.6.0-rc4-a9";export DUNEDAQ_ERS_INFO="erstrace,throttle,lstdout,protobufstream(monkafka.cern.ch:30092)";export DUNEDAQ_ERS_WARNING="erstrace,throttle,lstdout,protobufstream(monkafka.cern.ch:30092)";export DUNEDAQ_APPLICATION_NAME="runp04srv031tde";cd /nfs/sw/dunedaq/dunedaq-fddaq-v5.6.0-rc4-dev-k8s ; mkdir -p ${XDG_RUNTIME_DIR:-/tmp}/drunc ; source /nfs/sw/dunedaq/dunedaq-fddaq-v5.6.0-rc4-dev-k8s/install/daq_app_rte.sh;daq_application -s np02-release-testing-pm -k np02-session -n runp04srv031tde -c rest://np04-srv-031:0 -d oksconflibs:ehn1-daqconfigs/sessions/np02-session.data.xml &> /log/log_np04daq_np02-release-testing-pm_runp04srv031tde_2026-04-30-17-41-49.txt & PID=$! ; echo '{"pid": '$PID', "hostname": "np04-srv-031", "user": "np04daq", "started_at": '$(date +%s)', "tree_id": "0.2.2.1", "role": "unknown", "name": "runp04srv031tde"}' > ${XDG_RUNTIME_DIR:-/tmp}/drunc/metadata_669098d7-6937-4257-92b4-97684d6816df.json ; wait $PID
np04daq  3219819  0.0  0.0   6416  2048 pts/38   S+   17:44   0:00 grep --color=auto runp04srv031tde



[np04daq@np04-srv-024 ~]$ cat /proc/3195806/status
Name:   ssh
Umask:  0022
State:  S (sleeping)
Tgid:   3195806
Ngid:   0
Pid:    3195806
PPid:   3193744
TracerPid:      0
Uid:    106200  106200  106200  106200
Gid:    2841    2841    2841    2841
FDSize: 128
Groups: 2841 
NStgid: 3195806
NSpid:  3195806
NSpgid: 3195806
NSsid:  3195806
VmPeak:    12416 kB
VmSize:    12416 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:      8192 kB
VmRSS:      8192 kB
RssAnon:            1280 kB
RssFile:            6912 kB
RssShmem:              0 kB
VmData:      992 kB
VmStk:       320 kB
VmExe:       508 kB
VmLib:      6256 kB
VmPTE:        64 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
untag_mask:     0xffffffffffffffff
Threads:        1
SigQ:   0/766153
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001003
SigCgt: 0000000008014004
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       thread vulnerable
SpeculationIndirectBranch:      conditional enabled
Cpus_allowed:   ffffffff
Cpus_allowed_list:      0-31
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list:      0-1
voluntary_ctxt_switches:        128
nonvoluntary_ctxt_switches:     4


ps -p 3195806 -o pid,ppid,stat,comm,args,pcpu,pmem
    PID    PPID STAT COMMAND         COMMAND                     %CPU %MEM
3195806 3193744 Ss   ssh             /usr/bin/ssh np04daq@np04-s  0.0  0.0

back in the unified shell

                                                              Terminated process
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ session                 ┃ friendly name                 ┃ user    ┃ host         ┃ uuid                                 ┃ alive ┃ exit-code ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ np02-release-testing-pm │ root-controller               │ np04daq │ np04-srv-024 │ 846ab1da-9e6b-4cc4-9f44-a866c3606e78 │ False │ 0         │
│ np02-release-testing-pm │   df-controller               │ np04daq │ np04-srv-024 │ 117038dc-0457-47c8-93e0-7ba40c327522 │ False │ 0         │
│ np02-release-testing-pm │     df-s04-d0                 │ np04daq │ np04-srv-004 │ 553d4206-37d6-4295-85fb-59be59362403 │ False │ 0         │
│ np02-release-testing-pm │     dfo-01                    │ np04daq │ np04-srv-011 │ fa3a8050-dfe6-444c-8945-d084f2c6d760 │ False │ 134       │
│ np02-release-testing-pm │     tp-stream-writer          │ np04daq │ np04-srv-001 │ a2321131-07c1-45e5-b830-ab13bd7ed1ec │ False │ 0         │
│ np02-release-testing-pm │   tpc-controller              │ np04daq │ np04-srv-024 │ 350c4c48-bf87-4ea5-9321-638e1d5656c4 │ False │ 0         │
│ np02-release-testing-pm │     crp23-controller          │ np04daq │ np04-srv-024 │ 0468f378-0ff1-44e1-8887-5ac910969953 │ False │ 0         │
│ np02-release-testing-pm │       crp23-crates-np04srv031 │ np04daq │ np04-srv-013 │ 5f74c1bd-9c22-4595-8755-cc554e35c5d3 │ False │ 0         │
│ np02-release-testing-pm │       runp04srv031tde         │ np04daq │ np04-srv-031 │ 669098d7-6937-4257-92b4-97684d6816df │ False │ 0         │
│ np02-release-testing-pm │     crp4-controller           │ np04daq │ np04-srv-024 │ bde0e777-e6e6-4de0-9c3d-0f790586d3a7 │ False │ 0         │
│ np02-release-testing-pm │       crp4-wiec               │ np04daq │ np04-srv-012 │ 1aa698db-8d2c-4167-ac36-d9075820a575 │ False │ 0         │
│ np02-release-testing-pm │       runp02srv003eth0        │ np04daq │ np02-srv-003 │ e1e3c777-297c-4fbd-8687-52b6939d459b │ False │ 137       │
│ np02-release-testing-pm │     crp5-controller           │ np04daq │ np04-srv-024 │ 9c72c34f-5e80-4159-957b-9b76cf15a92e │ False │ 0         │
│ np02-release-testing-pm │       crp5-wiec               │ np04daq │ np04-srv-012 │ 681e09ce-3357-4698-800a-d41d1bd342c4 │ False │ 0         │
│ np02-release-testing-pm │       runp02srv003eth1        │ np04daq │ np02-srv-003 │ 0c350558-f7d6-4286-a654-6912cad91d54 │ False │ 137       │
│ np02-release-testing-pm │     tc-maker-tpc              │ np04daq │ np04-srv-018 │ 52d41663-ee28-444b-87ec-69ca3e17fa40 │ False │ 137       │
│ np02-release-testing-pm │   trg-controller              │ np04daq │ np04-srv-024 │ 58569948-75f2-4636-8e95-8a89c7e9a49e │ False │ 0         │
│ np02-release-testing-pm │     mlt                       │ np04daq │ np04-srv-018 │ eb31591a-bd39-4b07-85d6-d88425d58d31 │ False │ 137       │
└─────────────────────────┴───────────────────────────────┴─────────┴──────────────┴──────────────────────────────────────┴───────┴───────────┘
[2026/04/30 15:46:50 UTC] INFO       shell_utils.py:135                       drunc.utils.ShellContext                           You will not be able to issue commands to the controller anymore.
[2026/04/30 15:46:50 UTC] INFO       shell_utils.py:137                       drunc.utils.ShellContext                           Controller driver has been deleted.

The terminate at the end was reported correctly. The apps have non-zero exit codes, which may be solved by your PR. There were no error logs after the terminate. After terminate, the ps command did not return anything, neither did the cat command.