Restart a group of containers within a docker stack if one container is updated

[HeyItsJono]

I was just hoping to get some advice on if what I want to do is possible with DockCheck.
I run the *arr suite of containers (Radarr, Sonarr, etc) which are routed via Gluetun, a VPN tunnel container.
They all depend on Gluetun because they use it as a network, when it goes down or restarts, unless they are also restarted they lose internet connectivity.
They are all defined in one big docker compose file which also defines many containers that aren't routed through the Gluetun network.

Currently I use DockCheck to update containers on a nightly basis, including Gluetun.
If DockCheck updates Gluetun and one of the *arr containers (e.g. Radarr), then Radarr and Gluetun will continue to work normally since they both have been restarted.
However all the containers which did not receive an update but are routed through the Gluetun network then lose internet connectivity until I manually figure out they've gone down and restart them.

Is there any way to define a group of containers that should all be restarted if one of their dependencies is updated and restarted?

[mag37]

Thank you for raising the question.

There's currently a (not that graceful) way to solve this, with forcing the whole stack to restart through docker compose down && docker compose up -d.

The not so graceful bit is that will be done once for every container being updated in the stack, so if first gluetun got an update, that will restart the stack, then some other non related containers in between, then radarr comes along and that will restart the stack again.

Though it's not disruptive or cause for breakage, just inconvenient.

This can be activated either by the flag -f -f Force stop+start stack after update. Caution: restarts once for every updated container within stack. .

Or by setting the label in the compose to only activate it on the gluetun stack (and maybe only gluetun specifically).

    labels:
      mag37.dockcheck.restart-stack: true

Though it's also being worked upon and improved in this PR, which just grew quite a lot in scope so might take a couple of weeks to hit a main release.
#262

[HeyItsJono]

Thanks for the response! I'll await the new PR :)

[mnijman]

I have a similar / related issue, also in relation to gluetun, which is combined with deluge.
network_mode: "container:gluetun"

Which breaks if the gluetun container is updated and requires the deluge container to be recreated.
The gluetun container has the label set:

• "mag37.dockcheck.restart-stack=true"

But this is the result:

`Updating container(s):
gluetun

Now updating (1/1): gluetun
Using default tag: latest
latest: Pulling from qmcgaw/gluetun
d49a2dee86fb: Pull complete
ad5a3427bb90: Pull complete
82e18f6ff710: Pull complete
Digest: sha256:495cdc65ace4c110cf4de3d1f5f90e8a1dd2eb0f8b67151d1ad6101b2a02a476
Status: Downloaded newer image for qmcgaw/gluetun:latest
docker.io/qmcgaw/gluetun:latest

Done pulling updates.
Recreating updated containers.

Now recreating (1/1): gluetun
[+] up 2/2
✔ Container gluetun Recreated 0.9s
✔ Container deluge Running 0.0s

All updates done!`

The deluge container is not recreated, and I found 'stop & up' in the code for this scenario,
where 'down & up' would solve this issue I guess.

[mag37]

You're correct!
I just did some testing on this and must have come to the wrong conclusions in earlier testing.
Now when I ran through a few specific tests it seems like stop is not enough to force the rest of the stack to restart/recreate! A down is required for that.

Good call and should be patched! Depending on the #262 PR I might just push this fix prior to that.
Until then - you could always just replace the stop with a down on lines 319 and 325 edit: line 681 to see if it works as intended.

Thank you for commenting!

[mnijman]

Seems to work, impressive!

Now updating (1/1): gluetun
Using default tag: latest
latest: Pulling from qmcgaw/gluetun
Digest: sha256:495cdc65ace4c110cf4de3d1f5f90e8a1dd2eb0f8b67151d1ad6101b2a02a476
Status: Downloaded newer image for qmcgaw/gluetun:latest
docker.io/qmcgaw/gluetun:latest

Done pulling updates.
Recreating updated containers.

Now recreating (1/1): gluetun
[+] down 2/2
✔ Container deluge Removed 3.7s
✔ Container gluetun Removed 0.6s
[+] up 2/2
✔ Container gluetun Created 0.2s
✔ Container deluge Created 0.2s

All updates done!

Changed the following lines:

651: [[ "$ContOnlySpecific" == "null" ]] && ContOnlySpecific=""
681: ${DockerBin} ${CompleteConfs} down; ${DockerBin} ${CompleteConfs} ${ContEnvs} up -d || { printf "\n%bDocker error, exiting!%b\n" "$c_red" "$c_reset" ; exit 1; }

The change in 651 was required to ensure ContRestartStack was not overridden again, so I guess this was a small bug as well, but maybe I'm using it wrong.

[mag37]

Sorry, dont know what I was looking at with the line numbers 😅 but you found the correct one.

And nice find with the 651 error! Gotta check that it wasn't intended for some reason, but seems like its just a typo/mixup.

[mnijman]

Created a pull request (first ever, so don't hesitate to correct me in any way) to fix it.

[HeyItsJono]

Was just thinking, and could labels somehow be used to define a specific group of containers that would depend on another container and also be downed/upped if their dependency is updated? Like if the labels are applied:

gluetun:
  labels:
    - mag37.dockcheck.restart-stack: true
    - mag37.dockcheck.group: 'gluetun'

radarr:
  labels:
    - mag37.dockcheck.group: 'gluetun'

Then if DockCheck ever updates a container with the label restart-stack set to true, it would then check if that container is also labelled with a group, and if so then restart all containers labelled with that group? Would be more graceful than having to restart the whole stack.

[mag37]

Oh that's a great suggestion @HeyItsJono !

I'll explore this some, and see where the rest of current work will land and if this could fit in.

I need to check and test if this is partly already built in to docker compose or if we could use existing things partly to work with. But I think the group suggestion is a solid idea.

Looking at this, which I've had suggested doesn't solve the problem but not properly tested myself:
https://docs.docker.com/compose/how-tos/startup-order/

[mag37]

So @HeyItsJono I just did some brief testing with the depends_on option (and the sub-option restart: true).

This was the compose file:

---
services:
  nginx:
    image: lscr.io/linuxserver/nginx:latest
    container_name: web2_nginx
    depends_on:
      mariadb:
        condition: service_started
        restart: true
    ports:
      - 8080:80
    volumes:
      - ./config:/config
    restart: unless-stopped

  mariadb:
    image: lscr.io/linuxserver/mariadb:latest
    container_name: web2_mariadb
    volumes:
      - ./config:/config
    ports:
      - 3306:3306
    restart: unless-stopped

And this an excerpt of the output:

Done pulling updates.
Recreating updated containers.

Now recreating (1/1): web2_mariadb
[+] up 2/2
 ✔ Container web2_nginx   Stopped                     3.5s
 ✔ Container web2_mariadb Recreated                   3.3s

nginx shows as stopped in this output but its restarted again after the mariadb got recreated.

And then a docker ps (excuse the format):

NAME    CREATED    STATUS
web2_mariadb    20 seconds ago    Up 16 seconds
web2_nginx    5 minutes ago    Up 16 seconds       

So it looks like it'll recreate the mariadb container and then due to the depends_on together with restated: true option - the nginx container is restarted (not recreated).

[HeyItsJono]

I think depends_on might only work to restart dependencies with the restart command, I don't think docker compose down would cause dependant containers to also go down, likewise with docker compose up. In this instance if my understanding is correct, if dockcheck updated the mariadb container and then ran docker compose down mariadb -> docker compose up mariadb, the nginx container would not restart.

FWIW all my containers have had the restart: true flag on since before I created this issue