From 36a479739a287e15636ded28561c551b94034e59 Mon Sep 17 00:00:00 2001 From: Garrett Thornburg Date: Tue, 24 Aug 2021 07:57:00 -0600 Subject: [PATCH] Check rabbitmq connection status and reconnect when disconnected The underlying client should be good about reconnecting, but we are seeing behavior where a runtime (in this case JRuby) is using a lot of mem and then fails to disconnect despite every other connection (thousands of others) working just fine. Perhaps it's safer to reconnect and let the publishers figure it out? Right now we see: ``` com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.io.EOFException ``` This is called when the in-mem publisher tries to create a new channel (when a consumer thread is created). TODO: Will this potentially leak a connection? Should we explicitly disconnect the old connection? --- lib/active_publisher/connection.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/active_publisher/connection.rb b/lib/active_publisher/connection.rb index 9575a50..b3abd58 100644 --- a/lib/active_publisher/connection.rb +++ b/lib/active_publisher/connection.rb @@ -10,7 +10,8 @@ def self.connected? def self.connection CONNECTION_MUTEX.synchronize do - return @connection if @connection + # Connection must be a valid object and connected. Otherwise, reconnect. + return @connection if @connection && @connection.connected? @connection = create_connection end end