System initialization error when trying to send InitializeLogger message to the default logger

[vonnagy]

It appears as though the DefaultLogger does not reply to InitializeLogger on line 200 of the LoggingBus.

import akka.pattern.ask

val response = try Await.result(actor ? InitializeLogger(this), timeout.duration) catch {
      case _: TimeoutException ⇒
        publish(Warning(logName, this.getClass, "Logger " + name + " did not respond      within " + timeout + " to InitializeLogger(bus)"))
        "[TIMEOUT]"
    }

This issue is due to the actor system 'statsd' being created in the class AkkaIOStatsdCounterInterface. The issue only occurs when system agents are being monitored e.g. includeSystemAgents: true.

Here is the scenario:

• Make sure includeSystemAgents is turned on.
• When the system starts, Akka creates the default logger and sends an InitializeLogger message to it.
• This triggers the around pointcut and then the AkkaIOStatsdCounterInterface is created.
• When the "statsd" actor system is created, Akka tries to create another default loggger.
• The initial send of the InitializeLogger message times out because of the time it takes to create the new actor system and the thread that is creating the AkkaIOStatsdCounterInterface.

As an experiment to prove this out, I first changed the creation of the 'statsd' system to lazy instantiation see #90. I then wrapped the logic in theCounterInterface methods in a Future. This disconnected the logic from the running thread which, in turn, allowed all messages to be received properly.

The test above is not necessarily a good long-term solution, but does point out the issue and possible ideas to its resolution.

[janm399]

Do you think we can close it now? I think we'll soon come to a time when we'll all need to have a hangout and discuss any refactorings we may want to apply.

[vonnagy]

This is still an issue when includeSystemAgents is turned on and someone is using AkkaIOStatsdCounterInterface. As we know it is a threading issue because the application's actor system is starting it's logger at the same time the 'statsd' is being doing the same. Setting the logger-startup-timeout has no effect as the issue stems from a threading conflict.

This would be a candidate for some further discussion on a possible refactor as I don't believe the use of Future as outlined above is the right direction. Perhaps and instance of the CounterInterface is created per actor system and the 'statsd' actor is created within that system.