Messenger Instrumentation - The root span seems never ended

[jvocampings]

Hi

My configuration for instrumentation:

    instrumentation:
        http_kernel:
            type: auto
            tracing:
                enabled: true
        http_client:
            tracing:
                enabled: true
        messenger:
            type: auto
            tracing:
                enabled: true
        console:
            type: auto
            tracing:
                enabled: true

I'm using the main branch.

I run bin/console messenger:consume my_transport --limit=1.

When I check traces on my Grafana (from Tempo data source), I see a trace with <root span not yet received>.

And in my app logs

[2025-04-17T10:45:44.401272+00:00] open_telemetry.DEBUG: Starting span "454bfad0716aba41" 
[2025-04-17T10:45:44.404044+00:00] open_telemetry.DEBUG: Activating new scope "877" 
[2025-04-17T10:45:44.644812+00:00] open_telemetry.DEBUG: Using scope "877" 
..................... 
SOME OTHER APPLICATIVE LOGS
..................... 
[2025-04-17T10:45:45.078182+00:00] open_telemetry.DEBUG: Detaching scope "499"
[2025-04-17T10:45:45.078334+00:00] open_telemetry.DEBUG: Ending span "2720d0ecbb384553"

The span 454bfad0716aba41, which is the first one (then the root span), is never ended. Then never sent to the collector.
I don't have any issue with other command. I tested with bin/console assets:install.

I added some debug in TraceableConsoleEventSubscriber:

• 454bfad0716aba41 (the root span) is started by the method startSpan when the event ConsoleEvents::COMMAND occurs
• 2720d0ecbb384553 (the last span in my logs) is ended by the method endSpan when the event ConsoleEvents::TERMINATE occurs.

I don't know why it happens like this. I assume it is related to the command messenger:consume which is designed to run indefinitely and something strange happen.

So maybe we should do this ?

• always exclude messenger:consume from Command instrumentation
• listen Messenger events to encapsulate each message consumed in a span instead of using the Messenger Middleware ?

[gaelreyrol]

Hi @jvocampings,

I don't know why it happens like this. I assume it is related to the command messenger:consume which is designed to run indefinitely and something strange happen.

You are absolutely right, it comes from the commands auto instrumentation from TraceableConsoleEventSubscriber.

This command should definitely be excluded from the instrumentation. You can do it manually by specifying the exclude_commands configuration option:

open_telemetry:
  console:
    type: auto
      tracing:
        enabled: true
        exclude_commands:
          - messenger:consume

Still I think it should be excluded by default directly in the TraceableConsoleEventSubscriber.

The instrumentation of dispatched, consumed and handled messages should be done through an event subscriber via the following events:

• Symfony\Component\Messenger\Event\SendMessageToTransportsEvent
• Symfony\Component\Messenger\Event\WorkerMessageFailedEvent
• Symfony\Component\Messenger\Event\WorkerMessageHandledEvent
• Symfony\Component\Messenger\Event\WorkerMessageReceivedEvent
• and maybe more

I can work on it soon but feel free to submit a PR! 🙂

[jvocampings]

I'm working on a PR

[kochen]

Hi @jvocampings,

I don't know why it happens like this. I assume it is related to the command messenger:consume which is designed to run indefinitely and something strange happen.

You are absolutely right, it comes from the commands auto instrumentation from TraceableConsoleEventSubscriber.

This command should definitely be excluded from the instrumentation. You can do it manually by specifying the exclude_commands configuration option:

open_telemetry:
console:
type: auto
tracing:
enabled: true
exclude_commands:
- messenger:consume
Still I think it should be excluded by default directly in the TraceableConsoleEventSubscriber.

The instrumentation of dispatched, consumed and handled messages should be done through an event subscriber via the following events:

• Symfony\Component\Messenger\Event\SendMessageToTransportsEvent
• Symfony\Component\Messenger\Event\WorkerMessageFailedEvent
• Symfony\Component\Messenger\Event\WorkerMessageHandledEvent
• Symfony\Component\Messenger\Event\WorkerMessageReceivedEvent
• and maybe more

I can work on it soon but feel free to submit a PR! 🙂

Is excluding the consume command really a solution?