Skip to content

OTEL Tracing - Hierarchy inconsitency #4607

@ThibaultCoudertSephora

Description

@ThibaultCoudertSephora

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient
triaging. Requests without complete information may be rejected / deprioritized.
If an item is not applicable to you - please mark it as N/A

Describe the Bug:
When activating OTEL instrumentation (I used CloudTraceSpanExporter in my project), the parent_span_id management doesn't seem consistant across processing : the first LLM Call is linked to Agent, the Tool Execution is linked to Agent but the second LLM Call (post tool execution to format final answer) is linked to first LLM Call.

Steps to Reproduce:

  1. Install google-adk 1.25.0 ; opentelemetry-api 1.38.0 ; opentelemetry-exporter-gcp-logging 1.11.0a0
  2. Create an agent with a dummy tool and explain him when to call it.
  3. Setup OTEL with span processor CloudTraceSpanExporter and activate BigQueryAgentAnalyticsPlugin
  4. Launch adk web and start the conversation

Expected Behavior:
Let's say we have the following span generated:

  • Span 1 = Agent invoked
  • Span 2 = LLM Request
  • Span 3 = Call LLM (with user input)
  • Span 4 = Generate Content (Decide to use the tool)
  • Span 5 = Tool Request
  • Span 6 = Execute Tool
  • Span 7 = LLM Request
  • Span 8 = Call LLM (with tool output)
  • Span 9 = Generate Content (Final answer to user)

I would either expect the Span 1 (Agent Invoked) to be the parent of Span 2 / Span 5 / Span 7 (fork as below) or Span 1 parent of Span 2 parent of Span 5 parent of Span 7 (full hierarchy)

Span 1 = Agent Invoked
|__ Span 2 = LLM Request
     |__ Span 3 = Call LLM
          |__ Span 4 = Generate Content
|__ Span 5 = Tool Request
     |__ Span 6 = Execute Tool
|__ Span 7 = LLM Request
     |__ Span 8 = Call LLM
          |__ Span 9 = Generate Content

Observed Behavior:
The second LLM Request (post tool invocation) is attached to first LLM Request, which lead to an inconsistent timeframe in Cloud Trace:

Image
Span 1 = Agent Invoked
|__ Span 2 = LLM Request
     |__ Span 3 = Call LLM
          |__ Span 4 = Generate Content
     |__ Span 7 = LLM Request
          |__ Span 8 = Call LLM
               |__ Span 9 = Generate Content
|__ Span 5 = Tool Request
     |__ Span 6 = Execute Tool

Environment Details:

  • ADK Library Version (pip show google-adk): 1.25.0
  • Desktop OS:** [e.g., macOS, Linux, Windows] : Mac
  • Python Version (python -V): Python 3.12

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-3-flash-preview

🟡 Optional Information

N/A*

Screenshots / Video:
N/A*

Additional Context:
N/A*

Minimal Reproduction Code:
N/A*

How often has this issue occurred?:

  • Always (100%)

Metadata

Metadata

Labels

tracing[Component] This issue is related to OpenTelemetry tracing

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions