Span Hierarchy and Duration Inconsistency in BigQueryAgentAnalyticsPlugin

[haiyuan-eng-google]

This summary organizes Alexander McIntosh's feedback into a formal GitHub issue format, detailing the span hierarchy and duration bugs encountered when using the BigQueryAgentAnalyticsPlugin.

Description

When the BigQueryAgentAnalyticsPlugin is enabled alongside the opentelemetry-instrumentation-vertexai package, the OpenTelemetry span hierarchy becomes corrupted. The plugin appears to inject unexpected spans and re-parent existing ones, leading to broken ancestry and incorrect span durations in observability backends.

Current Behavior (Bug)

When the plugin is enabled, the following issues occur:

• Incorrect Hierarchy: An additional llm_request span is injected.
• Improper Reparenting: The execute_tool span, which is expected to be a child of call_llm, is being re-parented elsewhere in the tree.
• Duration Mismatch: Parent spans are showing shorter durations than their children (e.g., a child span of 12.52s existing inside a parent span of 2.342s). This breaks the logic for synchronous/sequential flow visualization.

Observed Ancestry:
invocation -> invoke_agent <agent_name> -> llm_request (Injected) -> call_llm -> generate_content

Expected Behavior

The span ancestry should remain clean and follow the standard execution flow without orphaned or miscalculated durations.

Standard Ancestry:
invocation -> invoke_agent <agent_name> -> call_llm -> generate_content (with execute_tool nested under call_llm).

Environment Information

• Plugin: BigQueryAgentAnalyticsPlugin
• Conflicting Package: opentelemetry-instrumentation-vertexai
• Service: eva-agent-dev-supervisor

Supporting Evidence

• Image 1: Shows the initial invocation and BigQuery table interactions.
• Image 2: Clearly demonstrates the duration bug (child span > parent span) and the injection of the llm_request layer.
• Image 3: Highlights the re-parenting of the execute_tool and generate_content spans.