feat: Add BigQuery vs Cloud Logging comparison demo #4079
Conversation
This demo compares approaches for logging and analyzing ADK agent telemetry: 1. BigQuery Agent Analytics Plugin - purpose-built for agent analytics 2. Cloud Trace via OTel - standard observability with manual BigQuery export 3. Cloud Logging Export - limited data capture Key findings: - BigQuery Plugin: 10 lines setup, real-time, 100% data coverage - Cloud Trace → BQ: 500+ lines ETL code, manual export, ~80% coverage - Cloud Logging → BQ: 150 lines setup, ~20% data coverage Includes: - Demo agent with 4 tools (weather, trip cost, flaky API, sentiment) - SQL queries for token usage, tool analytics, latency analysis - AI-powered analytics with Gemini (LLM-as-Judge, jailbreak detection) - Complete Cloud Trace to BigQuery export pipeline - Comprehensive comparison documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Summary of ChangesHello @caohy1988, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request delivers a comprehensive demonstration and comparison of different strategies for collecting and analyzing telemetry data from ADK agents. It aims to highlight the superior capabilities of the BigQuery Agent Analytics Plugin, particularly its seamless integration with AI for advanced insights and its efficiency in data capture and setup. The demo meticulously contrasts this plugin with the more conventional Cloud Logging and Cloud Trace methods, illustrating their limitations in terms of data completeness, real-time processing, and the significant manual effort required to achieve comparable analytical depth. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
adk-bot
added
the
tracing
|
Response from ADK Triaging Agent Hello @caohy1988, thank you for your contribution! This looks like a great new feature. According to our contribution guidelines, all new features should have an associated GitHub issue. Could you please create one for this pull request? |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive demonstration comparing BigQuery and Cloud Logging for ADK agent telemetry. The addition is substantial, providing runnable code, setup scripts, and detailed documentation. The overall structure and implementation are well-executed for a demonstration. My review has identified a few areas for improvement: an incorrect dependency noted in a script comment, a data corruption bug within the trace export script, and an inconsistent query in one of the execution scripts. I have included specific suggestions to address these points. With these changes, this will be an excellent resource for developers.
| input_tokens = int(labels.get("gen_ai.usage.input_tokens", 0)) or None | ||
| output_tokens = int(labels.get("gen_ai.usage.output_tokens", 0)) or None |
There was a problem hiding this comment.
There's a bug in the way token counts are parsed. The expression int(labels.get(..., 0)) or None incorrectly converts a token count of 0 to None, which leads to data loss. This occurs because int(0) evaluates to 0, and 0 or None results in None. The logic should be updated to correctly handle cases where the token count is zero.
| input_tokens = int(labels.get("gen_ai.usage.input_tokens", 0)) or None | |
| output_tokens = int(labels.get("gen_ai.usage.output_tokens", 0)) or None | |
| input_tokens_str = labels.get("gen_ai.usage.input_tokens") | |
| input_tokens = int(input_tokens_str) if input_tokens_str is not None else None | |
| output_tokens_str = labels.get("gen_ai.usage.output_tokens") | |
| output_tokens = int(output_tokens_str) if output_tokens_str is not None else None |
| JSON_EXTRACT_SCALAR(content, '$.usage.total_token_count') as tokens, | ||
| JSON_EXTRACT_SCALAR(content, '$.usage.prompt_token_count') as prompt_tokens, | ||
| JSON_EXTRACT_SCALAR(content, '$.usage.candidates_token_count') as response_tokens |
There was a problem hiding this comment.
The JSON paths for token counts in this query (total_token_count, prompt_token_count, candidates_token_count) are inconsistent with the data schema documented in README.md and the queries in bq_analysis_queries.sql. Other files in this demo use total, prompt, and completion. To ensure this example query functions correctly with the demo's data, please update the JSON paths for consistency.
| JSON_EXTRACT_SCALAR(content, '$.usage.total_token_count') as tokens, | |
| JSON_EXTRACT_SCALAR(content, '$.usage.prompt_token_count') as prompt_tokens, | |
| JSON_EXTRACT_SCALAR(content, '$.usage.candidates_token_count') as response_tokens | |
| JSON_EXTRACT_SCALAR(content, '$.usage.total') as tokens, | |
| JSON_EXTRACT_SCALAR(content, '$.usage.prompt') as prompt_tokens, | |
| JSON_EXTRACT_SCALAR(content, '$.usage.completion') as response_tokens |
| # pip install opentelemetry-exporter-gcp-logging \ | ||
| # opentelemetry-exporter-gcp-monitoring \ | ||
| # opentelemetry-exporter-otlp-proto-grpc |
There was a problem hiding this comment.
The comment suggests opentelemetry-exporter-otlp-proto-grpc as a prerequisite. However, ADK's OTLP exporters utilize the HTTP protocol, making opentelemetry-exporter-otlp-proto-http the correct dependency. Please update the comment to guide users to install the correct package.
| # pip install opentelemetry-exporter-gcp-logging \ | |
| # opentelemetry-exporter-gcp-monitoring \ | |
| # opentelemetry-exporter-otlp-proto-grpc | |
| # pip install opentelemetry-exporter-gcp-logging \ | |
| # opentelemetry-exporter-gcp-monitoring \ | |
| # opentelemetry-exporter-otlp-proto-http |
Address peer review feedback to make the document more resilient: 1. Change "IMPOSSIBLE" to "High Friction / Non-Native" framing - More accurate engineering trade-off language - Acknowledges Log Analytics can technically access AI 2. Add "Schema Stability" as key advantage - BigQuery Plugin provides agent-aware, documented contract - OTel attribute names may change without notice 3. Acknowledge Cloud Logging Log Analytics feature - Note it doesn't solve the "missing data" problem - Tokens, full responses still not captured 4. Clarify "Real-Time" nuances - "Operational real-time" (Live Tail) vs "Analytical real-time" - Cloud Trace wins for "debug this call now" - BigQuery wins for "what's our trend?" 5. Add OTel attribute size limits section - 128KB span limit affects large payloads - Multi-modal content won't fit - BigQuery Plugin handles via GCS offloading 6. Give Cloud Trace credit for visualization - Gantt charts are unmatched for latency debugging - Acknowledge "Trace ID: Cloud Trace wins" 7. Update conclusion to be balanced - "Use Both in Production" recommendation - Complementary tools, not competing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add a concise executive summary at the beginning that: - Positions BigQuery Plugin as "Intelligence Layer" - Positions Cloud Trace/Logging as "Observation Layer" - Provides quick decision table for common questions - Emphasizes complementary nature of both approaches - Highlights schema stability as key differentiator 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
3 participants
Summary
test-project-0728-467323Key Findings
Files Added (14 files, ~3,800 lines)
Agent & Runners:
agent.py- Demo agent with 4 tools (weather, trip cost, flaky API, sentiment)run_with_bq_analytics.py- BigQuery plugin runnerrun_cloud_logging_demo.py- Cloud Trace/Logging runnerBigQuery Plugin Queries:
bq_analysis_queries.sql- 30+ SQL queries for token usage, tool analytics, latencybq_ai_powered_analytics.sql- AI-powered queries (LLM-as-Judge, jailbreak detection, sentiment)Cloud Trace Export Pipeline:
setup_trace_to_bigquery.sh- Setup script for trace exportexport_traces_to_bigquery.py- ETL script to export traces to BigQuerytrace_export_queries.sql- SQL queries for exported trace dataCloud Logging Export:
setup_cloud_logging_export.sh- Cloud Logging to BQ export setupcloud_logging_export_queries.sql- Queries for exported logsDocumentation:
README.md- Comprehensive comparison one-pager with real resultsTest plan
🤖 Generated with Claude Code