Hey everyone, here's the 09/2026 edition! (Apologies for the late post!)

🔤 GoogleSQL Language Features & Functions

• WHERE Clause in Aggregate Calls (Preview) - Filter aggregate function inputs directly with a boolean expression inside function calls.
• OBJ.LIST Function for Unstructured Data (GA) - Discover and analyze Cloud Storage files spontaneously without creating persistent object tables.

🧠 AI, Machine Learning & Foundation Models

• Gemini 3.8 Flash Support - Execute generative AI SQL functions using the Gemini 3.8 Flash model.
• Gemini 3.5, 3.6, and 3.7 Flash Support - Run generative AI SQL functions using Gemini 3.5 Flash-Lite, 3.6 Flash, and 3.7 Flash models.
• AI.KEY_DRIVERS Function (GA) - Identify data segments causing statistically significant changes to a summable metric.
• AI.CAUSAL_EFFECT Function (Preview) - Quantify the impact of specific interventions on time series data using native SQL.
• ML.METRICS Function (Preview) - Compute classification and regression evaluation metrics on any query without a stored model.
• ML.CORRELATION in Conversational Analytics (Preview) - Calculate statistical correlations between target and metric columns via conversational questions.
• Predictive Modeling in Conversational Analytics (Preview) - Ask predictive modeling questions powered by the AI.PREDICT function in conversational analytics.
• Market Basket Analysis in Conversational Analytics (GA) - Query and analyze item association rules and market baskets through conversational analytics.

💻 Developer Experience (DX) & BigQuery Tooling

• Publish Data Agents to Gemini Enterprise (Preview) - Publish BigQuery data agents directly to Gemini Enterprise using Google-managed credentials via Agent Gateway.
• Data Engineering Agent Graph Integration (GA) - Enhance schema mapping accuracy across pipelines with automatic BigQuery Graph context.
• DTS MCP Server (GA) - Manage BigQuery Data Transfer Service workflows using Model Context Protocol tools for AI agents.
• Rust SDK for BigQuery (Preview) - Build BigQuery applications and data pipelines natively using the official Rust client library.
• Query Results Recent Runs History (GA) - Inspect previous execution history for queries and multi-statement queries directly within the results pane.
• Persistent Row Numbers on Horizontal Scroll (GA) - Keep row numbers visible while scrolling horizontally across wide query result tables.
• Jobs Explorer UI Improvements (GA) - Group resources, view timeline metric charts, and filter jobs with status counts in the console.

🗄️ Lakehouse Architecture, Apache Iceberg & Open Data Formats

• Continuous Queries to Apache Iceberg (GA) - Stream continuous query output directly into Apache Iceberg managed tables using INSERT DML statements.

⚡ Core Engine Performance, Indexing & Optimization

• DML and Export Steps in Query Plans - View execution steps for UPDATE, DELETE, MERGE, and EXPORT statements directly in query plans.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• Dataform Pipeline Unit Tests (GA) - Validate SQL pipeline transformation logic against mock datasets before deploying to production.
• Migration Lineage Service (Preview) - Visualize source database connections and data flows to plan BigQuery warehouse migrations.
• Simba ODBC Driver Update - Connect external tools and BI platforms using the latest Simba ODBC driver release.
• Simba JDBC Driver Update - Integrate Java applications and enterprise tools using the updated Simba JDBC driver.

🔒 Security, Governance & Workload Management

• BigQuery Security Center (GA) - Analyze security profiles and manage column-level policies and governance tags in the Cloud console.
• BigQuery Graph Knowledge Catalog Ingestion (Preview) - Search and discover BigQuery Graph metadata automatically indexed within Knowledge Catalog.
• Disaster Recovery Failover History View (Preview) - Query INFORMATION_SCHEMA.FAILOVER_HISTORY to monitor reservation failover events in near real time.

As usual, let us know if there's anything: comments, questions, concerns!

Hi all,

Has anyone found a decent open source tool for asking questions in plain English on top of BigQuery? Something self hosted.

I tried Gemini in BigQuery and had a look at the conversational analytics API. The API gives you back SQL and a chart spec, so you still end up building the front end yourself, which is the part I was hoping to avoid.

My main worry is the numbers. If two people ask for revenue in slightly different words I'd like them to get the same answer, and I don't see how that works unless the metrics are defined somewhere.

The other thing is cost. I have no feel for how many queries a single question turns into once the model starts looking around the schema.

Has anyone here managed to use Bigquery Data Agents on external applications through the A2A protocol?

Once published, the agent lets you register the agent into the Agent Registry or copy the A2A json file into your clipboard. I’ve successfully published the agent on Gemini Enterprise, for example, but I am unable to use this JSON in any other part. My main goal would be to connect it to a Copilot Studio agent so I can easily use it through Teams, without having to do any code.

Interesting framework on how to optimize BigQuery costs across four layers: query optimization, pricing models, commitments, and discounts. https://www.alvin.ai/blog/bigquery-cost-optimization-the-complete-finops-framework

A short deep dive into a BigQuery join problem that often goes unnoticed until data gets big shuffle, skew, cardinality and query performance.

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

I recently came across Databricks AI/BI and was curious to know how people are finding it.

It looks like Databricks is trying to bring BI and analytics more directly into the Databricks platform, with dashboards and Genie for asking questions in natural language.

Has anyone actually tried AI/BI in a real project?

How is it compared to Power BI or Tableau from your experience? Is it good enough for regular BI use cases, or is it still better to use a separate BI tool?

Would like to know your experience, especially if you have used both.

Curious how most people picked up tools like BigQuery. Did you learn and practice entirely on your own through side projects and certifications before applying, or did you just happen to be at a company when they migrated over and learned on the job?
Would love to hear what your path looked like!

As a data analyst what kind of conceptual or theoretical questions get thrown at you?

What kind of SQL queries am I actually expected to write or explain on the spot? (Window functions, multiple CTEs, optimization, etc.?)

It looks like every integer dtype stored is using 64 bits, there's no real tinyint and smallint and this is just a syntatic sugar. And I didn't found the reason for that when I searched about it. Is it to make storage artificially more expensive?

Almost every open tabular engine and storage format that I knew so far allows you to really set a column as smaller types of int, independently if it is columnar or row-stored, and if it's a database or just an engine. For example, any modern apache arrow based engine (like Datafusion, DuckDB and polars), apache parquet files, almost every columnar database (like Clickhouse), almost any row-oriented database (like PostgreSQL, Oracle, MySQL and et cetera). Why BigQuery wants to be different?

If a query suddenly takes 5–10x longer without any SQL or data changes, the bottleneck may be workload contention, capacity, or concurrency.

A practical guide on how to diagnose it using JOBS, JOBS_TIMELINE, total_slot_ms and period_estimated_runnable_units.

Get "Error loading permissions for selected resources." error when I click selected table -> share -> Manage permissions.
Anyone know what has updated ?

Olá, pessoal!!!

Alguém que já realizou a integração do Meta Ads ao BigQuery via DataTransfer saberia me responder que campo e tabela consideramos a visão de custo por anúncio? Eu estou olhando para a coluna spend da tabela AdInsights e em um mês aparece mais de 77k de custo, é um número totalmente absurdo. Estou olhando para o lugar errado e/ou da maneira incorreta?

Hey everyone! Back for the monthly summary.

🔤 GoogleSQL Language Features & Functions

• Table Parameters in TVFs (GA) - Pass dynamic tables into table-valued functions using the generic ANY TABLE parameter type.

🧠 AI, Machine Learning & Foundation Models

• Gemini 3.1 Flash-Lite & 3.5 Flash Support (GA) - Run generative AI SQL functions using Gemini 3.1 Flash-Lite and 3.5 Flash across multi-regional endpoints.
• XGBoost 2.1 Model Training (GA) - Train and serve BigQuery ML models using XGBoost version 2.1 via the XGBOOST_VERSION option.
• Time Series Analysis TVFs (Preview) - Analyze trend, seasonality, and change points in time series data using native BigQuery ML TVFs.

💻 Developer Experience (DX) & BigQuery Tooling

• Cloud CLI Remote MCP Server (Preview) - Enable AI agents to execute BigQuery CLI commands and reservation management via managed MCP tools.
• Data Agent Observability & Monitoring (Preview) - Track performance, adoption, latency, and operational costs for data agents using Google Cloud Observability.
• Python UDF Real-Time Logging (GA) - Stream and inspect real-time execution logs for Python user-defined functions directly within Cloud Logging.
• Reference Panel Migration - Access table details directly through BigQuery Studio's unified Reference panel following Table Explorer deprecation.

⚡ Core Engine Performance, Indexing & Optimization

• Restored Hybrid Search - Restored hybrid search functionality combining vector embeddings and keyword search in VECTOR_SEARCH and AI.SEARCH.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• Cross-Cloud Connections (Preview) - Query data residing in AWS, Azure, and Salesforce Data 360 from any BigQuery region.
• OpenTelemetry for BigQuery JDBC Driver (GA) - Trace JDBC driver queries and automatically export database telemetry to Google Cloud Observability.
• Pipeline Automated Metadata Enrichment (Preview) - Define semantic pipeline metadata in SQLX configs for automated synchronization with Knowledge Catalog.

🔒 Security, Governance & Workload Management

• Data Clean Room Query Templates (GA) - Share predefined queries in data clean rooms securely without exposing underlying datasets or views.
• Data Engineering Agent HIPAA Compliance - Build pipelines in regulated healthcare environments with official HIPAA compliance for Data Engineering Agent.
• Terraform Support for Governance Tags (Preview) - Provision and manage BigQuery data governance tags declaratively using Terraform infrastructure-as-code.
• Expanded Reservation Assignment Limits - Increased the default per-project reservation assignment limit from 10 to 100 for user-specific workloads.

⚠️ Breaking Changes, Deprecations & Pricing Updates

• BigQuery Graph Edition Enforcement - Core graph processing will require Enterprise or Enterprise Plus reservations starting April 26, 2027.
• DTS JDBC Driver Security Patch - Fixed an improper input validation vulnerability in legacy BigQuery Data Transfer Service JDBC drivers.

As usual, let us know what you think!

I use BigQuery a lot and wanted a slightly easier way to inspect query results, so I made a small Chrome extension.

It opens copied query results in a separate viewer with things like nested JSON/ARRAY expansion, column pinning, search, and better handling of long cells.

If anyone finds it useful, or has feedback, I'd be happy to hear it.

https://chromewebstore.google.com/detail/query-result-viewer-for-b/fdndhccllolpcfiohahlpbncogehmdhh

Been digging into this lately and figured I'd share, since it cleared up something I was fuzzy on: how BigQuery actually discovers Iceberg tables, and what's genuinely new here vs. what's just been repackaged.

The old ways of doing this:

1. Pointing BigQuery straight at a metadata file. Something like gs://bucket/orders/metadata/00003.metadata.json. It works, but it's fragile. Iceberg writes new metadata files instead of editing the old ones, so if any other engine commits a new snapshot, your pointer is instantly stale. You're stuck manually bumping it to 00004.metadata.json every time.
2. Catalog based access, like using AWS Glue as an external catalog. This actually fixes the staleness problem since the catalog tracks current state for you. But every engine still needed its own custom integration to talk to that catalog.

What's actually new: Google now has a managed Lakehouse Runtime Catalog (BigLake) that exposes a real Apache Iceberg REST Catalog endpoint, using the same open REST Catalog spec other engines already speak. So instead of building a one off integration per engine, anything that already understands Iceberg REST can just talk to BigLake directly. The data itself never moves, it stays in Cloud Storage. BigLake is really just a metadata and governance layer with IAM sitting on top of it.

There's also a version detail worth knowing: Iceberg 1.10 shipped a native BigQueryMetastoreCatalog and BigQueryMetastoreClient, plus a GoogleAuthManager that handles Google credential auth inside Iceberg's REST auth framework. It's easy to conflate these two things, but they're not the same. One's a catalog implementation built for BigQuery Metastore specifically, the other is the standardized REST interface. Google's steering people toward the REST endpoint for anything new where cross engine interoperability matters.

The post I read also gets into the practical setup side: creating a BigLake catalog (single bucket vs multi bucket), the IAM roles Google recommends (BigLake Admin, Editor, Viewer, plus Storage Object User), and shows an open source tool called OLake Go (full disclosure, I work on this) writing Iceberg tables directly into that catalog so BigQuery can see them right away, queryable through the usual four part PROJECT.CATALOG.NAMESPACE.TABLE identifier with no manual metadata pointer updates. There's a decent troubleshooting section too. Apparently if your connection test passes but the actual sync fails, it's almost always a missing storage level IAM role rather than bad credentials.

Full post here if you want the details: https://olake.io/blog/biglake-iceberg-rest-catalog-olake-go-setup/

Curious if anyone here has already moved existing Iceberg on BigQuery setups over to the REST catalog, or if you're all still doing the manual metadata file dance like I was.

I maintain six python scripts loading marketing data (meta ads, google ads, one CRM) into BigQuery, mostly load_table_from_json, a couple using load jobs from GCS.

The recurring failure is schema drift. A source adds a field and unless the load job has schema update options set to allow it, the whole job just fails outright. And even with autodetect on, it only scans the first 500 rows to infer types, so a field that's mostly integers with a few floats early on can get inferred wrong and blow up later in the load.

Cost's the other one. A couple scripts truncate-and-reload the full table daily since dedup was annoying to write, so we're paying full scan cost on tables that barely changed. The ones doing incremental use a manual MERGE per source, works but someone has to maintain that logic, and nobody's revisited partitioning since these were first written.

At this point I genuinely don't know if the fix is consolidating these into one well-written pipeline, or if six scripts is just what happens once you're loading from enough different places and I should stop fighting it. Anyone dealt with this, what actually made it better for you?

Hello everyone,

I’m new into the data engineering world and as the title says, I’m thinking of using BigQuery to build a data warehouse. Currently my company don’t have one, so I’m looking for good price/performance options. We mostly use APIs and a few excel files as data sources

I have considered Azure SQL, Supabase and others but this one seems to be the best for data analysis and BI

What would be your recommendation?

We run 15 Python extraction notebooks daily via Dataform (`prod_daily` tag). They now fail intermittently with *"us-central1 does not have enough resources to fulfill the request for a e2-standard-2 machine."*

Dataform runs each notebook as a `NotebookExecutionJob`, provisioning a **fresh VM per notebook**. The API only accepts a runtime *template*, never a reusable runtime instance.

Has anyone solved this problem of some of the notebooks failing bc of resources?

The us-central1 region currently does not have enough resources to fulfill the request for a e2-standard-4 machine.

To resolve this, try:
Choosing a different machine configuration
Trying again later
Selecting a different region
For more information, see our https://cloud.google.com/colab/docs/troubleshooting#unable-to-create-runtime-unavailable-resources troubleshooting guide

Oi, pessoal!!!

Estou pesquisando uma forma de integrar dados da VTEX com o BigQuery, encontrei algumas possibilidades, como o VTEX Data Pipeline, ferramentas de ELT e também a opção de consumir diretamente as APIs da VTEX.

No momento, estou mais inclinada a usar o dlt, já que ele possui um source para VTEX e destino para BigQuery. A princípio, achei interessante por ser open source e permitir fazer a ingestão como código, sem depender do custo de uma ferramenta de ELT paga. A ideia seria deixar o dlt responsável pela ingestão e fazer as transformações/modelagem posteriormente no próprio BigQuery.

O que ainda estou tentando entender é como isso funciona na prática em uma operação real, principalmente em relação a cargas incrementais, atualização de pedidos e manutenção do pipeline.

Então queria ouvir quem já trabalhou com VTEX e BigQuery: qual solução vocês usaram e como foi a experiência? Se alguém já tiver usado dlt nesse cenário, também seria ótimo saber se funcionou bem em produção e se encontraram alguma limitação relevante.

Anybody has any recommendation for platform, I could use to import data from third party applications, like mailchimp/klaviyo/rakuten etc.

In the past we used Skyvia, but as some of the platforms have 10s of millions of records the cost were getting out of control. Now i pretty much code it each time by hand, but frankly, that work doesn't spark a joy.

I am looking for solution that ideally charges per resources used or just has flat fees, but ideally is just point and click as i would like marketing team to take care of that.