Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces robust support for BigLake managed Iceberg tables within the BigQuery analytics plugin. It enables users to configure their analytics to write data to Iceberg format in Google Cloud Storage, leveraging BigQuery's capabilities. The changes include necessary schema adjustments for Iceberg compatibility, standardized connection handling, and flexible partitioning options, ensuring seamless integration and data consistency across different analytics engines.

Highlights

• BigLake Iceberg Support: Added a new biglake_storage_uri configuration option to BigQueryLoggerConfig to enable the creation of BigLake managed Iceberg tables.
• Schema Transformation: Implemented automatic replacement of JSON schema fields with STRING types, as BigLake Iceberg does not support the JSON data type.
• BigLake Configuration: Ensured that BigLakeConfiguration (including connection_id, storage_uri, file_format=PARQUET, and table_format=ICEBERG) is correctly set during BigLake table creation.
• Connection ID Normalization: Introduced logic to normalize connection_id to its full resource path format (projects/{p}/locations/{l}/connections/{c}), accepting various short forms.
• Time Partitioning Control: Configured BigLake Iceberg tables to skip time partitioning by default (as it's a preview feature), with an opt-in mechanism via biglake_time_partitioning.
• Documentation Update: Added a note documenting the Storage Write API latency caveat for Iceberg metadata refresh, advising on potential delays for cross-engine visibility.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Response from ADK Triaging Agent

Hello @caohy1988, thank you for your contribution!

To proceed with the review, could you please address the following points from our contribution guidelines:

1. Sign the Contributor License Agreement (CLA): It looks like the CLA check has failed. Please visit https://cla.developers.google.com/ to sign it.
2. Associate an Issue: For new features, we require an associated GitHub issue. Could you please create one that describes this feature and link it to this PR?

Completing these steps will help us move forward with the review process. Thanks!

I looked into the Spark BigQuery connector path as a way to validate the “BigLake Iceberg supports high throughput streaming using the Storage Write API” claim.

Short conclusion: yes, the Spark connector is a valid proof path for Storage Write API -> BigLake Iceberg, but it is not a good in-process replacement for the current Python plugin writer.

Why:

• The BigLake docs explicitly position Storage Write API support through connectors like Spark and Dataflow, not through a documented low-level Python AppendRows sample:
  https://docs.cloud.google.com/biglake/docs/biglake-iceberg-tables-in-bigquery
• The Spark BigQuery connector supports direct writes via Storage Write API using writeMethod=direct:
  https://github.com/GoogleCloudDataproc/spark-bigquery-connector

Example shape:

(
    df.write
    .format("bigquery")
    .option("writeMethod", "direct")
    .option("writeAtLeastOnce", "true")
    .mode("append")
    .save("project.dataset.biglake_iceberg_table")
)

Recommendation:

1. Use Spark/Dataflow as a standalone validation path or external ingestion backend.
2. Do not try to swap the current BigQueryAgentAnalyticsPlugin writer to Spark in-process. That would be a major architecture change: JVM/Spark runtime, connector jar management, process startup cost, and a very different operational model from the current lightweight Python callback path.
3. If we want to prove whether Storage Write API can work for this use case, the next practical step is a small standalone Spark or Dataflow repro that writes rows shaped like the plugin events into an existing BigLake Iceberg table.
4. If that repro succeeds while the current raw Python Storage Write API path still fails with _colidentifier_iceberg_1, then the correct conclusion is likely: connector-supported path works, but the current low-level direct client path used by the plugin is unsupported or at least undocumented.

One additional caveat from the docs: even when streamed writes succeed, Iceberg metadata visibility for open-source engines may lag by up to ~90 minutes, so this should not be treated as immediate cross-engine freshness.

Given that, I would not change the plugin implementation to Spark. I would treat Spark/Dataflow as:

• a validation harness for the product capability, and
• potentially a separate ingestion architecture if BigLake Iceberg is a hard requirement.

After looking at the documented support surface and the current E2E result, my recommendation is to keep this PR as a minimal MVP for BigLake support.

Recommended default behavior:

• native BigQuery table -> storage_write_api
• BigLake Iceberg table -> legacy_streaming

Why I think this is the right scope for this PR:

1. It solves the actual user problem in [Feature Request] Support writing analytics events to BigLake tables in BigQuery Agent Analytics plugin #4747: support writing analytics events to BigLake tables.
2. It avoids blocking the feature on the current raw Python Storage Write API failure (_colidentifier_iceberg_1).
3. It keeps the change small and easy to reason about instead of introducing multiple new backends / batching strategies / ingestion modes in one PR.
4. It preserves the current best path for native BigQuery tables, where storage_write_api is already the right default.

Why this split makes sense technically:

• For native BigQuery tables, storage_write_api is the existing fast path and should remain the default.
• For BigLake Iceberg, we have evidence that some supported write paths work (legacy_streaming, DML, load jobs), while the current low-level direct Storage Write API path does not.
• The BigLake docs mention Storage Write API support through connectors like Spark/Dataflow, but that is not the same thing as the current raw Python AppendRows path used by this plugin.

So for this PR, I would explicitly avoid expanding scope into:

• multiple BigLake write backends
• load-job batching
• DML fallback logic
• trying to force raw Storage Write API support to work

Those can all be follow-up work if needed. For MVP, the cleanest path is:

• add BigLake Iceberg table support
• route BigLake writes to legacy_streaming
• document that raw Python Storage Write API remains unresolved for BigLake Iceberg

That gives users a working feature now, keeps the PR minimal, and avoids overfitting to an undocumented / currently failing backend path.

I think this should be tracked as a Google-internal / product bug, separate from this PR.

Reason:

• On the same BigLake Iceberg table, the following write paths work: legacy_streaming, DML INSERT, and batch load / LOAD DATA.
• The raw Python Storage Write API path used by this plugin fails with:
  INVALID_ARGUMENT: Input schema has missing required field: _colidentifier_iceberg_1
• That strongly suggests the issue is not "BigLake Iceberg cannot be written", but specifically that the low-level direct AppendRows path is either unsupported, partially supported, or currently broken for managed BigLake Iceberg tables.

I would recommend filing a product bug with a minimal repro like this:

Title:
Raw BigQuery Storage Write API AppendRows to managed BigLake Iceberg table fails with _colidentifier_iceberg_1

Repro summary:

1. Create a managed BigLake Iceberg table in BigQuery with BigLakeConfiguration:
   • storage_uri=gs://...
   • file_format=PARQUET
   • table_format=ICEBERG
   • valid BigLake connection_id
2. Use google-cloud-bigquery-storage Python client (BigQueryWriteAsyncClient) to append rows to:
   projects/{project}/datasets/{dataset}/tables/{table}/_default
3. Send a standard Arrow AppendRowsRequest with a schema matching the user-visible table schema.
4. Observe failure:
   INVALID_ARGUMENT: Input schema has missing required field: _colidentifier_iceberg_1
5. Compare against the same table using:
   • legacy streaming inserts
   • DML INSERT
   • batch load / LOAD DATA
     All of those succeed.

Questions for product team:

1. Is raw direct Storage Write API AppendRows to managed BigLake Iceberg tables expected to be supported?
2. If yes, is _colidentifier_iceberg_1 a backend bug or is there an undocumented client requirement?
3. If no, can docs be clarified to distinguish connector-mediated Storage Write API support (Spark/Dataflow) from raw direct client support?

Given that, I would not block this PR on raw Storage Write API. I would keep the PR minimal and use:

• native BigQuery table -> storage_write_api
• BigLake Iceberg table -> legacy_streaming

That gives users a working MVP now, while the raw AppendRows behavior is investigated with the product team.

Response from ADK Triaging Agent

Hello @caohy1988, thank you for your contribution!

To proceed with the review, could you please address the following point from our contribution guidelines:

• Sign the Contributor License Agreement (CLA): It looks like the CLA check has failed. Please visit https://cla.developers.google.com/ to sign it.

Completing this step will help us move forward with the review process. Thanks!

Updated backend gap summary after the latest BigLake create_views=True validation.

The earlier comment I left is now partially outdated on one point: the BigLake view support gap is no longer just an unvalidated concern. The latest E2E update demonstrates that the auto-created analytics views do work on the current BigLake schema, including extraction from STRING columns using JSON_VALUE(...) / JSON_QUERY(...).

This remains a code-level comparison of the current plugin implementation, not a broader product statement about BigQuery vs BigLake in general.

Current backend comparison

Important update from latest E2E validation

The BigLake backend now has evidence that the current auto-created analytics views are usable:

• create_views=True enabled in the BigLake sample configs
• local BigLake E2E validates all auto-created views are queryable
• spot-check confirms typed extraction like tool_name from STRING columns works

So the previous "BigLake views may not work" concern should now be treated as resolved for the current implementation, assuming the current view SQL patterns.

Remaining confirmed gaps vs native backend

The biggest remaining BigLake backend gaps are now:

1. no Storage Write API path
2. no structured JSON / RECORD physical schema parity
3. no write_client / write_stream compatibility parity
4. different default partitioning behavior
5. lower-fidelity physical storage for nested / multi-modal payloads

Recommended next tasks to close the BigLake gap, ordered by priority

1. Resolve the write-transport gap

   • Goal: determine whether BigLake can ever get parity with the native Storage Write API path.
   • Practical next step: keep the product bug / repro active for raw AppendRows against BigLake Iceberg.
   • Why this is P0: this is the largest architectural gap between the two backends and the main reason BigLake is on a distinct code path.

2. Define the target for schema-fidelity parity

   • Goal: decide whether BigLake should stay as "logical parity, lower physical fidelity" or whether we want a higher-fidelity representation for JSON / RECORD payloads.
   • Practical next step: write a small design on options for preserving more structure without breaking BigLake ingestion.
   • Why this is P1: today BigLake works, but the physical schema is materially less expressive than native.

3. Characterize throughput / latency tradeoffs of legacy streaming vs native Storage Write API

   • Goal: quantify the operational cost of the BigLake fallback path.
   • Practical next step: benchmark representative event volume and flush settings on both backends.
   • Why this is P1: if BigLake is going to be production-worthy, we should know the ingestion cost/perf envelope.

4. Decide whether compatibility properties should be upgraded or explicitly documented as native-only

   • Scope: plugin.write_client and plugin.write_stream.
   • Practical next step: either keep them intentionally native-only and document that, or introduce a BigLake-specific equivalent abstraction later.
   • Why this is P2: not a functional blocker, but still a visible behavioral difference.

5. Revisit default partitioning policy for BigLake once preview/feature maturity is clearer

   • Goal: decide whether BigLake should eventually converge toward native default partitioning behavior.
   • Practical next step: keep current opt-in behavior for now, but treat it as a documented difference rather than a final design.
   • Why this is P2: important for table layout and cost, but not the main blocker to BigLake support.

Bottom line

With the latest view validation, the BigLake backend is in better shape than before: it now has working ingestion, working analytics views, and working E2E verification.

The remaining gap is no longer "can BigLake basically work?" It is now mainly about parity:

• transport parity
• schema fidelity parity
• operational/performance parity

If the PR goal remains a minimal working BigLake MVP, the current implementation is reasonable. The next phase should focus on reducing the parity gap rather than questioning whether the current BigLake path is viable at all.