fix(bigquery): relax metadata columns to NULLABLE to unblock loads (v0.4.1)

[anaselmhamdi]

Problem

All BigQuery batch (bigquery) pipelines were failing with:

400 Provided Schema does not match Table ...:bizon_anthropic.skills.
Field _source_record_id has changed mode from NULLABLE to REQUIRED

Root cause

get_bigquery_schema() declared the metadata columns (_source_record_id, _source_timestamp, _bizon_extracted_at, _bizon_loaded_at, _bizon_id) as mode="REQUIRED", and _build_load_job_config() passes that schema on every load (load_table_from_uri). The production tables already have those columns as NULLABLE, and BigQuery does not allow promoting an existing NULLABLE column to REQUIRED — only the reverse (relaxation) is permitted. So the load is irreconcilable without dropping/recreating the table. The failure is most visible in stream mode, where the load targets the final table directly (temp_table_id == table_id).

The REQUIRED mode has existed since Dec 2024; it surfaced now because these loads run against pre-existing NULLABLE tables.

Fix

• Relax the metadata columns to NULLABLE so the load schema matches existing tables. bizon always populates these fields, so this drops only the BQ-level NOT NULL constraint, not data.
• Add ALLOW_FIELD_RELAXATION + ALLOW_FIELD_ADDITION to the load job so any table created REQUIRED by the old code self-heals on its next load (no manual migration), and additive source-schema drift is tolerated.
• Scoped to the failing batch bigquery connector only (bigquery_streaming / bigquery_streaming_v2 unchanged).
• Regression test asserting NULLABLE modes + relaxation options.
• Bump to v0.4.1 + CHANGELOG entry.

Recovery for the already-stalled pipelines

bizon is at-least-once with cursor checkpointing, so recovery is: deploy this fix → re-run each pipeline. Each resumes from the last successful cursor, re-fetches the failed range, and loads into the (now matching) NULLABLE table; duplicate writes are expected and acceptable. No table surgery is needed for the NULLABLE tables (the common case). Tables created REQUIRED by the old code self-heal via ALLOW_FIELD_RELAXATION on the next load.

Note: for incremental/full_refresh, loads land in fresh _temp/_incremental tables (NULLABLE) and are copy_table'd into main. A copy job has no relaxation option, so a pre-existing REQUIRED main table in those modes needs a one-time ALTER TABLE ... ALTER COLUMN <col> DROP NOT NULL. The reported failures are stream mode, fixed by this PR directly.

Tests

• New tests/connectors/destinations/bigquery/test_bigquery_schema.py — passes.
• Existing async-load and unit tests pass. The 2 failing *_live tests + test_bigquery_client / test_bigquery_factory errors are pre-existing (require live BigQuery credentials / an unrelated SyncMetadata test bug) — verified identical on unmodified source.

🤖 Generated with Claude Code