Skip to content

[Snowflake] Support schema filtering in Snowflake metadata connector - #1161

Merged
kaxuna merged 2 commits into
google:mainfrom
jin-jj:snowflake_schema_parameter
Aug 3, 2026
Merged

[Snowflake] Support schema filtering in Snowflake metadata connector#1161
kaxuna merged 2 commits into
google:mainfrom
jin-jj:snowflake_schema_parameter

Conversation

@jin-jj

@jin-jj jin-jj commented Jul 24, 2026

Copy link
Copy Markdown
Contributor
  1. Add support for --schema "schema1, schema2"
  2. Fix a minor issue that we add extra database where condition for information_schema query.

@mshetairy

Copy link
Copy Markdown

Thanks for working on this! The overall wiring for @RespectsArgumentSchemaPredicate and the WHERE filters on ACCOUNT_USAGE / INFORMATION_SCHEMA looks great.

I did notice two critical risks around how SHOW EXTERNAL TABLES is handled, along with some comment cleanup, that we should address before merging:

1. Cartesian Product & Task Failures (addTasksTo, lines 366-378)

When both --database and --schema are provided, the loop creates a separate task for every possible pair (SHOW EXTERNAL TABLES IN SCHEMA "DB_i"."SCHEMA_j").

  • Task Failures: In Snowflake, running SHOW ... IN SCHEMA "DB1"."SCHEMA_B" throws a SQL compilation error if SCHEMA_B belongs to DB2 and doesn't exist in DB1. Because this pairs every database with every schema in the filter lists, any schema not present in every filtered database will cause JDBC task failures and pollute the error logs.
  • Scaling: A filter of 10 databases and 20 schemas explodes into 200 separate JDBC tasks.
  • Suggestion: Instead of generating schema-scoped SHOW commands in a nested loop, keep executing SHOW EXTERNAL TABLES IN DATABASE "DB_i" (one task per database, as before) and filter the resulting rows by schema in memory using the HeaderTransformer.

2. Unqualified Schema Scoping when --database is omitted (addTasksTo, lines 355-365)

When --database is empty and only --schema is used, this generates SHOW EXTERNAL TABLES IN SCHEMA "SCHEMA1". Because "SCHEMA1" isn't qualified with a database prefix, Snowflake will only search inside the currently selected default database of the JDBC session. If no default database is set on the connection—or if "SCHEMA1" resides in a different database—this query will fail or silently miss external tables across the rest of the account.

3. Comment Cleanup (SnowflakeMetadataConnector.java)

It looks like there are a few AI assistant comment artifacts left over from generating the code modifications (e.g., // Changed to EMPTY_WHERE_CONDITION and // Changed to String literal). Let's be sure to clean those inline comments up before merging so the codebase stays clean!

1. Add --schema to support passing in array of schemas like --schema "schema1, schema2"
2. Minor fix that when query db1.information_schema we still adding database name in the where condition.
@kaxuna

kaxuna commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

lint fails. apply spotlessApply

@jin-jj
jin-jj force-pushed the snowflake_schema_parameter branch 6 times, most recently from e9a047c to e28acac Compare July 29, 2026 01:08
@jin-jj

jin-jj commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author
  1. Added a in-memory filter/predict mechanism for jdbc task
  2. Did the spotless apply

@kaxuna
kaxuna merged commit d126a70 into google:main Aug 3, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants