feat(module-postgres): initial snapshot filters for Postgres sources - #754
Draft
henriquekraemer wants to merge 3 commits into
Draft
feat(module-postgres): initial snapshot filters for Postgres sources#754henriquekraemer wants to merge 3 commits into
henriquekraemer wants to merge 3 commits into
Conversation
Port of the Postgres-relevant subset of powersync-ja#502 onto current main, for testing against a Postgres source and Postgres storage. Adds a global initial_snapshot_filters section to the sync config yaml, mapping table patterns (including schema and table wildcards) to a SQL WHERE clause applied during the initial snapshot. The filter is applied in all three snapshot query types, including the chunked resume path and row re-fetches during streaming. Includes Postgres tests, which the original PR does not have yet: filtered chunked snapshot with resume, and a bare table name pattern matching the same table across multiple schemas.
… sync plan With config.edition: 3, deploying sync rules persists a compiled sync plan, and replication restores the config from that plan instead of re-parsing the YAML (parsePersistedSyncConfigContent). Initial snapshot filters were not part of the serialized plan, so they were silently dropped on that path: getInitialSnapshotFilter() always returned undefined and initial snapshots read entire tables. Store the filters alongside the plan - like event descriptors - as an ordered [pattern, filter] list (matching is first-match-wins and JSONB does not preserve object key order), and restore them when parsing a persisted plan. Plans stored before this field existed are treated as having no filters. The new module-postgres regression test reloads the sync config from storage before replicating (like the replicator does on startup) and uses a stream that selects all rows, so the bucket row count directly measures whether the filter reached the snapshot query.
🦋 Changeset detectedLatest commit: f515abc The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
This is the Postgres subset of the prefiltering work in #502, which we've been running in our staging environment (schema-per-tenant, 4 tenant schemas, largest table 13.6M rows). It follows the direction from that discussion: filters are global rather than per-definition, keyed by table patterns with the same wildcard support as bucket queries, and
sqlas the database-specific key so other syntaxes can be added later.What this does
Adds a global
initial_snapshot_filterssection to the sync config, mapping table patterns (including schema and table wildcards) to a SQL WHERE clause applied during the initial snapshot:Matching is first-match-wins, so specific tables go before wildcard patterns. The filter is applied in all snapshot query types, including the chunked resume path and row re-fetches during streaming. Rows excluded by the filter are skipped by the source database instead of being read, evaluated and discarded by the service, and never enter
current_data.The second commit fixes an interaction with compiled sync plans: with
config.edition: 3the deploy persists a compiled plan, and replication rebuilds the config from the stored plan (parsePersistedSyncConfigContent), which did not restoreinitialSnapshotFilters. The filters parsed and validated fine but never reached the snapshot. They are now persisted alongside the plan (the same treatmenteventDescriptorsalready get) as an ordered list of{pattern, filter}pairs, since matching is first-match-wins and JSONB storage does not preserve object key order. Plans stored before this field existed are treated as having no filters.Results
On our largest tenant the initial snapshot of a 13.6M row table went from reading all rows to only the ~282k matching the filter (~2% working set), and the full multi-tenant snapshot completes in minutes instead of hours.
One operational note that may be worth a line in the docs: a selective filter on a big table needs an index matching the predicate, since the chunked snapshot query is
WHERE <filter> ORDER BY pk LIMIT n. Without one, our first chunk on that table took 8m38s, which blows the snapshot socket timeout and leaves the table retrying. With a partial index on(id)using the same predicate, 2.6s.Tests