[MongoDB Storage] Incremental parameter index compacting

[rkistner]

#752 made compacting of bucket data incremental under MongoDB V3 storage. This does the same for parameter indexes, under both MongoDB v1 and v3 storage. See the initial proposal here: https://github.com/orgs/powersync-ja/discussions/537

For background, parameter index entries are essentially an attribute (lookup) -> value index , but we store prior versions to allow finding the value at a point-in-time. So we store an auto-incrementing _id, and index by a variation of (lookup, key, _id). When we read the values, we only use the last value for every (lookup, key) combination.

V3 tweaked the storage format slightly, but did not fundamentally change it. The index is different to allow more efficient reads, and we split the collections by parameter index.

Conceptually, this makes incremental parameter index compaction is much simpler than bucket data compaction:

1. Unlike for bucket data, there is no scheduling / throttling needed: We can just keep track of the last compacted position globally, and continue from there.
2. The process is just iterating through all the parameter index entries added since the last compact, and delete all the older matching entries.

Previously, we did a full scan through the collection on each compact, and tracked seen entries in memory to de-duplicate. This rewrites the process to match the above.

Implementation

We store a new field parameter_compaction.compacted_before to track the position.

The naive implementation here would be:

1. Scan through all entries after compacted_before.
2. For each entry, delete all prior entries matching (lookup, key) using a deleteMany.
3. If it's a delete tombstone, include the entry itself.

This has some issues:

1. deleteMany is not atomic by itself, and can execute the deletes in any order. If the deleteMany removes a tombstone before its prior entries, that can result in a read returning inconsistent data.
2. The deleteMany is not cheap for V1 storage: It partially uses the ('key.g': 1, lookup: 1, _id: 1) index, but if there are hundreds of keys with the same lookup, this is inefficient.

We cater for (1) by explicitly deleting prior entries before deleting the tombstone itself, in separate batches.
We cater for (2) by:

1. For ranges we have scanned, we delete directly by _id.
2. For earlier ranges, we group keys together and do a batched deleteMany.

We use a LRU cache to keep track of the last _id we've seen for each (lookup, key), safely falling back to the deleteMany approach for evicted entries.

The implementation is shared for MongoDB v1 and v3, with mostly minor differences related to the v3 collection splits, and the exact fields used for lookups.

Compact fence

The compact could cause consistency issues in rare cases. It could specifically happen when:

1. An API process is under load and falls behind in reading changes to parameter indexes.
2. A parameter index compact runs, deleting a parameter index entry shortly after it is created.

That could cause the entry to be deleted before the API process' change detection has read it. It does not affect reads directly, but does affect change detection triggering the reads.

In the case of an entry update, the new checkpoint would typically see the update and recover. But in the case of a delete, it could miss it completely, and the client could continue syncing buckets that should have been removed, until reconnecting, or until another change is detected that triggers a refresh.

In practice it would be very rare to have an impact: The specific conditions above are rare to trigger, and most cases would automatically recover shortly. It is nonetheless a consistency issue, and this PR fixes it.

In theory we could fix it by only compacting up to a prior checkpoint, say one that is 5 minutes old. That would then only trigger the issue if an API process is more than 5 minutes behind. However, we currently have no way to find prior checkpoints with a specific age, and we have no theoretical upper limit on how far behind an API process can be.

The fix here instead persists an checkpoint_changes_invalid_before invalidation fence:

1. Before compacting a batch, we set this to the "target checkpoint": the upper limit of what we'll compact.
2. When the API process reads a checkpoint, it reads that fence. If the prior checkpoint is older than that, it returns invalidateParameterBuckets: true instead of tracking the incremental changes. This is already a supported path to triggers a refresh of all parameter queries, instead of a more targeted refresh based on changes.

Persisting progress

The implementation described above already avoids repeating work from previous jobs on new runs, since the job persists the last compacted position. However, there can still be cases where the job can take a long time to run, such as on the first run after updating to this version, the first run after a large snapshot, or when the job has a long interval between runs. To cover for those cases, we'd like to continue where we left off if the job was interrupted.

For v1 storage, we scan through a single collection. That gives a simple way to persist progress: We can update compacted_before after each batch. To avoid updating sync_rules too frequently and potentially causing write conflicts with the replication process, we throttle this to max once per minute.

For v3 storage, this is slightly more complicated, since there are multiple collections, and we only persist a single compacted_before. Theoretically we can persist separate progress indicators for each collection, but I'd like to avoid that as far as possible. So instead, we process the collections in lock-step. For each step:

1. Pick the collection that this the furthest behind (we keep track in-memory).
2. Process a batch on that collection.
3. If that advanced the "frontier" (the oldest position compacted over all collections), advance the persisted compacted_before (similarly throttled to once per minute).
4. Repeat.

While each collection can be arbitrarily far ahead in terms of it's max _id in the sequence, that would be at most 1x batch in that collection. That means if the process is interrupted, we lose progress of at most one batch per defined parameter index (in addition to some further progress lost due to throttling the position advancement).

The exact same implementation works with V1 storage - it effectively collapses to the simpler process described above, since the same single collection is picked on each step.

I initially considered splitting this change out into a separate stacked PR, since it's conceptually an incremental change on top of the other work. In practice, this change resulted in a diff covering a very large portion of the implementation, so it doesn't give significant review gains. The specific changes can still be seen in the commit history.

AI Usage

Implemented and reviewed using both Codex gpt-5.6 and Claude Opus 5. Manually designed the approach, with input from Codex and Claude.

[changeset-bot]

🦋 Changeset detected

Latest commit: e4de973

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-mongodb-storage	Minor
@powersync/service-core	Minor
@powersync/service-module-postgres-storage	Patch
@powersync/service-schema	Minor
@powersync/service-module-convex	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Minor
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR