[Project] Incremental Compacting Implementation

[rkistner]

Status

Updated 2026-08-20

Incremental bucket data compacting:

• Implemented for MongoDB V3 storage in Restructure MongoDB V3 compacting #752
• Postgres support pending; depends on restructuring the storage schema to match MongoDB V3.
• MongoDB V1 storage not planned: The implementation requires storage changes present in V3.

Incremental parameter index compacting:

• Implementation for MongoDB V1 and V3 storage in [MongoDB Storage] Incremental parameter index compacting #759
• Postgres support: Proof-of-concept implementation in [Postgres Storage] Parameter index compacting #761

Background

Currently we recommend running a compact job once per day. It then iterates through existing bucket_data and bucket_parameters, and removes duplicates in short.

We've went through multiple iterations on how exactly this is implemented, but currently:

1. Bucket data (bucket_data collections) is compacted by iterating through each bucket in bucket_state, checking if it meets the compact threshold requirements, then compacts the data.
2. Parameter index entries (bucket_parameters collections) are compacted by iterating through each lookup value, then removing duplicates.

In the past, we've had a variation where we only iterated through buckets in bucket_state with more than 10 new operations since the previous compact. An issue here was that you could have large buckets that continuously get new operations, and then the compact job repeatedly re-compacts the same bucket.

A further issue is that the compact job is not resumeable: If it fails in the middle, you need to restart the process from scratch. It is also not feasible to do more regular compacting without adding a lot of overhead.

The effect of these issues are often not present until you reach high data volumes. We do however have cases of hundreds of millions of buckets on a single instance, where these issues become quite prevalent.

Proposal

The goal is to reduce the base overhead of both compact processes to make it possible to run incrementally: It should be possible to run say once per hour instead of once per day, without significant more overhead.

Bucket data

Implemented a new design in #752 - see that PR for a description. Old proposal kept below.

For bucket_data, we split the process into two steps:

1. Compute and persist the list of buckets to compact since the last run.
2. Iterate through the list of buckets and compact them.

For step 1, we utilize the index on bucket_state.last_op: This is already being used to track the buckets updated after any checkpoint. We can therefore keep track of the last checkpoint for which we computed the list of buckets to compact, and then use this to find any new buckets that we may need to compact. This gives us flexibility in what info we use to decide whether or not to compact a bucket, without needing to change indexes. We then store these buckets in a new collection, for example buckets_to_compact.

Step 2 is then simple:

1. Iterate through buckets_to_compact, in any order.
2. Compact each bucket, and remove it from buckets_to_compact.

This split approach means we can keep minimal state in bucket_state, keeping the replication write path clean. Any compact-specific state tracking can happen in the separate buckets_to_compact collection.

We could also do a form of compact progress tracking using the buckets_to_compact collection.

In the future, we could extend this to allow multiple concurrent compact processes: All we'd need additionally is some form of "locking" on the buckets_to_compact collection.

Parameter index entries

Implemented for MongoDB V1 and V3 storage in #759.

The bucket_parameters documents are already indexed by op_id sequence, correlated with checkpoints. The compact process can iterate through all the documents added since the last run, and compact on those (lookup, key) values only.

Splitting compact

Implemented in #569.

We should add new CLI options to compact either only bucket_data, or only bucket_parameters. Splitting this will help make it more robust: At the moment, we have cases of bucket_data compact never finishing, which then means bucket_parameters is never compacted, causing performance issues. By splitting these, we can ensure each gets a chance to compact.

The default option for the compact command should remain to compact both.

Postgres storage

Proof-of-concept implementation: #761

On Postgres storage, we currently only compact bucket_data. We should extend this to also compact bucket_parameters.

MongoDB Storage: Read from secondaries

Due to interleaved reads and writes, this is more complex than initially expected. Furthermore, S3 storage offloading significantly reduces the storage database load here. Abandoned POC in #570.

The compact can produce a high read load on the storage database. We can make the readPreference for compact jobs configurable, potentially defaulting to readPreference: secondaryPreferred.

Right now it may be possible to set readPreference in the connection URI, but it would make more sense to configure it for compacting specifically.