[APP-970] stored proc chunking - #993
Open
ericbuckley wants to merge 18 commits into
Open
Conversation
Introduce PostProcessingProperties to configure max batch size for post-processing operations with validation ensuring non-negative values.
Add UidChunker utility to support deduplication and chunking of collection UIDs for post-processing operations.
Update PostProcessingService topic processing methods to chunk IDs and condition codes according to the configured maximum batch size using the UidChunker utility.
Generalize `listToParameterString` in `PostProcessingService` to accept `Collection<?>` instead of `Collection<Long>`, enabling reuse for processing topic condition code chunks (`Collection<String>`).
Update ProcessDatamartData.processMetricEventDatamart to chunk event metric IDs according to the configured maximum batch size, matching the chunking behavior used across other post-processing operations.
Add unit test verifying that PostProcessingService retries the whole entity after a later chunk fails, ensuring correct retry cache handling and metric updates.
Update ProcessDatamartData.executeDmProc to chunk single-list datamart procedure input UIDs according to the configured maximum batch size using the UidChunker utility, and add unit tests.
Update ProcessDatamartData to chunk multi-id datamarts, covid vaccination parameters, and dynamic datamart outputs according to the configured maximum batch size, and add unit tests for multi-chunking behavior.
Replace verbose and large parameter string logs with structured chunk progress and metadata including chunk numbers, total chunks, distinct value counts, and max batch sizes. Standardize chunk iteration loops and remove redundant log constants across post-processing and datamart services.
ericbuckley
changed the title
Move the map chunking helper from ProcessDatamartData into UidChunker as an overloaded chunkDistinct method. This centralizes all UID and map chunking logic in UidChunker and adds comprehensive unit test coverage.
ericbuckley
marked this pull request as ready for review
August 12, 2026 02:33
ericnagel
reviewed
Aug 12, 2026
ericnagel
reviewed
Aug 12, 2026
Adding short-circuits so we skip chunking when the set size is smaller than the max Co-authored-by: Eric Nagel <eric@ericnagel.com>
Fix UidChunker to correctly evaluate total distinct value count across all entity lists instead of map key count when determining if values fit within max batch size. Also clean up code formatting.
mpeels
reviewed
Aug 13, 2026
Comment on lines
153
to
+155
| service: | ||
| post-processing: | ||
| max-batch-size: ${POST_PROCESSING_MAX_BATCH_SIZE:1000} |
Contributor
There was a problem hiding this comment.
Thoughts on adding this to the application-test.yaml with a very low size, maybe even 1 so we force a lot of chunking when the functional tests run?
Contributor
Author
There was a problem hiding this comment.
I ran a couple of tests, one with a batch size of 1 and another with a batch size of 100. Both appear to have worked.
ericnagel
previously approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces batch chunking for large ID lists and dynamic datamart inputs during reporting pipeline post-processing and datamart processing. By breaking down high-volume ID sets into manageable batches.
Related Issues
Additional Notes
Implemented batch chunking across post-processing and datamart services to ensure robust handling of high-volume data without exceeding parameter limits.
UidChunkerutility to split large ID collections into configurable batch sizes.PostProcessingPropertiesandapplication.yaml.listToParameterStringto support batched parameter string generation.Checklist