APP-926: optimize sp_morbidity_report_datamart_postprocessing - #980
Open
nullflux wants to merge 17 commits into
Open
APP-926: optimize sp_morbidity_report_datamart_postprocessing#980nullflux wants to merge 17 commits into
nullflux wants to merge 17 commits into
Conversation
The #MORB_EVENT_INIT build (the SP's worst query per the attached exec plan) is optimized two ways, both in RDB_MODERN: 1. Covering index IX_EVENT_METRIC_EVENT_UID on EVENT_METRIC(EVENT_UID) INCLUDE(...). The build LEFT JOINs EVENT_METRIC on EVENT_UID, but the clustered PK is (EVENT_TYPE, EVENT_UID) so it scanned the entire table. This is the plan's own missing-index hint (32.7% impact). New migration tables/263-... + changelog entry app-926-add_ix_event_metric_event_uid. 2. Split materialization in the SP. The WHERE had seven OR'd col IN (STRING_SPLIT(@PARAM,',')). Each @*_uids list is now parsed once into a typed bigint PK-indexed temp and probed with EXISTS. CAST (not TRY_CONVERT) preserves the original's throw-on-malformed-input behavior exactly. Result on a 5M-EVENT_METRIC / 50k-report repro: full SP ~9,273ms -> ~450ms (~20x); the target statement ~7,700ms -> ~370ms; EVENT_METRIC logical reads 100,847 -> 828. Output is byte-identical to the original across 8 scenarios (obs/pat/prov/org/inv/mixed/empty/junk) - see the equivalence harness under testing-tools/app-926-morbidity-equivalence (AC #2). SP file is runOnChange:true (AC #3). Further gain (not applied, needs MORBIDITY_REPORT_EVENT 1:1 validation): a UNION-of-seeks pre-filter would stop processing every ACTIVE report per call and close the last gap to <300ms.
Adds a #cand pre-filter before the #MORB_EVENT_INIT projection: a superset of candidate MORB_RPT_KEYs collected via seek-driven UNION branches (each anchored on a small #uid_* temp), then INNER JOIN #cand into the projection. The original WHERE (7 OR'd EXISTS + MORB_RPT_KEY<>1 + RECORD_STATUS='ACTIVE') is left UNCHANGED, so the per-(MR,MRE)-row filter still decides output -> equivalent even for reports with multiple events. This stops the SP from seek-joining every active report per call; runtime now scales with batch size, not total report count or dimension size. Also forces MAXDOP 1 on the #MORB_EVENT_FINAL build (it was taking a parallel plan burning ~140ms CPU on 251 rows). #MORB_EVENT_INIT ~370ms -> 16.6ms; full SP ~450ms -> ~114ms (~82x from the 9,273ms baseline). Equivalence harness extended to a 9th scenario: a report with two MORBIDITY_REPORT_EVENT rows where only one matches the filter (the case a key-grain rewrite would break). All 9 scenarios byte-identical (orig vs optimized, full-row EXCEPT both directions), determinism check clean. Remaining ~114ms is dominated by #MORB_EVENT_FINAL (~45ms, intrinsic 150-col projection CPU, already batch-scoped); reaching 30ms would require projection-logic surgery with its own equivalence risk.
Converts the equivalence scenarios into the repo's golden-file unit-test framework (src/test/resources/testData/unit/, run by DataDrivenUnitTests). 8 cases, one per qualifying branch plus edges: obs, pat, prov, org, inv, mixed, multievent, empty. Each case: setup.sql seeds a tiny RDB_MODERN fixture (distinct 92xxxxx key range per case) and EXECs sp_morbidity_report_datamart_postprocessing; query.sql selects the resulting MORBIDITY_REPORT_DATAMART rows; expected.json is the golden. Goldens were captured and verified to match the ORIGINAL (pre-optimization) SP, so they assert behavior, not just the current output. The multievent case (one report, two MORBIDITY_REPORT_EVENT rows, only one matching) guards the batch-scoping equivalence. The empty case asserts COUNT=0 (the runner requires a row-returning query). Full suite passes 8/8. These are the forward-looking CI regression guards; the original-vs-optimized diff harness remains under testing-tools/app-926-morbidity-equivalence for one-off certification + perf.
The forward-looking regression coverage lives in the DataDrivenUnitTests cases (testData/unit/app926_morbidity_datamart_*). The original-vs-optimized comparison harness was a one-time certification aid and is no longer needed in the repo.
nullflux
force-pushed
the
aw/app-926/optimize-morbidity-datamart-sp
branch
from
August 5, 2026 18:39
93af69c to
f755dd0
Compare
ericnagel
requested changes
Aug 5, 2026
…litting @obs_uids in the inactive lookup step.
…YS. This targets the later join on LAB_RPT_LOCAL_ID in the lab results step.
…ning access pattern (group by MORBIDITY_REPORT_KEY + joins filtered by row_num).
…n that groups by MORBIDITY_REPORT_KEY and joins by row_num.
…ned with the update intent for existing target rows only (src.DML_IND = 'U').
ericnagel
previously approved these changes
Aug 6, 2026
4 tasks
…ze-morbidity-datamart-sp # Conflicts: # reporting-pipeline-service/src/main/resources/db/changelog/migrations/v7.13/rdb/rdb.changelog-7.13.yaml
…ze-morbidity-datamart-sp # Conflicts: # reporting-pipeline-service/src/main/resources/db/changelog/migrations/v7.13/rdb/rdb.changelog-7.13.yaml
ericbuckley
reviewed
Aug 10, 2026
…ze-morbidity-datamart-sp # Conflicts: # reporting-pipeline-service/src/main/resources/db/changelog/migrations/v7.13/rdb/rdb.changelog-7.13.yaml
ericbuckley
approved these changes
Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Optimizes
sp_morbidity_report_datamart_postprocessing(>48h in KY). Three RDB_MODERN changes to the#MORB_EVENT_INITbuild (the plan's worst query):EVENT_METRIC(EVENT_UID)— the plan's own missing-index hint; turns a full-table scan into a seek.EXISTS, replacing seven per-rowSTRING_SPLITre-parses.#candpre-filter that scopes the projection to the batch's candidate reports instead of every active report (superset of keys + the original WHERE left unchanged).On a 5M-row / 50k-report repro: ~9,273 ms → ~114 ms (~82x), output byte-identical.
Related Issue
APP-926
Additional Notes
CAST(notTRY_CONVERT) keeps the original throw-on-malformed-input behavior; the WHERE is unchanged, so multi-event reports stay correct.tables/263-…); the SP migration isrunOnChange: true.#MORB_EVENT_FINALprojection (result-producing logic), left out to keep output guaranteed identical.Checklist
DataDrivenUnitTestscases (one per qualifying branch + edges), passing 8/8.