Skip to content

[APP-974] scheduled sp locks - #999

Open
ericbuckley wants to merge 8 commits into
rel-7.13from
eb/app-974/scheduled-sp-locks
Open

[APP-974] scheduled sp locks#999
ericbuckley wants to merge 8 commits into
rel-7.13from
eb/app-974/scheduled-sp-locks

Conversation

@ericbuckley

Copy link
Copy Markdown
Contributor

Description

This PR coordinates scheduled event-metric and LAB100 cleanup across service pods with independent SQL Server session application locks, returning clear completed, skipped, or failed statuses. Cleanup locks use exclusive zero-timeout acquisition, release on success and failure, and allow the two different cleanup jobs to run concurrently.

Related Issues

Additional Notes

  • Updated scheduled cleanup stored procedures and reporting pipeline service logic to coordinate execution via SQL Server application locks, preventing race conditions in multi-pod deployments.
  • Technical details:
    • Added sys.sp_getapplock (@LockTimeout = 0) and sys.sp_releaseapplock to sp_event_metric_cleanup_postprocessing and sp_lab100_cleanup.
    • Introduced ScheduledExecutionStatus enum to handle return codes (1/0 completion, -2 skipped, -1 failure).
    • Added comprehensive concurrency, release, and unit tests (ScheduledCleanupAppLockConcurrencyTest, ScheduledCleanupAppLockReleaseTest, etc.).
    • Updated README.md with documentation on scheduled cleanup coordination behavior and lock resource identifiers.

Checklist

  • I have ensured that the pull request is of a manageable size, allowing it to be reviewed within a single session.
  • I have reviewed my changes to ensure they are clear, concise, and well-documented.
  • I have updated the documentation, if applicable.
  • I have added or updated test cases to cover my changes, if applicable.

Update scheduled cleanup procedures (event metric and lab 100) to capture
and evaluate their return execution status codes. Previously, procedures
were executed without inspecting return values. Now, status codes are mapped
via ScheduledExecutionStatus to log skips when already running, throw on
failures, and support legacy completion codes.
Prevent concurrent execution of the event metric cleanup stored
procedure by acquiring an exclusive application lock using
`sys.sp_getapplock`. Update return code on completion to 1 and
adjust unit test expectations accordingly.
Prevent concurrent execution of the lab 100 cleanup stored procedure by
acquiring an exclusive application lock using sys.sp_getapplock. Update
return code on completion to 1 and update repository tests.
Ensure application locks are released in the CATCH block of event metric
and lab 100 cleanup stored procedures if an error occurs after the lock
has been acquired. Add integration tests verifying lock release on
failure.
Update cleanup tests for EventMetricCleanup and Lab100Cleanup to assert
exact log messages, verify repository interactions, and check for
DataProcessingException on failure. Also fix test naming typo.
Add concurrency tests for scheduled cleanup application locks.
Verify that event metric and lab 100 cleanup procedures skip execution
when their respective application locks are held without writing normal
job flow logs, and ensure locks do not block each other.
Add documentation for scheduled cleanup coordination using SQL Server
application locks. Explain that event-metric and LAB100 cleanup procedures
use separate locks with zero timeout to prevent duplicate execution across
service pods without blocking the scheduler.
@ericbuckley ericbuckley self-assigned this Aug 12, 2026
Remove legacy handling for return code 0 in scheduled execution status,
treating it as an unexpected return code.
@ericbuckley
ericbuckley marked this pull request as ready for review August 12, 2026 21:19
@ericbuckley
ericbuckley requested a review from a team as a code owner August 12, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant