Repository navigation
Fix deadlock from taking db lock while holding authz lock in Authz() - #571
Open
wallrj-cyberark wants to merge 1 commit into
Open
wallrj-cyberark wants to merge 1 commit into
wallrj-cyberark wants to merge 1 commit into
Conversation
Authz() held the authorization's write lock while it looked up the account, which takes the MemoryStore read lock. FindValidAuthorization takes the same two locks in the opposite order: it holds the MemoryStore read lock while it read-locks every authorization. Once a writer such as AddAccount queues on the MemoryStore lock, the three goroutines wait on each other and every later request that touches the db hangs. Authz() also read-locked the order while holding the authz lock, which is the reverse of the order used by Order.GetStatus. Do the order and account lookups before taking the authz lock. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Richard Wall <richard.wall@cyberark.com>
wallrj-cyberark
commented
Nov 4, 2026
wallrj-cyberark
left a comment
Author
There was a problem hiding this comment.
Self-review, carried over from #564.
|
|
||
| authz.Lock() | ||
| defer authz.Unlock() | ||
| authz.Order.RLock() |
Author
There was a problem hiding this comment.
This reads authz.Order without holding the authz lock. That is safe because makeAuthorizations sets Order before AddAuthorization publishes the authz under m.Lock(), and nothing assigns it afterwards. GetAuthorizationByID reads the map under m.RLock(), so it happens after that write.
updateChallenge() (line 2638) still takes authz.RLock() then authz.Order.RLock(). That is also the reverse of Order.GetStatus(), but both are read locks, so a cycle needs writers queued on both the order and the authz at once. I have not seen it hang and left it out to keep this change small. I can apply the same change there if you prefer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #563. This replaces #564, which I closed by mistake and GitHub will not let me reopen. It includes the change @aarongable asked for in his review of #564.
Authz()held the authorization's write lock while it looked up the account, which takes theMemoryStoreread lock.FindValidAuthorizationtakes the same locks in the opposite order. Once a writer such asAddAccountqueues on the store lock, the whole store wedges. #563 has the details and a goroutine dump.This change does both lookups before
authz.Lock():validPOSTAsGETorgetAcctByKey, straight afterverifyPOST. The only behaviour change is that a request with an unknown account and an unknown authz now gets the account error rather than a 404.authz.Orderis set when the authorization is created and never changes, so it is safe to read before taking the authz lock. Reading it first also removes the authz-then-order lock order, which is the reverse ofOrder.GetStatus().The authz lock now only covers reading and updating the authorization itself.
Testing
mainhung within 1 second in each of 3 runs. With this change, rebased onto b8fcbe4, it ran for 400 seconds without hanging: 19,835 accounts and 99,061 orders.go test -race ./...passes. Thewfepackage has no tests, so the stress client is the only test of this path.[with Claude]