Skip to content

Fix deadlock from taking db lock while holding authz lock in Authz() - #571

Open
wallrj-cyberark wants to merge 1 commit into
letsencrypt:mainfrom
wallrj-cyberark:fix-authz-lock-order-deadlock-v2
Open

wallrj-cyberark wants to merge 1 commit into
letsencrypt:mainfrom
wallrj-cyberark:fix-authz-lock-order-deadlock-v2

Conversation

@wallrj-cyberark

Copy link
Copy Markdown

Fixes #563. This replaces #564, which I closed by mistake and GitHub will not let me reopen. It includes the change @aarongable asked for in his review of #564.

Authz() held the authorization's write lock while it looked up the account, which takes the MemoryStore read lock. FindValidAuthorization takes the same locks in the opposite order. Once a writer such as AddAccount queues on the store lock, the whole store wedges. #563 has the details and a goroutine dump.

This change does both lookups before authz.Lock():

  • The requesting account, via validPOSTAsGET or getAcctByKey, straight after verifyPOST. The only behaviour change is that a request with an unknown account and an unknown authz now gets the account error rather than a 404.
  • The order's account ID. authz.Order is set when the authorization is created and never changes, so it is safe to read before taking the authz lock. Reading it first also removes the authz-then-order lock order, which is the reverse of Order.GetStatus().

The authz lock now only covers reading and updating the authorization itself.

Testing

[with Claude]

Authz() held the authorization's write lock while it looked up the
account, which takes the MemoryStore read lock. FindValidAuthorization
takes the same two locks in the opposite order: it holds the MemoryStore
read lock while it read-locks every authorization. Once a writer such as
AddAccount queues on the MemoryStore lock, the three goroutines wait on
each other and every later request that touches the db hangs.

Authz() also read-locked the order while holding the authz lock, which
is the reverse of the order used by Order.GetStatus.

Do the order and account lookups before taking the authz lock.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>

@wallrj-cyberark wallrj-cyberark left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review, carried over from #564.

Comment thread wfe/wfe.go

authz.Lock()
defer authz.Unlock()
authz.Order.RLock()

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads authz.Order without holding the authz lock. That is safe because makeAuthorizations sets Order before AddAuthorization publishes the authz under m.Lock(), and nothing assigns it afterwards. GetAuthorizationByID reads the map under m.RLock(), so it happens after that write.

updateChallenge() (line 2638) still takes authz.RLock() then authz.Order.RLock(). That is also the reverse of Order.GetStatus(), but both are read locks, so a cycle needs writers queued on both the order and the authz at once. I have not seen it hang and left it out to keep this change small. I can apply the same change there if you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authz() holds the authz lock while taking the db lock, which can deadlock the whole store

1 participant