Repository navigation
Fix deadlock from taking db lock while holding authz lock in Authz() - #564
wallrj-cyberark wants to merge 1 commit into
Conversation
wallrj-cyberark
left a comment
There was a problem hiding this comment.
Self-review: one note on why the order is read without the authz lock, and one place I left alone.
Pebble can deadlock when a new-order, an authorization and a new-account request arrive at the same time. After that it accepts connections but never answers, so every ACME e2e test in the run times out. See letsencrypt/pebble#563. The new commit is current upstream Pebble main, plus the existing Ed25519 patch, plus the fix from letsencrypt/pebble#564. Also correct the comment: we build from cert-manager's fork of Pebble, not @inteon's, and it now carries two patches. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Richard Wall <richard.wall@cyberark.com>
Authz() held the authorization's write lock while it looked up the account, which takes the MemoryStore read lock. FindValidAuthorization takes the same two locks in the opposite order: it holds the MemoryStore read lock while it read-locks every authorization. Once a writer such as AddAccount queues on the MemoryStore lock, the three goroutines wait on each other and every later request that touches the db hangs. Authz() also read-locked the order while holding the authz lock, which is the reverse of the order used by Order.GetStatus. Do the order and account lookups before taking the authz lock. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Richard Wall <richard.wall@cyberark.com>
f81564c to
2eced24
Compare
Pebble can deadlock when a new-order, an authorization and a new-account request arrive at the same time. After that it accepts connections but never answers, so every ACME e2e test in the run times out. See letsencrypt/pebble#563. The new commit is current upstream Pebble main, plus the existing Ed25519 patch, plus the fix from letsencrypt/pebble#564. Also correct the comment: we build from cert-manager's fork of Pebble, not @inteon's, and it now carries two patches. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Richard Wall <richard.wall@cyberark.com>
|
@aarongable could you take another look, please? I moved the account lookup to straight after I have also rebased onto current |
|
I closed this PR to make GitHub pick up my rebased commit, without realising I would not be able to reopen it. I am sorry for the noise. The work continues in #571. It has the same fix, rebased onto current |
Fixes #563.
Authz()held the authorization's write lock while it looked up the account, which takes theMemoryStoreread lock.FindValidAuthorizationtakes the same locks in the opposite order. Once a writer such asAddAccountqueues on the store lock, the whole store wedges. #563 has the details and a goroutine dump.This change does both lookups before
authz.Lock():authz.Orderis set when the authorization is created and never changes, so it is safe to read before taking the authz lock. Reading it first also removes the authz-then-order lock order, which is the reverse ofOrder.GetStatus().validPOSTAsGETorgetAcctByKey. Both branches did this lookup first anyway, so responses and error order do not change.The authz lock now only covers reading and updating the authorization itself.
Testing
mainhung within 1 second in each of 3 runs. With this change it ran for 10 minutes and 264,962 operations without hanging.go test -race ./...passes. Thewfepackage has no tests, so the stress client is the only test of this path.[with Claude]