fix: close the race in create_certificate that lets duplicate certificates through - #2667
fix: close the race in create_certificate that lets duplicate certificates through#2667suparikoli wants to merge 3 commits into
Conversation
…cates through create_certificate() checks is_certified() and only inserts if that comes back empty, but nothing locks between the check and the insert. A doubled click on "Get Certificate" is enough for two requests to both read "not certified yet" before either one's save() has committed, so both go on to create a certificate for the same member+course (frappe#2408). Added a unique constraint on (member, course, batch_name) - a data patch dedupes existing rows first (keeping the earliest of each duplicate set) before adding it, and after_install adds it directly for fresh installs, which skip patches. create_certificate now catches the UniqueValidationError this produces when it loses the race and returns the certificate the other request created instead of letting the error surface. Went with all three columns rather than just (member, course): a Certificate can be earned via a batch instead of a course, and course is blank on those rows, so a two-column constraint on just (member, course) would have treated every batch certificate a member holds as a duplicate of the others. Added two tests. One just checks that calling create_certificate twice in a row is idempotent - unremarkable on its own, but it's the existing is_certified() fast path and it was untested. The other forces the actual race window by mocking is_certified() to return None on the second call even though the first call's certificate is already in the database, which is what makes the new except branch run instead of the fast path. Fixes frappe#2408 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (2): Last reviewed commit: "fix: lock the course row, and fix a NULL..." | Re-trigger Greptile |
| for row in rows[1:]: | ||
| frappe.delete_doc("LMS Certificate", row.name, ignore_permissions=True, delete_permanently=True) | ||
|
|
||
| frappe.db.add_unique("LMS Certificate", ["member", "course", "batch_name"]) |
There was a problem hiding this comment.
Patch recreates existing index
The patch calls add_unique unconditionally, so rerunning it after index creation but before patch completion is recorded can fail migration recovery because the index already exists.
Context Used: Guidelines for reviewing Frappe Framework applicat... (source)
Knowledge Base Used: LMS App Lifecycle: Hooks, Auth, Permissions, and Install
…creation-race-condition # Conflicts: # lms/install.py
…traint Two corrections to the previous commit, both found before this landed anywhere - one from Greptile's review on the PR, one from upstream/develop having grown a sibling fix for the same shape of bug in the meantime. 1. course/batch_name save as SQL NULL, not "", on a plain frappe.get_doc() that never touches whichever of the two doesn't apply - and NULL != NULL as far as a unique index is concerned, so two course-only certificates for the same member+course, both sitting on batch_name=NULL, would not have collided with each other. The constraint would have quietly done nothing for exactly the case frappe#2408 is about. validate() now normalizes both to "" before every save, and the patch normalizes existing NULLs the same way before it groups rows to dedupe them - a mix of NULL and "" across old rows would otherwise put what should be one duplicate group into two. 2. While rebasing onto develop to pick up the unrelated security patches merged there, lms/lms/enrollment_constraints.py landed - the equivalent fix for LMS Enrollment and LMS Batch Enrollment, added at some point after I opened this PR. Its approach is a FOR UPDATE lock on the parent row before the check, with the unique constraint kept as a backstop for whatever bypasses the controller. That's a more direct fix than mine was: catching UniqueValidationError after an insert that didn't need to be attempted still works, but it doesn't stop the second request from getting as far as trying. create_certificate() now takes the same lock on the course row that enroll_in_course() already does, before its own is_certified() check, so a second request for the same member+course waits for the first to commit instead of racing it. Kept the try/except UniqueValidationError as the backstop, same framing as enrollment_constraints.py uses for its own indexes. Rewrote the tests around both: one confirming the lock actually happens before the read (adapted from test_course_enrollment_locks_the_course_before_reading in the new tests/test_enrollment_races.py), one confirming the database constraint holds even with both the fast path and the controller's own duplicate check bypassed (adapted from test_duplicate_batch_enrollment_is_refused_by_the_database in the same file), and one confirming an ordinary sequential duplicate - no race involved - still gets the existing friendly error rather than a raw database one. The original race-recovery test is corrected to bypass validate_duplicate_certificate() too, not just is_certified(): patching only is_certified() left the second call reaching the *existing*, already-working validate_course_duplicates() check first, which raises its own plain ValidationError before ever reaching the code this PR added - so the test was passing without actually exercising it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Pushed an update after Greptile's review and after rebasing onto Greptile was right that Separately, Also corrected the race-recovery test: it patched |
Fixes #2408.
create_certificate()callsis_certified(course)and only builds a new certificate if that comes back empty - there's nothing between the check and thesave()that stops two requests from both reading "not certified yet." A doubled click on "Get Certificate" is realistically enough to hit that window, and the result is twoLMS Certificaterows for the same member and course.There was already a PR for this (#2409), closed a few months back after going stale. @raizasafeel's review on it laid out the right shape for the fix - a DB-level unique constraint is the only thing that actually closes the window, since an
if existscheck has the same race whether it's in Python or invalidate(); keep the fast path for the normal case, but catch the constraint violation on the way out and hand back the certificate that won instead of erroring. This picks that up.Added
frappe.db.add_unique("LMS Certificate", ["member", "course", "batch_name"]), applied through a patch (existing databases can already have duplicates sitting around, so it dedupes first, keeping the earliest of each set) and also directly inafter_installfor fresh sites, which don't run patches. That part mirrorsdelete_duplicate_course_progressalmost exactly, which hit the same shape of bug forLMS Course Progressa few months ago.One thing I changed from the (member, course) framing in that review comment: a certificate can also be earned through a batch instead of a course, and
courseis blank on those rows. A two-column constraint on just (member, course) would treat every batch certificate a member holds as a duplicate of the next one, since they'd all share the same blank course. Made it (member, course, batch_name) instead, which keeps course-certs and batch-certs distinguishable from each other while still catching a real duplicate of either kind.create_certificate()now wraps thesave()in a try/except forfrappe.UniqueValidationErrorand, on that path, looks up whichever certificate actually won and returns it - same shape as what a normalis_certified()hit returns, so nothing downstream needs to care whether it was the fast path or the recovered-from-a-lost-race path.The patch does delete rows (the duplicate certificates, keeping the earliest per member+course/batch), so it's worth a closer look than the rest of the diff - flagging that up front rather than leaving it for someone to notice.
For tests, added two to what was an empty
test_lms_certificate.py. The first just callscreate_certificatetwice in a row and checks it's idempotent - not new behavior, but the existing fast path had no coverage either. The second is the one that actually exercises the fix: it mocksis_certifiedto returnNoneon the second call even though the first call's certificate already exists, which is the only way I could find to force the actual race window deterministically in a single-threaded test rather than just re-proving the fast path works.I wasn't able to run this against a live site from where I'm working - no bench set up in this environment - so I traced it carefully against current
developinstead: pulled the realcreate_certificate/is_certifiedsource, followedUniqueValidationErrorandadd_uniqueintofrappe/model/base_document.pyandfrappe/database/mariadb/database.pyto confirm exactly when Frappe raises that exception and whatadd_uniquedoes at the SQL level, and checked theenable_certificationdefault onLMS Course(it's 0) so the test explicitly turns it on rather than assuming the fixture already has it. Ranruff checkandruff format --checkagainst the changed files, both clean. If there's a lighter way to get this running against an actual site that I'm missing, happy to go do that too.