feat: bulk unenroll all learners from courses via support API (LP-860)

[djoseph-apphelix]

Summary

Adds a bulk-unenroll API to the support app: global staff upload a single-column CSV of course ids, review a dry-run preview, confirm, then watch progress. Every active enrollment in each listed course is deactivated asynchronously.

Jira ticket

LP-860. Backend only — the support-tools MFE that drives it ships separately.

Front end PR :

PR-18

Endpoints

All under /api/support/v1/, global staff only (IsAdminUser).

Method	Path	Purpose
POST	bulk_unenroll/	Upload CSV → validated batch + per-course active counts + row-numbered errors. Mutates nothing.
GET	bulk_unenroll/	List batches (?state=, ?page=).
GET	bulk_unenroll/<batch_id>/	Batch status + per-course rows (?state=, ?page=).
POST	bulk_unenroll/<batch_id>/confirm/	Queue the work.
POST	bulk_unenroll/<batch_id>/cancel/	Stop a run in flight.
POST	bulk_unenroll/<batch_id>/retry/	Re-run only the failed courses.

Upload and confirm are separate on purpose: nothing is deactivated until an operator has seen the counts and supplied a reason.

Engine

Three Celery levels, so no request and no single task holds a whole batch:

1. bulk_unenroll_batch — queues one task per pending course.
2. bulk_unenroll_course — streams the course's active enrollment ids and fans them out in BULK_UNENROLL_CHUNK_SIZE groups. Never materialises a large course in memory.
3. bulk_unenroll_chunk — the only place enrollments are mutated. Deactivates via CourseEnrollment.unenroll(..., skip_refund=True) and writes a ManualEnrollmentAudit per learner.

Correctness properties:

• Re-runnable. Every level filters on is_active=True and claims each learner with select_for_update, so a redelivered task re-does harmless work; a learner already inactive is counted, not removed twice.
• Counted once. BulkUnenrollChunk gives each chunk a durable identity and a single atomic pending -> finished claim, so an at-least-once redelivery cannot push chunks_finished to chunks_total while another chunk has never run.
• Generation-safe. Retry bumps attempt; chunks carry the attempt they were queued for, so a straggler from a superseded run is discarded instead of claiming an identity the current run needs.
• Bounded. A chunk out of soft-time-limit hands its untouched tail to a fresh chunk inside the same transaction that counts it.
• Recoverable. Every level publishes to the broker after claiming its work in the DB, which is what a redelivery cannot repair. Each now fails the work it could not queue, so the batch settles into a state retry accepts rather than sitting running forever.
• Honest status. A chunk stops — up front and periodically mid-pass — if the batch was cancelled, its attempt was superseded, or its course already settled. Enrollments are never mutated after the API reports the batch finished.

Models

BulkUnenrollBatch (one per upload), BulkUnenrollCourseState (one per valid CSV row; unit of progress and retry), BulkUnenrollChunk (completion ledger). Two migrations, no changes to existing tables.

Settings

BULK_UNENROLL_* in lms/envs/common.py: file/row limits, chunk size, rate limit, soft time limit, cancel-check interval, routing key.

Before a large production run, deployment must set BULK_UNENROLL_ROUTING_KEY to an isolated queue with controlled worker concurrency, and BULK_UNENROLL_CHUNK_RATE_LIMIT to a safe value. Defaults use the shared LMS queue with no rate limit.

Known limitation

There is no durable record of a chunk's learner payload before it is published, so a worker killed (OOM, eviction) between claiming and publishing can still strand a course. This matches the publication window in existing Celery features here, including bulk_email. Closing it needs an outbox plus a reconciler; worth deciding before the first large run. Broker failures that raise are handled — those settle into a retryable state.

Tests

pytest common/djangoapps/student/tests/test_api.py lms/djangoapps/support/ lms/djangoapps/support/rest_api/v1/tests/test_views.py → 336 passed.

Covers CSV parsing and limits, permissions, every lifecycle transition, duplicate delivery at all three levels, soft-time-limit continuation, cancellation mid-chunk, retry generations, broker-failure recovery, and a query-count assertion that per-course work is independent of enrollment count.

[Copilot]

Pull request overview

This PR adds a backend-only “bulk unenroll” capability to the Support API, allowing global staff to upload a CSV of course IDs, preview active enrollment counts, confirm with a reason, and then asynchronously deactivate all active enrollments per course with progress tracking, cancellation, and retry support.

Changes:

• Introduces new bulk-unenroll persistence models (BulkUnenrollBatch, BulkUnenrollCourseState, BulkUnenrollChunk) and migrations to track staged uploads and asynchronous progress.
• Implements the asynchronous Celery engine (dispatcher → per-course fanout → per-chunk mutation) with idempotency/at-least-once safety and operational controls via settings.
• Adds Support API endpoints/serializers for upload (dry-run), listing, status, confirm, cancel, and retry, plus a shared CSV parsing utility and extensive test coverage.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
lms/envs/common.py	Adds BULK_UNENROLL_* settings for upload limits and Celery engine throttling/routing.
lms/djangoapps/support/models.py	Adds batch/course/chunk models to persist staged uploads and execution progress.
lms/djangoapps/support/migrations/0007_bulkunenrollbatch_bulkunenrollcoursestate_and_more.py	Creates the new bulk-unenroll tables.
lms/djangoapps/support/migrations/0008_alter_bulkunenrollcoursestate_state.py	Extends course-state choices to include cancelled.
common/djangoapps/student/api.py	Adds a CSV parser and typed exceptions for bulk-unenroll uploads.
common/djangoapps/student/tests/test_api.py	Adds unit tests for the CSV parser behavior and limits.
lms/djangoapps/support/tasks.py	Implements the Celery bulk-unenroll engine and its safety/finalization primitives.
lms/djangoapps/support/rest_api/serializers.py	Adds serializers for batch and per-course status rows.
lms/djangoapps/support/rest_api/v1/views.py	Adds the upload/list/status/confirm/cancel/retry endpoints and shared state filter parsing.
lms/djangoapps/support/rest_api/v1/urls.py	Wires up the new bulk-unenroll routes under the support v1 API.
lms/djangoapps/support/rest_api/v1/tests/test_views.py	Adds extensive API tests for permissions, validation, pagination/filtering, and lifecycle behaviors.
lms/djangoapps/support/tests/test_models.py	Adds model defaults/constraints tests for the new bulk-unenroll tables.
lms/djangoapps/support/tests/test_bulk_unenroll_tasks.py	Adds comprehensive tests for idempotency, retries, cancellation, timeouts, broker failure handling, and query-scaling.

Suppressed comments (1)

lms/djangoapps/support/rest_api/v1/views.py:1098

• Same timestamp issue as confirm: this rollback uses QuerySet.update(...) so modified won’t change even though the batch state is being restored. Since the API exposes modified, include modified=now() here so operators can trust the timestamp when a retry fails to queue.

            BulkUnenrollBatch.objects.filter(
                pk=batch.pk, state=BulkUnenrollBatch.State.PENDING,
            ).update(state=previous_state)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lms/djangoapps/support/rest_api/v1/views.py:1097

• In the retry endpoint’s dispatch-failure rollback, the previously-reset course rows are set back to failed but their error (and finished) fields remain cleared. That makes the failure opaque to operators (status/row serializers expose error), and leaves “failed” rows without a terminal timestamp.

Consider restoring a minimal failure message + finished when rolling the rows back to failed (even if you don’t restore the original pre-retry values).

            BulkUnenrollCourseState.objects.filter(
                pk__in=reset_pks, state=BulkUnenrollCourseState.State.PENDING,
            ).update(state=BulkUnenrollCourseState.State.FAILED, modified=now())

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (3)

lms/djangoapps/support/rest_api/v1/views.py:1100

• In the retry rollback path (when _dispatch_bulk_unenroll raises), the compensating update only flips courses back to failed but does not restore the fields cleared by the retry reset (error, counters, chunk tracking, timestamps). That means a transient broker failure can permanently erase the previous failure details and progress metrics while leaving the course in a failed state, which makes the status endpoint misleading and makes subsequent retries harder to diagnose.

            log.exception("bulk_unenroll: failed to queue retry for batch %s", batch.uuid)
            BulkUnenrollCourseState.objects.filter(
                pk__in=reset_pks, state=BulkUnenrollCourseState.State.PENDING,
            ).update(state=BulkUnenrollCourseState.State.FAILED, modified=now())
            BulkUnenrollBatch.objects.filter(
                pk=batch.pk, state=BulkUnenrollBatch.State.PENDING,
            ).update(state=previous_state, modified=now())

lms/djangoapps/support/rest_api/v1/views.py:734

• This aggregate count queryset will also inherit CourseEnrollment’s default ordering (('user', 'course')), which can add an unnecessary ORDER BY (and sort) to a grouped count over potentially large enrollment tables. Clearing ordering improves performance without changing results.

        counts = dict(
            CourseEnrollment.objects
            .filter(course_id__in=course_keys, is_active=True)
            .values_list("course_id")
            .annotate(active=Count("id"))

lms/djangoapps/support/tasks.py:550

• CourseEnrollment has a default Meta.ordering = ('user', 'course') (common/djangoapps/student/models/course_enrollment.py:356-359). This queryset inherits that ordering, which can force an unnecessary sort over potentially large enrollment tables before streaming user_ids. Clearing ordering here avoids a costly ORDER BY while keeping semantics unchanged (chunking doesn’t rely on order).

    active_user_ids = (
        CourseEnrollment.objects
        .filter(course_id=course_key, is_active=True)
        .values_list("user_id", flat=True)
    )

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

lms/djangoapps/support/rest_api/v1/views.py:1100

• In the retry endpoint’s dispatch-failure rollback, only state is restored. Since the reset just cleared error/counters and finished, this leaves courses in failed with an empty error (and no finished timestamp), which makes the API/UI misleading and also loses the original failure context.

At minimum, set an error and finished during the rollback so operators can see why the retry didn’t start.

            BulkUnenrollCourseState.objects.filter(
                pk__in=reset_pks, state=BulkUnenrollCourseState.State.PENDING,
            ).update(state=BulkUnenrollCourseState.State.FAILED, modified=now())
            BulkUnenrollBatch.objects.filter(
                pk=batch.pk, state=BulkUnenrollBatch.State.PENDING,
            ).update(state=previous_state, modified=now())

[abhalsod-sonata]

Notes: This PR is quite large (4,000+ lines changed), which makes it difficult to review thoroughly.