Bbeggs/server side learner pathways - #281
Open
macdiesel wants to merge 5 commits into
Open
Conversation
Moves learner pathway generation out of the learner-portal MFE into enterprise-access, built on the existing apps/workflow pattern (ADR 0025) as two workflows behind two endpoints. POST /api/v1/learner-pathways/careers/ intake -> career candidates POST /api/v1/learner-pathways/pathway/ selected career -> 5 courses Both are behind LEARNER_PATHWAYS_SERVER_PIPELINE_ENABLED (default False, so they 404) and an RBAC role. No MFE calls them. Why this shape, and what the measurements changed: - Step records already persist per-execution input, output, timing and failure, so the evaluation harness reads them instead of running a parallel implementation. That means we measure the real pipeline. - Both Algolia indexes AND every query word with no fallback configured, so an 8-word query returns zero hits rather than poor ones. Retrieval sends removeWordsIfNoResults=allOptional and persists the query and hit count, because relaxation buys volume rather than relevance. - The catalog's skill vocabulary is Lightcast-canonical: "Python" is not a facet value, "Python (Programming Language)" is. Skill terms are resolved against the catalog's own vocabulary rather than matched exactly. The facet snapshot is truncated at Algolia's 1,000-value cap, so a conditional facet-search pass recovers the long tail. - Pathway structure is deterministic, not delegated to a model: five courses under a level quota and a per-provider cap. This addresses the reported "two 101 courses but zero 102 courses" defect, which measured as a selection problem rather than a content gap. - Only topical relevance goes to a model, through an adapter selected by configuration so model comparison is a query over step records. - Quality has a written bar in three tiers: correctness gates that are bugs rather than judgements, a ship bar as per-persona pass/fail, and tracked-not-gating metrics. Adds ADR 0037 with a mermaid diagram of the composition, and docs/references/algolia_search.md recording the measured index behaviour the decisions rest on. The measured bar is NOT met: 23% recall@20 and 0% on technology. This lands the shape and the instrumentation, not a solved quality problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (94.95%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #281 +/- ##
==========================================
+ Coverage 87.72% 88.75% +1.02%
==========================================
Files 157 182 +25
Lines 13489 15725 +2236
Branches 1312 1574 +262
==========================================
+ Hits 11833 13956 +2123
- Misses 1351 1432 +81
- Partials 305 337 +32 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Seed migration -------------- Without a candidate_rerank prompt row, XpertBackend raises, re-ranking degrades to retrieval order, and the pipeline runs *without* its model step while still returning a valid pathway. That failure is silent by design -- losing the ordering beats losing the recommendation -- so a run in that state reads as "the model does not help" when it was never called. prompts/migrations/0003 seeds it so a fresh environment works. The migration is idempotent (get_or_create on prompt_type), never clobbers an admin edit, and reverses only if the row is still the seeded text. Its text is inlined rather than imported: migrations are frozen history, so importing a constant that will legitimately change would make the migration mean something different later. The prompt asks for topical relevance ONLY, and says so explicitly. Level spread, provider spread and de-duplication are guaranteed deterministically by pathway_assembly; asking a model to reproduce arithmetic invites a disagreement someone then has to adjudicate. It also asks for every key to be ranked rather than dropping poor fits, because assembly fills each rung from the front of the window -- so last place is how the model says "poor fit", and dropping loses that signal. OpenAI backend -------------- A third backend alongside xpert and claude, so the harness has two metered backends to compare and "is there a model-class difference?" becomes a query over persisted traces rather than an assertion. Lazily imported like anthropic, so a Xpert-only deployment never loads it. It requests response_format=json_object: every prompt here requires JSON and OpenAI can enforce that server-side, removing a failure mode rather than handling it. Verified live against gpt-4o with a candidate set containing two deliberately off-topic courses. Both ranked last, all six keys ranked exactly once, no fabricated keys, and the rationales named the mismatch plainly. 780/231 tokens, 5.3s -- the first real latency data point for the budget ADR 0037 flags as outstanding. Two pre-existing prompt tests asserted a whole-table row count and now see the seeded row; they assert per prompt_type instead, which is what the constraint under test actually is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by the first live end-to-end run of PathwayAssemblyWorkflow, executed on the host against the pinned 2U catalog and gpt-4o. The re-rank prompt asks for JSON but never names ``ordered_keys`` -- that field name exists only in the output schema, which ``build_system_prompt`` appends from the prompt's database row. The direct backends (claude, openai) have no row, so they were sending the prompt text alone. gpt-4o duly returned a valid, well-reasoned ranking under field names of its own choosing, ``parse_rerank_response`` recognised none of them, and the run degraded to retrieval order. That degradation is silent by design -- losing the ordering beats losing the pathway -- so the only evidence was one log line reading "no ordered_keys list", next to a complete, apparently healthy 2/2/1 pathway. Nothing raised and no test failed: every existing test supplies the response shape it expects, so none of them could observe a prompt that failed to ask for that shape. Extracts the composition rule as ``compose_system_prompt`` and uses it on both paths, so a prompt validated on one backend is the prompt the other sends. Verified live after the fix: 19 of 19 candidate keys ranked, zero fabricated, and the ordering now changes which courses are selected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the LEARNER_PATHWAYS_SERVER_PIPELINE_ENABLED Django setting with a waffle switch, so the pipeline can be turned on and off in Django admin without a deploy. Switches rather than flags, deliberately. A waffle flag is request-scoped: enableable per-user, by percentage, and -- with WAFFLE_OVERRIDE -- by a ?flag_name=1 query string. A switch is one global boolean an administrator sets, and waffle never honours a query-string override for one. So nobody can enable an unreleased pipeline that spends money per call by crafting a URL. It also reads with no request at all, which a flag cannot express off-request, so the harness and management commands honour the same toggle as the endpoints. test_toggles.py asserts the type to stop a silent downgrade. Adds a second switch as a re-rank kill switch, with inverted polarity: default off means re-ranking runs. An enable-style switch defaulting to off would mean that turning the pipeline on produced pathways with no model input at all -- retrieval order, no error, a well-formed five-course response. That is the silent degradation this pipeline was bitten by in b266cc5, and it must not be reachable by forgetting a second switch. Turning it on is a supported degradation for cost, latency or a failing provider, and it overrides the workflow's own per-run input so it stops harness spend too. A switch that has never been created reads as off, so the previous safe default holds in every environment with nothing configured. Verified: both switches toggled via ./manage.py waffle_switch against a real database and read back correctly through the helpers; a live end-to-end assembly run still re-ranks 19 of 19 candidates with the kill switch at its default. 2731 tests pass; isort, pycodestyle, pylint and pii_check clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Add a description of your changes here.
Jira:
ENT-XXXX
Merge checklist:
./manage.py makemigrationshas been runPost merge: