diff --git a/docs/architecture-patterns.md b/docs/architecture-patterns.md index 7cb65649..3b981905 100644 --- a/docs/architecture-patterns.md +++ b/docs/architecture-patterns.md @@ -90,6 +90,30 @@ Uses `edx-rbac` for fine-grained permissions with: ### 15. Use ddt to parameterize unit tests - Improve test DRYness by using the `ddt` packages `@data` and `@unpack` decorators. +### 16. Evaluation harnesses own no domain logic +- An eval app (`apps/pathway_eval`) loads fixtures, calls production code, scores results, reports +- Pipeline logic — retrieval, translation, ranking — stays production code elsewhere; if the + harness reimplements any of it, the harness is what gets measured +- Fixtures validate against the **real** request serializer, not a copy of its rules, so a + contract change fails at load time rather than mid-run +- Ground-truth fixtures declare their own provenance in a machine-readable field + (`ground_truth_status: placeholder | expert_authored`). Authoring ground truth outlasts + building the harness, so "is this a real expectation?" must be queryable — a comment in a + fixture cannot keep a placeholder out of a headline metric +- Incomplete ground truth **loads** and reports itself as unscoreable. It is neither a + validation error (it must be visible and chaseable) nor a scored failure +- Diagnostics name their epistemic limits in the result. Where a credential can only probe + rather than enumerate, the outcome is `NOT_FOUND_IN_INDEX`, not "absent" — an over-claimed + negative drives the most expensive decisions + +### 17. Search clients bind a credential to an index, not to a service +- Where two indexes need different credentials, make the wrong pairing unconstructable rather + than validated: separate methods per index, and no parameter for the credential that must + not be used there +- Guard against a *misconfigured* credential too, not just a miscalled one +- A degraded mode (unscoped search) requires both a settings flag and an explicit call-site + argument, and is never reached by fallback — missing or expired scoping raises + ### Key Takeaways for Implementation: - Check permissions early using `@permission_required` decorator - Use separate serializers for request/response diff --git a/docs/conf.py b/docs/conf.py index 2a0a3d61..06ea39b2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -59,6 +59,7 @@ def get_version(*file_paths): # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ + 'sphinxcontrib.mermaid', 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', diff --git a/docs/decisions/0037-server-side-learner-pathway-pipeline.rst b/docs/decisions/0037-server-side-learner-pathway-pipeline.rst new file mode 100644 index 00000000..87c48e34 --- /dev/null +++ b/docs/decisions/0037-server-side-learner-pathway-pipeline.rst @@ -0,0 +1,267 @@ +0037 Moving the learner pathway pipeline server-side +***************************************************** + +Status +====== +**In progress** (September 2026) + +Context +======= +Learner pathway generation — turning a learner's stated goals into an ordered set of +recommended courses — currently runs in the ``frontend-app-learner-portal-enterprise`` +MFE. The MFE calls Xpert to derive intent, searches the Lightcast jobs index for careers, +searches the enterprise catalog index for courses, and assembles a recommendation, all +client-side. + +That placement has four consequences we want to remove: + +* **No trace.** A bad recommendation cannot be diagnosed after the fact. There is no + record of what was asked of Xpert, what the index returned, or which stage went wrong. +* **No evaluation.** Quality cannot be measured, so it cannot be improved deliberately. + The pipeline's output has never been scored against expert judgement. +* **Prompts outside versioning.** One Xpert call (the skill-translation refinement) has no + server-side home at all, so it sits outside prompt versioning and rate limiting. +* **Retrieval papered over with a ladder.** The MFE tries four progressively broader + queries and stops at the first that returns anything. The ladder exists because the + first query is too narrow, and it hides *why* a result set was thin. + +Before building, we measured the pipeline against the production indexes. Five findings +shaped every decision below, and each is recorded with its evidence in +``docs/references/algolia_search.md``: + +1. **Both Algolia indexes AND every query word**, with no ``removeWordsIfNoResults`` + configured. An eight-word query returns *zero* hits rather than poor ones. On the jobs + index a single common word ("Become") is enough to return nothing. +2. **Retrieval, not ranking, is the binding constraint.** Best measured recall@20 against + expert-authored ground truth was 23% overall and **0% for every technology persona**. +3. **The catalog's skill vocabulary is Lightcast-canonical.** ``Python`` does not exist as + a facet value; ``Python (Programming Language)`` does. Exact-match grounding therefore + drops the most in-demand technical skills silently. +4. **31% of courses carry no skill tags at all** (1,272 of 4,094), making them + structurally unreachable by any skill-facet query. +5. **Relevance ranking is heavily introductory at rank 5 and recovers by rank 20.** A + ``data analyst`` query returns five introductory courses in its top five and 16/3/1 + across introductory/intermediate/advanced in its top twenty. + +Decision +======== +We decided to move the pipeline into ``enterprise-access`` and build it on the existing +``enterprise_access.apps.workflow`` pattern (ADR 0025) rather than a new framework, as two +workflows behind two endpoints. + +**Why the existing workflow pattern.** Step records already persist ``input_data``, +``output_data``, ``succeeded_at``, ``failed_at`` and ``exception_message`` per execution. +That is exactly the trace the evaluation harness needed, so the harness reads step records +instead of running a parallel implementation — which means we measure the real pipeline +rather than something that resembles it. This was the single strongest argument, and it +made the harness substantially smaller than planned. + +Two workflows, because the learner's career selection splits the flow: + +.. mermaid:: + + flowchart TB + subgraph CD["CareerDiscoveryWorkflow — POST /learner-pathways/careers/"] + direction TB + EI["ExtractIntentStep
Xpert: learner_intent prompt"] + RC["RetrieveCareersStep
Lightcast jobs index"] + EI --> RC + end + + LEARNER(["Learner chooses a career"]) + + subgraph PA["PathwayAssemblyWorkflow — POST /learner-pathways/pathway/"] + direction TB + SF["SnapshotCatalogFacetsStep
one zero-hit search"] + TC["TranslateToCatalogStep
skill vocabulary resolution"] + RCa["RetrieveCandidatesStep
~20 candidates, one broad query"] + RR["RerankCandidatesStep
topical relevance only"] + AP["AssemblePathwayStep
5 courses, deterministic"] + ER["EnrichRationaleStep
Xpert: recommendations_feedback"] + SF --> TC --> RCa --> RR --> AP --> ER + end + + CD --> LEARNER --> PA + + classDef conditional stroke-dasharray: 5 5 + class TC,RR,ER conditional + +Steps drawn with a dashed border are **conditional** — they decide at run time that they +have nothing to do. That capability does not exist in ``apps/workflow``, so we added +``AbstractConditionalWorkflow`` locally in the pathways app (see *Divergences* below). + +Six decisions inside that shape are worth recording, because each was driven by a +measurement rather than a preference. + +**1. One broad query, then curate — replacing the retrieval ladder.** +``RetrieveCandidatesStep`` issues a single search with ``removeWordsIfNoResults: +allOptional`` and retrieves **20** candidates, not 5. The ladder existed to compensate for +a too-narrow query; widening once and selecting afterwards addresses the cause. Because +relaxation buys *volume* rather than relevance, the step persists the query and hit count +so a report can tell a good retrieval from a padded one without re-running anything. + +**2. Skill terms are resolved against the catalog's own vocabulary, not matched exactly.** +``TranslateToCatalogStep`` reads the scoped catalog's facet values and resolves each term +onto them, recovering ``Python`` → ``Python (Programming Language)`` without a +hand-maintained alias map or a model call. High-confidence matches become hard filters; +weaker containment matches become boosts, because a wrong hard filter returns nothing. + +The facet snapshot is capped by Algolia at 1,000 values and the live skill facets return +exactly 1,000 — i.e. they are truncated. A snapshot-only design therefore loses the long +tail silently, which is disproportionately the non-technology vocabulary. So a +**conditional** second pass over Algolia's facet-search endpoint recovers what the +snapshot could not serve, at one request per unresolved term. Facet-search results are +re-validated under the same rules, which is what stops ``AWS`` resolving to "AWS Certified +Solutions Architect Associate" merely because that is the highest-count candidate. + +**3. Pathway structure is deterministic; only topical relevance goes to a model.** +A pathway is five courses, ordered roughly by difficulty, with no duplicates. Product +reported the defect that motivated this: *"you'll get 2 intro courses from different +providers, so 2 101 courses but zero 102 courses."* + +That is a **selection** defect, not a content gap — the intermediate courses exist, they +sit below the rank-5 cut. ``AssemblePathwayStep`` therefore selects five from the twenty +under a level quota (2 introductory / 2 intermediate / 1 advanced) and a cap of two +courses per provider. It is pure computation, so it is directly testable, and none of it +is delegated to a model: asking a model to reproduce arithmetic invites a disagreement +someone then has to adjudicate. + +Two non-obvious properties came out of live measurement: + +* **The scarcest rung claims provider capacity first.** A single relevance-ordered pass + lets the most plentiful rung spend the scarce resource: ``Data Analyst`` returned 17 + candidates spanning all three rungs and still assembled to 5/0/0, because the two + introductory picks used one provider's entire allowance and every intermediate candidate + belonged to that provider. Filling rungs in ascending order of availability fixed it. +* **A strict skill filter can make the level mix worse**, by narrowing the window before + assembly can span it. When the strict set is thin or sits on a single rung, a second + unfiltered search runs and its hits are **appended, not substituted** — the precise + courses keep their rank and assembly gets the width it needs. + +**4. Rationale generation is its own step, reusing the live prompt.** +``EnrichRationaleStep`` reuses the existing ``recommendations_feedback`` prompt read-only, +exactly as ``ExtractIntentStep`` reuses ``learner_intent``. It runs on the delivered five +rather than the candidate twenty. We initially folded rationales into the re-rank response +and reversed that: it would have taken learner-facing wording from a prompt chosen for +*ordering*, and paid to explain four courses for every one that ships. + +**5. Model access goes through an adapter, selected by configuration.** +``apps/pathways/model_backends/`` presents one interface over Xpert and a direct +reasoning-model call, both returning content, token counts and elapsed milliseconds. Model +comparison is therefore a query over persisted step records rather than a bespoke rig. +Callers ask for a backend by configuration, never by import, so switching for a comparison +run needs no deploy. Unreported token counts are ``None`` rather than ``0``: zero is a +measurement, and a cost report that silently treats unknown as free is worse than one that +says it does not know. + +**6. Quality has a written bar, in three tiers, committed before the next run.** +A single aggregate recall number is the wrong shape for a ship decision here, for reasons +that are properties of this evaluation rather than opinions: eight scoreable personas +quantise any aggregate at 12.5 percentage points; averaging hides a measured 0%-vs-40% +technology gap that an aggregate bar of 30% could be met *around*; and recall measures +agreement with the ground-truth author, not learner value. + +So: **Tier 1** gates correctness (exactly five courses or none, valid keys, no duplicates, +in the pinned catalog, English, no more than two per provider) — these are bugs, not +quality judgements, and a run that fails them is not scored at all. **Tier 2** is the ship +bar, as per-persona pass/fail plus a passing count and a no-split-scores-zero rule. +**Tier 3** is tracked and never gating. Level mix is deliberately Tier 3: a rung can be +genuinely empty in the catalog, and ``level_type`` disagrees with course titles in 19–36% +of cases, so a gate on it would measure the metadata's noise. + +Divergences from ADR 0025's pattern +=================================== +**Conditional execution, added locally.** ``AbstractWorkflow.process_input`` iterates +``self.steps`` unconditionally. Three steps in this pipeline need to opt out at run time, +so ``AbstractConditionalWorkflow`` in the pathways app adds a ``should_execute`` +classmethod defaulting to ``True``. It lives here rather than in ``apps/workflow`` because +provisioning is live and it must be impossible for this feature to alter its behaviour. If +it earns its place, upstreaming it is a later conversation with that code's owner. + +This turned out to require more than the anticipated few lines. The parent builds its +generated input/output classes with ``field(type=step_class.output_class, default=None)`` +— the default is ``None`` but the declared type is not ``Optional``, so ``cattrs`` emits +unconditional dereferences. That is safe while every step runs and fatal once one can +skip, so ``input_class`` and ``output_class`` are overridden to declare +``Optional[...]``. A test pins that a skipped step's output round-trips as null. + +**The retrieval ladder was not given framework support.** "Try N strategies in order, stop +at the first satisfying a predicate" would turn the step list into a step tree, changing +the generated IO classes, the ``preceding_step_uuid`` linkage and the accumulated-output +threading. The ladder only existed because retrieval was too narrow; decision 1 above +removes the need for it, so building for it would have meant optimising something the fix +deletes. + +**Latency is unchanged and unbudgeted.** Steps run inline and synchronously, so a +learner-facing pathway request costs the same wall clock as today's client-side flow. +ADR 0025 notes async-via-Celery is envisioned but unbuilt. The endpoints are behind an +admin-toggled switch and unwired from any frontend, so this is not yet a user-facing +concern — but it needs a budget before it becomes one. One measured data point: a live +five-step assembly run with enrichment disabled took 9.7–15.2 seconds, dominated by the +single model call. + +Consequences +============ +* Every pathway generation leaves an inspectable per-step trace, queryable in Django admin + and by the evaluation harness, without a separate tracing layer. +* Re-running a failed workflow skips already-succeeded steps, so a failed enrichment does + not re-run intent extraction. During evaluation, where runs are counted in hundreds, + that matters more than in production. +* Quality is measurable. ``run_pathway_harness`` produces traces and + ``report_pathway_harness`` scores them into the three tiers, so a re-score never needs a + re-run — which matters because a run costs money. +* The endpoints are gated by the ``enterprise_access.learner_pathways_server_pipeline`` + waffle **switch** (off by default, so they 404) and by an RBAC role. No MFE calls them; + rollback is flipping the switch, not reverting behaviour. +* **The gates are switches, not flags, and that is a decision rather than a detail.** A + waffle flag is request-scoped: enableable per-user, by percentage, and -- with + ``WAFFLE_OVERRIDE`` -- by a ``?flag_name=1`` query string. A switch is one global + boolean an administrator sets in Django admin and can never be overridden from a + request. Two properties follow. Nobody can enable an unreleased pipeline that spends + money per call by crafting a URL. And because a switch needs no request, the harness and + the management commands honour the same toggle as the endpoints, which a flag could not + express off-request. ``enterprise_access/tests/test_toggles.py`` asserts the type, so + this cannot be silently downgraded. +* **Re-ranking has its own kill switch, with inverted polarity.** + ``enterprise_access.learner_pathways_disable_candidate_rerank`` defaults to off, meaning + re-ranking runs. An enable-style switch defaulting to off would mean enabling the + pipeline yielded pathways with no model input at all -- retrieval order, no error, a + well-formed five-course response. That silent degradation has already occurred once in + this pipeline (see the ``ordered_keys`` note under Divergences), so it must not be + reachable by forgetting a second switch. Turning the kill switch on is a supported + degradation for cost, latency or provider failure: deterministic assembly still produces + a valid pathway, and it overrides the workflow's own per-run input so it stops harness + spend too. +* **The measured bar is not met.** 23% recall@20 and 0% on technology is far below + anything worth putting in front of a learner. This ADR records the shape and the + instrumentation; it does not claim the quality problem is solved. Two Tier 2 numbers + remain unsigned by product, and the pipeline has not yet been run end to end against + live personas. + +Alternatives Considered +======================= +* **A new pipeline framework.** Rejected: ``apps/workflow`` already provides persisted, + composable, resumable multi-step execution with two ADRs behind it and production usage + in provisioning. Composition work here is writing steps, not writing a framework. +* **Keeping the pipeline in the MFE and adding client-side telemetry.** Rejected: it does + not address prompt versioning, cannot make retrieval measurable against ground truth, + and leaves the one unhosted Xpert call unhosted. +* **Asking the model to produce the whole pathway, structure included.** Rejected on the + measurement in decision 3: the structural constraints are arithmetic over retrieved + candidates and are cheaper, testable and more reliable as deterministic code. It also + keeps the model's contribution isolated enough to be measured as a delta — a run with + re-ranking disabled is a meaningful baseline, not a broken run. +* **Configurable composition** (``steps`` backed by an admin row, so reordering and A/B-ing + is configuration rather than a deploy). Deferred, not rejected. It is the right shape for + the model-comparison experiments, but ``input_class`` and ``output_class`` are cached + properties derived from ``self.steps``, so stored ``input_data`` is only meaningful + against the composition that produced it. That needs versioned composition rows and a + stamp on each run, and it is worth doing once the pipeline shape settles. + +References +========== +* ADR 0025 — abstract workflow pattern +* ADR 0028 — why attrs for workflow IO +* ``docs/references/algolia_search.md`` — the measured index behaviour behind the + decisions above +* ``docs/references/career_discovery_workflow.md`` — the career-side query shape diff --git a/docs/decisions/README.rst b/docs/decisions/README.rst index cd113924..c790cf53 100644 --- a/docs/decisions/README.rst +++ b/docs/decisions/README.rst @@ -266,3 +266,15 @@ and related business logic. Accepted Sept 2025, this ADR describes the rational for identifying SSP Stripe products based on ``lookup_key`` instead of the Stripe price id. + +`<0037-server-side-learner-pathway-pipeline.rst>`_ +*************************************************** +*Feature: Learner Pathways* + +In progress September 2026, this ADR describes moving learner pathway +generation out of the learner-portal MFE into this service, built on the +abstract workflow pattern as two workflows behind two endpoints. It records +the measured index behaviour the design rests on -- both Algolia indexes AND +every query word, the Lightcast-canonical skill vocabulary, and 31% of +courses carrying no skill tags -- and is explicit that the measured quality +bar is not yet met. diff --git a/docs/references/algolia_search.md b/docs/references/algolia_search.md new file mode 100644 index 00000000..739e982d --- /dev/null +++ b/docs/references/algolia_search.md @@ -0,0 +1,210 @@ +# Algolia search from enterprise-access + +How to query the catalog and jobs indexes, and the index behaviours that will bite you. +Measured against the production indexes on 2026-09-09. + +Client: `enterprise_access/apps/api_client/algolia_client.py`. + +## Two indexes, two credentials, not interchangeable + +| Index | Credential | Why | +| --- | --- | --- | +| Catalog (`enterprise_catalog_incremental_prod`) | Enterprise-scoped **secured** key | Scoping is the point — it keeps results inside the learner's catalog | +| Jobs / Lightcast taxonomy (`prod_taxonomy`) | Plain search key | Secured keys can't read it | + +A secured key sets `restrictIndices` to the catalog index and its replicas +(enterprise-catalog's `generate_secured_api_key`), so sending one to the jobs index fails +with an opaque Algolia error. The learner portal MFE encodes the same constraint as +`unsupportedSecuredAlgoliaIndices = [ALGOLIA_INDEX_NAME_JOBS]`. `search_jobs_index()` +refuses before issuing the request, including when the *configured* search key turns out +to be a secured key (they base64-decode to a querystring containing `restrictIndices`). + +**The write key must never be configured here.** enterprise-catalog's `AlgoliaSearchClient` +is an indexing/administration client built on `ALGOLIA.API_KEY` and has no `search()`. +This client is search-only by construction. + +### Secured keys are vended per user, which constrains where you can use them + +`get_secured_algolia_api_key()` lives only on `EnterpriseCatalogUserV1ApiClient`, a +`BaseUserApiClient` — it forwards user context, and the generated key carries a +`userToken`. So a secured key is reachable from a request-backed code path and **not** +from a management command or a Celery task. Offline tooling has to either accept unscoped +results (`ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH`, off by default) or go through +enterprise-catalog. + +Note also that `bffs.api.get_and_cache_secured_algolia_search_keys` caches for a fixed +`SECURED_ALGOLIA_API_KEY_CACHE_TIMEOUT` and does **not** parse `valid_until`, despite its +docstring saying so. `SecuredAlgoliaKey.is_expired()` checks it properly; treat a key with +no `valid_until` as expired rather than assuming validity. + +## A search-only key cannot enumerate an index + +* No `browse` ACL — `/browse` returns 403. +* Pagination is capped: with `hitsPerPage: 1000` the catalog index reports `nbPages: 1` + against `nbHits: 4094`. Page 1 and beyond return nothing. + +So you cannot build the set of course keys in an index with a search key. Establishing +that a course is *absent* requires a browse-scoped key or enterprise-catalog's +`contains_content_items`. Anything else is a probe, not a proof. + +`key` is also neither filterable (`filters: 'key:"IBM+DA0101EN"'` → 0 hits for a course +that exists) nor searchable (`restrictSearchableAttributes: ['key']` → HTTP 400). + +## Scoping to an enterprise customer needs no secured key + +`enterprise_customer_uuids` is in the catalog index's `attributesForFaceting` +(`enterprise-catalog`'s `apps/catalog/algolia_utils.py`), so this works with the plain +search key: + +``` +filters: 'content_type:course AND enterprise_customer_uuids:' +``` + +It is *also* in `unretrievableAttributes`, which means it never appears on a hit — but +**that does not block faceting on it.** `facets: ['enterprise_customer_uuids']` returns +values and counts, so customer UUIDs are enumerable (capped at 1,000 like any facet) and a +candidate UUID can be verified by its hit count. + +Two consequences. Evaluation and diagnostic work can be scoped to a real customer without +the secured-key machinery, which needs a request and a user token and so cannot run from a +management command. And `unretrievableAttributes` should not be read as "private" — it +hides the value from a *hit*, not from an aggregate. + +Measured 2026-09-10: the broadest customers see 4,057 of 4,094 courses (99.1%), so +per-skill counts scoped to one differ from unscoped by about a single course. Academy +customers are the exception at 13–16 courses. + +## Course keys are `+`, not run keys + +The catalog index's `key` field holds `HarvardX+ER22.1x`, `IBM+DA0101EN`, +`CodeSignal+164`. A `course-v1:...` **course-run** key appears nowhere in the index. This +is a silent failure mode: run keys look like course identifiers, so code or ground-truth +data that uses them matches nothing and reads as a relevance problem. + +`aggregation_key` is `course:`; `objectID` is `course--customer-uuids-` and +is not derivable from the course key. + +## Query semantics: every word is ANDed, and there is no fallback configured + +This is the single most surprising behaviour. Measured on one persona's goal text, +filtered to `content_type:course`: + +| Query words | Hits | +| --- | --- | +| 1 (`Move`) | 533 | +| 4 (`Move into a data`) | 90 | +| 5 | 4 | +| 6 | 1 | +| 8 or more | **0** | + +`removeWordsIfNoResults` is not configured on the index, so a verbose query returns +**zero hits, not poor hits**. A five-word career title (`Medical Surgical Registered +Nurse Manager`) returns 0. Passing `removeWordsIfNoResults: 'allOptional'` turns that +24-word query into 348 hits and the career title into 121. (`'lastWords'` does not +rescue a long query.) + +Two consequences: + +1. Any natural-language query — a learner's free text, or a verbose model-generated + `condensed_algolia_query` — silently returns nothing. This is the mechanism behind a + retrieval ladder always descending to its widest step. +2. **Relaxing the query is worth a lot, but it is not sufficient.** Against + product-authored ground truth, `removeWordsIfNoResults: 'allOptional'` moved + recall@20 from **12% to 23%** overall and **21% to 40%** for non-technology personas. + One search parameter is the cheapest available improvement. + + It is still not evidence of success on its own: four personas went from 0 hits to 20 + hits with recall unchanged at 0%. **Measure expected-key recall, never hit count** — + a full result set of the wrong courses is the same pathology as a scope-only fallback + wearing a different hat. + +Keep text queries to a few words. In the diagnostic, the *shortest* strategy (a bare +career title) was the only one that retrieved anything at all without `allOptional`. + +## The skill facet vocabulary is Lightcast-canonical, and short names are absent + +`skill_names` holds disambiguated Lightcast forms. The short name a learner or a model +would produce is usually **not a facet value at all**: + +| What you'd write | Hits | What the index actually holds | Hits | +| --- | --- | --- | --- | +| `Python` | 0 | `Python (Programming Language)` | 95 | +| `SQL` | 0 | `SQL (Programming Language)` | 42 | +| `Java` | 0 | `Java (Programming Language)` | 27 | +| `Excel` | 0 | `Microsoft Excel` | 26 | + +Verified by direct `facetFilters` counts, not by reading the facet list — the facet +vocabulary response is capped at `maxValuesPerFacet` (1000), so "missing from the list" +is not evidence of absence. + +Some names *are* canonical as-is (`Data Analysis`, `Machine Learning`, `Project +Management`, `Nursing`, `Leadership`). So the mismatch is systematic but not uniform, and +it has a predictable shape: `X` → `X (Programming Language)` / `X (Python Package)` / a +vendor-qualified form. + +Exact-match grounding against a facet snapshot therefore drops the most in-demand +technical skills silently. That is a vocabulary-normalisation problem with a mechanical +fix, not an LLM-paraphrasing problem. + +## Facet counts are inflated ~75-80x; `nbHits` under `facetFilters` is exact + +The catalog index de-duplicates at query time (`distinct` on `aggregation_key`) but +**facet counts are computed before de-duplication**. The `subjects` facet reports 79,392 +for "Business & Management" against a real 1,015 courses. Any code or dashboard that +displays a raw facet count is displaying a wrong number, roughly 75-80x too high. + +Use `nbHits` under `facetFilters` with `hitsPerPage: 0` instead — that is exact +(verified against distinct keys returned on slices of 8, 35 and 1,015). + +The same caution applies to the facet-*search* endpoint's `count` field, which is +additionally unfiltered by `content_type`. Treat those as candidate vocabulary only. + +## 31% of courses carry no skills at all + +A full census of all 4,094 courses found **1,272 (31.1%) with both `skill_names` and +`skills` empty**; the median tagged course carries 5 skills. + +Do not sample by relevance rank to measure this. An earlier pass using the top 1,000 +hits reported 13.1%, less than half the true rate, because relevance rank is +popularity-biased toward well-tagged content. Enumerate by slicing on a facet instead. + +Concentration matters more than the average: **130 of 141 "Artificial Intelligence" +subject courses (92%) are untagged**, as are 115 of 122 Google Cloud courses and all 62 +CodeSignal courses. Those courses cannot be retrieved by any skill-facet query, whatever +the vocabulary handling. + +## Two thirds of jobs have no skills, deterministically + +In a full census of 43,513 English-language jobs, **29,525 (67.9%) have an empty +`skills` array**. The rule is exact, with no exceptions observed: a job carries skills +**iff** it carries `job_sources: course_skill`. All 13,988 such jobs have skills; all +29,525 industry-only jobs have none. + +So "the career resolved to a Lightcast entry" and "that entry has usable skills" are two +separate gates, and for two thirds of careers skill-based course retrieval has nothing to +work with. Check `job_sources` before relying on `skills.name`. + +`prod_taxonomy` also supports numeric filters on `id` (`filters: 'id >= X AND id <= Y'`), +which makes full enumeration by bisection feasible there despite the pagination cap. + +## Settings + +```python +ALGOLIA_APP_ID = '' +ALGOLIA_SEARCH_API_KEY = '' # plain, search ACL only +ALGOLIA_CATALOG_INDEX_NAME = '' +ALGOLIA_JOBS_INDEX_NAME = '' +ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH = False # diagnostics only +``` + +Index names and the plain search key are configured per environment in +`edx-internal/frontends/frontend-app-learner-portal-enterprise/*_config.yml`; the search +key is search-ACL-only and already ships in the MFE bundle. + +## Local development + +The devstack `enterprise-access` container has **no network egress** — DNS fails and even +a raw-IP connection times out — so live Algolia calls cannot be made from inside it, and +`algoliasearch` cannot be `pip install`ed there without staging the wheel via +`docker cp`. Unit tests mock the client and are unaffected; anything that needs a real +Algolia response has to run outside the container. diff --git a/docs/references/career_discovery_workflow.md b/docs/references/career_discovery_workflow.md new file mode 100644 index 00000000..67d8eba2 --- /dev/null +++ b/docs/references/career_discovery_workflow.md @@ -0,0 +1,84 @@ +# Career discovery workflow + +`POST /api/v1/learner-pathways/careers/` — learner intake in, career candidates out, as a +persisted two-step workflow. Code: `enterprise_access/apps/pathways/` (steps, workflow, +domain API) and `enterprise_access/apps/api/v1/views/pathways.py` (endpoint). + +``` +intake (4 fields) -> ExtractIntentStep (Xpert, learner_intent prompt) + -> RetrieveCareersStep (Algolia jobs index) + -> [{external_id, name, skills, industries}] +``` + +Gated on `LEARNER_PATHWAYS_SERVER_PIPELINE_ENABLED` (default `False` → 404) and on the +existing `LEARNER_PATHWAYS_LEARNER_ROLE` via a new +`LEARNER_PATHWAYS_CAREER_DISCOVERY_PERMISSION`. + +## Why a workflow rather than a view function + +The step records *are* the trace. Each carries its own input, output, timing and failure, +so diagnosing a bad run is a query rather than a reproduction, and re-executing a workflow +skips the steps that already succeeded. That is also why nothing needed a bespoke tracing +layer for the evaluation harness — the response returns `workflow_uuid` and the harness +reads the records. + +`CareerDiscoveryWorkflow` subclasses `AbstractConditionalWorkflow`, not +`AbstractWorkflow`. Neither of its steps defines `should_execute` today, so execution is +identical; the base is there because the pathway workflows that extend this pipeline do +have steps that opt out, and because its `Optional`-typed dynamic IO classes are what let +a skipped step round-trip as `null`. + +## Skills are boosts, industries are hard filters + +Ported from the MFE's `careerRetrieval.ts`, and the asymmetry is the load-bearing part: + +| Signal | Algolia parameter | Why | +| --- | --- | --- | +| Required skills (max 4) | `optionalFilters`, unscored | An unmatched *hard* skill filter returns zero hits and says nothing about why | +| Preferred skills (max 2) | `optionalFilters`, `` | Weaker signal, so a weaker boost | +| Industries, job sources | `filters` | Caller is expected to have grounded these against the index already | + +Compound artifacts (`"SQL & Python"`, `"Excel + Tableau"`) are dropped before filtering — +they match nothing and spend a filter slot. + +**The intake's `interested_industries` is deliberately *not* piped into the hard filter.** +It is learner free text ("healthcare, technology"), and a hard filter on a value that is +not a facet value returns zero hits silently. Free text belongs in the text query, where +partial matching applies. `RetrieveCareersInput.industries` exists for a caller that has +grounded real facet values first, and the endpoint leaves it empty. + +## Careers are identified by `external_id`, and carry no match percentage + +`external_id` is the Lightcast job id (`ET` + 16 hex, e.g. `ETE78CD2CDFFFAC66B`). Taxonomy +names are neither unique nor stable, so a name cannot be a key — and the evaluation +personas record expected careers as `external_id`s for the same reason. A hit missing +either its `external_id` or its name is dropped rather than given a placeholder: a +fabricated id would corrupt the harness's ground-truth comparison. + +There is no match-percentage field anywhere in the pipeline or the response. The +client-side POC hardcoded `0.95` on every card; the MFE removed it deliberately, because +no verified compatible domain value exists. + +## What the step output records, and why + +`RetrieveCareersOutput` persists `query` and `hit_count` alongside the careers. A full +result set is not evidence that retrieval worked — relaxing a query buys volume, not +relevance — so a report needs both numbers to tell a real retrieval from a padded one +without re-running the search. + +## Gotchas found while building + +* **A per-action `throttle_scope` needs a class-level sentinel.** DRF's `as_view()` + rejects any `@action` initkwarg that is not also an attribute on the viewset class, so + `throttle_scope: str | None = None` on the class is load-bearing, not decoration. Without + it the router raises `TypeError` at import time and every URL in the service fails to + resolve. +* **Two viewsets can share a router prefix.** `learner-pathways` is registered twice, with + different basenames, so the careers endpoint sits beside the prompt endpoints without + touching them. Neither viewset has a `list` route, so nothing collides. +* **The Xpert conversation ID is keyed on the step record, not the request ID.** A step can + be re-executed outside the request that created it, and the step UUID is the one + identifier that ties an Xpert conversation back to a persisted trace either way. +* **Step tables persist learner-authored free text** (the intake), with no user + identifier. They are annotated `.. no_pii:` on that basis; if a user linkage is ever + added, the retirement pipeline has to be part of that change. diff --git a/enterprise_access/apps/api/serializers/__init__.py b/enterprise_access/apps/api/serializers/__init__.py index 16249615..1f984f63 100644 --- a/enterprise_access/apps/api/serializers/__init__.py +++ b/enterprise_access/apps/api/serializers/__init__.py @@ -38,8 +38,14 @@ TransactionsListResponseSerializer ) from .learner_pathways import ( + CareerCandidateSerializer, + CareerDiscoveryRequestSerializer, + CareerDiscoveryResponseSerializer, LearningIntentRequestSerializer, LearningIntentResponseSerializer, + PathwayCourseSerializer, + PathwayRequestSerializer, + PathwayResponseSerializer, RecommendationFeedbackRequestSerializer, RecommendationFeedbackResponseSerializer ) diff --git a/enterprise_access/apps/api/serializers/learner_pathways.py b/enterprise_access/apps/api/serializers/learner_pathways.py index f898432d..f61c6992 100644 --- a/enterprise_access/apps/api/serializers/learner_pathways.py +++ b/enterprise_access/apps/api/serializers/learner_pathways.py @@ -42,3 +42,106 @@ class RecommendationFeedbackResponseSerializer(serializers.Serializer): # pylin Validates and serializes the HTTP 200 response for the recommendation-feedback endpoint. """ reasons = serializers.DictField(child=serializers.CharField()) + + +class CareerDiscoveryRequestSerializer(LearningIntentRequestSerializer): # pylint: disable=abstract-method + """ + Validates the request body for the career-discovery endpoint. + + Subclasses the learning-intent request rather than restating its four fields: the + server-side pipeline sends the same intake to the same prompt, and the evaluation + harness validates its personas against ``LearningIntentRequestSerializer`` directly. + A divergence between the two contracts would only show up as unexplained differences + between harness runs and live requests. + """ + + +class CareerCandidateSerializer(serializers.Serializer): # pylint: disable=abstract-method + """ + Serializes one career candidate. + + ``external_id`` is the Lightcast job identifier and is the career's identity for + every downstream call; ``name`` is for display only, because taxonomy names are + neither unique nor stable. + + There is deliberately no match-percentage field. The client-side POC hardcoded 0.95 + on every card and the MFE has since removed it, on the grounds that no verified + compatible domain value exists. Adding one here would reintroduce a fabricated number + into the one place a consumer would trust it. + """ + external_id = serializers.CharField() + name = serializers.CharField() + skills = serializers.ListField(child=serializers.CharField(), required=False) + industries = serializers.ListField(child=serializers.CharField(), required=False) + + +class CareerDiscoveryResponseSerializer(serializers.Serializer): # pylint: disable=abstract-method + """ + Serializes the HTTP 200 response for the career-discovery endpoint. + + ``workflow_uuid`` is returned so a caller can retrieve the full per-step trace -- + input, output, timing and failure -- for a run it has already made, without the + endpoint having to embed any of it in the response. + """ + workflow_uuid = serializers.UUIDField() + careers = CareerCandidateSerializer(many=True) + + +class PathwayRequestSerializer(serializers.Serializer): # pylint: disable=abstract-method + """ + Validates the request body for the pathway endpoint. + + Takes a career the client already chose, plus the skills that career carries. The + learner's choice sits *between* career discovery and pathway assembly, so this is a + second request rather than a continuation of the first -- and the client is the only + party that knows which card was clicked. + + ``career_skills`` is required and non-empty because two thirds of Lightcast careers + carry no skills at all, and a pathway built from no skills is a keyword search wearing + a pathway's clothes. A caller holding a skill-less career should not reach here. + """ + career_name = serializers.CharField(allow_blank=False) + career_external_id = serializers.CharField(allow_blank=False) + career_skills = serializers.ListField( + child=serializers.CharField(allow_blank=False), + allow_empty=False, + ) + skills_required = serializers.ListField( + child=serializers.CharField(allow_blank=False), required=False, default=list, + ) + skills_preferred = serializers.ListField( + child=serializers.CharField(allow_blank=False), required=False, default=list, + ) + # Passed through to the existing ``recommendations_feedback`` prompt, which is what + # generates the per-course rationale. Optional: a pathway without it is still a + # pathway, just explained more generically. + learner_profile = serializers.DictField(required=False, default=dict) + + +class PathwayCourseSerializer(serializers.Serializer): # pylint: disable=abstract-method + """ + Serializes one course in a delivered pathway. + + Order is the pathway's order -- roughly easiest first. "Roughly" is honest: the + catalog's ``level_type`` disagrees with course titles in 19-36% of cases, so the + ordering combines it with title cues and is best-effort rather than guaranteed. + """ + key = serializers.CharField() + title = serializers.CharField() + level_type = serializers.CharField(allow_blank=True) + partner = serializers.CharField(allow_blank=True) + rationale = serializers.CharField(allow_blank=True) + + +class PathwayResponseSerializer(serializers.Serializer): # pylint: disable=abstract-method + """ + Serializes the HTTP 200 response for the pathway endpoint. + + ``unfilled_rungs`` is part of the contract rather than an internal detail: a pathway + that could not reach an advanced course is materially different from one that did, + and the catalog genuinely has skills with no advanced content. Telling the client + lets it say so instead of implying a progression that is not there. + """ + workflow_uuid = serializers.UUIDField() + courses = PathwayCourseSerializer(many=True) + unfilled_rungs = serializers.ListField(child=serializers.CharField(), required=False) diff --git a/enterprise_access/apps/api/v1/tests/test_pathway_endpoint_views.py b/enterprise_access/apps/api/v1/tests/test_pathway_endpoint_views.py new file mode 100644 index 00000000..8984b63d --- /dev/null +++ b/enterprise_access/apps/api/v1/tests/test_pathway_endpoint_views.py @@ -0,0 +1,367 @@ +""" +Tests for the pathway assembly endpoint. + +HTTP-layer behaviour only: the feature flag, validation, permissions, error mapping and +response shape. Assembly and query-construction behaviour is tested in +``enterprise_access.apps.pathways.tests``. + +The distinction this file cares about most is 200-with-no-courses versus 500. "We could +not build a pathway for this career" and "something broke" lead to different client +behaviour, and the catalog genuinely contains careers with no matching courses. +""" +import uuid +from unittest import mock + +import ddt +from django.core.cache import cache as django_cache +from django.test import TestCase +from edx_rest_framework_extensions.auth.jwt.authentication import JwtAuthentication +from edx_toggles.toggles.testutils import override_waffle_switch +from rest_framework import permissions, status +from rest_framework.reverse import reverse +from rest_framework.test import APIClient +from rest_framework.throttling import ScopedRateThrottle + +from enterprise_access.apps.api.v1.views.pathways import PathwayViewSet +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.core.constants import LEARNER_PATHWAYS_LEARNER_ROLE, SYSTEM_ENTERPRISE_LEARNER_ROLE +from enterprise_access.apps.core.models import EnterpriseAccessFeatureRole, EnterpriseAccessRoleAssignment +from enterprise_access.apps.core.tests.factories import UserFactory +from enterprise_access.apps.pathways.models import AssemblePathwayStep, PathwayAssemblyWorkflow +from enterprise_access.apps.prompts.api import PromptError +from enterprise_access.toggles import LEARNER_PATHWAYS_SERVER_PIPELINE +from test_utils import APITest + +PATCH_SNAPSHOT = 'enterprise_access.apps.pathways.catalog_translation.snapshot_catalog_facets' +PATCH_RETRIEVE = 'enterprise_access.apps.pathways.course_retrieval.retrieve_candidate_courses' +PATCH_RERANK = 'enterprise_access.apps.pathways.reranking.rerank_candidates' +# Patched because the payload's skills deliberately do not all resolve against the +# snapshot, which is what makes the conditional refinement pass fire. Without this the +# tests would reach a real Algolia client. +PATCH_REFINE = 'enterprise_access.apps.pathways.catalog_translation.refine_unmatched_skills' +PATCH_ENRICH = 'enterprise_access.apps.pathways.models.pathways_api.enrich_rationales' + +_PATHWAY_URL_NAME = 'api:v1:pathway-pathway' + +_VALID_PAYLOAD = { + 'career_name': 'Welder', + 'career_external_id': 'ETE78CD2CDFFFAC66B', + 'career_skills': ['Welding', 'Blueprint Reading'], + 'skills_required': ['Welding'], + 'skills_preferred': ['Metallurgy'], +} + + +def course_hit(key, *, level='Introductory', partner='edX'): + return { + 'key': key, + 'title': f'Course {key}', + 'short_description': 'short', + 'full_description': 'long', + 'level_type': level, + 'partners': [{'name': partner}], + 'language': 'English', + } + + +def spanning_hits(): + """Six candidates able to fill a 2/2/1 quota across four providers.""" + return [ + course_hit('A+1', partner='P1'), + course_hit('A+2', partner='P1'), + course_hit('A+3', partner='P2'), + course_hit('B+1', level='Intermediate', partner='P3'), + course_hit('B+2', level='Intermediate', partner='P4'), + course_hit('C+1', level='Advanced', partner='P4'), + ] + + +def retrieval_result(courses=None, **overrides): + hits = spanning_hits() if courses is None else courses + return { + 'query': 'Welder Welding', + 'hit_count': len(hits), + 'courses': hits, + 'strict_filters_applied': ['Welding'], + 'strict_hit_count': len(hits), + 'strict_rungs_spanned': len({h['level_type'] for h in hits}), + 'broadened': False, + 'zero_hits': not hits, + **overrides, + } + + +class PathwayAPITestMixin: + """Shared set-up: patched externals and an authorized learner.""" + + def setUp(self): + super().setUp() + self.addCleanup(django_cache.clear) + self.url = reverse(_PATHWAY_URL_NAME) + + self.snapshot_patcher = mock.patch(PATCH_SNAPSHOT, return_value={ + 'skill_names': ['Welding'], 'skills.name': [], 'subjects': [], 'truncated': [], + }) + self.mock_snapshot = self.snapshot_patcher.start() + self.addCleanup(self.snapshot_patcher.stop) + + self.refine_patcher = mock.patch(PATCH_REFINE, return_value={ + 'recovered': [], 'unresolved': ['Blueprint Reading', 'Metallurgy'], 'errors': [], + }) + self.mock_refine = self.refine_patcher.start() + self.addCleanup(self.refine_patcher.stop) + + self.retrieve_patcher = mock.patch(PATCH_RETRIEVE, return_value=retrieval_result()) + self.mock_retrieve = self.retrieve_patcher.start() + self.addCleanup(self.retrieve_patcher.stop) + + self.enrich_patcher = mock.patch(PATCH_ENRICH, return_value={ + 'reasons': {}, 'prompt_revision': '', + }) + self.mock_enrich = self.enrich_patcher.start() + self.addCleanup(self.enrich_patcher.stop) + + self.rerank_patcher = mock.patch(PATCH_RERANK, return_value={ + 'ordered_keys': [], 'rationales': {}, 'fabricated_keys': [], + 'prompt_revision': '', 'trace': {}, + }) + self.mock_rerank = self.rerank_patcher.start() + self.addCleanup(self.rerank_patcher.stop) + + self.authenticate_as_enterprise_learner() + + def authenticate_as_enterprise_learner(self): + self.set_jwt_cookie([{ + 'system_wide_role': SYSTEM_ENTERPRISE_LEARNER_ROLE, + 'context': str(uuid.uuid4()), + }]) + + def post_pathway(self, payload=None): + body = _VALID_PAYLOAD if payload is None else payload + return self.client.post(self.url, data=body, format='json') + + +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestPathwaySuccess(PathwayAPITestMixin, APITest): + """Tests for a successful pathway request.""" + + def test_a_selected_career_returns_five_ordered_courses(self): + """Scenario: A pathway is returned end to end.""" + response = self.post_pathway() + + assert response.status_code == status.HTTP_200_OK + assert len(response.data['courses']) == 5 + + def test_courses_carry_the_display_fields_a_client_needs(self): + response = self.post_pathway() + + course = response.data['courses'][0] + for name in ('key', 'title', 'level_type', 'partner', 'rationale'): + assert name in course + + def test_rationales_reach_the_response(self): + self.mock_enrich.return_value = { + 'reasons': {'A+1': 'a solid starting point'}, 'prompt_revision': '4', + } + + response = self.post_pathway() + + rationales = {c['key']: c['rationale'] for c in response.data['courses']} + self.assertEqual(rationales.get('A+1'), 'a solid starting point') + + def test_a_failed_enrichment_still_returns_the_pathway(self): + """Losing the explanations is a far smaller loss than losing the recommendation.""" + self.mock_enrich.side_effect = PromptError('no prompt configured') + + response = self.post_pathway() + + assert response.status_code == status.HTTP_200_OK + assert len(response.data['courses']) == 5 + + def test_the_response_carries_the_trace_handle(self): + response = self.post_pathway() + + assert PathwayAssemblyWorkflow.objects.filter( + uuid=response.data['workflow_uuid'], + ).exists() + + def test_unfilled_rungs_are_reported_to_the_client(self): + """ + A pathway that could not reach an advanced course differs materially from one + that did, and some skills genuinely have no advanced content. + """ + self.mock_retrieve.return_value = retrieval_result( + [course_hit(f'A+{i}', partner=f'P{i}') for i in range(6)], + ) + + response = self.post_pathway() + + assert response.status_code == status.HTTP_200_OK + assert 'Intermediate' in response.data['unfilled_rungs'] + assert 'Advanced' in response.data['unfilled_rungs'] + + def test_every_execution_leaves_an_inspectable_trace(self): + response = self.post_pathway() + + record = AssemblePathwayStep.objects.filter( + workflow_record_uuid=response.data['workflow_uuid'], + ).first() + assert record is not None + assert record.output_data + assert record.succeeded_at is not None + + def test_explicit_db_role_assignment_is_allowed(self): + self.client.logout() + self.client.cookies.clear() + user = UserFactory(is_active=True) + role, _ = EnterpriseAccessFeatureRole.objects.get_or_create(name=LEARNER_PATHWAYS_LEARNER_ROLE) + EnterpriseAccessRoleAssignment.objects.create( + user=user, role=role, enterprise_customer_uuid=uuid.uuid4(), + ) + self.client.force_authenticate(user=user) + + assert self.post_pathway().status_code == status.HTTP_200_OK + + +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestPathwayNoCoverage(PathwayAPITestMixin, APITest): + """A career with no catalog coverage is an answer, not an error.""" + + def test_no_candidates_returns_200_with_no_courses(self): + self.mock_retrieve.return_value = retrieval_result([], broadened=True) + + response = self.post_pathway() + + assert response.status_code == status.HTTP_200_OK + assert response.data['courses'] == [] + + def test_too_few_candidates_returns_no_courses_rather_than_a_short_pathway(self): + """Never pad, and never return four courses as though that were a pathway.""" + self.mock_retrieve.return_value = retrieval_result([ + course_hit('A+1'), course_hit('B+1', level='Intermediate'), + ]) + + response = self.post_pathway() + + assert response.status_code == status.HTTP_200_OK + assert response.data['courses'] == [] + + def test_a_skill_less_career_is_rejected_at_validation(self): + """ + Two thirds of Lightcast careers carry no skills, and a pathway built from none is + a keyword search wearing a pathway's clothes. + """ + response = self.post_pathway({**_VALID_PAYLOAD, 'career_skills': []}) + + assert response.status_code == status.HTTP_400_BAD_REQUEST + + +@ddt.ddt +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestPathwayAuthorization(PathwayAPITestMixin, APITest): + """Authorization and validation tests.""" + + def test_unauthenticated_caller_is_rejected(self): + self.client.logout() + self.client.cookies.clear() + + assert self.post_pathway().status_code in ( + status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN, + ) + + def test_authenticated_non_enterprise_learner_is_rejected(self): + self.client.logout() + self.client.cookies.clear() + self.client.force_authenticate(user=UserFactory(is_active=True)) + + assert self.post_pathway().status_code == status.HTTP_403_FORBIDDEN + + @ddt.data( + {}, + {'career_name': 'Welder'}, + {'career_name': '', 'career_external_id': 'X', 'career_skills': ['Welding']}, + {'career_name': 'Welder', 'career_external_id': '', 'career_skills': ['Welding']}, + {'career_name': 'Welder', 'career_external_id': 'X', 'career_skills': ['']}, + ) + def test_invalid_payload_is_rejected(self, payload): + assert self.post_pathway(payload).status_code == status.HTTP_400_BAD_REQUEST + + def test_optional_skill_lists_may_be_omitted(self): + response = self.post_pathway({ + 'career_name': 'Welder', + 'career_external_id': 'ETE78CD2CDFFFAC66B', + 'career_skills': ['Welding'], + }) + + assert response.status_code == status.HTTP_200_OK + + def test_get_is_rejected(self): + assert self.client.get(self.url).status_code == status.HTTP_405_METHOD_NOT_ALLOWED + + +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestPathwayFailures(PathwayAPITestMixin, APITest): + """A broken dependency is a 500, and never a silently empty pathway.""" + + def test_an_algolia_failure_returns_500_without_partial_results(self): + self.mock_snapshot.side_effect = AlgoliaSearchError('boom') + + response = self.post_pathway() + + assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR + assert 'courses' not in response.data + + def test_a_retrieval_failure_returns_500_rather_than_no_coverage(self): + """ + The distinction that matters: a transport failure must not be reported as "this + career has no courses", which is what an empty 200 would say. + """ + self.mock_retrieve.side_effect = AlgoliaSearchError('boom') + + response = self.post_pathway() + + assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR + + +class TestPathwayFeatureFlag(PathwayAPITestMixin, APITest): + """The endpoint 404s while the pipeline is disabled.""" + + @override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, False) + def test_disabled_pipeline_returns_404(self): + assert self.post_pathway().status_code == status.HTTP_404_NOT_FOUND + + @override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, False) + def test_disabled_pipeline_returns_404_for_an_invalid_payload_too(self): + """A disabled endpoint must be indistinguishable from one that does not exist.""" + assert self.post_pathway({}).status_code == status.HTTP_404_NOT_FOUND + + @override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) + def test_enabled_pipeline_serves_the_endpoint(self): + assert self.post_pathway().status_code == status.HTTP_200_OK + + +class TestPathwayRouteConfig(TestCase): + """Route-level configuration for the pathway action.""" + + def test_url_reverses_under_learner_pathways(self): + assert reverse(_PATHWAY_URL_NAME).endswith('/learner-pathways/pathway/') + + def test_route_does_not_shadow_the_other_pathway_endpoints(self): + assert reverse('api:v1:career-discovery-careers') != reverse(_PATHWAY_URL_NAME) + assert reverse('api:v1:learner-pathways-learning-intent') != reverse(_PATHWAY_URL_NAME) + + def test_post_is_routed(self): + response = APIClient().post(reverse(_PATHWAY_URL_NAME), data={}, format='json') + assert response.status_code != status.HTTP_405_METHOD_NOT_ALLOWED + + def test_action_configuration(self): + # pylint: disable=no-member # DRF @action adds .kwargs at decoration time. + action_kwargs = PathwayViewSet.pathway.kwargs + assert JwtAuthentication in action_kwargs['authentication_classes'] + assert permissions.IsAuthenticated in action_kwargs['permission_classes'] + assert ScopedRateThrottle in action_kwargs['throttle_classes'] + assert action_kwargs['throttle_scope'] == 'learner_pathways_pathway' + + def test_no_class_level_throttle_classes(self): + assert 'throttle_classes' not in PathwayViewSet.__dict__ + assert PathwayViewSet.throttle_scope is None diff --git a/enterprise_access/apps/api/v1/tests/test_pathways_views.py b/enterprise_access/apps/api/v1/tests/test_pathways_views.py new file mode 100644 index 00000000..b8042dc8 --- /dev/null +++ b/enterprise_access/apps/api/v1/tests/test_pathways_views.py @@ -0,0 +1,298 @@ +""" +Tests for the career discovery endpoint. + +HTTP-layer behaviour only: the feature flag, validation, permissions, error mapping and +response shape. Workflow and query-construction behaviour is tested in +``enterprise_access.apps.pathways.tests``. + +Xpert and Algolia are mocked in every test; nothing here issues a network call. +""" +import uuid +from unittest import mock + +import ddt +from django.core.cache import cache as django_cache +from django.test import TestCase +from edx_rest_framework_extensions.auth.jwt.authentication import JwtAuthentication +from edx_toggles.toggles.testutils import override_waffle_switch +from rest_framework import permissions, status +from rest_framework.reverse import reverse +from rest_framework.test import APIClient +from rest_framework.throttling import ScopedRateThrottle + +from enterprise_access.apps.api.v1.views.pathways import CareerDiscoveryViewSet +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.core.constants import LEARNER_PATHWAYS_LEARNER_ROLE, SYSTEM_ENTERPRISE_LEARNER_ROLE +from enterprise_access.apps.core.models import EnterpriseAccessFeatureRole, EnterpriseAccessRoleAssignment +from enterprise_access.apps.core.tests.factories import UserFactory +from enterprise_access.apps.pathways.models import CareerDiscoveryWorkflow, ExtractIntentStep, RetrieveCareersStep +from enterprise_access.apps.prompts.api_client import XpertAPIRequestError, XpertResponseMessage +from enterprise_access.apps.prompts.models import PromptType, XpertLearnerPathwaysSystemPrompt +from enterprise_access.apps.prompts.tests.factories import XpertLearnerPathwaysSystemPromptFactory +from enterprise_access.toggles import LEARNER_PATHWAYS_SERVER_PIPELINE +from test_utils import APITest + +PATCH_XPERT_CLIENT = 'enterprise_access.apps.prompts.api.XpertAPIClient' +PATCH_ALGOLIA_CLIENT = 'enterprise_access.apps.pathways.api.AlgoliaSearchClient' + +_CAREERS_URL_NAME = 'api:v1:career-discovery-careers' + +_VALID_PAYLOAD = { + 'selected_goals': 'move into data analysis', + 'free_text': 'I report on spreadsheets all day and want to automate it', + 'known_context': 'operations analyst, five years', + 'interested_industries': 'healthcare, technology', +} + +_XPERT_CONTENT = ( + '{"skills_required": ["SQL"], "skills_preferred": ["Tableau"], ' + '"condensed_algolia_query": "data analyst"}' +) + +_JOBS_RESPONSE = { + 'hits': [{ + 'external_id': 'ETE78CD2CDFFFAC66B', + 'name': 'Data Analyst', + 'skills': [{'name': 'SQL (Programming Language)'}], + 'industry_names': ['Health Care'], + }], + 'nbHits': 1, +} + + +class CareerDiscoveryAPITestMixin: + """Shared set-up: a configured prompt, mocked Xpert and Algolia, an authorized learner.""" + + @classmethod + def setUpTestData(cls): + super().setUpTestData() + XpertLearnerPathwaysSystemPromptFactory(prompt_type=PromptType.LEARNER_INTENT) + + def setUp(self): + super().setUp() + self.addCleanup(django_cache.clear) + self.url = reverse(_CAREERS_URL_NAME) + + self.xpert_patcher = mock.patch(PATCH_XPERT_CLIENT) + self.mock_xpert = self.xpert_patcher.start().return_value + self.mock_xpert.send_message.return_value = XpertResponseMessage( + role='assistant', + content=_XPERT_CONTENT, + ) + self.addCleanup(self.xpert_patcher.stop) + + self.algolia_patcher = mock.patch(PATCH_ALGOLIA_CLIENT) + self.mock_algolia = self.algolia_patcher.start().return_value + self.mock_algolia.search_jobs_index.return_value = _JOBS_RESPONSE + self.addCleanup(self.algolia_patcher.stop) + + def authenticate_as_enterprise_learner(self): + self.set_jwt_cookie([{ + 'system_wide_role': SYSTEM_ENTERPRISE_LEARNER_ROLE, + 'context': str(uuid.uuid4()), + }]) + + def post_careers(self, payload=None): + body = _VALID_PAYLOAD if payload is None else payload + return self.client.post(self.url, data=body, format='json') + + +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestCareerDiscoverySuccess(CareerDiscoveryAPITestMixin, APITest): + """Tests for a successful career discovery request.""" + + def setUp(self): + super().setUp() + self.authenticate_as_enterprise_learner() + + def test_intake_returns_careers(self): + response = self.post_careers() + + assert response.status_code == status.HTTP_200_OK + assert response.json()['careers'] == [{ + 'external_id': 'ETE78CD2CDFFFAC66B', + 'name': 'Data Analyst', + 'skills': ['SQL (Programming Language)'], + 'industries': ['Health Care'], + }] + + def test_match_percentage_is_absent_rather_than_fabricated(self): + response = self.post_careers() + + career = response.json()['careers'][0] + assert 'match_percentage' not in career + assert not any('match' in key for key in career) + + def test_response_carries_the_trace_handle(self): + response = self.post_careers() + + workflow = CareerDiscoveryWorkflow.objects.get() + assert response.json()['workflow_uuid'] == str(workflow.uuid) + + def test_every_execution_leaves_a_trace(self): + self.post_careers() + + workflow = CareerDiscoveryWorkflow.objects.get() + assert workflow.succeeded_at is not None + + step_records = [ + ExtractIntentStep.objects.get(workflow_record_uuid=workflow.uuid), + RetrieveCareersStep.objects.get(workflow_record_uuid=workflow.uuid), + ] + for step_record in step_records: + assert step_record.input_data is not None + assert step_record.output_data + assert step_record.succeeded_at is not None + + def test_explicit_db_role_assignment_is_allowed(self): + self.client.logout() + self.client.cookies.clear() + user = UserFactory(is_active=True) + role, _ = EnterpriseAccessFeatureRole.objects.get_or_create(name=LEARNER_PATHWAYS_LEARNER_ROLE) + EnterpriseAccessRoleAssignment.objects.create( + user=user, + role=role, + enterprise_customer_uuid=uuid.uuid4(), + ) + self.client.force_authenticate(user=user) + + assert self.post_careers().status_code == status.HTTP_200_OK + + +@ddt.ddt +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestCareerDiscoveryAuthorization(CareerDiscoveryAPITestMixin, APITest): + """Authorization and validation tests.""" + + def test_unauthenticated_caller_is_rejected(self): + self.client.logout() + self.client.cookies.clear() + + response = self.post_careers() + + assert response.status_code in (status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN) + assert not CareerDiscoveryWorkflow.objects.exists() + self.mock_xpert.send_message.assert_not_called() + + def test_authenticated_non_enterprise_learner_is_rejected(self): + self.client.force_authenticate(user=UserFactory(is_active=True)) + + response = self.post_careers() + + assert response.status_code == status.HTTP_403_FORBIDDEN + assert not CareerDiscoveryWorkflow.objects.exists() + self.mock_xpert.send_message.assert_not_called() + + @ddt.data( + {}, + {**_VALID_PAYLOAD, 'free_text': ''}, + {key: value for key, value in _VALID_PAYLOAD.items() if key != 'known_context'}, + ) + def test_invalid_payload_is_rejected(self, payload): + self.authenticate_as_enterprise_learner() + + response = self.post_careers(payload) + + assert response.status_code == status.HTTP_400_BAD_REQUEST + # No workflow record for a request that never ran. + assert not CareerDiscoveryWorkflow.objects.exists() + self.mock_xpert.send_message.assert_not_called() + + def test_get_is_rejected(self): + self.authenticate_as_enterprise_learner() + + assert self.client.get(self.url).status_code == status.HTTP_405_METHOD_NOT_ALLOWED + + +class TestCareerDiscoveryFeatureFlag(CareerDiscoveryAPITestMixin, APITest): + """The flag defaults off, so the endpoint must behave as though it does not exist.""" + + def setUp(self): + super().setUp() + self.authenticate_as_enterprise_learner() + + def test_disabled_pipeline_returns_404(self): + response = self.post_careers() + + assert response.status_code == status.HTTP_404_NOT_FOUND + assert not CareerDiscoveryWorkflow.objects.exists() + self.mock_xpert.send_message.assert_not_called() + + def test_disabled_pipeline_returns_404_for_an_invalid_payload_too(self): + # The flag is checked before validation, so a disabled endpoint cannot be + # distinguished from a missing one by probing it with a bad body. + assert self.post_careers({}).status_code == status.HTTP_404_NOT_FOUND + + @override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) + def test_enabled_pipeline_serves_the_endpoint(self): + assert self.post_careers().status_code == status.HTTP_200_OK + + +@override_waffle_switch(LEARNER_PATHWAYS_SERVER_PIPELINE, True) +class TestCareerDiscoveryFailures(CareerDiscoveryAPITestMixin, APITest): + """A failed step returns 500 and leaves the failure on the record.""" + + def setUp(self): + super().setUp() + self.authenticate_as_enterprise_learner() + + def test_xpert_failure_returns_500_without_partial_results(self): + self.mock_xpert.send_message.side_effect = XpertAPIRequestError('xpert exploded') + + response = self.post_careers() + + assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR + assert 'careers' not in response.json() + + workflow = CareerDiscoveryWorkflow.objects.get() + intent_step = ExtractIntentStep.objects.get(workflow_record_uuid=workflow.uuid) + assert intent_step.failed_at is not None + assert 'xpert exploded' in intent_step.exception_message + assert not RetrieveCareersStep.objects.exists() + + def test_algolia_failure_returns_500_and_marks_the_step_failed(self): + self.mock_algolia.search_jobs_index.side_effect = AlgoliaSearchError('algolia exploded') + + response = self.post_careers() + + assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR + assert 'careers' not in response.json() + + careers_step = RetrieveCareersStep.objects.get() + assert careers_step.failed_at is not None + assert 'algolia exploded' in careers_step.exception_message + + def test_missing_prompt_returns_500(self): + XpertLearnerPathwaysSystemPrompt.objects.all().delete() + + response = self.post_careers() + + assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR + self.mock_algolia.search_jobs_index.assert_not_called() + + +class TestCareerDiscoveryRouteConfig(TestCase): + """Route-level configuration for the careers action.""" + + def test_url_reverses_under_learner_pathways(self): + url = reverse(_CAREERS_URL_NAME) + assert url.endswith('/learner-pathways/careers/') + + def test_route_does_not_shadow_the_prompt_endpoints(self): + assert reverse('api:v1:learner-pathways-learning-intent') != reverse(_CAREERS_URL_NAME) + + def test_post_is_routed(self): + response = APIClient().post(reverse(_CAREERS_URL_NAME), data={}, format='json') + assert response.status_code != status.HTTP_405_METHOD_NOT_ALLOWED + + def test_action_configuration(self): + # pylint: disable=no-member # DRF @action adds .kwargs at decoration time. + action_kwargs = CareerDiscoveryViewSet.careers.kwargs + assert JwtAuthentication in action_kwargs['authentication_classes'] + assert permissions.IsAuthenticated in action_kwargs['permission_classes'] + assert ScopedRateThrottle in action_kwargs['throttle_classes'] + assert action_kwargs['throttle_scope'] == 'learner_pathways_careers' + + def test_no_class_level_throttle_classes(self): + assert 'throttle_classes' not in CareerDiscoveryViewSet.__dict__ + assert CareerDiscoveryViewSet.throttle_scope is None diff --git a/enterprise_access/apps/api/v1/urls.py b/enterprise_access/apps/api/v1/urls.py index fe1a2e9f..cad62e13 100644 --- a/enterprise_access/apps/api/v1/urls.py +++ b/enterprise_access/apps/api/v1/urls.py @@ -12,6 +12,8 @@ router.register("testimonials", views.TestimonialViewSet, "testimonials") router.register('learner-pathways', views.LearnerPathwaysViewSet, 'learner-pathways') +router.register('learner-pathways', views.CareerDiscoveryViewSet, 'career-discovery') +router.register('learner-pathways', views.PathwayViewSet, 'pathway') router.register("policy-redemption", views.SubsidyAccessPolicyRedeemViewset, 'policy-redemption') router.register("policy-allocation", views.SubsidyAccessPolicyAllocateViewset, 'policy-allocation') router.register("subsidy-access-policies", views.SubsidyAccessPolicyViewSet, 'subsidy-access-policies') diff --git a/enterprise_access/apps/api/v1/views/__init__.py b/enterprise_access/apps/api/v1/views/__init__.py index db04c27f..1a36b317 100644 --- a/enterprise_access/apps/api/v1/views/__init__.py +++ b/enterprise_access/apps/api/v1/views/__init__.py @@ -23,6 +23,7 @@ SspProductViewSet, StripeEventSummaryViewSet ) +from .pathways import CareerDiscoveryViewSet, PathwayViewSet from .prompt import LearnerPathwaysViewSet from .provisioning import ProvisioningCreateView, SubscriptionPlanOLIUpdateView from .subsidy_access_policy import ( diff --git a/enterprise_access/apps/api/v1/views/pathways.py b/enterprise_access/apps/api/v1/views/pathways.py new file mode 100644 index 00000000..31a6f60e --- /dev/null +++ b/enterprise_access/apps/api/v1/views/pathways.py @@ -0,0 +1,233 @@ +""" +REST API viewsets for the server-side learner pathways pipeline. + +Distinct from ``views/prompt.py``, which owns the two live single-shot prompt endpoints. +These endpoints run a persisted, multi-step workflow instead, so every request leaves a +trace that can be inspected -- or re-serialized -- afterwards without re-running it. +""" +import logging + +from drf_spectacular.utils import extend_schema +from edx_rbac.decorators import permission_required +from edx_rest_framework_extensions.auth.jwt.authentication import JwtAuthentication +from rest_framework import permissions, status +from rest_framework.decorators import action +from rest_framework.exceptions import APIException, NotFound +from rest_framework.request import Request +from rest_framework.response import Response +from rest_framework.throttling import ScopedRateThrottle +from rest_framework.viewsets import ViewSet + +from enterprise_access.apps.api import serializers as api_serializers +from enterprise_access.apps.api.serializers.learner_pathways import LEARNER_PATHWAYS_API_TAG +from enterprise_access.apps.core import constants +from enterprise_access.apps.pathways.models import CareerDiscoveryWorkflow, PathwayAssemblyWorkflow +from enterprise_access.apps.workflow.exceptions import UnitOfWorkException +from enterprise_access.toggles import learner_pathways_server_pipeline_enabled + +logger = logging.getLogger(__name__) + + +class CareerDiscoveryException(APIException): + """ + Raised when the career discovery workflow fails. + + Deliberately carries no partial results: a failed run's step records hold the input, + output and failure of every stage, so the diagnosis lives in the trace rather than in + a half-populated response body a client might render. + """ + + status_code = status.HTTP_500_INTERNAL_SERVER_ERROR + default_detail = 'Career discovery failed.' + default_code = 'career_discovery_error' + + +class CareerDiscoveryViewSet(ViewSet): + """ + Endpoint for deriving career candidates from a learner's intake. + + Registered alongside ``LearnerPathwaysViewSet`` under the same ``learner-pathways`` + prefix. Each action defines its own authentication, permissions and throttling + explicitly; nothing is configured at the class level. + """ + + # DRF's ``as_view()`` rejects an ``@action`` initkwarg that is not also a class + # attribute, so the per-action ``throttle_scope`` needs this declaration to exist. + throttle_scope: str | None = None + + @extend_schema( + tags=[LEARNER_PATHWAYS_API_TAG], + summary='Derive career candidates from learner intake.', + description=( + 'Runs the server-side career discovery workflow: derives learning intent from the ' + 'learner\'s intake via Xpert, then searches the careers taxonomy for matching roles. ' + 'Every execution persists a workflow record and one record per executed step, so the ' + 'returned workflow_uuid can be used to inspect the full trace afterwards.' + ), + request=api_serializers.CareerDiscoveryRequestSerializer, + responses={ + status.HTTP_200_OK: api_serializers.CareerDiscoveryResponseSerializer, + status.HTTP_400_BAD_REQUEST: None, + status.HTTP_401_UNAUTHORIZED: None, + status.HTTP_403_FORBIDDEN: None, + status.HTTP_404_NOT_FOUND: None, + status.HTTP_429_TOO_MANY_REQUESTS: None, + status.HTTP_500_INTERNAL_SERVER_ERROR: None, + }, + ) + @permission_required(constants.LEARNER_PATHWAYS_CAREER_DISCOVERY_PERMISSION) + @action( + detail=False, + methods=['post'], + url_path='careers', + url_name='careers', + authentication_classes=(JwtAuthentication,), + permission_classes=(permissions.IsAuthenticated,), + throttle_classes=(ScopedRateThrottle,), + throttle_scope='learner_pathways_careers', + ) + def careers(self, request: Request) -> Response: + """ + Derive career candidates from the learner's stated goals, free text and context. + + Returns HTTP 404 when the server-side pipeline is disabled. + Returns HTTP 400 for invalid request input. + Returns HTTP 401/403 when the caller is unauthenticated or not an enterprise learner. + Returns HTTP 429 when the per-endpoint rate limit is exceeded. + Returns HTTP 500 when any step of the workflow fails, with no partial results. + """ + # Checked before validation so a disabled pipeline is indistinguishable from an + # endpoint that does not exist, whatever the payload. + if not learner_pathways_server_pipeline_enabled(): + raise NotFound() + + request_serializer = api_serializers.CareerDiscoveryRequestSerializer(data=request.data) + request_serializer.is_valid(raise_exception=True) + + workflow = CareerDiscoveryWorkflow.objects.create( + input_data=CareerDiscoveryWorkflow.generate_input_dict(request_serializer.validated_data), + ) + logger.info('Created CareerDiscoveryWorkflow (uuid=%s)', workflow.uuid) + + try: + workflow.execute() + except UnitOfWorkException as exc: + logger.exception('CareerDiscoveryWorkflow (uuid=%s) failed: %s', workflow.uuid, exc) + raise CareerDiscoveryException( + detail=f'Error in career discovery workflow: {exc}', + ) from exc + + response_serializer = api_serializers.CareerDiscoveryResponseSerializer({ + 'workflow_uuid': workflow.uuid, + 'careers': workflow.career_candidates(), + }) + return Response(response_serializer.data, status=status.HTTP_200_OK) + + +class PathwayAssemblyException(APIException): + """ + Raised when the pathway assembly workflow fails. + + Carries no partial results, for the same reason ``CareerDiscoveryException`` does not: + the step records hold every stage's input, output and failure, so a client is better + served by a clean error and a workflow_uuid than by half a pathway it might render. + """ + + status_code = status.HTTP_500_INTERNAL_SERVER_ERROR + default_detail = 'Pathway assembly failed.' + default_code = 'pathway_assembly_error' + + +class PathwayViewSet(ViewSet): + """ + Endpoint for assembling a pathway for a career the learner has chosen. + + A separate request from career discovery rather than a continuation of it: the + learner's choice sits between the two, and only the client knows which career was + picked. + """ + + # See CareerDiscoveryViewSet -- DRF requires the class attribute to exist for the + # per-action throttle_scope initkwarg to be accepted. + throttle_scope: str | None = None + + @extend_schema( + tags=[LEARNER_PATHWAYS_API_TAG], + summary='Assemble a course pathway for a selected career.', + description=( + 'Runs the server-side pathway assembly workflow: reads the catalog\'s skill ' + 'vocabulary, translates the career\'s skills into it, retrieves a candidate ' + 'window, optionally re-ranks it with a model, and selects five courses that ' + 'span difficulty levels without over-representing one provider. Returns HTTP ' + '200 with an empty course list when the catalog has no pathway for the career.' + ), + request=api_serializers.PathwayRequestSerializer, + responses={ + status.HTTP_200_OK: api_serializers.PathwayResponseSerializer, + status.HTTP_400_BAD_REQUEST: None, + status.HTTP_401_UNAUTHORIZED: None, + status.HTTP_403_FORBIDDEN: None, + status.HTTP_404_NOT_FOUND: None, + status.HTTP_429_TOO_MANY_REQUESTS: None, + status.HTTP_500_INTERNAL_SERVER_ERROR: None, + }, + ) + @permission_required(constants.LEARNER_PATHWAYS_PATHWAY_PERMISSION) + @action( + detail=False, + methods=['post'], + url_path='pathway', + url_name='pathway', + authentication_classes=(JwtAuthentication,), + permission_classes=(permissions.IsAuthenticated,), + throttle_classes=(ScopedRateThrottle,), + throttle_scope='learner_pathways_pathway', + ) + def pathway(self, request: Request) -> Response: + """ + Assemble a five-course pathway for the selected career. + + Returns HTTP 404 when the server-side pipeline is disabled. + Returns HTTP 400 for invalid request input. + Returns HTTP 401/403 when the caller is unauthenticated or not an enterprise learner. + Returns HTTP 429 when the per-endpoint rate limit is exceeded. + Returns HTTP 500 when any step of the workflow fails, with no partial results. + + A career with no catalog coverage is **not** an error: it returns 200 with an + empty ``courses`` list. That distinction matters because "we could not build this" + and "something broke" lead to different client behaviour, and the catalog + genuinely has careers with no matching courses. + """ + if not learner_pathways_server_pipeline_enabled(): + raise NotFound() + + request_serializer = api_serializers.PathwayRequestSerializer(data=request.data) + request_serializer.is_valid(raise_exception=True) + validated = request_serializer.validated_data + + workflow = PathwayAssemblyWorkflow.objects.create( + input_data=PathwayAssemblyWorkflow.generate_input_dict( + career_name=validated['career_name'], + career_skills=validated['career_skills'], + skills_required=validated.get('skills_required') or [], + skills_preferred=validated.get('skills_preferred') or [], + learner_profile=validated.get('learner_profile') or {}, + ), + ) + logger.info('Created PathwayAssemblyWorkflow (uuid=%s)', workflow.uuid) + + try: + workflow.execute() + except UnitOfWorkException as exc: + logger.exception('PathwayAssemblyWorkflow (uuid=%s) failed: %s', workflow.uuid, exc) + raise PathwayAssemblyException( + detail=f'Error in pathway assembly workflow: {exc}', + ) from exc + + assembled = workflow.pathway() or {} + response_serializer = api_serializers.PathwayResponseSerializer({ + 'workflow_uuid': workflow.uuid, + 'courses': assembled.get('courses') or [], + 'unfilled_rungs': assembled.get('unfilled_rungs') or [], + }) + return Response(response_serializer.data, status=status.HTTP_200_OK) diff --git a/enterprise_access/apps/api_client/algolia_client.py b/enterprise_access/apps/api_client/algolia_client.py new file mode 100644 index 00000000..e86887e4 --- /dev/null +++ b/enterprise_access/apps/api_client/algolia_client.py @@ -0,0 +1,348 @@ +""" +Search-only Algolia client for enterprise-access. + +enterprise-catalog owns an ``AlgoliaSearchClient`` too, but it is an *indexing and +administration* client built on the write-scoped ``ALGOLIA.API_KEY`` and it exposes no +``search()``. That key must never reach this service, so this module is a separate, +deliberately read-only client rather than a reuse of that one. + +Two indexes, two credential paths, and they are not interchangeable: + +* **Catalog index** — queried with an *enterprise-scoped secured key* vended by + enterprise-catalog. That scoping is the whole point: it is what keeps results inside + the learner's catalog without this service reimplementing catalog filtering. +* **Jobs (Lightcast taxonomy) index** — queried with the plain search key. Secured keys + set ``restrictIndices`` to the catalog index and its replicas only + (``enterprise-catalog``'s ``generate_secured_api_key``), so a secured key *cannot* + read the jobs index. The learner portal MFE encodes the same constraint as + ``unsupportedSecuredAlgoliaIndices = [ALGOLIA_INDEX_NAME_JOBS]``. + +Sending a secured key to the jobs index therefore fails at Algolia with an opaque error. +``search_jobs_index()`` refuses before issuing the request instead. +""" +import base64 +import binascii +import logging +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Any + +from algoliasearch.exceptions import AlgoliaException, AlgoliaUnreachableHostException, RequestException +from algoliasearch.search_client import SearchClient +from django.conf import settings +from django.utils.dateparse import parse_datetime + +logger = logging.getLogger(__name__) + + +class AlgoliaClientError(Exception): + """Base class for every error raised by this module.""" + + +class AlgoliaConfigurationError(AlgoliaClientError): + """ + Raised when a search cannot be issued safely: missing settings, or a credential + that is wrong for the requested index. + + Distinct from ``AlgoliaSearchError`` because it is never transient — retrying will + not help, and no request was sent. + """ + + +class AlgoliaSearchError(AlgoliaClientError): + """ + Raised when a search request fails in transit. + + Wraps ``AlgoliaException`` and friends so callers never have to import + ``algoliasearch`` to handle a failure. + """ + + +def looks_like_secured_api_key(api_key: str) -> bool: + """ + Whether ``api_key`` is an Algolia *secured* key rather than a plain search key. + + A secured key is ``base64(hmac_sha256_hex + urlencoded_restrictions)``, so decoding + one yields a querystring carrying ``restrictIndices`` and/or ``validUntil``. A plain + search key is a 32-character hex string that does not base64-decode to anything of + that shape. + + Used to catch a misconfiguration — a secured key handed to the client as its plain + search key — before it reaches the jobs index, where ``restrictIndices`` guarantees + it will fail with an opaque Algolia error. + """ + if not api_key: + return False + try: + decoded = base64.b64decode(api_key, validate=True).decode('utf-8', errors='replace') + except (binascii.Error, ValueError): + return False + return 'restrictIndices=' in decoded or 'validUntil=' in decoded + + +@dataclass(frozen=True) +class SecuredAlgoliaKey: + """ + An enterprise-scoped secured Algolia API key and the instant it stops working. + + Built from enterprise-catalog's ``secured-algolia-api-key`` response, whose + ``valid_until`` is an ISO-8601 UTC timestamp. + """ + + api_key: str + valid_until: datetime | None + + @classmethod + def from_response_payload(cls, payload: dict[str, Any]) -> 'SecuredAlgoliaKey': + """ + Build a key from an enterprise-catalog ``get_secured_algolia_api_key()`` payload. + + Raises: + AlgoliaConfigurationError: If the payload carries no secured key. + """ + algolia_payload = (payload or {}).get('algolia') or {} + # The BFF serializes this field as ``secured_algolia_api_key``; the raw + # enterprise-catalog response uses ``secured_api_key``. Accept either. + api_key = ( + algolia_payload.get('secured_api_key') or + algolia_payload.get('secured_algolia_api_key') + ) + if not api_key: + raise AlgoliaConfigurationError( + 'enterprise-catalog returned no secured Algolia API key; ' + 'refusing to fall back to an unscoped search key.' + ) + + raw_valid_until = algolia_payload.get('valid_until') + valid_until = parse_datetime(raw_valid_until) if raw_valid_until else None + if valid_until is not None and valid_until.tzinfo is None: + valid_until = valid_until.replace(tzinfo=timezone.utc) + + return cls(api_key=api_key, valid_until=valid_until) + + def is_expired(self, now: datetime | None = None) -> bool: + """ + Whether this key is past ``valid_until``. + + A key with no ``valid_until`` is treated as expired: an unknown expiry is not + evidence of validity, and the safe failure here is to re-vend. + """ + if self.valid_until is None: + return True + return (now or datetime.now(timezone.utc)) >= self.valid_until + + +class AlgoliaSearchClient: + """ + Issues read-only queries against the catalog and jobs Algolia indexes. + + One instance holds no credentials of its own beyond the configured application ID + and plain search key. The per-enterprise secured key is passed in per call, because + it is vended per user by enterprise-catalog and expires. + + This class deliberately exposes only ``search``-shaped methods. It has no write, + index-management, or key-management surface. + """ + + def __init__(self, app_id: str | None = None, search_api_key: str | None = None): + self._app_id = app_id or settings.ALGOLIA_APP_ID + self._search_api_key = search_api_key or settings.ALGOLIA_SEARCH_API_KEY + + if not self._app_id: + raise AlgoliaConfigurationError('ALGOLIA_APP_ID is not configured.') + + @property + def catalog_index_name(self) -> str: + return settings.ALGOLIA_CATALOG_INDEX_NAME + + @property + def jobs_index_name(self) -> str: + return settings.ALGOLIA_JOBS_INDEX_NAME + + def _search(self, index_name: str, api_key: str, query: str, search_params: dict[str, Any]) -> dict[str, Any]: + """ + Issue one search and return the raw Algolia response body. + + Mirrors the calling shape already used in enterprise-catalog's CSV export view: + ``index.search(query, {facetFilters, attributesToRetrieve, hitsPerPage, page})``. + + Raises: + AlgoliaConfigurationError: If ``index_name`` or ``api_key`` is missing. + AlgoliaSearchError: On any transport or API failure. + """ + if not index_name: + raise AlgoliaConfigurationError('Cannot search Algolia without an index name.') + if not api_key: + raise AlgoliaConfigurationError(f'Cannot search Algolia index {index_name!r} without an API key.') + + client = SearchClient.create(self._app_id, api_key) + try: + return client.init_index(index_name).search(query, search_params) + except (AlgoliaException, AlgoliaUnreachableHostException, RequestException) as exc: + # The query text is safe to log (it is derived from learner intent, not + # credentials); the API key is never logged. + logger.exception( + 'Algolia search failed for index=%r, query=%r.', + index_name, query, + ) + raise AlgoliaSearchError(f'Algolia search failed for index {index_name!r}: {exc}') from exc + finally: + client.close() + + def search_catalog_index( + self, + query: str, + *, + secured_key: SecuredAlgoliaKey | None = None, + allow_unscoped: bool = False, + index_name: str | None = None, + **search_params: Any, + ) -> dict[str, Any]: + """ + Search the enterprise catalog index. + + Normally requires a live enterprise-scoped ``secured_key``; that is what makes + results catalog-correct for the learner. + + ``allow_unscoped=True`` opts out of scoping and searches the whole index with the + plain search key. It exists for the offline retrieval diagnostic, which runs as a + management command with no request and therefore no per-user secured key to vend. + It is gated on ``settings.ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH`` so it cannot be + reached in an environment that has not deliberately enabled it, and it is never a + fallback — a missing or expired secured key raises rather than silently widening + the search. + + Raises: + AlgoliaConfigurationError: If no usable secured key is supplied, or if + ``allow_unscoped`` is requested but not enabled by settings. + AlgoliaSearchError: On any transport or API failure. + """ + resolved_index_name = index_name or self.catalog_index_name + api_key = self._resolve_catalog_api_key(secured_key, allow_unscoped, resolved_index_name) + return self._search(resolved_index_name, api_key, query, search_params) + + def _resolve_catalog_api_key( + self, + secured_key: SecuredAlgoliaKey | None, + allow_unscoped: bool, + index_name: str, + ) -> str: + """ + Decide which key may be used for a catalog read, or refuse. + + Shared by every catalog-reading method so the scoping rules exist in exactly one + place — a second copy of this logic is a second place for it to be wrong. + """ + if allow_unscoped: + if not getattr(settings, 'ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH', False): + raise AlgoliaConfigurationError( + 'Unscoped catalog search requires ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH to be enabled.' + ) + logger.warning( + 'Issuing an UNSCOPED Algolia catalog search against index=%r. ' + 'Results are not restricted to any enterprise catalog.', + index_name, + ) + return self._search_api_key + + if secured_key is None: + raise AlgoliaConfigurationError( + 'A secured Algolia API key is required to search the catalog index. ' + 'Pass allow_unscoped=True only for offline diagnostics.' + ) + if secured_key.is_expired(): + raise AlgoliaConfigurationError( + 'The supplied secured Algolia API key is expired; re-vend it before searching. ' + 'Refusing to issue an unscoped query.' + ) + + return secured_key.api_key + + def search_facet_values( + self, + facet_name: str, + facet_query: str, + *, + secured_key: SecuredAlgoliaKey | None = None, + allow_unscoped: bool = False, + index_name: str | None = None, + max_facet_hits: int = 20, + **search_params: Any, + ) -> list[dict[str, Any]]: + """ + Search the *values* of one facet on the catalog index. + + Answers "what does this index actually call this?" — e.g. a ``skill_names`` + facet query for ``Python`` returns ``Python (Programming Language)``. That is the + cheap, always-current alternative to a hand-maintained alias map for grounding a + skill name against the catalog's Lightcast-canonical vocabulary. + + Returns a list of ``{'value': str, 'count': int, 'highlighted': str}``. + + **The counts are not course counts.** Catalog records are duplicated per customer + group (``objectID`` looks like ``course--customer-uuids-9``) and this + endpoint is not restricted to ``content_type:course``, so a count here can be two + orders of magnitude above the number of matching courses. Treat a value as + *candidate vocabulary* and confirm it against a scoped facet snapshot before + relying on it. + + Raises: + AlgoliaConfigurationError: Same credential rules as ``search_catalog_index``. + AlgoliaSearchError: On any transport or API failure. + """ + resolved_index_name = index_name or self.catalog_index_name + api_key = self._resolve_catalog_api_key(secured_key, allow_unscoped, resolved_index_name) + + if not resolved_index_name: + raise AlgoliaConfigurationError('Cannot search Algolia facets without an index name.') + + client = SearchClient.create(self._app_id, api_key) + try: + response = client.init_index(resolved_index_name).search_for_facet_values( + facet_name, + facet_query, + {'maxFacetHits': max_facet_hits, **search_params}, + ) + except (AlgoliaException, AlgoliaUnreachableHostException, RequestException) as exc: + logger.exception( + 'Algolia facet search failed for index=%r, facet=%r.', + resolved_index_name, facet_name, + ) + raise AlgoliaSearchError( + f'Algolia facet search failed for {facet_name!r} on {resolved_index_name!r}: {exc}' + ) from exc + finally: + client.close() + + return response.get('facetHits', []) + + def search_jobs_index( + self, + query: str, + *, + index_name: str | None = None, + **search_params: Any, + ) -> dict[str, Any]: + """ + Search the Lightcast jobs/taxonomy index with the plain search key. + + There is no secured-key variant. Secured keys restrict themselves to the catalog + index and its replicas, so one would fail here; the parameter is absent rather + than validated so that no caller can construct that request at all. + + Raises: + AlgoliaConfigurationError: If the jobs index or plain search key is unset. + AlgoliaSearchError: On any transport or API failure. + """ + resolved_index_name = index_name or self.jobs_index_name + if resolved_index_name and resolved_index_name == self.catalog_index_name: + raise AlgoliaConfigurationError( + f'Refusing to search index {resolved_index_name!r} as a jobs index: ' + 'it is the configured catalog index, which requires a secured key.' + ) + if looks_like_secured_api_key(self._search_api_key): + raise AlgoliaConfigurationError( + 'The configured Algolia search key is a secured key. Secured keys are ' + f'restricted to the catalog index and cannot read {resolved_index_name!r}.' + ) + return self._search(resolved_index_name, self._search_api_key, query, search_params) diff --git a/enterprise_access/apps/api_client/tests/test_algolia_client.py b/enterprise_access/apps/api_client/tests/test_algolia_client.py new file mode 100644 index 00000000..df1caf7b --- /dev/null +++ b/enterprise_access/apps/api_client/tests/test_algolia_client.py @@ -0,0 +1,301 @@ +""" +Tests for the search-only Algolia client. +""" +from datetime import datetime, timedelta, timezone +from unittest import mock + +import ddt +from algoliasearch.exceptions import AlgoliaException, AlgoliaUnreachableHostException, RequestException +from algoliasearch.search_client import SearchClient +from django.test import TestCase, override_settings + +from enterprise_access.apps.api_client.algolia_client import ( + AlgoliaConfigurationError, + AlgoliaSearchClient, + AlgoliaSearchError, + SecuredAlgoliaKey, + looks_like_secured_api_key +) + +CATALOG_INDEX = 'enterprise_catalog_incremental' +JOBS_INDEX = 'stage_taxonomy' +APP_ID = 'TESTAPPID' +PLAIN_SEARCH_KEY = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + +ALGOLIA_SETTINGS = { + 'ALGOLIA_APP_ID': APP_ID, + 'ALGOLIA_SEARCH_API_KEY': PLAIN_SEARCH_KEY, + 'ALGOLIA_CATALOG_INDEX_NAME': CATALOG_INDEX, + 'ALGOLIA_JOBS_INDEX_NAME': JOBS_INDEX, + 'ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH': False, +} + +SEARCH_CLIENT_PATH = 'enterprise_access.apps.api_client.algolia_client.SearchClient' + + +def _live_secured_key(api_key='secured-key-value'): + return SecuredAlgoliaKey( + api_key=api_key, + valid_until=datetime.now(timezone.utc) + timedelta(hours=1), + ) + + +@override_settings(**ALGOLIA_SETTINGS) +@ddt.ddt +class TestAlgoliaSearchClient(TestCase): + """ + Tests for ``AlgoliaSearchClient``. + + Every test patches ``SearchClient`` — this suite must never issue a real Algolia + request. Assertions are written against the credential actually handed to + ``SearchClient.create`` and the index actually handed to ``init_index``, because + "which key reached which index" is the property this client exists to guarantee. + """ + + def setUp(self): + super().setUp() + self.mock_index = mock.MagicMock() + self.mock_index.search.return_value = {'hits': [], 'nbHits': 0} + + def _patched_search_client(self): + """Patch ``SearchClient`` and return (patcher_mock, index_mock).""" + patcher = mock.patch(SEARCH_CLIENT_PATH) + mock_search_client = patcher.start() + self.addCleanup(patcher.stop) + mock_search_client.create.return_value.init_index.return_value = self.mock_index + return mock_search_client + + def _assert_searched_with(self, mock_search_client, expected_key, expected_index): + mock_search_client.create.assert_called_once_with(APP_ID, expected_key) + mock_search_client.create.return_value.init_index.assert_called_once_with(expected_index) + + # -- Configuration ---------------------------------------------------------------- + + @override_settings(ALGOLIA_APP_ID='') + def test_missing_app_id_raises(self): + with self.assertRaisesRegex(AlgoliaConfigurationError, 'ALGOLIA_APP_ID'): + AlgoliaSearchClient() + + @override_settings(ALGOLIA_JOBS_INDEX_NAME='') + def test_missing_index_name_raises_before_request(self): + mock_search_client = self._patched_search_client() + + with self.assertRaisesRegex(AlgoliaConfigurationError, 'without an index name'): + AlgoliaSearchClient().search_jobs_index('data analyst') + + mock_search_client.create.assert_not_called() + + @override_settings(ALGOLIA_SEARCH_API_KEY='') + def test_missing_search_key_raises_before_request(self): + mock_search_client = self._patched_search_client() + + with self.assertRaisesRegex(AlgoliaConfigurationError, 'without an API key'): + AlgoliaSearchClient().search_jobs_index('data analyst') + + mock_search_client.create.assert_not_called() + + # -- Catalog index: enterprise scoping -------------------------------------------- + + def test_catalog_search_uses_the_secured_key(self): + """Scenario: Catalog search is scoped to the enterprise.""" + mock_search_client = self._patched_search_client() + secured_key = _live_secured_key() + + result = AlgoliaSearchClient().search_catalog_index( + 'python', + secured_key=secured_key, + hitsPerPage=20, + ) + + self.assertEqual(result, {'hits': [], 'nbHits': 0}) + self._assert_searched_with(mock_search_client, secured_key.api_key, CATALOG_INDEX) + self.mock_index.search.assert_called_once_with('python', {'hitsPerPage': 20}) + # The plain, unscoped search key must not have been used. + self.assertNotIn( + mock.call(APP_ID, PLAIN_SEARCH_KEY), + mock_search_client.create.mock_calls, + ) + + def test_catalog_search_without_secured_key_raises(self): + mock_search_client = self._patched_search_client() + + with self.assertRaisesRegex(AlgoliaConfigurationError, 'secured Algolia API key is required'): + AlgoliaSearchClient().search_catalog_index('python') + + mock_search_client.create.assert_not_called() + + @ddt.data( + # An expiry in the past. + datetime(2020, 1, 1, tzinfo=timezone.utc), + # No expiry at all: unknown validity is not evidence of validity. + None, + ) + def test_expired_secured_key_raises_and_never_falls_back(self, valid_until): + """Scenario: Expired secured keys are refreshed / no unscoped query is issued.""" + mock_search_client = self._patched_search_client() + expired = SecuredAlgoliaKey(api_key='stale', valid_until=valid_until) + + with self.assertRaisesRegex(AlgoliaConfigurationError, 'expired'): + AlgoliaSearchClient().search_catalog_index('python', secured_key=expired) + + mock_search_client.create.assert_not_called() + + # -- Catalog index: the explicit unscoped diagnostic path ------------------------- + + def test_unscoped_catalog_search_is_refused_by_default(self): + mock_search_client = self._patched_search_client() + + with self.assertRaisesRegex(AlgoliaConfigurationError, 'ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH'): + AlgoliaSearchClient().search_catalog_index('python', allow_unscoped=True) + + mock_search_client.create.assert_not_called() + + @override_settings(ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH=True) + def test_unscoped_catalog_search_uses_plain_key_when_enabled(self): + mock_search_client = self._patched_search_client() + + AlgoliaSearchClient().search_catalog_index('python', allow_unscoped=True, hitsPerPage=20) + + self._assert_searched_with(mock_search_client, PLAIN_SEARCH_KEY, CATALOG_INDEX) + + @override_settings(ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH=True) + def test_enabling_unscoped_search_does_not_weaken_the_scoped_path(self): + """Even with the escape hatch on, a caller that does not ask for it still needs a key.""" + mock_search_client = self._patched_search_client() + + with self.assertRaises(AlgoliaConfigurationError): + AlgoliaSearchClient().search_catalog_index('python') + + mock_search_client.create.assert_not_called() + + # -- Jobs index ------------------------------------------------------------------- + + def test_jobs_search_uses_the_plain_key(self): + """Scenario: Jobs index uses the plain search key.""" + mock_search_client = self._patched_search_client() + self.mock_index.search.return_value = { + 'hits': [{ + 'name': 'Data Analyst', + 'external_id': 'ET123', + 'skills': [{'name': 'SQL'}], + 'industry_names': ['Finance and Insurance'], + }], + } + + result = AlgoliaSearchClient().search_jobs_index( + 'data analyst', + facetFilters=[['skills.name:SQL']], + hitsPerPage=10, + ) + + self._assert_searched_with(mock_search_client, PLAIN_SEARCH_KEY, JOBS_INDEX) + self.mock_index.search.assert_called_once_with( + 'data analyst', + {'facetFilters': [['skills.name:SQL']], 'hitsPerPage': 10}, + ) + hit = result['hits'][0] + self.assertEqual(hit['name'], 'Data Analyst') + self.assertEqual(hit['skills'], [{'name': 'SQL'}]) + self.assertEqual(hit['industry_names'], ['Finance and Insurance']) + + def test_jobs_search_refuses_a_secured_key(self): + """Scenario: A secured key is never sent to the jobs index.""" + secured_key = SearchClient.generate_secured_api_key( + PLAIN_SEARCH_KEY, + {'restrictIndices': [CATALOG_INDEX], 'validUntil': 4102444800}, + ) + mock_search_client = self._patched_search_client() + + with override_settings(ALGOLIA_SEARCH_API_KEY=secured_key): + with self.assertRaisesRegex(AlgoliaConfigurationError, 'secured key'): + AlgoliaSearchClient().search_jobs_index('data analyst') + + mock_search_client.create.assert_not_called() + + @override_settings(ALGOLIA_JOBS_INDEX_NAME=CATALOG_INDEX) + def test_jobs_search_refuses_the_catalog_index(self): + """A misconfigured jobs index name must not become an unscoped catalog search.""" + mock_search_client = self._patched_search_client() + + with self.assertRaisesRegex(AlgoliaConfigurationError, 'requires a secured key'): + AlgoliaSearchClient().search_jobs_index('python') + + mock_search_client.create.assert_not_called() + + # -- Transport failures ----------------------------------------------------------- + + @ddt.data( + AlgoliaException('boom'), + AlgoliaUnreachableHostException('unreachable'), + RequestException('500 server error', 500), + ) + def test_transport_failures_are_typed(self, raised_exception): + """Scenario: Transport failures are typed.""" + self._patched_search_client() + self.mock_index.search.side_effect = raised_exception + + with self.assertRaises(AlgoliaSearchError): + AlgoliaSearchClient().search_jobs_index('data analyst') + + def test_client_is_closed_even_when_the_search_fails(self): + mock_search_client = self._patched_search_client() + self.mock_index.search.side_effect = AlgoliaException('boom') + + with self.assertRaises(AlgoliaSearchError): + AlgoliaSearchClient().search_jobs_index('data analyst') + + mock_search_client.create.return_value.close.assert_called_once() + + +@ddt.ddt +class TestSecuredAlgoliaKey(TestCase): + """ + Tests for ``SecuredAlgoliaKey`` and secured-key detection. + """ + + @ddt.data('secured_api_key', 'secured_algolia_api_key') + def test_from_response_payload_accepts_both_field_names(self, field_name): + """enterprise-catalog returns ``secured_api_key``; the BFF renames it.""" + key = SecuredAlgoliaKey.from_response_payload({ + 'algolia': {field_name: 'abc123', 'valid_until': '2099-01-01T00:00:00Z'}, + }) + + self.assertEqual(key.api_key, 'abc123') + self.assertEqual(key.valid_until, datetime(2099, 1, 1, tzinfo=timezone.utc)) + self.assertFalse(key.is_expired()) + + @ddt.data( + {}, + {'algolia': {}}, + {'algolia': {'secured_api_key': ''}}, + None, + ) + def test_from_response_payload_without_a_key_raises(self, payload): + with self.assertRaisesRegex(AlgoliaConfigurationError, 'no secured Algolia API key'): + SecuredAlgoliaKey.from_response_payload(payload) + + def test_naive_valid_until_is_treated_as_utc(self): + key = SecuredAlgoliaKey.from_response_payload({ + 'algolia': {'secured_api_key': 'abc', 'valid_until': '2099-01-01T00:00:00'}, + }) + + self.assertEqual(key.valid_until.tzinfo, timezone.utc) + + def test_is_expired_at_the_boundary(self): + boundary = datetime(2026, 9, 9, 12, 0, tzinfo=timezone.utc) + key = SecuredAlgoliaKey(api_key='abc', valid_until=boundary) + + self.assertTrue(key.is_expired(now=boundary)) + self.assertFalse(key.is_expired(now=boundary - timedelta(seconds=1))) + + def test_looks_like_secured_api_key_detects_a_real_secured_key(self): + secured = SearchClient.generate_secured_api_key( + PLAIN_SEARCH_KEY, + {'restrictIndices': [CATALOG_INDEX], 'validUntil': 4102444800}, + ) + + self.assertTrue(looks_like_secured_api_key(secured)) + + @ddt.data('', None, PLAIN_SEARCH_KEY, '68c192f7c3aeca5d488c9e1a8ee15966', 'not-base64!!') + def test_looks_like_secured_api_key_is_false_for_plain_keys(self, api_key): + self.assertFalse(looks_like_secured_api_key(api_key)) diff --git a/enterprise_access/apps/core/constants.py b/enterprise_access/apps/core/constants.py index bdff0322..76978ba4 100644 --- a/enterprise_access/apps/core/constants.py +++ b/enterprise_access/apps/core/constants.py @@ -52,6 +52,8 @@ LEARNER_PATHWAYS_LEARNER_ROLE = 'enterprise_access_learner_pathways_learner' LEARNER_PATHWAYS_LEARNING_INTENT_PERMISSION = 'learner_pathways.has_learning_intent_access' LEARNER_PATHWAYS_RECOMMENDATION_FEEDBACK_PERMISSION = 'learner_pathways.has_recommendation_feedback_access' +LEARNER_PATHWAYS_CAREER_DISCOVERY_PERMISSION = 'learner_pathways.has_career_discovery_access' +LEARNER_PATHWAYS_PATHWAY_PERMISSION = 'learner_pathways.has_pathway_access' ALL_ACCESS_CONTEXT = '*' diff --git a/enterprise_access/apps/core/rules.py b/enterprise_access/apps/core/rules.py index 5c4897e0..3ef2f2de 100644 --- a/enterprise_access/apps/core/rules.py +++ b/enterprise_access/apps/core/rules.py @@ -509,6 +509,14 @@ def has_explicit_access_to_learner_pathways_for_any_context(user, *args, **kwarg has_learner_pathways_learning_intent_access ) +has_learner_pathways_career_discovery_access = ( + has_learner_pathways_learning_intent_access +) + +has_learner_pathways_pathway_access = ( + has_learner_pathways_learning_intent_access +) + ############################################### # Map permissions to consolidated predicates. # ############################################### @@ -659,3 +667,15 @@ def has_explicit_access_to_learner_pathways_for_any_context(user, *args, **kwarg constants.LEARNER_PATHWAYS_RECOMMENDATION_FEEDBACK_PERMISSION, has_learner_pathways_recommendation_feedback_access, ) + +# Grants permission to discover careers via the Learner Pathways API. +rules.add_perm( + constants.LEARNER_PATHWAYS_CAREER_DISCOVERY_PERMISSION, + has_learner_pathways_career_discovery_access, +) + +# Grants permission to assemble a pathway via the Learner Pathways API. +rules.add_perm( + constants.LEARNER_PATHWAYS_PATHWAY_PERMISSION, + has_learner_pathways_pathway_access, +) diff --git a/enterprise_access/apps/core/tests/factories.py b/enterprise_access/apps/core/tests/factories.py index ec520e94..82b6f01e 100644 --- a/enterprise_access/apps/core/tests/factories.py +++ b/enterprise_access/apps/core/tests/factories.py @@ -25,7 +25,10 @@ class UserFactory(factory.django.DjangoModelFactory): is_active = True is_staff = False is_superuser = False - lms_user_id = factory.LazyAttribute(lambda x: FAKER.pyint()) + # A sequence, not a random int: FAKER.pyint() draws from 0-9999, so a suite that + # builds a few users per test collides often enough to break any view looking a + # user up by lms_user_id. Offset past the ids tests hardcode (max 98123). + lms_user_id = factory.Sequence(lambda n: 10000000 + n) class Meta: model = User diff --git a/enterprise_access/apps/pathway_eval/__init__.py b/enterprise_access/apps/pathway_eval/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/enterprise_access/apps/pathway_eval/apps.py b/enterprise_access/apps/pathway_eval/apps.py new file mode 100644 index 00000000..c0f34514 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/apps.py @@ -0,0 +1,17 @@ +""" +App configuration for the learner pathway evaluation harness. +""" +from django.apps import AppConfig + + +class PathwayEvalConfig(AppConfig): + """ + Evaluation harness for learner pathway recommendation quality. + + This app owns *no* pipeline logic. Retrieval, translation, re-ranking and + assembly are production code elsewhere in the service; the harness only calls + them, scores what comes back, and reports. If the harness reimplements any part + of the pipeline, the harness is what gets measured. + """ + name = 'enterprise_access.apps.pathway_eval' + verbose_name = 'Learner Pathway Evaluation' diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/README.md b/enterprise_access/apps/pathway_eval/fixtures/personas/README.md new file mode 100644 index 00000000..bc15c65c --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/README.md @@ -0,0 +1,131 @@ +# Evaluation personas + +One YAML file per persona. These files are the harness's ground truth: every quality +metric is computed relative to what they claim. Authoring them is expert work, not +engineering work, and it is the long pole of the whole evaluation. + +## Schema + +```yaml +id: p001-example-slug # required, unique, kebab-case; also the filename +domain: technology # required; technology | business | healthcare | trades | ... +tier: core # core | edge (edge = deliberately thin coverage) + +inputs: # required; validated by LearningIntentRequestSerializer + selected_goals: "..." # all four are required and must be non-blank + free_text: "..." + known_context: "..." + interested_industries: "..." + +expected: + ground_truth_status: placeholder # placeholder | expert_authored + expect_no_coverage: false # true = the catalog genuinely cannot serve this + careers: + - external_id: "ETE78CD2CDFFFAC66B" # Lightcast id, REQUIRED + name: "Data Analyst Consultant" # optional, for humans + courses: + - key: "IBM+DA0101EN" # Algolia catalog course key, REQUIRED + title: "Analyzing Data with Python" # optional, for humans + note: "why an expert picks this" # optional + +catalog: + enterprise_uuid: "..." # optional until an enterprise is pinned + snapshot_date: 2026-09-09 # YAML date; when the expectations were authored + +notes: "..." # optional free text +``` + +## The four rules that will reject your file + +**1. Courses are identified by catalog key, never by title.** Duplicate titles under +different keys are one of the defects being measured, so a title cannot identify a +course. Titles are welcome *alongside* the key. + +**2. A course key is `+`, not a course-run key.** The catalog index's `key` +field holds `HarvardX+ER22.1x`, `IBM+DA0101EN`, `CodeSignal+164`. A `course-v1:...` run +key is a valid platform identifier but it appears nowhere in the index, so it can never +match a hit and would score as a miss no matter how good retrieval is. The loader +rejects run keys explicitly for this reason. + +**3. Careers are identified by Lightcast `external_id`** — `ET` followed by 16 hex +digits. Career titles are neither unique nor stable in the taxonomy index. + +**4. `expect_no_coverage: true` and a list of expected courses are contradictory.** A +persona cannot both be uncoverable and have a correct answer. + +## `ground_truth_status` — read this before adding a persona + +`placeholder` means the expectations are not expert judgement. Placeholders exist so the +harness can be exercised before ground-truth authoring finishes; they are excluded from +headline metrics. **Do not promote a persona to `expert_authored` because it looks +plausible** — only because someone qualified in that domain chose those courses. + +The one honest exception is a persona whose expectation is *absence* +(`expect_no_coverage: true`) verified directly against the index. Absence is a fact about +the catalog, not a judgement, so it can be `expert_authored` on the strength of the +verification alone. `p010-welder` is that case. + +## Where the shipped set came from + +`p001`–`p009` were imported from `Learner Rec Persona Testing.xlsx`, authored by the +product team, which records intake inputs, expected careers, expected courses and notes +from three observed runs. Expected-course *titles* were resolved to catalog course keys +against the live index on 2026-09-09; **only exact title matches were kept**. `p010` is a +deliberately-added zero-coverage case. + +Three things were deliberately *not* carried over, and each is recorded in the affected +persona's `notes`: + +- **`Product Management (Professional Certificate)`** (p005) is a `content_type:program` + record, and programs carry a null `key`. It cannot be expressed as an expected course + key. Open product question: should a pathway be able to recommend a program? +- **`Foundations of Client Care 2: ...`** (p004) could not be resolved — the title looks + truncated in the source sheet, and the catalog's Osmosis courses are named + `Client Care: ` (`OsmosisFromElsevier+CC1`..`CC6`). Needs the author to confirm. +- **Three expected careers** (`Nurse Practitioner`, `Product Manager`, `Data Analyst`) + have **no exact entry in the Lightcast jobs index** — only qualified variants + (151, 372 and 314 of them respectively). Rather than substitute a near-miss and invent + ground truth, those personas ship with `careers: []` and an explanation. This is the + same canonical-vs-colloquial mismatch as the skills vocabulary, one level up, and it + independently confirms the p005 author note "No Product Manager career". + +`p006` and `p007` have no expected courses yet and therefore report +`has_ground_truth == False`. Keep them anyway: they are the input-shape boundary cases — +`p006` is ~1,100 characters of conversational prose (which returns **zero** Algolia hits, +because the index ANDs every query word), and `p007` is ~60 characters total. Neither +failure mode is visible from the other personas. + +## Coverage of the shipped set, and what it is missing + +Six domains: technology (4), finance, engineering, healthcare, business, trades. The +technology personas are the ones that matter most and score worst — 0% recall@20 under +every configuration — because 92% of courses tagged with the "Artificial Intelligence" +subject carry no skill tags at all. + +Known gaps worth filling: no persona has more than 5 expected courses (so recall is +coarse), `p008`/`p009` have 1 each, and there is no persona for the frontline retail, +transport or allied-health roles that the catalog analysis found unservable. + +## Finding the identifiers + +Both indexes are queryable read-only with the plain search key. To find a course key: + +```bash +curl -s -X POST "https://-dsn.algolia.net/1/indexes//query" \ + -H "X-Algolia-API-Key: " -H "X-Algolia-Application-Id: " \ + -H "Content-Type: application/json" \ + -d '{"query":"data analysis","filters":"content_type:course","hitsPerPage":10, + "removeWordsIfNoResults":"allOptional", + "attributesToRetrieve":["key","title","level_type"]}' +``` + +`removeWordsIfNoResults` matters: the index ANDs every query word, so searching a full +course title (8+ words) returns **zero** hits without it. Confirm the hit's `title` +matches what you meant before recording its key — with that flag on, Algolia will always +return *something*. + +Swap the index for the jobs index and retrieve `["name","external_id"]` to find a career. + +Note that a bare index query is **not** scoped to any enterprise catalog. Once an +enterprise is pinned, expectations should be re-checked against that enterprise's scoped +catalog, because a course that exists in the index may not be in the customer's catalog. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p001-insurance-to-financial-advisor.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p001-insurance-to-financial-advisor.yaml new file mode 100644 index 00000000..b5f2e02b --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p001-insurance-to-financial-advisor.yaml @@ -0,0 +1,42 @@ +# Persona 1 — Insurance adjustor → Financial advisor +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p001-insurance-to-financial-advisor +domain: finance +tier: core + +inputs: + selected_goals: >- + Become a certified financial advisor + free_text: >- + Make more money, have more prestige + known_context: >- + 9 years of insurance sales and insurance adjusting + interested_industries: >- + Finance, accounting + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: + - external_id: "ETF27C4D9C94A7F49A" + name: "Financial Advisor" + courses: + - key: "NYIF+ECS-IBFx" + title: "Essential Career Skills for Investment Banking and Finance" + - key: "UniversityofCambridge+2122edx010" + title: "Foundations of Corporate Finance" + - key: "UniversityofCambridge+FNFM" + title: "Finance for Non-finance Professionals" + - key: "OxfordX+OXFALG01" + title: "Finance Fundamentals: Classical and Behavioural Finance" + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth. All four expected courses resolved to exact title matches in the + catalog index. Observed-run note: "Two of the 5 courses are in different languages" -- + language contamination, seen independently in personas 2 and 5 as well. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p002-student-to-biomedical-engineer.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p002-student-to-biomedical-engineer.yaml new file mode 100644 index 00000000..67c6cade --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p002-student-to-biomedical-engineer.yaml @@ -0,0 +1,45 @@ +# Persona 2 — Student → Biomedical Engineer +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p002-student-to-biomedical-engineer +domain: engineering +tier: core + +inputs: + selected_goals: >- + Become a biomedical engineer for a major pharma company. + free_text: >- + I need a job when I graduate. + known_context: >- + 2 years of undergrad study in biochemistry + interested_industries: >- + Biomedical engineering, pharmaceuticals + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: + - external_id: "ETF5E6EF44E3582007" + name: "Biomedical Engineer" + courses: + - key: "BayreuthX+ubt202bio" + title: "Biomaterials and Biofabrication: Design, Engineering and Innovation" + - key: "DelftX+SGS1x" + title: "Biomedical Equipment: Repairing and Maintaining Biomedical Devices" + - key: "IsraelX+gabi" + title: "Essentials of Genomics and Biomedical Informatics" + - key: "HarvardX+MCB63X" + title: "Principles of Biochemistry" + - key: "TUGrazX+TUGXPHARM01" + title: "Technology for Continuous Production of Medicines" + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth; all five expected courses resolved exactly. Observed-run note: "3 + careers in spanish, which are identical to the ones above them in english. When you select + one, both are selected/highlighted" -- a duplicate-record defect on the careers surface, not + a retrieval miss. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p003-advisor-to-ai-superuser.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p003-advisor-to-ai-superuser.yaml new file mode 100644 index 00000000..5017b35a --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p003-advisor-to-ai-superuser.yaml @@ -0,0 +1,45 @@ +# Persona 3 — Student success advisor → AI superuser +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p003-advisor-to-ai-superuser +domain: technology +tier: core + +inputs: + selected_goals: >- + Become really good at AI so I can get a better job in my field + free_text: >- + Everyone is raving about AI and I want to get in on the action + known_context: >- + 18 months as student success advisor at a public 4 year university + interested_industries: >- + Education, tech, management + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: + - external_id: "ETF483B1F2DAC63AB7" + name: "Student Success Advisor" + courses: + - key: "GTx+AI4E102x" + title: "Chatbots for Instruction" + - key: "edX+BCT-prompt" + title: "Try It: Prompt Engineering" + - key: "DelftX+AIIP2x" + title: "AI in Practice: Applying AI" + - key: "Microsoft+WSWAI" + title: "AI for Workplace Productivity" + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth; all four expected courses resolved exactly. Observed-run note: this + learner does not want a career change but the careers page forces one, and the returned + courses are "high level scientific courses on artificial intelligence" rather than applied + ones. Relevant measurement: 92% of courses tagged with the "Artificial Intelligence" subject + carry no skills at all, so the applied AI content that would serve this learner is largely + unreachable by skill retrieval. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p004-medical-assistant-to-nurse-practitioner.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p004-medical-assistant-to-nurse-practitioner.yaml new file mode 100644 index 00000000..d31cc16f --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p004-medical-assistant-to-nurse-practitioner.yaml @@ -0,0 +1,51 @@ +# Persona 4 — The PA → NP career advancer +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p004-medical-assistant-to-nurse-practitioner +domain: healthcare +tier: core + +inputs: + selected_goals: >- + I want to become a nurse practitioner + free_text: >- + Advancing my career + known_context: >- + I'm a medical assistant with 3 years of experience in a large clinic, primarily in + radiology. + interested_industries: >- + Nursing, healthcare + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: [] + courses: + - key: "HKPolyUx+RSCPEx102x" + title: "Interdisciplinary Management of Cardiopulmonary Health and Disease - Cardiac Focus" + - key: "SDGAcademyX+GPH001" + title: "Global Public Health" + - key: "StanfordOnline+CME.20758" + title: "To Prescribe or Not To Prescribe? Antibiotics and Outpatient Infections" + - key: "StanfordOnline+CME-34506" + title: "Identifying and Responding to Developmental Delay in Young Children." + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth, with two deliberate omissions. + + (1) No expected career is recorded: "Nurse Practitioner" has NO exact entry in the Lightcast + jobs index -- only 151 qualified variants (Family Practice NP, Orthopedic NP, ...). + Substituting one would be inventing ground truth, so the field is left empty. + + (2) One expected course, "Foundations of Client Care 2: Workplace Safety, Emergency Care, + and Infection Control -", could not be resolved; the title appears truncated in the source + sheet and the catalog's Osmosis courses are named "Client Care: " + (OsmosisFromElsevier+CC1..CC6). Needs the author to confirm which course was meant. + + Observed-run note: "all the ones that do have pathways are assistant level, not helpful to + progressing this persona's career" -- a level-progression failure, not a relevance failure. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p005-developer-to-product-manager.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p005-developer-to-product-manager.yaml new file mode 100644 index 00000000..5a00f922 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p005-developer-to-product-manager.yaml @@ -0,0 +1,48 @@ +# Persona 5 — Computer science → Product Management +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p005-developer-to-product-manager +domain: technology +tier: core + +inputs: + selected_goals: >- + I want to become a product manager at my current job + free_text: >- + I want to change careers from my current one + known_context: >- + I have been a software developer for 5 years, and I am a certified scrum master. + interested_industries: >- + technology + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: [] + courses: + - key: "USMx+ENES608.1" + title: "Product Management Fundamentals" + - key: "IPL+PL407" + title: "Agile Product Management" + - key: "CodeSignal+249" + title: "Acing the Product Management Interview" + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth, with two deliberate omissions. + + (1) No expected career: the author noted "No Product Manager career", and that is confirmed + -- "Product Manager" has NO exact entry in the Lightcast jobs index (372 qualified variants + only). "Product Owner" does exist exactly (ETB480C279F0423146) but is a different role, so + it is not substituted. + + (2) "Product Management (Professional Certificate)" is a *program*, not a course -- program + records carry a null key -- so it cannot be expressed as an expected course key. Worth + raising as a product question: should pathways be able to recommend programs? + + Observed-run note: "Courses 2 and 4 are not relevant at all, and then the last one is in + spanish." diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p006-chatty-data-analyst.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p006-chatty-data-analyst.yaml new file mode 100644 index 00000000..b087f845 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p006-chatty-data-analyst.yaml @@ -0,0 +1,49 @@ +# Persona 6 - Chatty Cathy +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p006-chatty-data-analyst +domain: technology +tier: core + +inputs: + selected_goals: >- + I want to be a better team member to my current team, maybe getting a promotion since we + might have more budget opening up soon, but I also want to bulk up my resume if something + were to happen if I got laid off (god forbid). I want to learn more about how to use the + tools I use everyday better. + free_text: >- + So many things are motivating me! Ever since I was a girl I absolutely loved learning and + school. My favorite class was English but math was a close second. In college, I majored + in Math but wasn't really sure what I was going to do with it, but I want to spend my + whole life constantly learning. + known_context: >- + I'm a Data Analyst and I've worked at Big Data Corporation LLC for 5 years with some + really amazing people who are so lovely. The first couple years were hard, but I really + found my stride when I moved teams, and I got to work on mySQL more and became a certified + database wizard (in my own opinion). + interested_industries: >- + The big data corporation is technology but I'd be open to working in any industry and any + field, I'm very flexible and I have no idea where the world might take me! + +expected: + ground_truth_status: placeholder + expect_no_coverage: false + careers: [] + courses: [] + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + **No expected courses authored yet** -- loads, but reports has_ground_truth false and must + not be scored as a miss. + + Its value is as an input-shape test case: the four fields total ~1,100 characters of + conversational prose. That directly exercises the most consequential retrieval defect found + -- the catalog index ANDs every query word, so a query built from this input returns ZERO + hits (8+ words is already 0). Keep this persona even once ground truth is authored. + + No expected career recorded: "Data Analyst" has no exact Lightcast entry (314 qualified + variants only). diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p007-brief-ml-engineer.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p007-brief-ml-engineer.yaml new file mode 100644 index 00000000..937fe8e9 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p007-brief-ml-engineer.yaml @@ -0,0 +1,37 @@ +# Persona 7 - Brief Bob +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p007-brief-ml-engineer +domain: technology +tier: core + +inputs: + selected_goals: >- + Learn + free_text: >- + Job is making me take a course + known_context: >- + machine learning engineer + interested_industries: >- + tech + +expected: + ground_truth_status: placeholder + expect_no_coverage: false + careers: + - external_id: "ETFE217EE8D3363FC5" + name: "Machine Learning Engineer" + courses: [] + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + **No expected courses authored yet** -- loads, reports has_ground_truth false, must not be + scored as a miss. + + The deliberate opposite of persona 6: four fields totalling ~60 characters ("Learn", + "tech"). Minimal input is its own failure mode -- there is almost nothing for intent + extraction to work with. Keep as a boundary case. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p008-aimless-administrative-assistant.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p008-aimless-administrative-assistant.yaml new file mode 100644 index 00000000..d5747b8d --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p008-aimless-administrative-assistant.yaml @@ -0,0 +1,45 @@ +# Persona 8 - Innapropriate and aimless assistant +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p008-aimless-administrative-assistant +domain: business +tier: edge + +inputs: + selected_goals: >- + leave this shitty job and get a better one + free_text: >- + this fucking job is forcing me to take a course + known_context: >- + I'm an administrative assistant but I want to go into a career that actually makes me some + fucking money + interested_industries: >- + anything + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: [] + courses: + - key: "FullbridgeX+Career5x" + title: "Resume, Networking, and Interview Skills" + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth: one expected course, resolved exactly. + + Two things this persona tests that no other does: hostile/profane input (verbatim from the + author -- it is real learner-input behaviour and the pipeline should degrade gracefully, not + crash or moralise), and a learner with no career direction at all ("anything"). + + No expected career recorded -- the author left it blank, appropriately, since the persona + has no target role. + + Observed-run note: "Majority don't create pathways, and the only one this person might be + qualified for (warehouse sorter) has completely random courses." That matches the measured + catalog shape: frontline and warehouse roles are among the 20 of 43 sampled careers with no + serviceable pathway. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p009-ai-engineer-upskilling.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p009-ai-engineer-upskilling.yaml new file mode 100644 index 00000000..4673c6b9 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p009-ai-engineer-upskilling.yaml @@ -0,0 +1,41 @@ +# Persona 9 - Artificial Intelligence Engineer looking to upskill +# +# Imported from "Learner Rec Persona Testing.xlsx", authored by the product team. Expected- +# course titles were resolved to catalog course keys against the live index on 2026-09-09; only +# exact title matches were kept. +id: p009-ai-engineer-upskilling +domain: technology +tier: core + +inputs: + selected_goals: >- + Be knowledgable and informed + free_text: >- + I want to continue to stay on the cutting edge of my job, and be knowledgable about new + advancements in artificial intelligence, specifically generative. + known_context: >- + I've been an Artificial Intelligence Engineer for 4 years now, and I have a degree in + computer science + interested_industries: >- + Technology + +expected: + ground_truth_status: expert_authored + expect_no_coverage: false + careers: + - external_id: "ET2C4049255C630B08" + name: "Artificial Intelligence Engineer" + courses: + - key: "MITx+AGAI" + title: "Implementing Agentic AI: Building Your Organizational Playbook" + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Authored ground truth: one expected course, resolved exactly. Only one course was recorded, + so recall on this persona is a coarse signal. + + Observed-run note: "Run 1 was run a little bit after the rest in this doc, so it's slightly + improved results. But overall ok!" -- the one persona the author judged acceptable. Useful + as a positive control. diff --git a/enterprise_access/apps/pathway_eval/fixtures/personas/p010-welder.yaml b/enterprise_access/apps/pathway_eval/fixtures/personas/p010-welder.yaml new file mode 100644 index 00000000..ab6c94b6 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/fixtures/personas/p010-welder.yaml @@ -0,0 +1,45 @@ +# Trades / edge. The diagnostic's third outcome: the catalog genuinely cannot serve +# this learner. +# +# Verified against the catalog index on 2026-09-09, two ways: +# * a "welding" text query returns 2 hits, neither about welding (a Chinese-language +# manufacturing internship and a Spanish-language image-reading course); +# * facetFilters skill_names:Welding returns exactly 1 course -- +# TsinghuaX+THU2023102608 (机械制造实习), a Chinese-language general manufacturing +# internship carrying Welding as 1 of 11 skills. +# One tangentially-tagged, non-English course is not a pathway, so expect_no_coverage +# holds. This is a content-coverage fact, not a retrieval defect. +# +# This is the persona that makes "narrow the product to domains we can serve" a +# measurable decision instead of an argument -- so unlike the other placeholders, its +# expectation IS the finding, and it is scoreable as-is. +id: p010-welder +domain: trades +tier: edge + +inputs: + selected_goals: "Get certified as a welder" + free_text: >- + I want to get into welding and eventually get certified so I can work on + structural jobs instead of general labour. + known_context: >- + Two years of general construction labour. Some hands-on experience with a + MIG welder but no certification and no formal instruction. + interested_industries: "Construction" + +expected: + ground_truth_status: expert_authored + expect_no_coverage: true + careers: + - external_id: "ET7A98E5539C2B749C" + name: "Welder" + courses: [] + +catalog: + snapshot_date: 2026-09-09 + +notes: >- + Marked expert_authored because the claim being asserted is absence, and absence was + verified directly against the index rather than judged. The career exists in the + Lightcast taxonomy, which is the point: career retrieval will succeed and course + retrieval cannot. Any pathway returned for this persona is padding. diff --git a/enterprise_access/apps/pathway_eval/harness.py b/enterprise_access/apps/pathway_eval/harness.py new file mode 100644 index 00000000..d87556bf --- /dev/null +++ b/enterprise_access/apps/pathway_eval/harness.py @@ -0,0 +1,328 @@ +""" +The harness runner: personas through the pipeline, traces out. + +Much smaller than the plan originally allowed for, because **workflow step records +already are the trace.** Every step persists its input, output, timing and failure, so +nothing here builds a tracing layer -- it orchestrates personas x runs x career modes and +records which workflow uuid belongs to which cell. + +Two career modes, and the delta between them is the point +--------------------------------------------------------- +``auto`` follows whichever career the discovery workflow ranked first. ``oracle`` forces +the persona's expected career. Running both separates two failure modes that look +identical in a single-mode run: a bad pathway because the *career* was wrong, and a bad +pathway because the *courses* were wrong. Without the oracle arm, career-selection error +is silently charged to course retrieval. + +Cost is bounded by construction +------------------------------- +A run issues paid model calls, so ``max_calls`` is counted and enforced *before* each +call rather than checked afterwards, and ``dry_run`` reports the plan while issuing none. +Both exist because the harness is the one place in this codebase that can spend real money +in a loop. +""" +import logging +from dataclasses import dataclass, field + +from enterprise_access.apps.pathways.models import CareerDiscoveryWorkflow, ExtractIntentOutput, PathwayAssemblyWorkflow +from enterprise_access.apps.workflow.exceptions import UnitOfWorkException + +logger = logging.getLogger(__name__) + +CAREER_MODE_AUTO = 'auto' +CAREER_MODE_ORACLE = 'oracle' +CAREER_MODES = (CAREER_MODE_AUTO, CAREER_MODE_ORACLE) + +# Each cell costs one career-discovery workflow plus one pathway workflow, and the latter +# includes a model call when re-ranking is enabled. +CALLS_PER_CELL = 2 + + +@dataclass +class CellResult: + """One persona in one career mode on one run.""" + + persona_id: str + career_mode: str + run_index: int + career_workflow_uuid: str = '' + pathway_workflow_uuid: str = '' + career_name: str = '' + career_external_id: str = '' + course_keys: list = field(default_factory=list) + complete: bool = False + violations: list = field(default_factory=list) + unfilled_rungs: list = field(default_factory=list) + rationale_count: int = 0 + enrichment_error: str = '' + skipped_reason: str = '' + error: str = '' + + @property + def ran(self) -> bool: + """Whether this cell actually executed a pipeline.""" + return not self.skipped_reason and not self.error + + def to_dict(self) -> dict: + """Plain dict for JSON export and for the scorers.""" + return { + 'persona_id': self.persona_id, + 'career_mode': self.career_mode, + 'run_index': self.run_index, + 'career_workflow_uuid': str(self.career_workflow_uuid or ''), + 'pathway_workflow_uuid': str(self.pathway_workflow_uuid or ''), + 'career_name': self.career_name, + 'career_external_id': self.career_external_id, + 'course_keys': list(self.course_keys), + 'complete': self.complete, + 'violations': list(self.violations), + 'unfilled_rungs': list(self.unfilled_rungs), + 'rationale_count': self.rationale_count, + 'enrichment_error': self.enrichment_error, + 'skipped_reason': self.skipped_reason, + 'error': self.error, + } + + +@dataclass +class HarnessBudget: + """ + Counts paid calls and refuses to start a cell that would exceed the limit. + + Enforced before the call, not after: a limit checked afterwards has already spent the + money it was meant to prevent. + """ + + max_calls: int | None = None + calls_made: int = 0 + + def can_afford(self, calls: int = CALLS_PER_CELL) -> bool: + """Whether ``calls`` more calls are within budget.""" + if self.max_calls is None: + return True + return self.calls_made + calls <= self.max_calls + + def charge(self, calls: int = CALLS_PER_CELL) -> None: + """Record calls as spent.""" + self.calls_made += calls + + +class PathwayHarness: + """ + Runs personas through career discovery and pathway assembly. + + Holds no scoring logic: it produces traces, and ``scoring`` turns traces into metrics. + That separation is architecture pattern 16 -- the harness owns no domain logic, and it + also owns no judgement about what the results mean. + """ + + def __init__(self, *, runs: int = 1, career_modes=CAREER_MODES, max_calls: int | None = None, + dry_run: bool = False, customer_uuid: str = '', allow_unscoped: bool = False, + rerank_enabled: bool = True, enrich_enabled: bool = True): + self.runs = runs + self.career_modes = tuple(career_modes) + self.budget = HarnessBudget(max_calls=max_calls) + self.dry_run = dry_run + self.customer_uuid = customer_uuid + self.allow_unscoped = allow_unscoped + self.rerank_enabled = rerank_enabled + self.enrich_enabled = enrich_enabled + self._last_intent: dict = {} + self._last_profile: dict = {} + + def plan(self, personas) -> list: + """ + The cells a run would execute, without executing any. + + Returned as ``CellResult`` objects with ``skipped_reason`` already filled in where + a cell cannot run, so a dry run reports exactly the shape a real run would. + """ + cells = [] + for run_index in range(1, self.runs + 1): + for persona in personas: + for career_mode in self.career_modes: + cell = CellResult( + persona_id=persona.id, career_mode=career_mode, run_index=run_index, + ) + cell.skipped_reason = self.skip_reason(persona, career_mode) + cells.append(cell) + return cells + + @staticmethod + def skip_reason(persona, career_mode: str) -> str: + """ + Why this cell cannot run, or an empty string. + + The oracle mode needs an expected career to force; a persona without one has + nothing to be an oracle about, and running it as though it did would quietly make + it a second auto-mode run. + """ + if career_mode == CAREER_MODE_ORACLE and not persona.expected_careers: + return 'no expected career to force in oracle mode' + return '' + + def run(self, personas) -> dict: + """ + Execute the persona set and return the traces. + + Returns ``cells``, ``calls_made``, ``budget_exhausted`` and ``personas_completed``. + A cell that fails is recorded and the run continues: one persona's broken + dependency should not cost the other seven. + """ + cells = [] + budget_exhausted = False + completed_personas = set() + + for cell in self.plan(personas): + persona = next(p for p in personas if p.id == cell.persona_id) + + if cell.skipped_reason: + cells.append(cell) + continue + + if self.dry_run: + cell.skipped_reason = 'dry run' + cells.append(cell) + continue + + if not self.budget.can_afford(): + # Stop starting new cells, but keep the ones already recorded. + budget_exhausted = True + cell.skipped_reason = 'max calls reached' + cells.append(cell) + continue + + self.budget.charge() + self.execute_cell(cell, persona) + cells.append(cell) + if cell.ran: + completed_personas.add(cell.persona_id) + + return { + 'cells': cells, + 'calls_made': self.budget.calls_made, + 'budget_exhausted': budget_exhausted, + 'personas_completed': len(completed_personas), + 'personas_total': len(personas), + } + + def execute_cell(self, cell: CellResult, persona) -> None: + """Run one persona in one career mode, recording the outcome on ``cell``.""" + try: + career = self.resolve_career(cell, persona) + if career is None: + return + self.assemble(cell, career) + except UnitOfWorkException as exc: + cell.error = f'{type(exc).__name__}: {exc}' + logger.warning( + 'Harness cell failed (persona=%s mode=%s run=%d): %s', + cell.persona_id, cell.career_mode, cell.run_index, exc, + ) + + def resolve_career(self, cell: CellResult, persona): + """ + Pick the career for this cell, running discovery when the mode calls for it. + + Oracle mode still runs discovery -- the ``auto`` versus ``oracle`` delta is only + meaningful if both arms paid the same intake cost, and the discovery trace is also + what says whether the expected career was retrievable at all. + """ + workflow = CareerDiscoveryWorkflow.objects.create( + input_data=CareerDiscoveryWorkflow.generate_input_dict(persona.inputs), + ) + workflow.execute() + cell.career_workflow_uuid = workflow.uuid + # Stashed so ``assemble`` can use the derived skills without re-reading the trace. + self._last_intent = self.intent_from_workflow(workflow) + # The persona's intake *is* the learner profile the rationale prompt expects. + self._last_profile = dict(persona.inputs or {}) + + candidates = workflow.career_candidates() + + if cell.career_mode == CAREER_MODE_ORACLE: + expected = {career.external_id for career in persona.expected_careers} + match = next( + (c for c in candidates if c.get('external_id') in expected), None, + ) + if match is None: + # The expected career was not retrieved at all. Recorded as a skip rather + # than fabricated, because forcing a career the pipeline cannot find would + # measure a pathway no learner could ever reach. + cell.skipped_reason = 'expected career not present in retrieved candidates' + return None + return match + + if not candidates: + cell.skipped_reason = 'career discovery returned no candidates' + return None + return candidates[0] + + def assemble(self, cell: CellResult, career) -> None: + """Run pathway assembly for the chosen career.""" + cell.career_name = career.get('name') or '' + cell.career_external_id = career.get('external_id') or '' + + intent = self._last_intent or {} + workflow = PathwayAssemblyWorkflow.objects.create( + input_data=PathwayAssemblyWorkflow.generate_input_dict( + career_name=cell.career_name, + career_skills=self.career_skill_names(career), + skills_required=intent.get('skills_required', []), + skills_preferred=intent.get('skills_preferred', []), + customer_uuid=self.customer_uuid, + allow_unscoped=self.allow_unscoped, + rerank_enabled=self.rerank_enabled, + enrich_enabled=self.enrich_enabled, + learner_profile=self._last_profile, + ), + ) + workflow.execute() + cell.pathway_workflow_uuid = workflow.uuid + + output = (workflow.output_data or {}).get('assemble_pathway_output') or {} + cell.course_keys = [course.get('key') for course in output.get('courses') or []] + cell.complete = bool(output.get('complete')) + cell.violations = list(output.get('violations') or []) + cell.unfilled_rungs = list(output.get('unfilled_rungs') or []) + + # Recorded separately from the pathway: a pathway that shipped unexplained is a + # different (and much milder) problem than one that failed to assemble. + enrichment = (workflow.output_data or {}).get('enrich_rationale_output') or {} + cell.rationale_count = len(enrichment.get('reasons') or {}) + cell.enrichment_error = enrichment.get('error') or '' + + @staticmethod + def career_skill_names(career) -> list: + """ + Read skill names off a career candidate. + + The candidate's ``skills`` are already flattened to names by + ``career_candidate_from_hit``, but a raw hit carries dicts -- both shapes are + accepted so a caller can pass either without a conversion step. + """ + names = [] + for skill in career.get('skills') or []: + if isinstance(skill, dict): + name = skill.get('name') + else: + name = skill + if isinstance(name, str) and name.strip(): + names.append(name.strip()) + return list(dict.fromkeys(names)) + + @staticmethod + def intent_from_workflow(workflow) -> dict: + """ + The derived skills, read off the discovery workflow's persisted output. + + Reading the trace rather than re-deriving costs nothing and cannot disagree with + what the pipeline actually used. Empty when unavailable: the pathway workflow + treats intent skills as additive to the career's own, so a missing intent narrows + the query rather than breaking it. + """ + output = (workflow.output_data or {}).get(ExtractIntentOutput.KEY) or {} + return { + 'skills_required': list(output.get('skills_required') or []), + 'skills_preferred': list(output.get('skills_preferred') or []), + } diff --git a/enterprise_access/apps/pathway_eval/management/__init__.py b/enterprise_access/apps/pathway_eval/management/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/enterprise_access/apps/pathway_eval/management/commands/__init__.py b/enterprise_access/apps/pathway_eval/management/commands/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/enterprise_access/apps/pathway_eval/management/commands/report_pathway_harness.py b/enterprise_access/apps/pathway_eval/management/commands/report_pathway_harness.py new file mode 100644 index 00000000..1ec8d95f --- /dev/null +++ b/enterprise_access/apps/pathway_eval/management/commands/report_pathway_harness.py @@ -0,0 +1,229 @@ +""" +Management command for the Tier 1/2/3 harness report. + +Reads exported traces rather than re-running the pipeline, so a re-score is free. That +matters: a harness run costs money, and the shape of the report has changed several times +while the underlying traces have not. +""" +import json +from pathlib import Path + +from django.core.management.base import BaseCommand, CommandError + +from enterprise_access.apps.pathway_eval.personas import PersonaValidationError, load_personas +from enterprise_access.apps.pathway_eval.scoring import ( + MIN_EXPECTED_COURSES_IN_PATHWAY, + MIN_PASSING_PERSONAS, + SPLIT_NON_TECHNOLOGY, + SPLIT_TECHNOLOGY, + regression_verdict, + score_run +) + + +def _percent(value): + """Render a ratio, or say it is unavailable rather than printing a misleading zero.""" + return f'{value:.0%}' if value is not None else 'n/a' + + +class Command(BaseCommand): + """ + Score exported harness traces into the three tiers Decision 8 defines. + + Tier 1 is pass/fail correctness, Tier 2 is the ship bar, Tier 3 is tracked and never + gating. A run that fails Tier 1 is not scored on quality at all -- a quality number + computed over structurally invalid pathways is noise. + """ + + help = 'Score exported pathway-harness traces and report the Tier 1/2/3 verdict.' + + def add_arguments(self, parser): + parser.add_argument('traces', help='Path to a run_pathway_harness --output-json file.') + parser.add_argument( + '--previous', + help='A prior traces file to compare against, for the regression bar.', + ) + parser.add_argument( + '--persona-dir', + help='Directory of persona YAML files. Must be the set the run used.', + ) + parser.add_argument( + '--min-passing', type=int, default=MIN_PASSING_PERSONAS, + help=f'Tier 2 passing-persona threshold (default: {MIN_PASSING_PERSONAS}).', + ) + parser.add_argument('--output-json', help='Write the full scored report to this path.') + + def handle(self, *args, **options): + traces = self._load_traces(Path(options['traces'])) + + try: + personas = load_personas(fixture_dir=options.get('persona_dir')) + except PersonaValidationError as exc: + raise CommandError(str(exc)) from exc + + report = score_run(personas, traces['cells'], min_passing=options['min_passing']) + report['run_config'] = traces.get('run_config') or {} + + previous_report = None + if options.get('previous'): + previous_traces = self._load_traces(Path(options['previous'])) + previous_report = score_run( + personas, previous_traces['cells'], min_passing=options['min_passing'], + ) + report['regression'] = regression_verdict(report, previous_report) + + self._render(report) + + if options.get('output_json'): + path = Path(options['output_json']) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(report, indent=2, sort_keys=True, default=str)) + self.stdout.write(f' wrote report to {path}') + + @staticmethod + def _load_traces(path): + """Read one traces file, failing clearly rather than half-way through scoring.""" + if not path.exists(): + raise CommandError(f'{path} does not exist.') + try: + payload = json.loads(path.read_text()) + except json.JSONDecodeError as exc: + raise CommandError(f'{path} is not valid JSON: {exc}') from exc + if not isinstance(payload, dict) or 'cells' not in payload: + raise CommandError(f'{path} is not a harness traces file (no "cells" key).') + return payload + + def _render(self, report): + """Print the report.""" + write = self.stdout.write + config = report.get('run_config') or {} + + write('') + write('PATHWAY HARNESS REPORT') + if config.get('customer_uuid'): + write(f' enterprise customer: {config["customer_uuid"]}') + else: + write(self.style.WARNING( + ' NOT scoped to any enterprise customer -- every number is an upper bound.' + )) + write(f' model backend: {config.get("model_backend", "unknown")} ' + f're-rank: {"on" if config.get("rerank_enabled") else "off"} ' + f'runs: {config.get("runs", "?")}') + write('=' * 78) + + self._render_tier_one(report['tier_one']) + self._render_tier_two(report['tier_two']) + self._render_personas(report['personas']) + self._render_tier_three(report['tier_three']) + self._render_regression(report.get('regression')) + + write('') + write('=' * 78) + verdict = 'MEETS THE BAR' if report['shippable'] else 'DOES NOT MEET THE BAR' + style = self.style.SUCCESS if report['shippable'] else self.style.ERROR + write(style(f' {verdict}')) + write(' Tier 3 is tracked, never gating; the verdict is Tier 1 and Tier 2 only.') + + def _render_tier_one(self, tier_one): + """Print the Tier 1 correctness verdict.""" + write = self.stdout.write + write('') + write('TIER 1 -- correctness gates (bugs, not quality)') + if tier_one['passed']: + write(self.style.SUCCESS( + f' PASS ({tier_one["cells_checked"]} cells checked)' + )) + return + write(self.style.ERROR(' FAIL -- do not read the quality numbers below as meaningful.')) + for name, rows in tier_one['failures'].items(): + write(f' {name}: {len(rows)}') + for row in rows[:5]: + write(f' {row}') + + def _render_tier_two(self, tier_two): + """Print the Tier 2 ship bar.""" + write = self.stdout.write + write('') + write('TIER 2 -- the ship bar') + write(f' rule: a persona passes if >= {MIN_EXPECTED_COURSES_IN_PATHWAY} expected ' + 'course appears in the delivered pathway') + style = self.style.SUCCESS if tier_two['passed'] else self.style.ERROR + write(style( + f' {"PASS" if tier_two["passed"] else "FAIL"} ' + f'{tier_two["passing"]} of {tier_two["scoreable"]} scoreable personas pass ' + f'(bar: {tier_two["min_passing"]})' + )) + for split in (SPLIT_TECHNOLOGY, SPLIT_NON_TECHNOLOGY): + row = tier_two['per_split'][split] + flag = ' <-- ZERO' if row['zero'] else '' + write(f' {split:<18} {row["passing"]}/{row["scoreable"]} passing{flag}') + if tier_two['zero_splits']: + write(self.style.ERROR( + ' A split scored zero. Concentrated failure is not shippable regardless ' + 'of the total, which is what an aggregate metric cannot express.' + )) + + def _render_personas(self, persona_scores): + """Print the per-persona pass/fail detail.""" + write = self.stdout.write + write('') + write(' per persona:') + for score in persona_scores: + if not score['scoreable']: + write(f' {score["persona_id"]:<22} not scoreable ' + f'(expected courses: {score["expected_course_count"]})') + continue + mark = 'PASS' if score['passed'] else 'FAIL' + style = self.style.SUCCESS if score['passed'] else self.style.ERROR + note = ' [expects no coverage]' if score['expect_no_coverage'] else '' + write(style( + f' {score["persona_id"]:<22} {mark} {score["split"]:<16} ' + f'best recall={_percent(score["best_recall"])} ' + f'cells ran={score["cells_ran"]}/{score["cells_planned"]}{note}' + )) + + def _render_tier_three(self, tier_three): + """Print the tracked-not-gated metrics.""" + write = self.stdout.write + write('') + write('TIER 3 -- tracked, not gating') + write(f' cells ran: {tier_three["cells_ran"]} ' + f'pathways completed: {tier_three["cells_complete"]} ' + f'({_percent(tier_three["completion_rate"])})') + write(f' zero-hit rate: {_percent(tier_three["zero_hit_rate"])} ' + f'unexplained pathways: {_percent(tier_three["unexplained_pathway_rate"])}') + for split in (SPLIT_TECHNOLOGY, SPLIT_NON_TECHNOLOGY): + row = tier_three['splits'][split] + write(f' {split:<18} personas={row["personas"]:<3} ' + f'mean recall={_percent(row["mean_recall"])}') + delta = tier_three['technology_delta'] + write(f' technology delta: {f"{delta:+.0%}" if delta is not None else "n/a"}') + write(' unfilled rungs: ' + ' '.join( + f'{level}={_percent(rate)}' + for level, rate in tier_three['unfilled_rung_rate'].items() + )) + modes = tier_three['career_mode_delta'] + write(f' career mode: auto={modes["auto"]} oracle={modes["oracle"]} ' + f'delta={modes["delta"]:+d} ' + '(the gap is the cost of automatic career selection, not of retrieval)') + consistency = tier_three['cross_run_consistency'] + write(f' cross-run consistency: ' + f'{_percent(consistency["mean_jaccard"])} mean Jaccard over ' + f'{consistency["pairs_compared"]} pair(s)') + + def _render_regression(self, regression): + """Print the regression verdict against the previous run.""" + write = self.stdout.write + write('') + write('REGRESSION BAR -- engineering-owned, no product input') + if regression is None: + write(' no previous run supplied; nothing to compare.') + return + if regression['passed']: + write(self.style.SUCCESS(' PASS -- no tracked metric decreased.')) + else: + write(self.style.ERROR(' FAIL -- a tracked metric decreased:')) + for row in regression['regressions']: + write(f' {row["metric"]}: {row["from"]} -> {row["to"]}') + for row in regression['improvements']: + write(f' improved {row["metric"]}: {row["from"]} -> {row["to"]}') diff --git a/enterprise_access/apps/pathway_eval/management/commands/run_pathway_harness.py b/enterprise_access/apps/pathway_eval/management/commands/run_pathway_harness.py new file mode 100644 index 00000000..c7ef6ef3 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/management/commands/run_pathway_harness.py @@ -0,0 +1,202 @@ +""" +Management command for the pathway evaluation harness. + +Issues paid model calls in volume, so ``--dry-run`` and ``--max-calls`` are first-class +rather than conveniences, and the default is a *single* run of a *single* career mode -- +the expensive shape has to be asked for explicitly. +""" +import json +from pathlib import Path + +from django.conf import settings +from django.core.management.base import BaseCommand, CommandError + +from enterprise_access.apps.pathway_eval.harness import CAREER_MODES, PathwayHarness +from enterprise_access.apps.pathway_eval.personas import PersonaValidationError, load_personas +from enterprise_access.apps.pathway_eval.retrieval_diagnostic import validate_customer_uuid +from enterprise_access.apps.pathways.course_retrieval import eval_customer_uuid + + +class Command(BaseCommand): + """ + Run the persona set through career discovery and pathway assembly. + + Produces traces, not scores. ``report_pathway_harness`` turns the exported JSON into + the Tier 1/2/3 report, so a re-score never needs a re-run -- which matters when a run + costs money. + """ + + help = ( + 'Run the learner-pathway persona set through the server-side pipeline and export ' + 'per-cell traces as JSON for scoring.' + ) + + def add_arguments(self, parser): + parser.add_argument( + '--persona-dir', + help='Directory of persona YAML files. Defaults to the bundled fixtures.', + ) + parser.add_argument( + '--persona-id', action='append', dest='persona_ids', + help='Restrict the run to this persona id. Repeatable.', + ) + parser.add_argument( + '--runs', type=int, default=1, + help='How many times to run each persona (default: 1). More than one is what ' + 'makes cross-run consistency measurable.', + ) + parser.add_argument( + '--career-mode', action='append', dest='career_modes', choices=CAREER_MODES, + help='Career selection mode. Repeatable; defaults to both, which is what ' + 'separates career-selection error from course-retrieval error.', + ) + parser.add_argument( + '--customer-uuid', + help='Enterprise customer to scope catalog searches to. Defaults to ' + 'PATHWAYS_EVAL_CUSTOMER_UUID.', + ) + parser.add_argument( + '--unscoped', action='store_true', + help='Use the plain search key rather than a secured key. Required offline, ' + 'and also requires ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH.', + ) + parser.add_argument( + '--no-rerank', action='store_true', + help='Disable the model re-rank step. This is the baseline arm of the A/B: ' + 'deterministic assembly alone still produces a valid pathway.', + ) + parser.add_argument( + '--no-enrich', action='store_true', + help='Disable the per-course rationale step. Pathways still ship, unexplained.', + ) + parser.add_argument( + '--max-calls', type=int, + help='Stop before exceeding this many workflow executions. Checked before ' + 'each cell, so the limit is never overshot.', + ) + parser.add_argument( + '--dry-run', action='store_true', + help='Report the cells that would run and issue no calls.', + ) + parser.add_argument( + '--output-json', help='Write the full per-cell traces to this path.', + ) + + def handle(self, *args, **options): + try: + customer_uuid = validate_customer_uuid( + options.get('customer_uuid') or eval_customer_uuid() or None, + ) + except ValueError as exc: + raise CommandError(str(exc)) from exc + + try: + personas = load_personas( + fixture_dir=options.get('persona_dir'), + persona_ids=options.get('persona_ids'), + ) + except PersonaValidationError as exc: + raise CommandError(str(exc)) from exc + + if not personas: + raise CommandError('No personas found; nothing to run.') + + if options['runs'] < 1: + raise CommandError('--runs must be at least 1.') + + harness = PathwayHarness( + runs=options['runs'], + career_modes=tuple(options.get('career_modes') or CAREER_MODES), + max_calls=options.get('max_calls'), + dry_run=options['dry_run'], + customer_uuid=customer_uuid or '', + allow_unscoped=options['unscoped'], + rerank_enabled=not options['no_rerank'], + enrich_enabled=not options['no_enrich'], + ) + + result = harness.run(personas) + self._render(result, options, customer_uuid) + + if options.get('output_json'): + self._write_json(result, options, customer_uuid, Path(options['output_json'])) + + def _render(self, result, options, customer_uuid): + """Print the human-readable summary.""" + write = self.stdout.write + cells = result['cells'] + + write('') + write('PATHWAY HARNESS' + (' (DRY RUN -- no calls issued)' if options['dry_run'] else '')) + write(f' personas: {result["personas_total"]} runs: {options["runs"]} ' + f'modes: {", ".join(options.get("career_modes") or CAREER_MODES)}') + if customer_uuid: + write(f' scoped to enterprise customer {customer_uuid}') + else: + write(self.style.WARNING( + ' NOT scoped to any enterprise customer -- results are an upper bound ' + 'on what a real learner sees.' + )) + if not options['no_rerank'] and not options['dry_run']: + write(' model re-rank: enabled (paid calls)') + write('=' * 78) + + for cell in cells: + self._render_cell(cell) + + write('') + write('=' * 78) + write(f' cells: {len(cells)} ran: {len([c for c in cells if c.ran])} ' + f'skipped: {len([c for c in cells if c.skipped_reason])} ' + f'errors: {len([c for c in cells if c.error])}') + write(f' workflow executions: {result["calls_made"]}') + write(f' personas completed: {result["personas_completed"]} of {result["personas_total"]}') + if result['budget_exhausted']: + write(self.style.WARNING( + ' --max-calls was reached; the remaining cells were not started.' + )) + write('') + write('This command produces traces, not a verdict. Run report_pathway_harness ' + 'on the exported JSON for the Tier 1/2/3 report.') + + def _render_cell(self, cell): + """Print one cell's line.""" + write = self.stdout.write + label = f'{cell.persona_id:<22} {cell.career_mode:<7} run={cell.run_index}' + + if cell.skipped_reason: + write(f' {label} SKIPPED ({cell.skipped_reason})') + return + if cell.error: + write(self.style.ERROR(f' {label} ERROR {cell.error}')) + return + + state = 'pathway' if cell.complete else 'no pathway' + write(f' {label} {state:<11} career={cell.career_name!r} ' + f'courses={len(cell.course_keys)}') + if cell.violations: + write(self.style.ERROR(f' TIER 1 VIOLATIONS: {"; ".join(cell.violations)}')) + if cell.unfilled_rungs: + write(f' unfilled rungs: {", ".join(cell.unfilled_rungs)}') + + def _write_json(self, result, options, customer_uuid, path): + """Export the traces for scoring.""" + path.parent.mkdir(parents=True, exist_ok=True) + payload = { + 'run_config': { + 'runs': options['runs'], + 'career_modes': list(options.get('career_modes') or CAREER_MODES), + 'customer_uuid': customer_uuid or '', + 'rerank_enabled': not options['no_rerank'], + 'enrich_enabled': not options['no_enrich'], + 'dry_run': options['dry_run'], + 'model_backend': settings.PATHWAYS_MODEL_BACKEND, + }, + 'calls_made': result['calls_made'], + 'budget_exhausted': result['budget_exhausted'], + 'personas_completed': result['personas_completed'], + 'personas_total': result['personas_total'], + 'cells': [cell.to_dict() for cell in result['cells']], + } + path.write_text(json.dumps(payload, indent=2, sort_keys=True)) + self.stdout.write(f' wrote traces to {path}') diff --git a/enterprise_access/apps/pathway_eval/management/commands/run_retrieval_diagnostic.py b/enterprise_access/apps/pathway_eval/management/commands/run_retrieval_diagnostic.py new file mode 100644 index 00000000..0a28516e --- /dev/null +++ b/enterprise_access/apps/pathway_eval/management/commands/run_retrieval_diagnostic.py @@ -0,0 +1,282 @@ +""" +Management command for the retrieval diagnostic. + +Read-only. Issues Algolia searches and writes no database records, so it is safe to +run against production indexes. +""" +import json +from pathlib import Path + +from django.core.management.base import BaseCommand, CommandError + +from enterprise_access.apps.api_client.algolia_client import AlgoliaClientError, AlgoliaSearchClient +from enterprise_access.apps.pathway_eval.personas import PersonaValidationError, load_personas +from enterprise_access.apps.pathway_eval.retrieval_diagnostic import ( + DEFAULT_TOP_N, + Outcome, + RetrievalDiagnostic, + summarize, + validate_customer_uuid +) + + +class Command(BaseCommand): + """ + Report whether expert-picked courses appear in Algolia's top N for each persona. + + The output of this command is a *decision*, not a number: it fills in the gate table + that determines whether re-ranking is aimed at the right layer. + """ + + help = ( + 'Run the learner-pathway retrieval diagnostic over the persona set and report, ' + 'per persona and per technology split, whether expected courses are retrieved.' + ) + + def add_arguments(self, parser): + parser.add_argument( + '--persona-dir', + help='Directory of persona YAML files. Defaults to the bundled fixtures.', + ) + parser.add_argument( + '--persona-id', + action='append', + dest='persona_ids', + help='Restrict the run to this persona id. Repeatable.', + ) + parser.add_argument( + '--top-n', + type=int, + default=DEFAULT_TOP_N, + help=f'How many hits count as "retrieved" (default: {DEFAULT_TOP_N}).', + ) + parser.add_argument( + '--unscoped', + action='store_true', + help=( + 'Use the plain search key instead of a secured key. This governs the ' + 'CREDENTIAL, not the filter: a secured key is vended per request from a ' + 'user token, so a management command cannot obtain one and needs this to ' + 'search at all. Also requires ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH. ' + 'Combine with --customer-uuid to scope by filter instead.' + ), + ) + parser.add_argument( + '--customer-uuid', + help=( + 'Scope catalog searches to this enterprise customer UUID. Needs no ' + 'secured key -- enterprise_customer_uuids is a facetable attribute. The ' + 'scope is verified before the run, because a wrong-but-well-formed UUID ' + 'matches nothing and would report 0%% recall rather than an error.' + ), + ) + parser.add_argument( + '--relax-query', + action='store_true', + help=( + 'Send removeWordsIfNoResults=allOptional. The catalog index ANDs every ' + 'query word, so a verbose query returns zero hits rather than poor ones. ' + 'Run the diagnostic both ways: the delta measures how much of the quality ' + 'problem is query construction rather than ranking.' + ), + ) + parser.add_argument( + '--output-json', + help='Write the full per-persona results to this path as JSON.', + ) + + def handle(self, *args, **options): + # Validated first, and deliberately before any client is built: a bad argument + # should not be masked by missing credentials. + try: + customer_uuid = validate_customer_uuid(options.get('customer_uuid')) + except ValueError as exc: + raise CommandError(str(exc)) from exc + + try: + personas = load_personas( + fixture_dir=options.get('persona_dir'), + persona_ids=options.get('persona_ids'), + ) + except PersonaValidationError as exc: + raise CommandError(str(exc)) from exc + + if not personas: + raise CommandError('No personas found; nothing to diagnose.') + + try: + diagnostic = RetrievalDiagnostic( + algolia_client=AlgoliaSearchClient(), + top_n=options['top_n'], + allow_unscoped=options['unscoped'], + relax_query=options['relax_query'], + customer_uuid=customer_uuid, + ) + scoped_courses = self._verify_scope(diagnostic) + results = diagnostic.run(personas) + except AlgoliaClientError as exc: + raise CommandError(f'Algolia is not usable: {exc}') from exc + + summary = summarize(results) + self._render(results, summary, options, scoped_courses) + + if options.get('output_json'): + self._write_json(results, summary, Path(options['output_json'])) + + def _verify_scope(self, diagnostic): + """ + Confirm the scope holds courses, and fail loudly when it does not. + + An empty scope produces a full run of zeroes that reads exactly like a genuine + retrieval failure, so it has to be an error rather than a caveat in the report. + """ + if diagnostic.customer_uuid is None: + return None + scoped_courses = diagnostic.count_scoped_courses() + if not scoped_courses: + raise CommandError( + f'Enterprise customer {diagnostic.customer_uuid} has no courses in the ' + 'catalog index. Every persona would score zero, which is not a result. ' + 'Check that this is a customer UUID and not a catalog or catalog-query UUID.' + ) + return scoped_courses + + def _render(self, results, summary, options, scoped_courses=None): + """Print the human-readable report.""" + write = self.stdout.write + top_n = options['top_n'] + + write('') + write(f'RETRIEVAL DIAGNOSTIC (top_n={top_n}, ' + f'relax_query={"on" if options["relax_query"] else "off"})') + if options.get('customer_uuid'): + write(f'Scoped to enterprise customer {options["customer_uuid"]} ' + f'({scoped_courses} courses in scope).') + if not options['relax_query']: + write( + 'Queries use the index default (every word ANDed). Re-run with ' + '--relax-query to measure how much recall is lost to query construction.' + ) + if options['unscoped'] and not options.get('customer_uuid'): + write(self.style.WARNING( + 'Catalog searches were UNSCOPED. Results are not restricted to any ' + 'enterprise catalog, so they are an upper bound on what a real learner sees.' + )) + elif options.get('customer_uuid'): + write(self.style.WARNING( + 'Scoped by FILTER, not by secured key. Adequate for a read-only ' + 'diagnostic; not a substitute for a secured key on request-scoped ' + 'production traffic, where a forgotten filter would leak catalog breadth.' + )) + write('=' * 78) + + for result in results: + self._render_persona(result, top_n) + + write('') + write('=' * 78) + write('SUMMARY') + write(f' personas: {summary["total_personas"]} ' + f'({summary["expert_authored_personas"]} expert-authored, ' + f'{summary["placeholder_personas"]} placeholder)') + + for split_name in ('expert_authored_only', 'technology', 'non_technology'): + stats = summary[split_name] + recall = stats['mean_recall_at_top_n'] + recall_text = f'{recall:.0%}' if recall is not None else 'n/a (nothing scoreable)' + write(f' {split_name:<22} personas={stats["personas"]:<3} ' + f'scoreable={stats["scoreable"]:<3} mean recall={recall_text}') + for outcome, count in stats['outcomes'].items(): + write(f' {outcome}: {count}') + + if summary['errors']: + write(self.style.ERROR(f' {len(summary["errors"])} error(s) occurred:')) + for error in summary['errors'][:10]: + write(f' {error}') + + write('') + write(self.style.WARNING( + 'GATE: record the decision (proceed / re-aim / re-scope) in project-log.md. ' + 'A green run is not the deliverable -- the logged decision is.' + )) + if summary['expert_authored_personas'] == 0: + write(self.style.ERROR( + 'No persona carries expert-authored ground truth, so this run cannot ' + 'clear the gate. It only proves the diagnostic works.' + )) + write('') + + def _render_persona(self, result, top_n): + """Print one persona's strategies, per-course verdicts and outcome.""" + write = self.stdout.write + + write('') + label = f'{result.persona_id} [{result.domain}/{result.tier}]' + if result.ground_truth_status != 'expert_authored': + label += ' (placeholder ground truth)' + write(label) + + for strategy in result.strategy_results: + if strategy.error: + write(f' {strategy.strategy:<22} ERROR: {strategy.error}') + continue + rank = strategy.best_rank + position = f'first expected at rank {rank}' if rank else 'no expected course retrieved' + write(f' {strategy.strategy:<22} {len(strategy.returned_keys):>3} hits, {position}') + write(f' query: {strategy.query[:90]!r}') + + if result.expected_course_keys: + retrieved = result.retrieved_keys + for key in result.expected_course_keys: + if key in retrieved: + mark = self.style.SUCCESS('RETRIEVED') + elif result.probe_found.get(key): + mark = self.style.WARNING('in index, not retrieved') + else: + mark = self.style.ERROR('not found by probe') + write(f' - {key:<34} {mark}') + recall = result.recall_at_top_n + if recall is not None: + write(f' recall@{top_n}: {recall:.0%}') + + if result.incidental_hits: + keys = ', '.join(sorted({hit['key'] for hit in result.incidental_hits})[:5]) + write(f' returned anyway (expected no coverage): {keys}') + + write(f' OUTCOME: {result.outcome} -- {Outcome.CONSEQUENCES[result.outcome]}') + + def _write_json(self, results, summary, path): + """Write the machine-readable trace.""" + payload = { + 'summary': summary, + 'personas': [ + { + 'persona_id': result.persona_id, + 'domain': result.domain, + 'tier': result.tier, + 'is_technology': result.is_technology, + 'ground_truth_status': result.ground_truth_status, + 'expected_course_keys': result.expected_course_keys, + 'outcome': result.outcome, + 'best_rank': result.best_rank, + 'recall_at_top_n': result.recall_at_top_n, + 'probe_found': result.probe_found, + 'incidental_hits': result.incidental_hits, + 'errors': result.errors, + 'strategies': [ + { + 'strategy': strategy.strategy, + 'query': strategy.query, + 'returned_keys': strategy.returned_keys, + 'matched_ranks': strategy.matched_ranks, + 'error': strategy.error, + } + for strategy in result.strategy_results + ], + } + for result in results + ], + } + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, indent=2, sort_keys=True)) + self.stdout.write(f'Wrote {path}') diff --git a/enterprise_access/apps/pathway_eval/personas.py b/enterprise_access/apps/pathway_eval/personas.py new file mode 100644 index 00000000..465f25a5 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/personas.py @@ -0,0 +1,446 @@ +""" +Evaluation personas and their expected results, as version-controlled data. + +A persona is one learner intake payload plus the answer an expert would give for it. +Personas are the harness's ground truth, so this module's job is less "parse YAML" than +"refuse to load ground truth that cannot be scored against". + +Three validation rules carry their weight: + +* **Inputs must satisfy the real request contract.** ``inputs`` is validated by + ``LearningIntentRequestSerializer`` itself, not a copy of its rules, so a persona that + the live endpoint would reject can never enter a run. +* **Expected courses are Algolia catalog keys, never titles.** The defect being measured + is literally "same title, different key", so a title cannot identify a course. Titles + are still allowed *alongside* the key, as a human-readable note. +* **Expected careers are Lightcast ``external_id`` values.** Career titles are neither + unique nor stable in the taxonomy index. + +The catalog key format is enforced strictly because getting it wrong is silent: a +``course-v1:...`` *run* key looks like a course identifier, but the catalog index's +``key`` field holds the course key (``HarvardX+ER22.1x``), so run keys match nothing and +score as a miss no matter how good retrieval is. +""" +import re +from datetime import date +from pathlib import Path +from typing import Any + +import attrs +import yaml +from django.conf import settings + +from enterprise_access.apps.api.serializers.learner_pathways import LearningIntentRequestSerializer +from enterprise_access.apps.pathways.content_keys import is_course_run_key, is_valid_course_key + +# Default location of the version-controlled persona set. +PERSONA_FIXTURE_DIR = Path(__file__).parent / 'fixtures' / 'personas' + +# Lightcast job identifiers, e.g. "ETEA2F329D54D4142E". +LIGHTCAST_EXTERNAL_ID_PATTERN = re.compile(r'^ET[0-9A-F]{16}$') + +# The complete persona schema. Enforced strictly, because an unrecognised key is almost +# always a misspelt one -- and a persona whose ground truth silently failed to load has no +# expected courses, so it scores 0% and reads exactly like a retrieval failure. That is the +# worst way for a typo to present in an evaluation. +PERSONA_KEYS = frozenset({ + 'id', 'domain', 'tier', 'inputs', 'expected', 'catalog', 'notes', +}) +EXPECTED_KEYS = frozenset({ + 'careers', 'courses', 'expect_no_coverage', 'ground_truth_status', +}) + +# ``tier`` marks how much catalog coverage a persona is expected to have. ``edge`` +# personas are drawn from known-thin domains on purpose. +PERSONA_TIERS = frozenset({'core', 'edge'}) + +# Whether a persona's expectations are real ground truth yet. +# +# Authoring ground truth is expert work and the long pole of the whole evaluation, so the +# harness has to run before it is finished. That makes "is this a real expectation?" a +# question reports must be able to answer mechanically -- a comment in a YAML file cannot +# keep a placeholder out of a headline recall number. +GROUND_TRUTH_EXPERT_AUTHORED = 'expert_authored' +GROUND_TRUTH_PLACEHOLDER = 'placeholder' +GROUND_TRUTH_STATUSES = frozenset({GROUND_TRUTH_EXPERT_AUTHORED, GROUND_TRUTH_PLACEHOLDER}) + + +class PersonaValidationError(Exception): + """ + Raised when a persona file cannot be loaded as scoreable ground truth. + + Always names the offending persona and field: these files are authored by hand by + people who are not looking at this code, so the message is the whole interface. + """ + + +@attrs.frozen +class ExpectedCourse: + """One course an expert says belongs in this persona's pathway.""" + + key: str + title: str | None = None + note: str | None = None + + +@attrs.frozen +class ExpectedCareer: + """One career an expert says this persona's intake should surface.""" + + external_id: str + name: str | None = None + + +@attrs.frozen +class PersonaCatalogContext: + """ + Which catalog the ground truth was authored against. + + Both fields matter for the same reason: expected courses are only meaningful + relative to one enterprise's catalog at one point in time. The taxonomy took + thousands of skill updates in a single month, so a persona with no snapshot date + cannot be told apart from one whose answers have simply gone stale. + """ + + enterprise_uuid: str | None = None + snapshot_date: date | None = None + + +@attrs.frozen +class Persona: + """ + One evaluation persona: an intake payload plus its expected results. + """ + + id: str + domain: str + tier: str + inputs: dict[str, str] + expected_careers: tuple[ExpectedCareer, ...] = () + expected_courses: tuple[ExpectedCourse, ...] = () + expect_no_coverage: bool = False + ground_truth_status: str = GROUND_TRUTH_PLACEHOLDER + catalog: PersonaCatalogContext = attrs.field(factory=PersonaCatalogContext) + notes: str | None = None + source_path: Path | None = None + + @property + def is_expert_authored(self) -> bool: + """Whether this persona's expectations may drive a reported metric.""" + return self.ground_truth_status == GROUND_TRUTH_EXPERT_AUTHORED + + @property + def is_technology(self) -> bool: + """ + Whether this persona counts toward the technology split. + + The technology / non-technology delta is the sharpest quality signal available, + so the split is derived from one declared field rather than inferred per report. + """ + return self.domain == 'technology' + + @property + def expected_course_keys(self) -> tuple[str, ...]: + return tuple(course.key for course in self.expected_courses) + + @property + def expected_career_ids(self) -> tuple[str, ...]: + return tuple(career.external_id for career in self.expected_careers) + + @property + def has_ground_truth(self) -> bool: + """ + Whether this persona can contribute to a recall metric. + + A persona marked ``expect_no_coverage`` is scoreable *because* it has no expected + courses -- absence is the expected result. A persona with neither expected courses + nor that flag is simply unfinished, and reports must be able to say so rather + than counting it as a failure. + """ + return bool(self.expected_courses) or self.expect_no_coverage + + +def _require_mapping(value: Any, persona_id: str, field_name: str) -> dict: + """Assert that a persona sub-structure is a mapping, naming it if it is not.""" + if not isinstance(value, dict): + raise PersonaValidationError( + f'Persona {persona_id!r}: {field_name} must be a mapping, got {type(value).__name__}.' + ) + return value + + +def _validate_inputs(raw_inputs: Any, persona_id: str) -> dict[str, str]: + """ + Validate ``inputs`` against the live learning-intent request contract. + + Delegating to the serializer is deliberate: if the endpoint's contract changes, every + persona fails loudly here rather than at run time, three hundred model calls in. + """ + inputs = _require_mapping(raw_inputs, persona_id, 'inputs') + serializer = LearningIntentRequestSerializer(data=inputs) + if not serializer.is_valid(): + raise PersonaValidationError( + f'Persona {persona_id!r}: inputs do not satisfy LearningIntentRequestSerializer: ' + f'{serializer.errors}' + ) + return dict(serializer.validated_data) + + +def _validate_expected_courses(raw_courses: Any, persona_id: str) -> tuple[ExpectedCourse, ...]: + """ + Coerce and validate the expected-course list, rejecting anything that is not a key. + """ + if raw_courses is None: + return () + if not isinstance(raw_courses, list): + raise PersonaValidationError( + f'Persona {persona_id!r}: expected.courses must be a list.' + ) + + courses = [] + for position, entry in enumerate(raw_courses): + location = f'expected.courses[{position}]' + + if isinstance(entry, str): + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} is a bare string ({entry!r}). ' + 'Expected courses must be given as a mapping with a "key", because course ' + 'titles are not unique -- duplicate titles under different keys are one of ' + 'the defects being measured.' + ) + + entry = _require_mapping(entry, persona_id, location) + key = (entry.get('key') or '').strip() + title = entry.get('title') + + if not key: + titled = f' (title: {title!r})' if title else '' + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} has no "key"{titled}. ' + 'Ground truth must be recorded as Algolia catalog course keys, not titles.' + ) + if is_course_run_key(key): + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} key {key!r} is a course *run* key. ' + 'The Algolia catalog index keys courses as "+" ' + '(e.g. "HarvardX+ER22.1x"), so a run key can never match a hit.' + ) + if not is_valid_course_key(key): + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} key {key!r} is not a valid catalog ' + 'course key. Expected "+", e.g. "IBM+DA0101EN".' + ) + + courses.append(ExpectedCourse( + key=key, + title=title, + note=entry.get('note'), + )) + + return tuple(courses) + + +def _validate_expected_careers(raw_careers: Any, persona_id: str) -> tuple[ExpectedCareer, ...]: + """ + Coerce and validate the expected-career list, rejecting anything that is not an + ``external_id``. + """ + if raw_careers is None: + return () + if not isinstance(raw_careers, list): + raise PersonaValidationError( + f'Persona {persona_id!r}: expected.careers must be a list.' + ) + + careers = [] + for position, entry in enumerate(raw_careers): + location = f'expected.careers[{position}]' + + if isinstance(entry, str): + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} is a bare string ({entry!r}). ' + 'Expected careers must be given as a mapping with an "external_id", ' + 'because career titles are neither unique nor stable in the taxonomy index.' + ) + + entry = _require_mapping(entry, persona_id, location) + external_id = (entry.get('external_id') or '').strip() + name = entry.get('name') + + if not external_id: + named = f' (name: {name!r})' if name else '' + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} has no "external_id"{named}. ' + 'Ground truth must be recorded as Lightcast external_ids, not career titles.' + ) + if not LIGHTCAST_EXTERNAL_ID_PATTERN.match(external_id): + raise PersonaValidationError( + f'Persona {persona_id!r}: {location} external_id {external_id!r} does not ' + 'look like a Lightcast job id (expected "ET" followed by 16 hex digits).' + ) + + careers.append(ExpectedCareer(external_id=external_id, name=name)) + + return tuple(careers) + + +def _validate_catalog_context(raw_catalog: Any, persona_id: str) -> PersonaCatalogContext: + """Validate the optional catalog-provenance block.""" + if raw_catalog is None: + return PersonaCatalogContext() + + catalog = _require_mapping(raw_catalog, persona_id, 'catalog') + snapshot_date = catalog.get('snapshot_date') + if snapshot_date is not None and not isinstance(snapshot_date, date): + raise PersonaValidationError( + f'Persona {persona_id!r}: catalog.snapshot_date must be a YAML date ' + f'(YYYY-MM-DD), got {snapshot_date!r}.' + ) + + enterprise_uuid = catalog.get('enterprise_uuid') + return PersonaCatalogContext( + enterprise_uuid=str(enterprise_uuid) if enterprise_uuid else None, + snapshot_date=snapshot_date, + ) + + +def _reject_unknown_keys(mapping: dict, allowed: frozenset, persona_id: str, where: str) -> None: + """ + Fail on any key outside the schema. + + Deliberately strict rather than forgiving. ``expected_courses`` at the top level + instead of ``expected.courses`` is a plausible mistake, and ignoring it would produce + a persona with no ground truth that scores zero -- indistinguishable from a genuine + total retrieval failure. + """ + unknown = sorted(set(mapping) - allowed) + if unknown: + raise PersonaValidationError( + f'Persona {persona_id!r}: unrecognised {where} key(s) {unknown}. ' + f'Allowed: {sorted(allowed)}. A silently ignored key would leave this persona ' + 'with no ground truth, which scores zero and looks like a retrieval failure.' + ) + + +def persona_from_dict(data: Any, source_path: Path | None = None) -> Persona: + """ + Build and validate one ``Persona`` from parsed YAML. + + Raises: + PersonaValidationError: If the persona is not scoreable ground truth. + """ + where = str(source_path) if source_path else '' + if not isinstance(data, dict): + raise PersonaValidationError(f'{where}: persona file must contain a YAML mapping.') + + persona_id = (data.get('id') or '').strip() + if not persona_id: + raise PersonaValidationError(f'{where}: persona is missing a non-empty "id".') + + domain = (data.get('domain') or '').strip() + if not domain: + raise PersonaValidationError(f'Persona {persona_id!r}: "domain" is required.') + + tier = (data.get('tier') or 'core').strip() + if tier not in PERSONA_TIERS: + raise PersonaValidationError( + f'Persona {persona_id!r}: tier {tier!r} is not one of {sorted(PERSONA_TIERS)}.' + ) + + _reject_unknown_keys(data, PERSONA_KEYS, persona_id, 'persona') + + expected = data.get('expected') or {} + expected = _require_mapping(expected, persona_id, 'expected') + _reject_unknown_keys(expected, EXPECTED_KEYS, persona_id, 'expected') + + ground_truth_status = (expected.get('ground_truth_status') or GROUND_TRUTH_PLACEHOLDER).strip() + if ground_truth_status not in GROUND_TRUTH_STATUSES: + raise PersonaValidationError( + f'Persona {persona_id!r}: expected.ground_truth_status {ground_truth_status!r} is ' + f'not one of {sorted(GROUND_TRUTH_STATUSES)}.' + ) + + expect_no_coverage = bool(expected.get('expect_no_coverage', False)) + expected_courses = _validate_expected_courses(expected.get('courses'), persona_id) + + if expect_no_coverage and expected_courses: + raise PersonaValidationError( + f'Persona {persona_id!r}: expect_no_coverage is true but {len(expected_courses)} ' + 'expected course(s) are listed. A persona cannot both be uncoverable and have ' + 'a correct answer.' + ) + + return Persona( + id=persona_id, + domain=domain, + tier=tier, + inputs=_validate_inputs(data.get('inputs'), persona_id), + expected_careers=_validate_expected_careers(expected.get('careers'), persona_id), + expected_courses=expected_courses, + expect_no_coverage=expect_no_coverage, + ground_truth_status=ground_truth_status, + catalog=_validate_catalog_context(data.get('catalog'), persona_id), + notes=data.get('notes'), + source_path=source_path, + ) + + +def load_persona_file(path: Path) -> Persona: + """ + Load and validate a single persona YAML file. + + Raises: + PersonaValidationError: If the file is unparseable or not scoreable. + """ + try: + raw = yaml.safe_load(path.read_text()) + except yaml.YAMLError as exc: + raise PersonaValidationError(f'{path}: could not parse YAML: {exc}') from exc + return persona_from_dict(raw, source_path=path) + + +def load_personas( + fixture_dir: Path | str | None = None, + persona_ids: list[str] | None = None, +) -> list[Persona]: + """ + Load every persona in ``fixture_dir``, sorted by persona id. + + Args: + fixture_dir: Directory of ``*.yaml`` persona files. Defaults to + ``settings.PATHWAY_EVAL_PERSONA_DIR`` when set, else the bundled fixtures. + persona_ids: Optional allow-list. Every requested id must exist, so a typo in a + run invocation fails instead of silently scoring a smaller set. + + Raises: + PersonaValidationError: If the directory is missing, any persona is invalid, + two personas share an id, or a requested id is absent. + """ + resolved_dir = Path( + fixture_dir or + getattr(settings, 'PATHWAY_EVAL_PERSONA_DIR', None) or + PERSONA_FIXTURE_DIR + ) + if not resolved_dir.is_dir(): + raise PersonaValidationError(f'Persona fixture directory does not exist: {resolved_dir}') + + personas: dict[str, Persona] = {} + for path in sorted(resolved_dir.glob('*.yaml')): + persona = load_persona_file(path) + if persona.id in personas: + raise PersonaValidationError( + f'Duplicate persona id {persona.id!r} in {path} ' + f'(already defined by {personas[persona.id].source_path}).' + ) + personas[persona.id] = persona + + if persona_ids is not None: + missing = [persona_id for persona_id in persona_ids if persona_id not in personas] + if missing: + raise PersonaValidationError( + f'No persona found for id(s): {", ".join(sorted(missing))}. ' + f'Available: {", ".join(sorted(personas)) or "(none)"}.' + ) + return [personas[persona_id] for persona_id in persona_ids] + + return [personas[persona_id] for persona_id in sorted(personas)] diff --git a/enterprise_access/apps/pathway_eval/retrieval_diagnostic.py b/enterprise_access/apps/pathway_eval/retrieval_diagnostic.py new file mode 100644 index 00000000..557760ef --- /dev/null +++ b/enterprise_access/apps/pathway_eval/retrieval_diagnostic.py @@ -0,0 +1,407 @@ +""" +The retrieval diagnostic: does Algolia surface the courses an expert would pick? + +This answers one question, and the answer decides where the pipeline's quality problem +actually lives: + +============================ ================================================== +Result What it means +============================ ================================================== +Expected course in the top N Retrieval works. Re-ranking is the right fix. +Present but not in the top N The query is wrong, not the ranking. Fix intent + and query construction; re-ranking would be + polish on the wrong layer. +Not findable at all Content coverage, not engineering. +============================ ================================================== + +Deliberately does **not** call Xpert. Intent extraction is a separate stage with its own +failure modes, and mixing it in here would mean a bad result could always be blamed on +the prompt. Instead the diagnostic issues several *fixed* query strategies built directly +from the persona, and reports each one. If no simple strategy retrieves the expected +course, that is a stronger finding than one prompt underperforming; if a simple strategy +does, that bounds how much intent extraction can be worth. + +Known limitation on absence +--------------------------- +A search-only Algolia key cannot enumerate an index: it has no ``browse`` ACL, and +pagination is capped at 1000 hits (the catalog index holds ~4,100 courses). So absence is +established by *probe* -- a targeted title search -- not by enumeration, and the outcome +is named ``NOT_FOUND_IN_INDEX`` rather than "not in the catalog". Turning that into a +definitive answer needs either a browse-scoped key or enterprise-catalog's +``contains_content_items``. +""" +import logging +import uuid as uuid_module +from dataclasses import dataclass, field +from typing import Any + +from enterprise_access.apps.api_client.algolia_client import AlgoliaClientError, AlgoliaSearchClient +from enterprise_access.apps.pathway_eval.personas import Persona + +logger = logging.getLogger(__name__) + +# How many hits count as "retrieved". The quality doc's diagnostic is a top-20 question. +DEFAULT_TOP_N = 20 + +# Hits requested when probing whether a specific course exists in the index at all. +PROBE_HITS_PER_PAGE = 50 + +# Free text is a paragraph; Algolia queries are not. Truncated at a word boundary. +MAX_QUERY_CHARS = 200 + +# Attributes the diagnostic needs back. Kept minimal: this runs once per persona per +# strategy and the payload is otherwise dominated by descriptions. +COURSE_ATTRIBUTES = ['key', 'title', 'level_type', 'partners'] + +COURSE_SCOPE_FILTER = 'content_type:course' + +# Scoping the diagnostic to one enterprise customer needs no secured key: +# ``enterprise_customer_uuids`` is a facetable attribute, so it can be filtered with the +# plain search key. (It is also in the index's ``unretrievableAttributes``, which hides it +# from a hit but does not block filtering or faceting on it.) That matters because a +# secured key is vended per request from a user token and so cannot be obtained by a +# management command at all -- see ``docs/references/algolia_search.md``. +CUSTOMER_SCOPE_FACET = 'enterprise_customer_uuids' + +# The catalog index ANDs every query word, and has no ``removeWordsIfNoResults`` +# configured. Measured against the production index on 2026-09-09, for a single persona's +# goal text: 4 words -> 90 hits, 5 -> 4, 6 -> 1, 8 or more -> 0. Sending +# ``removeWordsIfNoResults=allOptional`` turned the same 24-word query from 0 hits into +# 348, and a 5-word career title from 0 into 121. +# +# This matters more than it looks: a verbose query does not return *bad* results, it +# returns *none*, which is what makes the POC's retrieval ladder descend to its widest +# step. Passing this parameter is therefore a candidate fix that the diagnostic has to be +# able to measure rather than assume, so it is a run-time option and not a constant. +RELAXED_QUERY_PARAMS = {'removeWordsIfNoResults': 'allOptional'} + + +def validate_customer_uuid(customer_uuid): + """ + Return ``customer_uuid`` normalised, or raise ``ValueError``. + + Module-level so a caller can reject a bad argument *before* building an Algolia + client: a malformed UUID is an input error and should not need working credentials to + surface. + + Algolia does not error on a filter that matches nothing, so an unvalidated typo would + not fail -- it would return zero hits for every persona and report 0% recall, which is + indistinguishable from a genuine total retrieval failure. + """ + if customer_uuid is None: + return None + try: + return str(uuid_module.UUID(str(customer_uuid))) + except (ValueError, AttributeError, TypeError) as exc: + raise ValueError( + f'{customer_uuid!r} is not a UUID. An enterprise customer scope has to be a ' + 'UUID; a malformed one would silently match no courses and report 0% recall ' + 'rather than failing.' + ) from exc + + +class Outcome: + """The mutually exclusive conclusions the diagnostic can reach about a persona.""" + + NO_GROUND_TRUTH = 'no_ground_truth' + EXPECTED_NO_COVERAGE = 'expected_no_coverage' + IN_TOP_N = 'in_top_n' + NOT_IN_TOP_N = 'not_in_top_n' + NOT_FOUND_IN_INDEX = 'not_found_in_index' + + #: Maps each outcome to the decision it implies, per the gate table. + CONSEQUENCES = { + NO_GROUND_TRUTH: 'Unscoreable -- ground truth not authored yet.', + EXPECTED_NO_COVERAGE: 'Absence is the expected result; verify nothing plausible was returned.', + IN_TOP_N: 'Retrieval is fine. Re-ranking is the right fix.', + NOT_IN_TOP_N: 'Problem is upstream: intent and query construction, not ranking.', + NOT_FOUND_IN_INDEX: 'Content coverage, not engineering. Re-scope to servable domains.', + } + + +def _truncate_query(text: str) -> str: + """Shorten a query to ``MAX_QUERY_CHARS`` without splitting a word.""" + collapsed = ' '.join((text or '').split()) + if len(collapsed) <= MAX_QUERY_CHARS: + return collapsed + return collapsed[:MAX_QUERY_CHARS].rsplit(' ', 1)[0] + + +def build_query_strategies(persona: Persona) -> dict[str, str]: + """ + Build the fixed set of queries to try for one persona. + + Each is a plausible, *deterministic* stand-in for what intent extraction produces. + Reporting all of them is what separates "the index cannot surface this" from "our + particular query cannot surface this". + + Returns a mapping of strategy name to query text, skipping strategies with no text. + """ + inputs = persona.inputs + strategies = { + # What the learner literally said they want. + 'goals_only': inputs.get('selected_goals', ''), + # Goals plus the free-text elaboration: the most information available without + # a model in the loop. + 'goals_and_free_text': f"{inputs.get('selected_goals', '')} {inputs.get('free_text', '')}", + # The POC's own last-resort query, and a useful floor. + 'career_title': ' '.join( + career.name for career in persona.expected_careers if career.name + ), + } + return { + name: _truncate_query(text) + for name, text in strategies.items() + if _truncate_query(text) + } + + +@dataclass +class StrategyResult: + """The outcome of one query strategy for one persona.""" + + strategy: str + query: str + returned_keys: list[str] = field(default_factory=list) + #: Expected course key -> 1-based rank within the returned hits. + matched_ranks: dict[str, int] = field(default_factory=dict) + error: str | None = None + + @property + def best_rank(self) -> int | None: + """Rank of the first expected course retrieved, or ``None`` if none were.""" + return min(self.matched_ranks.values()) if self.matched_ranks else None + + +@dataclass +class PersonaDiagnostic: + """Everything the diagnostic concluded about one persona.""" + + persona_id: str + domain: str + tier: str + is_technology: bool + ground_truth_status: str + expected_course_keys: list[str] = field(default_factory=list) + strategy_results: list[StrategyResult] = field(default_factory=list) + #: Expected key -> whether a targeted probe found it in the index at all. + probe_found: dict[str, bool] = field(default_factory=dict) + outcome: str = Outcome.NO_GROUND_TRUTH + #: For expect_no_coverage personas: what the index returned anyway, for eyeballing. + incidental_hits: list[dict[str, Any]] = field(default_factory=list) + errors: list[str] = field(default_factory=list) + + @property + def best_rank(self) -> int | None: + """Best rank achieved by any strategy, or ``None`` if nothing was retrieved.""" + ranks = [ + result.best_rank + for result in self.strategy_results + if result.best_rank is not None + ] + return min(ranks) if ranks else None + + @property + def retrieved_keys(self) -> set[str]: + """Expected keys retrieved by at least one strategy.""" + retrieved = set() + for result in self.strategy_results: + retrieved.update(result.matched_ranks) + return retrieved + + @property + def recall_at_top_n(self) -> float | None: + """Fraction of expected courses retrieved by at least one strategy.""" + if not self.expected_course_keys: + return None + return len(self.retrieved_keys) / len(self.expected_course_keys) + + +class RetrievalDiagnostic: + """ + Runs the diagnostic over a persona set. + + Holds no pipeline logic: it issues plain Algolia searches and compares keys. The + Algolia client is injected so tests never touch the network. + """ + + def __init__( + self, + algolia_client: AlgoliaSearchClient | None = None, + top_n: int = DEFAULT_TOP_N, + allow_unscoped: bool = False, + secured_key=None, + relax_query: bool = False, + customer_uuid: str | None = None, + ): + self.client = algolia_client or AlgoliaSearchClient() + self.top_n = top_n + self.allow_unscoped = allow_unscoped + self.secured_key = secured_key + self.relax_query = relax_query + self.customer_uuid = validate_customer_uuid(customer_uuid) + + @property + def catalog_filters(self) -> str: + """The Algolia ``filters`` expression applied to every catalog query.""" + if self.customer_uuid is None: + return COURSE_SCOPE_FILTER + return f'{COURSE_SCOPE_FILTER} AND {CUSTOMER_SCOPE_FACET}:"{self.customer_uuid}"' + + def count_scoped_courses(self) -> int: + """ + How many courses the current scope actually contains. + + Worth calling before a run: a UUID that is well-formed but wrong (a catalog UUID + where a customer UUID belongs, say) returns zero, and zero courses in scope makes + every recall number meaningless rather than bad. + """ + response = self.client.search_catalog_index( + '', + secured_key=self.secured_key, + allow_unscoped=self.allow_unscoped, + filters=self.catalog_filters, + hitsPerPage=0, + ) + return response.get('nbHits', 0) + + @property + def extra_search_params(self) -> dict[str, Any]: + """Search parameters applied to every query in this run.""" + return dict(RELAXED_QUERY_PARAMS) if self.relax_query else {} + + def _search_catalog(self, query: str, hits_per_page: int) -> dict[str, Any]: + return self.client.search_catalog_index( + query, + secured_key=self.secured_key, + allow_unscoped=self.allow_unscoped, + filters=self.catalog_filters, + hitsPerPage=hits_per_page, + attributesToRetrieve=COURSE_ATTRIBUTES, + **self.extra_search_params, + ) + + def _run_strategy(self, strategy: str, query: str, expected_keys: list[str]) -> StrategyResult: + """Issue one strategy's query and record where the expected courses landed.""" + result = StrategyResult(strategy=strategy, query=query) + try: + response = self._search_catalog(query, self.top_n) + except AlgoliaClientError as exc: + result.error = str(exc) + logger.warning('Diagnostic strategy %r failed: %s', strategy, exc) + return result + + result.returned_keys = [hit.get('key') for hit in response.get('hits', []) if hit.get('key')] + expected = set(expected_keys) + for rank, key in enumerate(result.returned_keys, start=1): + if key in expected and key not in result.matched_ranks: + result.matched_ranks[key] = rank + return result + + def _probe_for_course(self, course_key: str, title: str | None) -> bool: + """ + Ask whether one specific course is findable in the index at all. + + Searches its title, because the catalog index makes ``key`` neither searchable nor + filterable. A false result means "not findable by title probe", which is weaker + than proven absence -- see the module docstring. + """ + if not title: + return False + try: + response = self._search_catalog(_truncate_query(title), PROBE_HITS_PER_PAGE) + except AlgoliaClientError as exc: + logger.warning('Probe for %r failed: %s', course_key, exc) + return False + return any(hit.get('key') == course_key for hit in response.get('hits', [])) + + def _classify(self, diagnostic: PersonaDiagnostic, persona: Persona) -> str: + """Reduce a persona's results to exactly one outcome.""" + if persona.expect_no_coverage: + return Outcome.EXPECTED_NO_COVERAGE + if not persona.expected_courses: + return Outcome.NO_GROUND_TRUTH + if diagnostic.retrieved_keys: + return Outcome.IN_TOP_N + # Nothing was retrieved. Whether that is a query problem or a coverage problem + # depends on whether the courses exist in the index at all. + if any(diagnostic.probe_found.values()): + return Outcome.NOT_IN_TOP_N + return Outcome.NOT_FOUND_IN_INDEX + + def run_for_persona(self, persona: Persona) -> PersonaDiagnostic: + """Run every query strategy for one persona and classify the result.""" + diagnostic = PersonaDiagnostic( + persona_id=persona.id, + domain=persona.domain, + tier=persona.tier, + is_technology=persona.is_technology, + ground_truth_status=persona.ground_truth_status, + expected_course_keys=list(persona.expected_course_keys), + ) + + for strategy, query in build_query_strategies(persona).items(): + result = self._run_strategy(strategy, query, diagnostic.expected_course_keys) + diagnostic.strategy_results.append(result) + if result.error: + diagnostic.errors.append(f'{strategy}: {result.error}') + + if persona.expect_no_coverage: + # There is nothing to match, so record what came back instead. A pathway + # built from these hits is padding, and a human needs to see it to agree. + for result in diagnostic.strategy_results: + for key in result.returned_keys[:5]: + diagnostic.incidental_hits.append({'strategy': result.strategy, 'key': key}) + else: + unretrieved = [ + key for key in diagnostic.expected_course_keys + if key not in diagnostic.retrieved_keys + ] + titles = {course.key: course.title for course in persona.expected_courses} + for key in unretrieved: + diagnostic.probe_found[key] = self._probe_for_course(key, titles.get(key)) + + diagnostic.outcome = self._classify(diagnostic, persona) + return diagnostic + + def run(self, personas: list[Persona]) -> list[PersonaDiagnostic]: + return [self.run_for_persona(persona) for persona in personas] + + +def summarize(diagnostics: list[PersonaDiagnostic]) -> dict[str, Any]: + """ + Aggregate per-persona results into the report the gate decision needs. + + Splits technology from non-technology because that delta is the sharpest quality + signal available, and reports placeholder personas separately because scoring a + guess as if it were expert judgement is worse than reporting nothing. + """ + + def split_stats(subset: list[PersonaDiagnostic]) -> dict[str, Any]: + scoreable = [d for d in subset if d.recall_at_top_n is not None] + recalls = [d.recall_at_top_n for d in scoreable] + ranks = [d.best_rank for d in scoreable if d.best_rank is not None] + return { + 'personas': len(subset), + 'scoreable': len(scoreable), + 'mean_recall_at_top_n': (sum(recalls) / len(recalls)) if recalls else None, + 'personas_with_any_expected_retrieved': len(ranks), + 'median_best_rank': sorted(ranks)[len(ranks) // 2] if ranks else None, + 'outcomes': { + outcome: sum(1 for d in subset if d.outcome == outcome) + for outcome in sorted({d.outcome for d in subset}) + }, + } + + expert_authored = [d for d in diagnostics if d.ground_truth_status == 'expert_authored'] + + return { + 'total_personas': len(diagnostics), + 'expert_authored_personas': len(expert_authored), + 'placeholder_personas': len(diagnostics) - len(expert_authored), + 'overall': split_stats(diagnostics), + 'expert_authored_only': split_stats(expert_authored), + 'technology': split_stats([d for d in diagnostics if d.is_technology]), + 'non_technology': split_stats([d for d in diagnostics if not d.is_technology]), + 'errors': [error for d in diagnostics for error in d.errors], + } diff --git a/enterprise_access/apps/pathway_eval/scoring.py b/enterprise_access/apps/pathway_eval/scoring.py new file mode 100644 index 00000000..a78cd4d6 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/scoring.py @@ -0,0 +1,360 @@ +""" +Deterministic scoring of harness output, in the three tiers Decision 8 defines. + +Pure computation over the traces the harness produced. No network, no model, no judgement +calls that a person has not already written down -- which is the point of putting the +threshold in configuration rather than in a reviewer's head. + +Why three tiers rather than one number +-------------------------------------- +A single aggregate recall figure is the wrong shape for a ship decision, for three reasons +that are properties of *this* evaluation rather than opinions: + +1. **Resolution.** Eight scoreable personas quantise any aggregate at 12.5 percentage + points, and several personas have exactly one expected course -- so that persona scores + 0% or 100% and nothing between. A bar of "recall >= 40%" would have increments finer + than the instrument. +2. **Averaging hides the defect.** Measured technology recall is 0% against 40% for + non-technology. An aggregate bar of 30% can be met with technology still at zero, so an + aggregate metric is structurally unable to gate on the thing that is actually broken. +3. **Recall measures agreement with the author, not learner value.** Ground truth is the + courses product thought of. High recall is good evidence; *low* recall is ambiguous -- + bad retrieval, or thin ground truth. + +So: Tier 1 gates correctness (bugs, not quality, and no product input needed), Tier 2 is +the ship bar as per-persona pass/fail plus a count, and Tier 3 is tracked but never +gating. ``expect_no_coverage`` personas invert the Tier 2 rule rather than being excluded +from it -- for them, returning nothing *is* the correct answer. +""" +import logging +from collections import defaultdict + +from enterprise_access.apps.pathways.pathway_assembly import LEVEL_ORDER, PATHWAY_SIZE + +logger = logging.getLogger(__name__) + +# Tier 2's two product-owned numbers. Named constants rather than literals because they +# are the parameters of a decision, and a decision that is hard to find is hard to revise. +MIN_EXPECTED_COURSES_IN_PATHWAY = 1 +MIN_PASSING_PERSONAS = 6 + +SPLIT_TECHNOLOGY = 'technology' +SPLIT_NON_TECHNOLOGY = 'non_technology' + + +def cell_dicts(cells): + """Accept either ``CellResult`` objects or plain dicts.""" + return [cell if isinstance(cell, dict) else cell.to_dict() for cell in cells] + + +def score_persona(persona, cells) -> dict: + """ + Score one persona across all of its cells. + + A persona passes if **any** of its cells produced a passing pathway. That is + deliberate: the cells are repeat runs and career modes of the same question, and the + oracle arm existing at all is an admission that auto-mode career selection is a + separate problem. Requiring every cell to pass would conflate the two again. + """ + persona_cells = [c for c in cell_dicts(cells) if c['persona_id'] == persona.id] + ran = [c for c in persona_cells if not c['skipped_reason'] and not c['error']] + + expected = set(persona.expected_course_keys) + per_cell = [_score_cell(persona, cell, expected) for cell in ran] + + scoreable = persona.has_ground_truth and persona.is_expert_authored + passed = any(entry['passed'] for entry in per_cell) if per_cell else False + + return { + 'persona_id': persona.id, + 'domain': persona.domain, + 'split': SPLIT_TECHNOLOGY if persona.is_technology else SPLIT_NON_TECHNOLOGY, + 'scoreable': scoreable, + 'expect_no_coverage': persona.expect_no_coverage, + 'expected_course_count': len(expected), + 'cells_planned': len(persona_cells), + 'cells_ran': len(ran), + 'passed': passed if scoreable else None, + 'best_recall': max((e['recall'] for e in per_cell), default=None), + 'cells': per_cell, + } + + +def _score_cell(persona, cell, expected) -> dict: + """Score one cell against the persona's ground truth.""" + returned = set(cell['course_keys']) + matched = expected & returned + recall = (len(matched) / len(expected)) if expected else None + + if persona.expect_no_coverage: + # Scenario: expected absence is not counted as failure. Returning nothing is the + # right answer, so the rule inverts rather than being skipped. + passed = not cell['complete'] + else: + passed = len(matched) >= MIN_EXPECTED_COURSES_IN_PATHWAY + + return { + 'career_mode': cell['career_mode'], + 'run_index': cell['run_index'], + 'complete': cell['complete'], + 'returned_count': len(returned), + 'matched_keys': sorted(matched), + 'recall': recall, + 'passed': passed, + 'violations': list(cell['violations']), + } + + +def tier_one_gates(cells) -> dict: + """ + Apply the Tier 1 correctness gates across every cell. + + Each is a bug if it fires. ``passed`` being False means the run should not be scored + at all -- a quality number computed over structurally invalid pathways is noise. + + Most of the gates live in ``pathway_assembly.validate_pathway`` and arrive here as + persisted ``violations``; this function adds only the ones that need the *run* rather + than a single pathway to see. + """ + rows = cell_dicts(cells) + ran = [c for c in rows if not c['skipped_reason'] and not c['error']] + + assembly_violations = [ + {'persona_id': c['persona_id'], 'career_mode': c['career_mode'], + 'run_index': c['run_index'], 'violations': c['violations']} + for c in ran if c['violations'] + ] + + wrong_length = [ + {'persona_id': c['persona_id'], 'returned': len(c['course_keys'])} + for c in ran + if c['complete'] and len(c['course_keys']) != PATHWAY_SIZE + ] + + # A pathway reported incomplete must carry no courses at all: a partial set returned + # to a client would render as a pathway that nobody claimed was one. + partial_pathways = [ + {'persona_id': c['persona_id'], 'returned': len(c['course_keys'])} + for c in ran + if not c['complete'] and c['course_keys'] + ] + + failures = { + 'assembly_violations': assembly_violations, + 'wrong_length': wrong_length, + 'partial_pathways_returned': partial_pathways, + } + return { + 'passed': not any(failures.values()), + 'failures': {name: rows for name, rows in failures.items() if rows}, + 'cells_checked': len(ran), + } + + +def tier_two_bar(persona_scores, *, min_passing=MIN_PASSING_PERSONAS) -> dict: + """ + The ship bar: how many scoreable personas passed, and whether a split scored zero. + + The no-zero-split rule is what an aggregate cannot express. A run where every + technology persona fails is not shippable regardless of the total, because the failure + is concentrated in a domain rather than spread thin. + """ + scoreable = [s for s in persona_scores if s['scoreable']] + passing = [s for s in scoreable if s['passed']] + + per_split = {} + for split in (SPLIT_TECHNOLOGY, SPLIT_NON_TECHNOLOGY): + split_rows = [s for s in scoreable if s['split'] == split] + split_passing = [s for s in split_rows if s['passed']] + per_split[split] = { + 'scoreable': len(split_rows), + 'passing': len(split_passing), + # A split with no scoreable personas cannot fail this rule; it has nothing to + # say either way, and treating silence as failure would block on ground truth. + 'zero': bool(split_rows) and not split_passing, + } + + zero_splits = [name for name, row in per_split.items() if row['zero']] + + return { + 'scoreable': len(scoreable), + 'passing': len(passing), + 'min_passing': min_passing, + 'count_met': len(passing) >= min_passing, + 'no_zero_split': not zero_splits, + 'zero_splits': zero_splits, + 'passed': len(passing) >= min_passing and not zero_splits, + 'per_split': per_split, + 'passing_persona_ids': sorted(s['persona_id'] for s in passing), + 'failing_persona_ids': sorted( + s['persona_id'] for s in scoreable if not s['passed'] + ), + } + + +def tier_three_metrics(persona_scores, cells) -> dict: + """ + Tracked-not-gated numbers. + + Level mix is here rather than in Tier 1 on purpose: a rung can be genuinely empty in + the catalog (``Nursing`` has no advanced course, ``Welding`` has one course in total), + so a hard level quota would fail personas for a content reason indistinguishable from + a retrieval failure -- and ``level_type`` is 19-36% unreliable at the item level, so a + gate on it would be measuring the metadata's noise. + """ + rows = cell_dicts(cells) + ran = [c for c in rows if not c['skipped_reason'] and not c['error']] + complete = [c for c in ran if c['complete']] + + recalls_by_split = defaultdict(list) + for score in persona_scores: + if score['scoreable'] and score['best_recall'] is not None: + recalls_by_split[score['split']].append(score['best_recall']) + + splits = {} + for split in (SPLIT_TECHNOLOGY, SPLIT_NON_TECHNOLOGY): + values = recalls_by_split.get(split) or [] + splits[split] = { + 'personas': len(values), + 'mean_recall': (sum(values) / len(values)) if values else None, + } + delta = None + if splits[SPLIT_TECHNOLOGY]['mean_recall'] is not None and \ + splits[SPLIT_NON_TECHNOLOGY]['mean_recall'] is not None: + delta = splits[SPLIT_TECHNOLOGY]['mean_recall'] - splits[SPLIT_NON_TECHNOLOGY]['mean_recall'] + + # A pathway that shipped unexplained is a real but much milder defect than one that + # failed to assemble, so it is tracked rather than gated. + unexplained = [c for c in complete if not c.get('rationale_count')] + + return { + 'cells_ran': len(ran), + 'cells_complete': len(complete), + 'completion_rate': (len(complete) / len(ran)) if ran else None, + 'unexplained_pathway_rate': ( + len(unexplained) / len(complete) + ) if complete else None, + 'zero_hit_rate': ( + len([c for c in ran if not c['course_keys']]) / len(ran) + ) if ran else None, + 'unfilled_rung_rate': { + level: ( + len([c for c in complete if level in c['unfilled_rungs']]) / len(complete) + ) if complete else None + for level in LEVEL_ORDER + }, + 'splits': splits, + 'technology_delta': delta, + 'career_mode_delta': _career_mode_delta(persona_scores), + 'cross_run_consistency': _cross_run_consistency(ran), + } + + +def _career_mode_delta(persona_scores) -> dict: + """ + How many personas pass in each career mode. + + The gap is the cost of automatic career selection. Charging that cost to course + retrieval is the specific mistake the oracle arm exists to prevent. + """ + counts = {} + for mode in ('auto', 'oracle'): + passing = 0 + for score in persona_scores: + if not score['scoreable']: + continue + if any(c['passed'] for c in score['cells'] if c['career_mode'] == mode): + passing += 1 + counts[mode] = passing + counts['delta'] = counts['oracle'] - counts['auto'] + return counts + + +def _cross_run_consistency(ran_cells) -> dict: + """ + Set overlap across repeat runs of the same persona and mode. + + Reported as mean Jaccard similarity over the pairs available. ``None`` when there is + only one run, which is honest: a single run says nothing about stability, and + reporting 1.0 would claim perfect consistency from no evidence. + """ + grouped = defaultdict(list) + for cell in ran_cells: + grouped[(cell['persona_id'], cell['career_mode'])].append(set(cell['course_keys'])) + + similarities = [] + for key_sets in grouped.values(): + for index, first in enumerate(key_sets): + for second in key_sets[index + 1:]: + union = first | second + if union: + similarities.append(len(first & second) / len(union)) + return { + 'pairs_compared': len(similarities), + 'mean_jaccard': (sum(similarities) / len(similarities)) if similarities else None, + } + + +def score_run(personas, cells, *, min_passing=MIN_PASSING_PERSONAS) -> dict: + """ + Score a whole harness run into the three tiers. + + Returns ``tier_one``, ``tier_two``, ``tier_three`` and the per-persona detail. + ``shippable`` requires *both* Tier 1 and Tier 2 -- Tier 3 never gates. + """ + persona_scores = [score_persona(persona, cells) for persona in personas] + tier_one = tier_one_gates(cells) + tier_two = tier_two_bar(persona_scores, min_passing=min_passing) + tier_three = tier_three_metrics(persona_scores, cells) + + return { + 'personas': persona_scores, + 'tier_one': tier_one, + 'tier_two': tier_two, + 'tier_three': tier_three, + 'shippable': tier_one['passed'] and tier_two['passed'], + } + + +def regression_verdict(current, previous) -> dict: + """ + Compare a run against its predecessor. + + The regression bar is engineering-owned and needs no product input: no Tier 2 or Tier 3 + metric may fall run-over-run without an explicit note. It is the bar with immediate + value, because the ship bar will not be met for some time -- and without it there is + nothing to steer by in between. + + Returns ``None`` when there is no previous run, rather than inventing a baseline. + """ + if not previous: + return None + + regressions = [] + improvements = [] + + def compare(name, current_value, previous_value): + if current_value is None or previous_value is None: + return + if current_value < previous_value: + regressions.append({'metric': name, 'from': previous_value, 'to': current_value}) + elif current_value > previous_value: + improvements.append({'metric': name, 'from': previous_value, 'to': current_value}) + + compare('tier_two.passing', current['tier_two']['passing'], previous['tier_two']['passing']) + compare( + 'tier_three.completion_rate', + current['tier_three']['completion_rate'], previous['tier_three']['completion_rate'], + ) + for split in (SPLIT_TECHNOLOGY, SPLIT_NON_TECHNOLOGY): + compare( + f'tier_three.{split}.mean_recall', + current['tier_three']['splits'][split]['mean_recall'], + previous['tier_three']['splits'][split]['mean_recall'], + ) + + return { + 'passed': not regressions, + 'regressions': regressions, + 'improvements': improvements, + } diff --git a/enterprise_access/apps/pathway_eval/tests/__init__.py b/enterprise_access/apps/pathway_eval/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/enterprise_access/apps/pathway_eval/tests/test_harness.py b/enterprise_access/apps/pathway_eval/tests/test_harness.py new file mode 100644 index 00000000..f7c6b663 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/tests/test_harness.py @@ -0,0 +1,424 @@ +""" +Tests for the pathway harness runner. + +The budget and dry-run tests matter more than they look: this is the only place in the +codebase that can spend real money in a loop, so "the limit is never overshot" and "a dry +run issues nothing" are correctness properties, not conveniences. +""" +import json +import tempfile +from io import StringIO +from pathlib import Path +from unittest import mock + +import yaml +from django.core.management import call_command +from django.core.management.base import CommandError +from django.test import TestCase + +from enterprise_access.apps.pathway_eval.harness import ( + CALLS_PER_CELL, + CAREER_MODE_AUTO, + CAREER_MODE_ORACLE, + CellResult, + HarnessBudget, + PathwayHarness +) +from enterprise_access.apps.pathway_eval.personas import persona_from_dict +from enterprise_access.apps.workflow.exceptions import UnitOfWorkException + +PATCH_CAREER_WORKFLOW = 'enterprise_access.apps.pathway_eval.harness.CareerDiscoveryWorkflow' +PATCH_PATHWAY_WORKFLOW = 'enterprise_access.apps.pathway_eval.harness.PathwayAssemblyWorkflow' + +CAREER_ID = 'ETE78CD2CDFFFAC66B' + + +def persona_dict(persona_id='p001-test', *, careers=(CAREER_ID,), courses=('IBM+DA0101EN',), + domain='technology', expect_no_coverage=False): + """Build a persona in the on-disk schema, with ground truth nested under ``expected``.""" + return { + 'id': persona_id, + 'domain': domain, + 'tier': 'core', + 'inputs': { + 'selected_goals': 'change careers', + 'free_text': 'I want to work with data', + 'known_context': 'analyst, five years', + 'interested_industries': 'technology', + }, + 'expected': { + 'ground_truth_status': 'expert_authored', + 'expect_no_coverage': expect_no_coverage, + 'careers': [{'external_id': c} for c in careers], + 'courses': [{'key': k} for k in courses], + }, + } + + +def make_persona(*args, **kwargs): + """Build a validated persona.""" + return persona_from_dict(persona_dict(*args, **kwargs)) + + +def fake_career_workflow(candidates, uuid='c-uuid', intent=None): + """A stand-in CareerDiscoveryWorkflow class.""" + instance = mock.Mock() + instance.uuid = uuid + instance.career_candidates.return_value = candidates + instance.output_data = {'extract_intent_output': intent or { + 'skills_required': ['SQL'], 'skills_preferred': ['Tableau'], + }} + cls = mock.Mock() + cls.objects.create.return_value = instance + cls.generate_input_dict.return_value = {} + return cls, instance + + +def fake_pathway_workflow(output, uuid='p-uuid'): + """A stand-in PathwayAssemblyWorkflow class.""" + instance = mock.Mock() + instance.uuid = uuid + instance.output_data = {'assemble_pathway_output': output} + cls = mock.Mock() + cls.objects.create.return_value = instance + cls.generate_input_dict.return_value = {} + return cls, instance + + +def pathway_output(keys=('IBM+DA0101EN',), complete=True, violations=(), unfilled=()): + return { + 'courses': [{'key': key, 'title': key} for key in keys], + 'complete': complete, + 'violations': list(violations), + 'unfilled_rungs': list(unfilled), + } + + +class TestHarnessBudget(TestCase): + """ + Tests for ``HarnessBudget``. + """ + + def test_no_limit_always_affords(self): + self.assertTrue(HarnessBudget().can_afford(1000)) + + def test_the_limit_is_checked_before_spending_not_after(self): + """A limit checked afterwards has already spent what it was meant to prevent.""" + budget = HarnessBudget(max_calls=2) + + self.assertTrue(budget.can_afford(2)) + budget.charge(2) + self.assertFalse(budget.can_afford(1)) + + def test_a_cell_that_would_overshoot_is_refused_whole(self): + budget = HarnessBudget(max_calls=3) + budget.charge(CALLS_PER_CELL) + + self.assertFalse(budget.can_afford(CALLS_PER_CELL)) + + +class TestCellResult(TestCase): + """ + Tests for ``CellResult``. + """ + + def test_a_skipped_or_errored_cell_did_not_run(self): + self.assertFalse(CellResult('p', 'auto', 1, skipped_reason='dry run').ran) + self.assertFalse(CellResult('p', 'auto', 1, error='boom').ran) + self.assertTrue(CellResult('p', 'auto', 1).ran) + + def test_uuids_serialize_as_strings(self): + cell = CellResult('p', 'auto', 1, career_workflow_uuid=None) + + self.assertEqual(cell.to_dict()['career_workflow_uuid'], '') + + +class TestHarnessPlan(TestCase): + """ + Tests for ``PathwayHarness.plan``. + """ + + def test_the_plan_is_personas_times_modes_times_runs(self): + personas = [make_persona('p001'), make_persona('p002')] + + cells = PathwayHarness(runs=3).plan(personas) + + self.assertEqual(len(cells), 2 * 2 * 3) + + def test_oracle_mode_is_skipped_for_a_persona_with_no_expected_career(self): + """ + Running it anyway would quietly make the oracle arm a second auto-mode run, and + the delta between the arms is the entire point of having two. + """ + persona = make_persona(careers=()) + + cells = PathwayHarness().plan([persona]) + + oracle = next(c for c in cells if c.career_mode == CAREER_MODE_ORACLE) + auto = next(c for c in cells if c.career_mode == CAREER_MODE_AUTO) + self.assertIn('no expected career', oracle.skipped_reason) + self.assertEqual(auto.skipped_reason, '') + + def test_a_single_mode_can_be_selected(self): + cells = PathwayHarness(career_modes=(CAREER_MODE_AUTO,)).plan([make_persona()]) + + self.assertEqual([c.career_mode for c in cells], [CAREER_MODE_AUTO]) + + +class TestHarnessRun(TestCase): + """ + Tests for ``PathwayHarness.run``. + """ + + def setUp(self): + super().setUp() + self.candidates = [{'external_id': CAREER_ID, 'name': 'Data Analyst', + 'skills': ['SQL (Programming Language)']}] + self.career_cls = None + self.pathway_cls = None + self.career_instance = None + self.pathway_instance = None + + def _run(self, harness, personas, candidates=None, output=None): + """Run the harness with both workflow classes patched, and keep the stand-ins.""" + career_cls, self.career_instance = fake_career_workflow( + candidates if candidates is not None else self.candidates, + ) + pathway_cls, self.pathway_instance = fake_pathway_workflow( + output if output is not None else pathway_output(), + ) + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls), \ + mock.patch(PATCH_PATHWAY_WORKFLOW, pathway_cls): + self.career_cls = career_cls + self.pathway_cls = pathway_cls + return harness.run(personas) + + def test_both_career_modes_run_and_are_recorded(self): + """Scenario: Both career modes run.""" + result = self._run(PathwayHarness(), [make_persona()]) + + modes = {c.career_mode for c in result['cells'] if c.ran} + self.assertEqual(modes, {CAREER_MODE_AUTO, CAREER_MODE_ORACLE}) + + def test_a_dry_run_makes_no_calls(self): + """Scenario: A dry run makes no paid calls.""" + result = self._run(PathwayHarness(dry_run=True), [make_persona()]) + + self.career_cls.objects.create.assert_not_called() + self.pathway_cls.objects.create.assert_not_called() + self.assertEqual(result['calls_made'], 0) + self.assertTrue(all(c.skipped_reason for c in result['cells'])) + + def test_cost_is_bounded_and_the_shortfall_is_reported(self): + """Scenario: Cost is bounded.""" + personas = [make_persona(f'p00{i}') for i in range(1, 4)] + + result = self._run( + PathwayHarness(max_calls=CALLS_PER_CELL * 2, career_modes=(CAREER_MODE_AUTO,)), + personas, + ) + + self.assertTrue(result['budget_exhausted']) + self.assertEqual(result['calls_made'], CALLS_PER_CELL * 2) + self.assertEqual(result['personas_completed'], 2) + self.assertEqual(result['personas_total'], 3) + + def test_oracle_mode_forces_the_expected_career(self): + candidates = [ + {'external_id': 'ETOTHER0000000000', 'name': 'Wrong Career', 'skills': ['X']}, + {'external_id': CAREER_ID, 'name': 'Data Analyst', 'skills': ['SQL']}, + ] + + result = self._run( + PathwayHarness(career_modes=(CAREER_MODE_ORACLE,)), [make_persona()], candidates, + ) + + cell = result['cells'][0] + self.assertEqual(cell.career_external_id, CAREER_ID) + + def test_auto_mode_follows_the_top_ranked_career(self): + candidates = [ + {'external_id': 'ETOTHER0000000000', 'name': 'Top Career', 'skills': ['X']}, + {'external_id': CAREER_ID, 'name': 'Data Analyst', 'skills': ['SQL']}, + ] + + result = self._run( + PathwayHarness(career_modes=(CAREER_MODE_AUTO,)), [make_persona()], candidates, + ) + + self.assertEqual(result['cells'][0].career_name, 'Top Career') + + def test_an_unretrievable_expected_career_is_skipped_not_fabricated(self): + """ + Forcing a career the pipeline cannot find would measure a pathway no learner + could ever reach. + """ + candidates = [{'external_id': 'ETOTHER0000000000', 'name': 'Other', 'skills': ['X']}] + + result = self._run( + PathwayHarness(career_modes=(CAREER_MODE_ORACLE,)), [make_persona()], candidates, + ) + + self.assertIn('not present in retrieved candidates', result['cells'][0].skipped_reason) + self.pathway_cls.objects.create.assert_not_called() + + def test_no_career_candidates_skips_assembly(self): + result = self._run( + PathwayHarness(career_modes=(CAREER_MODE_AUTO,)), [make_persona()], candidates=[], + ) + + self.assertIn('no candidates', result['cells'][0].skipped_reason) + + def test_the_derived_intent_is_read_off_the_discovery_trace(self): + """Reading the trace cannot disagree with what the pipeline actually used.""" + self._run(PathwayHarness(career_modes=(CAREER_MODE_AUTO,)), [make_persona()]) + + kwargs = self.pathway_cls.generate_input_dict.call_args.kwargs + self.assertEqual(kwargs['skills_required'], ['SQL']) + self.assertEqual(kwargs['skills_preferred'], ['Tableau']) + + def test_the_pathway_outcome_is_recorded_on_the_cell(self): + result = self._run( + PathwayHarness(career_modes=(CAREER_MODE_AUTO,)), [make_persona()], + output=pathway_output(keys=('A+1', 'B+2'), unfilled=['Advanced']), + ) + + cell = result['cells'][0] + self.assertEqual(cell.course_keys, ['A+1', 'B+2']) + self.assertTrue(cell.complete) + self.assertEqual(cell.unfilled_rungs, ['Advanced']) + + def test_tier_one_violations_survive_onto_the_cell(self): + result = self._run( + PathwayHarness(career_modes=(CAREER_MODE_AUTO,)), [make_persona()], + output=pathway_output(violations=['3 courses from P1 exceeds the cap of 2']), + ) + + self.assertEqual(len(result['cells'][0].violations), 1) + + def test_a_failing_cell_does_not_abandon_the_run(self): + """One persona's broken dependency should not cost the other seven.""" + career_cls, _ = fake_career_workflow(self.candidates) + career_cls.objects.create.return_value.execute.side_effect = [ + UnitOfWorkException('boom'), None, + ] + pathway_cls, _ = fake_pathway_workflow(pathway_output()) + + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls), \ + mock.patch(PATCH_PATHWAY_WORKFLOW, pathway_cls): + result = PathwayHarness(career_modes=(CAREER_MODE_AUTO,)).run( + [make_persona('p001'), make_persona('p002')], + ) + + self.assertTrue(result['cells'][0].error) + self.assertTrue(result['cells'][1].ran) + + +class TestCareerSkillNames(TestCase): + """ + Tests for ``PathwayHarness.career_skill_names``. + """ + + def test_both_flattened_names_and_raw_dicts_are_accepted(self): + self.assertEqual( + PathwayHarness.career_skill_names({'skills': ['Welding']}), ['Welding'], + ) + self.assertEqual( + PathwayHarness.career_skill_names({'skills': [{'name': 'Welding'}]}), ['Welding'], + ) + + def test_blanks_and_duplicates_are_dropped(self): + names = PathwayHarness.career_skill_names( + {'skills': ['Welding', ' Welding ', '', None, {'name': ''}]}, + ) + + self.assertEqual(names, ['Welding']) + + def test_a_career_with_no_skills_yields_an_empty_list(self): + """Two thirds of Lightcast careers carry no skills.""" + self.assertEqual(PathwayHarness.career_skill_names({}), []) + + +class TestRunPathwayHarnessCommand(TestCase): + """ + Tests for the ``run_pathway_harness`` command. + + Writes its own persona fixtures rather than using the shipped set: those are real + product-authored ground truth that changes as it is edited, and coupling command tests + to a particular persona id makes those edits look like command regressions. + """ + + def setUp(self): + super().setUp() + self._tmpdir = tempfile.TemporaryDirectory() + self.addCleanup(self._tmpdir.cleanup) + self.persona_dir = Path(self._tmpdir.name) + (self.persona_dir / 'p001-test.yaml').write_text( + yaml.safe_dump(persona_dict('p001-test'), sort_keys=False), + ) + + def call(self, **kwargs): + stdout = StringIO() + kwargs.setdefault('persona_dir', str(self.persona_dir)) + call_command('run_pathway_harness', stdout=stdout, **kwargs) + return stdout.getvalue() + + def test_a_dry_run_reports_the_plan_without_calling_anything(self): + career_cls, _ = fake_career_workflow([]) + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls): + output = self.call(dry_run=True) + + self.assertIn('DRY RUN', output) + self.assertIn('p001-test', output) + career_cls.objects.create.assert_not_called() + + def test_an_unscoped_run_is_flagged(self): + career_cls, _ = fake_career_workflow([]) + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls): + output = self.call(dry_run=True) + + self.assertIn('NOT scoped', output) + + def test_a_scoped_run_names_the_customer(self): + career_cls, _ = fake_career_workflow([]) + uuid = '417306cb-b24a-4d06-b83c-fb2a61d7fb96' + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls): + output = self.call(dry_run=True, customer_uuid=uuid) + + self.assertIn(uuid, output) + + def test_a_malformed_customer_uuid_is_a_command_error(self): + with self.assertRaisesRegex(CommandError, 'not a UUID'): + self.call(dry_run=True, customer_uuid='2u') + + def test_zero_runs_is_a_command_error(self): + with self.assertRaisesRegex(CommandError, 'at least 1'): + self.call(dry_run=True, runs=0) + + def test_traces_are_exported_for_scoring(self): + career_cls, _ = fake_career_workflow( + [{'external_id': CAREER_ID, 'name': 'Data Analyst', 'skills': ['SQL']}], + ) + pathway_cls, _ = fake_pathway_workflow(pathway_output()) + + with tempfile.TemporaryDirectory() as tmpdir: + path = Path(tmpdir) / 'nested' / 'traces.json' + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls), \ + mock.patch(PATCH_PATHWAY_WORKFLOW, pathway_cls): + self.call(output_json=str(path)) + payload = json.loads(path.read_text()) + + self.assertIn('cells', payload) + self.assertIn('run_config', payload) + self.assertEqual(payload['cells'][0]['persona_id'], 'p001-test') + + def test_the_command_says_it_produces_traces_not_a_verdict(self): + """ + The separation is the point: a re-score must never need a re-run, because a run + costs money. + """ + career_cls, _ = fake_career_workflow([]) + with mock.patch(PATCH_CAREER_WORKFLOW, career_cls): + output = self.call(dry_run=True) + + self.assertIn('report_pathway_harness', output) diff --git a/enterprise_access/apps/pathway_eval/tests/test_personas.py b/enterprise_access/apps/pathway_eval/tests/test_personas.py new file mode 100644 index 00000000..4fa45655 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/tests/test_personas.py @@ -0,0 +1,398 @@ +""" +Tests for persona fixture loading and validation. +""" +import tempfile +from datetime import date +from pathlib import Path + +import ddt +import yaml +from django.test import TestCase, override_settings + +from enterprise_access.apps.pathway_eval.personas import ( + GROUND_TRUTH_EXPERT_AUTHORED, + GROUND_TRUTH_PLACEHOLDER, + PERSONA_FIXTURE_DIR, + PersonaValidationError, + load_persona_file, + load_personas, + persona_from_dict +) + +VALID_INPUTS = { + 'selected_goals': 'Move into a data analyst role', + 'free_text': 'I do spreadsheet reporting and want to work with real databases.', + 'known_context': 'Comfortable with Excel, no programming background.', + 'interested_industries': 'Finance and Insurance', +} + + +def make_persona_dict(**overrides): + """A minimal valid persona payload, with ``expected`` merged rather than replaced.""" + persona = { + 'id': 'p999-test', + 'domain': 'technology', + 'tier': 'core', + 'inputs': dict(VALID_INPUTS), + 'expected': { + 'ground_truth_status': GROUND_TRUTH_EXPERT_AUTHORED, + 'careers': [{'external_id': 'ETE78CD2CDFFFAC66B', 'name': 'Data Analyst Consultant'}], + 'courses': [{'key': 'IBM+DA0101EN', 'title': 'Analyzing Data with Python'}], + }, + } + expected_overrides = overrides.pop('expected', None) + persona.update(overrides) + if expected_overrides is not None: + persona['expected'] = {**persona['expected'], **expected_overrides} + return persona + + +@ddt.ddt +class TestPersonaFromDict(TestCase): + """ + Tests for ``persona_from_dict`` validation. + """ + + def test_a_persona_round_trips(self): + """Scenario: A persona round-trips.""" + persona = persona_from_dict(make_persona_dict()) + + self.assertEqual(persona.id, 'p999-test') + self.assertEqual(persona.domain, 'technology') + self.assertEqual(persona.tier, 'core') + # Inputs come back as the serializer validated them, so they are directly + # postable to the learning-intent endpoint. + self.assertEqual(persona.inputs, VALID_INPUTS) + self.assertEqual(persona.expected_course_keys, ('IBM+DA0101EN',)) + self.assertEqual(persona.expected_career_ids, ('ETE78CD2CDFFFAC66B',)) + self.assertTrue(persona.is_technology) + self.assertTrue(persona.has_ground_truth) + self.assertTrue(persona.is_expert_authored) + + # -- the schema is closed, because a misspelt key scores zero --------------------- + + @ddt.data( + 'expected_courses', + 'expected_careers', + 'ground_truth_status', + 'expcted', + ) + def test_an_unrecognised_top_level_key_is_rejected(self, bad_key): + """ + ``expected_courses`` at the top level instead of ``expected.courses`` is a + plausible mistake. Ignoring it would leave the persona with no ground truth, which + scores 0% and reads exactly like a total retrieval failure. + """ + payload = make_persona_dict() + payload[bad_key] = [] + + with self.assertRaisesRegex(PersonaValidationError, 'unrecognised persona key'): + persona_from_dict(payload) + + def test_an_unrecognised_expected_key_is_rejected(self): + payload = make_persona_dict() + payload['expected']['course'] = [{'key': 'IBM+DA0101EN'}] + + with self.assertRaisesRegex(PersonaValidationError, 'unrecognised expected key'): + persona_from_dict(payload) + + def test_the_documented_schema_is_accepted_in_full(self): + """Guards against the closed schema being narrower than the shipped fixtures.""" + payload = make_persona_dict( + catalog={'enterprise_uuid': '417306cb-b24a-4d06-b83c-fb2a61d7fb96'}, + notes='a note', + ) + + persona = persona_from_dict(payload) + + self.assertEqual(persona.notes, 'a note') + + # -- inputs must satisfy the real request contract -------------------------------- + + @ddt.data( + 'selected_goals', + 'free_text', + 'known_context', + 'interested_industries', + ) + def test_missing_required_input_field_is_rejected(self, field_name): + inputs = dict(VALID_INPUTS) + del inputs[field_name] + + with self.assertRaisesRegex(PersonaValidationError, 'LearningIntentRequestSerializer'): + persona_from_dict(make_persona_dict(inputs=inputs)) + + @ddt.data('', ' ') + def test_blank_input_field_is_rejected(self, blank_value): + inputs = dict(VALID_INPUTS, free_text=blank_value) + + with self.assertRaisesRegex(PersonaValidationError, 'LearningIntentRequestSerializer'): + persona_from_dict(make_persona_dict(inputs=inputs)) + + @ddt.data(None, 'a string', ['a', 'list']) + def test_non_mapping_inputs_is_rejected(self, inputs): + with self.assertRaisesRegex(PersonaValidationError, 'inputs must be a mapping'): + persona_from_dict(make_persona_dict(inputs=inputs)) + + # -- ground truth uses identifiers, not titles ------------------------------------ + + def test_course_given_as_a_title_only_is_rejected_naming_the_entry(self): + """Scenario: Ground truth uses identifiers not titles.""" + persona_dict = make_persona_dict( + expected={'courses': [{'title': 'Analyzing Data with Python'}]}, + ) + + with self.assertRaises(PersonaValidationError) as ctx: + persona_from_dict(persona_dict) + + message = str(ctx.exception) + self.assertIn('expected.courses[0]', message) + self.assertIn('Analyzing Data with Python', message) + self.assertIn('not titles', message) + + def test_course_given_as_a_bare_string_is_rejected(self): + persona_dict = make_persona_dict( + expected={'courses': ['Analyzing Data with Python']}, + ) + + with self.assertRaises(PersonaValidationError) as ctx: + persona_from_dict(persona_dict) + + self.assertIn('bare string', str(ctx.exception)) + self.assertIn('Analyzing Data with Python', str(ctx.exception)) + + def test_course_run_key_is_rejected_with_an_explanation(self): + """ + A ``course-v1:`` run key is the plausible-looking wrong answer: valid elsewhere + on the platform, absent from the catalog index, and therefore a guaranteed miss. + """ + persona_dict = make_persona_dict( + expected={'courses': [{'key': 'course-v1:HarvardX+ER22.1x+2T2019'}]}, + ) + + with self.assertRaises(PersonaValidationError) as ctx: + persona_from_dict(persona_dict) + + message = str(ctx.exception) + self.assertIn('course *run* key', message) + self.assertIn('+', message) + + @ddt.data( + 'IBM+DA0101EN', + 'HarvardX+ER22.1x', + 'CodeSignal+164', + 'MGH_Institute+MGH-RN101', + 'StanfordOnline+SOM-YCME0045', + ) + def test_real_catalog_course_keys_are_accepted(self, course_key): + """These are verbatim ``key`` values observed in the catalog index.""" + persona = persona_from_dict( + make_persona_dict(expected={'courses': [{'key': course_key}]}), + ) + + self.assertEqual(persona.expected_course_keys, (course_key,)) + + @ddt.data('no-plus-sign', '+missing-org', 'missing-number+', 'has space+X1') + def test_malformed_course_keys_are_rejected(self, course_key): + with self.assertRaisesRegex(PersonaValidationError, 'not a valid catalog'): + persona_from_dict(make_persona_dict(expected={'courses': [{'key': course_key}]})) + + def test_career_given_as_a_name_only_is_rejected(self): + persona_dict = make_persona_dict( + expected={'careers': [{'name': 'Data Analyst'}]}, + ) + + with self.assertRaises(PersonaValidationError) as ctx: + persona_from_dict(persona_dict) + + message = str(ctx.exception) + self.assertIn('expected.careers[0]', message) + self.assertIn('Data Analyst', message) + self.assertIn('not career titles', message) + + @ddt.data('Data Analyst', 'ET123', 'etea2f329d54d4142e', 'XXEA2F329D54D4142E') + def test_malformed_career_ids_are_rejected(self, external_id): + with self.assertRaises(PersonaValidationError): + persona_from_dict(make_persona_dict(expected={'careers': [{'external_id': external_id}]})) + + # -- known-uncoverable personas --------------------------------------------------- + + def test_expect_no_coverage_is_preserved_and_scoreable(self): + """Scenario: Known-uncoverable personas are marked.""" + persona = persona_from_dict(make_persona_dict( + expected={'expect_no_coverage': True, 'courses': []}, + )) + + self.assertTrue(persona.expect_no_coverage) + self.assertEqual(persona.expected_courses, ()) + # Absence *is* the expected answer, so this persona is scoreable with no + # expected courses -- scorers must not treat it as unfinished. + self.assertTrue(persona.has_ground_truth) + + def test_expect_no_coverage_with_expected_courses_is_contradictory(self): + with self.assertRaisesRegex(PersonaValidationError, 'cannot both be uncoverable'): + persona_from_dict(make_persona_dict( + expected={'expect_no_coverage': True, 'courses': [{'key': 'IBM+DA0101EN'}]}, + )) + + def test_persona_without_courses_or_the_flag_is_unfinished_not_invalid(self): + """ + Ground-truth authoring is unfinished by design at this stage. Such a persona must + load -- so it can be listed and chased -- while reporting no ground truth. + """ + persona = persona_from_dict(make_persona_dict( + expected={'expect_no_coverage': False, 'courses': []}, + )) + + self.assertFalse(persona.has_ground_truth) + + # -- placeholder vs expert-authored ----------------------------------------------- + + def test_ground_truth_status_defaults_to_placeholder(self): + persona_dict = make_persona_dict() + del persona_dict['expected']['ground_truth_status'] + + persona = persona_from_dict(persona_dict) + + self.assertEqual(persona.ground_truth_status, GROUND_TRUTH_PLACEHOLDER) + self.assertFalse(persona.is_expert_authored) + + def test_unknown_ground_truth_status_is_rejected(self): + with self.assertRaisesRegex(PersonaValidationError, 'ground_truth_status'): + persona_from_dict(make_persona_dict(expected={'ground_truth_status': 'probably-fine'})) + + # -- required identity fields ----------------------------------------------------- + + @ddt.data( + ({'id': ''}, 'non-empty "id"'), + ({'domain': ''}, '"domain" is required'), + ({'tier': 'medium'}, 'is not one of'), + ) + @ddt.unpack + def test_identity_field_validation(self, overrides, expected_message): + persona_dict = make_persona_dict(**overrides) + + with self.assertRaisesRegex(PersonaValidationError, expected_message): + persona_from_dict(persona_dict) + + def test_non_mapping_persona_is_rejected(self): + with self.assertRaisesRegex(PersonaValidationError, 'must contain a YAML mapping'): + persona_from_dict(['not', 'a', 'mapping']) + + # -- catalog provenance ----------------------------------------------------------- + + def test_catalog_context_is_parsed(self): + persona = persona_from_dict(make_persona_dict(catalog={ + 'enterprise_uuid': '11111111-2222-3333-4444-555555555555', + 'snapshot_date': date(2026, 9, 9), + })) + + self.assertEqual(persona.catalog.enterprise_uuid, '11111111-2222-3333-4444-555555555555') + self.assertEqual(persona.catalog.snapshot_date, date(2026, 9, 9)) + + def test_string_snapshot_date_is_rejected(self): + """YAML parses bare ``2026-09-09`` as a date; a quoted string is an authoring slip.""" + with self.assertRaisesRegex(PersonaValidationError, 'snapshot_date must be a YAML date'): + persona_from_dict(make_persona_dict(catalog={'snapshot_date': '2026-09-09'})) + + +class TestLoadPersonas(TestCase): + """ + Tests for loading persona files off disk. + """ + + def _write_persona(self, directory, filename, persona_dict): + path = Path(directory) / filename + path.write_text(yaml.safe_dump(persona_dict, sort_keys=False)) + return path + + def test_load_persona_file_records_its_source(self): + with tempfile.TemporaryDirectory() as tmpdir: + path = self._write_persona(tmpdir, 'p999-test.yaml', make_persona_dict()) + + persona = load_persona_file(path) + + self.assertEqual(persona.source_path, path) + + def test_unparseable_yaml_names_the_file(self): + with tempfile.TemporaryDirectory() as tmpdir: + path = Path(tmpdir) / 'broken.yaml' + path.write_text('id: p1\n bad: indentation\n:::\n') + + with self.assertRaisesRegex(PersonaValidationError, 'could not parse YAML'): + load_persona_file(path) + + def test_duplicate_persona_ids_are_rejected(self): + with tempfile.TemporaryDirectory() as tmpdir: + self._write_persona(tmpdir, 'a.yaml', make_persona_dict()) + self._write_persona(tmpdir, 'b.yaml', make_persona_dict()) + + with self.assertRaisesRegex(PersonaValidationError, 'Duplicate persona id'): + load_personas(fixture_dir=tmpdir) + + def test_missing_directory_is_rejected(self): + with self.assertRaisesRegex(PersonaValidationError, 'does not exist'): + load_personas(fixture_dir='/nonexistent/persona/dir') + + def test_persona_ids_filter_requires_every_requested_id(self): + with tempfile.TemporaryDirectory() as tmpdir: + self._write_persona(tmpdir, 'a.yaml', make_persona_dict(id='p001-a')) + + with self.assertRaisesRegex(PersonaValidationError, 'p002-typo'): + load_personas(fixture_dir=tmpdir, persona_ids=['p001-a', 'p002-typo']) + + def test_persona_ids_filter_preserves_requested_order(self): + with tempfile.TemporaryDirectory() as tmpdir: + self._write_persona(tmpdir, 'a.yaml', make_persona_dict(id='p001-a')) + self._write_persona(tmpdir, 'b.yaml', make_persona_dict(id='p002-b')) + + personas = load_personas(fixture_dir=tmpdir, persona_ids=['p002-b', 'p001-a']) + + self.assertEqual([p.id for p in personas], ['p002-b', 'p001-a']) + + def test_settings_override_selects_the_persona_directory(self): + with tempfile.TemporaryDirectory() as tmpdir: + self._write_persona(tmpdir, 'a.yaml', make_persona_dict(id='p001-from-settings')) + + with override_settings(PATHWAY_EVAL_PERSONA_DIR=tmpdir): + personas = load_personas() + + self.assertEqual([p.id for p in personas], ['p001-from-settings']) + + +class TestBundledPersonaFixtures(TestCase): + """ + The persona set that actually ships is loaded and checked here, so an authoring + slip in a YAML file fails CI rather than a run. + """ + + def setUp(self): + super().setUp() + self.personas = load_personas(fixture_dir=PERSONA_FIXTURE_DIR) + + def test_bundled_personas_all_load(self): + self.assertGreater(len(self.personas), 0) + self.assertEqual( + [p.id for p in self.personas], + sorted(p.id for p in self.personas), + ) + + def test_the_set_covers_both_sides_of_the_technology_split(self): + domains = {p.domain for p in self.personas} + + self.assertIn('technology', domains) + self.assertTrue(domains - {'technology'}, 'the set has no non-technology personas') + + def test_the_set_includes_a_known_uncoverable_persona(self): + """The diagnostic's third outcome has to be represented, not stumbled into.""" + uncoverable = [p for p in self.personas if p.expect_no_coverage] + + self.assertTrue(uncoverable, 'no persona exercises the expect_no_coverage path') + for persona in uncoverable: + self.assertEqual(persona.tier, 'edge') + + def test_every_bundled_persona_declares_a_snapshot_date(self): + """Without one, a stale expectation is indistinguishable from a regression.""" + for persona in self.personas: + with self.subTest(persona=persona.id): + self.assertIsNotNone(persona.catalog.snapshot_date) diff --git a/enterprise_access/apps/pathway_eval/tests/test_retrieval_diagnostic.py b/enterprise_access/apps/pathway_eval/tests/test_retrieval_diagnostic.py new file mode 100644 index 00000000..fe29d970 --- /dev/null +++ b/enterprise_access/apps/pathway_eval/tests/test_retrieval_diagnostic.py @@ -0,0 +1,454 @@ +""" +Tests for the retrieval diagnostic. +""" +import json +import tempfile +from io import StringIO +from pathlib import Path +from unittest import mock + +import ddt +import yaml +from django.core.management import call_command +from django.core.management.base import CommandError +from django.test import TestCase + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.pathway_eval.personas import persona_from_dict +from enterprise_access.apps.pathway_eval.retrieval_diagnostic import ( + MAX_QUERY_CHARS, + Outcome, + RetrievalDiagnostic, + build_query_strategies, + summarize +) +from enterprise_access.apps.pathway_eval.tests.test_personas import VALID_INPUTS, make_persona_dict + +EXPECTED_KEY = 'IBM+DA0101EN' +EXPECTED_TITLE = 'Analyzing Data with Python' +OTHER_KEY = 'HarvardX+CS109x' + + +def make_persona(**overrides): + return persona_from_dict(make_persona_dict(**overrides)) + + +def hits_for(*keys): + """An Algolia response body containing the given keys, in order.""" + return {'hits': [{'key': key, 'title': f'Title for {key}'} for key in keys]} + + +class FakeAlgoliaClient: + """ + Records every catalog search and replays scripted responses. + + Scripted by *query substring* rather than call order, because the diagnostic issues + several strategies and asserting on order would make these tests fragile. + """ + + def __init__(self, responses_by_query_substring=None, default=None, error=None): + self.responses = responses_by_query_substring or {} + self.default = default if default is not None else {'hits': []} + self.error = error + self.calls = [] + + def search_catalog_index(self, query, **kwargs): + """Stand in for ``AlgoliaSearchClient.search_catalog_index``.""" + self.calls.append({'query': query, **kwargs}) + if self.error: + raise self.error + for substring, response in self.responses.items(): + if substring.lower() in query.lower(): + return response + return self.default + + +@ddt.ddt +class TestBuildQueryStrategies(TestCase): + """ + Tests for deterministic query construction. + """ + + def test_strategies_are_built_from_persona_inputs(self): + strategies = build_query_strategies(make_persona()) + + self.assertEqual( + set(strategies), + {'goals_only', 'goals_and_free_text', 'career_title'}, + ) + self.assertEqual(strategies['goals_only'], VALID_INPUTS['selected_goals']) + self.assertIn(VALID_INPUTS['selected_goals'], strategies['goals_and_free_text']) + self.assertIn(VALID_INPUTS['free_text'], strategies['goals_and_free_text']) + self.assertEqual(strategies['career_title'], 'Data Analyst Consultant') + + def test_career_title_strategy_is_absent_without_a_named_career(self): + """A strategy with no text must be skipped, not issued as an empty query.""" + strategies = build_query_strategies(make_persona(expected={'careers': []})) + + self.assertNotIn('career_title', strategies) + + def test_long_free_text_is_truncated_at_a_word_boundary(self): + inputs = dict(VALID_INPUTS, free_text='word ' * 200) + + query = build_query_strategies(make_persona(inputs=inputs))['goals_and_free_text'] + + self.assertLessEqual(len(query), MAX_QUERY_CHARS) + self.assertFalse(query.endswith('wor')) + + def test_whitespace_is_collapsed(self): + inputs = dict(VALID_INPUTS, selected_goals='Move into\n\na data role') + + query = build_query_strategies(make_persona(inputs=inputs))['goals_only'] + + self.assertEqual(query, 'Move into a data role') + + +@ddt.ddt +class TestRetrievalDiagnostic(TestCase): + """ + Tests for ``RetrievalDiagnostic``. + """ + + def test_recall_and_rank_are_reported_per_persona(self): + """Scenario: Recall is reported per persona.""" + client = FakeAlgoliaClient(default=hits_for(OTHER_KEY, 'X+1', EXPECTED_KEY)) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(make_persona()) + + self.assertEqual(result.expected_course_keys, [EXPECTED_KEY]) + self.assertEqual(result.retrieved_keys, {EXPECTED_KEY}) + # Third in the hit list, so rank 3 -- 1-based, as a human reads a results page. + self.assertEqual(result.best_rank, 3) + self.assertEqual(result.recall_at_top_n, 1.0) + self.assertEqual(result.outcome, Outcome.IN_TOP_N) + + def test_partial_recall_is_reported_as_a_fraction(self): + persona = make_persona(expected={'courses': [ + {'key': EXPECTED_KEY, 'title': EXPECTED_TITLE}, + {'key': 'IBM+RP0321EN', 'title': 'R Data Science Capstone Project'}, + ]}) + client = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(persona) + + self.assertEqual(result.recall_at_top_n, 0.5) + # One expected course was retrieved, so retrieval demonstrably works here. + self.assertEqual(result.outcome, Outcome.IN_TOP_N) + + def test_top_n_bounds_what_counts_as_retrieved(self): + client = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + + RetrievalDiagnostic(algolia_client=client, top_n=5).run_for_persona(make_persona()) + + self.assertTrue(client.calls) + for call in client.calls: + # Probes deliberately use a wider window; strategy searches must honour top_n. + self.assertIn(call['hitsPerPage'], (5, 50)) + + # -- the three outcomes ----------------------------------------------------------- + + def test_outcome_not_in_top_n_when_the_course_exists_but_is_not_retrieved(self): + """Scenario: The three outcomes are distinguished (present, not retrieved).""" + client = FakeAlgoliaClient( + # The persona's own queries miss it... + default=hits_for(OTHER_KEY), + # ...but a targeted title probe finds it, so it is in the index. + responses_by_query_substring={EXPECTED_TITLE: hits_for(EXPECTED_KEY)}, + ) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(make_persona()) + + self.assertEqual(result.outcome, Outcome.NOT_IN_TOP_N) + self.assertEqual(result.probe_found, {EXPECTED_KEY: True}) + self.assertEqual(result.recall_at_top_n, 0.0) + self.assertIn('upstream', Outcome.CONSEQUENCES[result.outcome]) + + def test_outcome_not_found_in_index_when_even_the_probe_misses(self): + """Scenario: The three outcomes are distinguished (absent entirely).""" + client = FakeAlgoliaClient(default=hits_for(OTHER_KEY)) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(make_persona()) + + self.assertEqual(result.outcome, Outcome.NOT_FOUND_IN_INDEX) + self.assertEqual(result.probe_found, {EXPECTED_KEY: False}) + self.assertIn('coverage', Outcome.CONSEQUENCES[result.outcome]) + + def test_a_course_with_no_title_cannot_be_probed(self): + """The probe searches titles, so a key-only expectation is unprobeable.""" + persona = make_persona(expected={'courses': [{'key': EXPECTED_KEY}]}) + client = FakeAlgoliaClient(default=hits_for(OTHER_KEY)) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(persona) + + self.assertEqual(result.probe_found, {EXPECTED_KEY: False}) + + def test_expect_no_coverage_persona_records_incidental_hits(self): + persona = make_persona(expected={'expect_no_coverage': True, 'courses': []}) + client = FakeAlgoliaClient(default=hits_for('IRRELEVANT+1', 'IRRELEVANT+2')) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(persona) + + self.assertEqual(result.outcome, Outcome.EXPECTED_NO_COVERAGE) + # The padding a learner would have been shown, so a human can confirm it is padding. + self.assertEqual( + {hit['key'] for hit in result.incidental_hits}, + {'IRRELEVANT+1', 'IRRELEVANT+2'}, + ) + self.assertIsNone(result.recall_at_top_n) + + def test_persona_without_ground_truth_is_not_a_failure(self): + persona = make_persona(expected={'courses': [], 'expect_no_coverage': False}) + client = FakeAlgoliaClient(default=hits_for(OTHER_KEY)) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(persona) + + self.assertEqual(result.outcome, Outcome.NO_GROUND_TRUTH) + self.assertIsNone(result.recall_at_top_n) + + @ddt.data( + Outcome.NO_GROUND_TRUTH, + Outcome.EXPECTED_NO_COVERAGE, + Outcome.IN_TOP_N, + Outcome.NOT_IN_TOP_N, + Outcome.NOT_FOUND_IN_INDEX, + ) + def test_every_outcome_states_its_consequence(self, outcome): + """The gate is a decision, so each outcome must name the decision it implies.""" + self.assertIn(outcome, Outcome.CONSEQUENCES) + self.assertTrue(Outcome.CONSEQUENCES[outcome].strip()) + + # -- credential handling ---------------------------------------------------------- + + def test_unscoped_flag_is_passed_through_to_the_client(self): + client = FakeAlgoliaClient() + + RetrievalDiagnostic(algolia_client=client, allow_unscoped=True).run_for_persona(make_persona()) + + self.assertTrue(client.calls) + for call in client.calls: + self.assertTrue(call['allow_unscoped']) + self.assertIsNone(call['secured_key']) + + def test_searches_are_scoped_to_course_content(self): + client = FakeAlgoliaClient() + + RetrievalDiagnostic(algolia_client=client).run_for_persona(make_persona()) + + for call in client.calls: + self.assertEqual(call['filters'], 'content_type:course') + + # -- enterprise customer scoping -------------------------------------------------- + + def test_customer_uuid_is_added_to_the_filters(self): + client = FakeAlgoliaClient() + customer_uuid = '91dc5e6c-7166-4c24-9514-cd871bc46deb' + + RetrievalDiagnostic( + algolia_client=client, customer_uuid=customer_uuid, + ).run_for_persona(make_persona()) + + self.assertTrue(client.calls) + for call in client.calls: + self.assertEqual( + call['filters'], + f'content_type:course AND enterprise_customer_uuids:"{customer_uuid}"', + ) + + @ddt.data('not-a-uuid', '', '852eac48-b5a9-4849', 12345) + def test_a_malformed_customer_uuid_is_rejected_rather_than_filtered_on(self, bad_uuid): + """ + Algolia does not error on a filter that matches nothing, so a typo would report + 0% recall for every persona -- a wrong answer that looks exactly like a real one. + """ + with self.assertRaisesRegex(ValueError, 'not a UUID'): + RetrievalDiagnostic(algolia_client=FakeAlgoliaClient(), customer_uuid=bad_uuid) + + def test_no_customer_uuid_leaves_the_filters_unscoped(self): + client = FakeAlgoliaClient() + + diagnostic = RetrievalDiagnostic(algolia_client=client) + + self.assertIsNone(diagnostic.customer_uuid) + self.assertEqual(diagnostic.catalog_filters, 'content_type:course') + + def test_count_scoped_courses_reads_nbhits_under_the_scope(self): + client = FakeAlgoliaClient(default={'hits': [], 'nbHits': 4057}) + + count = RetrievalDiagnostic( + algolia_client=client, + customer_uuid='91dc5e6c-7166-4c24-9514-cd871bc46deb', + ).count_scoped_courses() + + self.assertEqual(count, 4057) + self.assertEqual(client.calls[0]['hitsPerPage'], 0) + + # -- failures --------------------------------------------------------------------- + + def test_a_failing_strategy_is_recorded_not_raised(self): + """One flaky search must not abandon the whole persona set.""" + client = FakeAlgoliaClient(error=AlgoliaSearchError('boom')) + + result = RetrievalDiagnostic(algolia_client=client).run_for_persona(make_persona()) + + self.assertTrue(result.errors) + for strategy in result.strategy_results: + self.assertIn('boom', strategy.error) + # No expected course was retrieved and the probe also failed, so the honest + # classification is "not found" -- with the errors attached alongside. + self.assertEqual(result.outcome, Outcome.NOT_FOUND_IN_INDEX) + + +class TestSummarize(TestCase): + """ + Tests for report aggregation. + """ + + def _run(self, personas, client): + return RetrievalDiagnostic(algolia_client=client).run(personas) + + def test_results_are_split_by_technology(self): + """Scenario: Results split by domain.""" + tech = make_persona(id='p001-tech', domain='technology') + non_tech = make_persona(id='p002-nurse', domain='healthcare') + client = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + + summary = summarize(self._run([tech, non_tech], client)) + + self.assertEqual(summary['technology']['personas'], 1) + self.assertEqual(summary['non_technology']['personas'], 1) + self.assertEqual(summary['total_personas'], 2) + + def test_placeholder_personas_are_reported_separately(self): + """ + Scoring a search-picked guess as though it were expert judgement is worse than + reporting nothing, so the split has to be mechanical. + """ + expert = make_persona(id='p001-expert', expected={'ground_truth_status': 'expert_authored'}) + placeholder = make_persona(id='p002-placeholder', expected={'ground_truth_status': 'placeholder'}) + client = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + + summary = summarize(self._run([expert, placeholder], client)) + + self.assertEqual(summary['expert_authored_personas'], 1) + self.assertEqual(summary['placeholder_personas'], 1) + self.assertEqual(summary['expert_authored_only']['personas'], 1) + + def test_mean_recall_is_none_when_nothing_is_scoreable(self): + persona = make_persona(expected={'courses': [], 'expect_no_coverage': False}) + client = FakeAlgoliaClient() + + summary = summarize(self._run([persona], client)) + + self.assertIsNone(summary['overall']['mean_recall_at_top_n']) + self.assertEqual(summary['overall']['scoreable'], 0) + + def test_outcome_counts_are_reported_per_split(self): + client = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + personas = [ + make_persona(id='p001-a', domain='technology'), + make_persona(id='p002-b', domain='technology'), + ] + + summary = summarize(self._run(personas, client)) + + self.assertEqual(summary['technology']['outcomes'][Outcome.IN_TOP_N], 2) + + +class TestRunRetrievalDiagnosticCommand(TestCase): + """ + Tests for the ``run_retrieval_diagnostic`` management command. + + These write their own persona fixtures to a temporary directory rather than running + against the shipped set. The shipped personas are real, product-authored ground truth + that changes as it is edited; coupling command tests to a particular persona id makes + those edits look like command regressions. + """ + + def setUp(self): + super().setUp() + self._tmpdir = tempfile.TemporaryDirectory() + self.addCleanup(self._tmpdir.cleanup) + self.persona_dir = Path(self._tmpdir.name) + (self.persona_dir / 'p001-test.yaml').write_text( + yaml.safe_dump(make_persona_dict(id='p001-test'), sort_keys=False) + ) + + def call(self, **kwargs): + stdout = StringIO() + kwargs.setdefault('persona_dir', str(self.persona_dir)) + call_command('run_retrieval_diagnostic', stdout=stdout, **kwargs) + return stdout.getvalue() + + @mock.patch('enterprise_access.apps.pathway_eval.management.commands.run_retrieval_diagnostic.AlgoliaSearchClient') + def test_command_reports_the_persona_set(self, mock_client_class): + mock_client_class.return_value = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + + output = self.call() + + self.assertIn('RETRIEVAL DIAGNOSTIC', output) + self.assertIn('SUMMARY', output) + self.assertIn('GATE:', output) + self.assertIn('p001-test', output) + + @mock.patch('enterprise_access.apps.pathway_eval.management.commands.run_retrieval_diagnostic.AlgoliaSearchClient') + def test_unscoped_run_is_flagged_in_the_output(self, mock_client_class): + mock_client_class.return_value = FakeAlgoliaClient() + + output = self.call(unscoped=True) + + self.assertIn('UNSCOPED', output) + + @mock.patch('enterprise_access.apps.pathway_eval.management.commands.run_retrieval_diagnostic.AlgoliaSearchClient') + def test_json_output_is_written(self, mock_client_class): + mock_client_class.return_value = FakeAlgoliaClient(default=hits_for(EXPECTED_KEY)) + + with tempfile.TemporaryDirectory() as tmpdir: + path = Path(tmpdir) / 'nested' / 'out.json' + self.call(persona_ids=['p001-test'], output_json=str(path)) + + payload = json.loads(path.read_text()) + + self.assertEqual(len(payload['personas']), 1) + self.assertEqual(payload['personas'][0]['persona_id'], 'p001-test') + self.assertIn('summary', payload) + self.assertIn('strategies', payload['personas'][0]) + + @mock.patch('enterprise_access.apps.pathway_eval.management.commands.run_retrieval_diagnostic.AlgoliaSearchClient') + def test_customer_scoped_run_reports_the_scope_and_its_size(self, mock_client_class): + mock_client_class.return_value = FakeAlgoliaClient( + default={**hits_for(EXPECTED_KEY), 'nbHits': 4057}, + ) + customer_uuid = '91dc5e6c-7166-4c24-9514-cd871bc46deb' + + output = self.call(customer_uuid=customer_uuid) + + self.assertIn(customer_uuid, output) + self.assertIn('4057 courses in scope', output) + # Scoped by filter, so the blanket "upper bound" caveat no longer applies... + self.assertNotIn('Results are not restricted', output) + # ...but the weaker guarantee a filter gives is stated instead. + self.assertIn('not a substitute for a secured key', output) + + @mock.patch('enterprise_access.apps.pathway_eval.management.commands.run_retrieval_diagnostic.AlgoliaSearchClient') + def test_an_empty_customer_scope_is_an_error_not_a_run_of_zeroes(self, mock_client_class): + """ + A well-formed UUID that is not a customer's -- a catalog UUID, say -- matches no + courses. Reporting that as 0% recall would be a false negative on the whole + pipeline, so the command refuses to run. + """ + mock_client_class.return_value = FakeAlgoliaClient(default={'hits': [], 'nbHits': 0}) + + with self.assertRaisesRegex(CommandError, 'no courses in the catalog index'): + self.call(customer_uuid='91dc5e6c-7166-4c24-9514-cd871bc46deb') + + def test_a_malformed_customer_uuid_is_a_command_error(self): + with self.assertRaisesRegex(CommandError, 'not a UUID'): + self.call(customer_uuid='2u') + + def test_unknown_persona_id_is_a_command_error(self): + with self.assertRaisesRegex(CommandError, 'p999-nope'): + self.call(persona_ids=['p999-nope']) + + def test_missing_persona_dir_is_a_command_error(self): + with self.assertRaisesRegex(CommandError, 'does not exist'): + self.call(persona_dir='/nonexistent/persona/dir') diff --git a/enterprise_access/apps/pathway_eval/tests/test_scoring.py b/enterprise_access/apps/pathway_eval/tests/test_scoring.py new file mode 100644 index 00000000..6744bafb --- /dev/null +++ b/enterprise_access/apps/pathway_eval/tests/test_scoring.py @@ -0,0 +1,609 @@ +""" +Tests for the deterministic scorers. + +Three properties get the most attention, because each is a claim about *why* the report +has the shape Decision 8 gave it: + +* Tier 1 failing must invalidate the quality numbers, not sit alongside them. +* A split scoring zero must fail Tier 2 even when the total clears the bar -- that is + exactly what an aggregate metric cannot express. +* ``expect_no_coverage`` personas must invert the rule, not be excluded from it. +""" +import json +import tempfile +from io import StringIO +from pathlib import Path + +import yaml +from django.core.management import call_command +from django.core.management.base import CommandError +from django.test import TestCase + +from enterprise_access.apps.pathway_eval.personas import persona_from_dict +from enterprise_access.apps.pathway_eval.scoring import ( + SPLIT_NON_TECHNOLOGY, + SPLIT_TECHNOLOGY, + regression_verdict, + score_persona, + score_run, + tier_one_gates, + tier_three_metrics, + tier_two_bar +) +from enterprise_access.apps.pathways.pathway_assembly import PATHWAY_SIZE + +CAREER_ID = 'ETE78CD2CDFFFAC66B' + + +def persona_dict(persona_id, *, domain='technology', courses=('IBM+DA0101EN',), + expect_no_coverage=False, status='expert_authored'): + return { + 'id': persona_id, + 'domain': domain, + 'tier': 'core', + 'inputs': { + 'selected_goals': 'change careers', + 'free_text': 'I want a new job', + 'known_context': 'analyst', + 'interested_industries': 'technology', + }, + 'expected': { + 'ground_truth_status': status, + 'expect_no_coverage': expect_no_coverage, + 'careers': [{'external_id': CAREER_ID}], + 'courses': [{'key': k} for k in courses], + }, + } + + +def make_persona(persona_id, **kwargs): + return persona_from_dict(persona_dict(persona_id, **kwargs)) + + +# A complete pathway is exactly PATHWAY_SIZE courses, so a realistic cell has five keys. +# Padding here rather than in each test keeps the Tier 1 length gate meaningful: a fixture +# that returned one course marked complete would trip it in every test. +FILLER_KEYS = ('Filler+1', 'Filler+2', 'Filler+3', 'Filler+4', 'Filler+5') + + +def five_keys(*keys): + """Pad ``keys`` out to a full pathway with filler that matches no ground truth.""" + padded = list(keys) + [k for k in FILLER_KEYS if k not in keys] + return tuple(padded[:PATHWAY_SIZE]) + + +def cell(persona_id, *, mode='auto', run=1, keys=None, complete=True, + violations=(), unfilled=(), skipped='', error='', rationale_count=5): + """Build one harness cell trace, padded to a valid pathway length by default.""" + if keys is None: + keys = five_keys('IBM+DA0101EN') + elif complete and len(keys) not in (0, PATHWAY_SIZE): + keys = five_keys(*keys) + return { + 'persona_id': persona_id, + 'career_mode': mode, + 'run_index': run, + 'career_workflow_uuid': 'c', + 'pathway_workflow_uuid': 'p', + 'career_name': 'Data Analyst', + 'career_external_id': CAREER_ID, + 'course_keys': list(keys), + 'complete': complete, + 'violations': list(violations), + 'unfilled_rungs': list(unfilled), + 'rationale_count': rationale_count, + 'skipped_reason': skipped, + 'error': error, + } + + +class TestScorePersona(TestCase): + """ + Tests for ``score_persona``. + """ + + def test_a_matched_expected_course_passes(self): + persona = make_persona('p001') + + score = score_persona(persona, [cell('p001')]) + + self.assertTrue(score['passed']) + self.assertEqual(score['best_recall'], 1.0) + + def test_no_matched_course_fails(self): + persona = make_persona('p001') + + score = score_persona(persona, [cell('p001', keys=five_keys('Other+1'))]) + + self.assertFalse(score['passed']) + self.assertEqual(score['best_recall'], 0.0) + + def test_one_of_several_expected_courses_is_enough_to_pass(self): + """ + The bar is deliberately a floor. At 23% recall the thing to establish first is + that the pipeline works at all for a domain. + """ + persona = make_persona('p001', courses=('A+1', 'B+2', 'C+3')) + + score = score_persona(persona, [cell('p001', keys=five_keys('A+1'))]) + + self.assertTrue(score['passed']) + self.assertAlmostEqual(score['best_recall'], 1 / 3) + + def test_any_passing_cell_passes_the_persona(self): + """ + Cells are repeat runs and career modes of the same question. Requiring all of + them to pass would fold career-selection error back into the course score, which + is what the oracle arm exists to separate. + """ + persona = make_persona('p001') + + score = score_persona(persona, [ + cell('p001', mode='auto', keys=five_keys('Other+1')), + cell('p001', mode='oracle', keys=five_keys('IBM+DA0101EN')), + ]) + + self.assertTrue(score['passed']) + + def test_expect_no_coverage_inverts_the_rule(self): + """Scenario: Expected absence is not counted as failure.""" + persona = make_persona('p001', courses=(), expect_no_coverage=True) + + passing = score_persona(persona, [cell('p001', keys=(), complete=False)]) + failing = score_persona(persona, [cell('p001', keys=five_keys('A+1'), complete=True)]) + + self.assertTrue(passing['passed']) + self.assertFalse(failing['passed']) + + def test_a_placeholder_persona_is_not_scoreable(self): + """Placeholder ground truth must not pollute a headline metric.""" + persona = make_persona('p001', status='placeholder') + + score = score_persona(persona, [cell('p001')]) + + self.assertFalse(score['scoreable']) + self.assertIsNone(score['passed']) + + def test_a_persona_with_no_ground_truth_is_not_scoreable(self): + persona = make_persona('p001', courses=()) + + self.assertFalse(score_persona(persona, [cell('p001')])['scoreable']) + + def test_skipped_and_errored_cells_are_not_scored(self): + persona = make_persona('p001') + + score = score_persona(persona, [ + cell('p001', skipped='dry run'), + cell('p001', error='boom'), + ]) + + self.assertEqual(score['cells_ran'], 0) + self.assertEqual(score['cells_planned'], 2) + self.assertFalse(score['passed']) + + def test_the_split_follows_the_domain(self): + self.assertEqual( + score_persona(make_persona('p1', domain='technology'), [])['split'], + SPLIT_TECHNOLOGY, + ) + self.assertEqual( + score_persona(make_persona('p2', domain='healthcare'), [])['split'], + SPLIT_NON_TECHNOLOGY, + ) + + +class TestTierOneGates(TestCase): + """ + Tests for ``tier_one_gates``. + """ + + def test_clean_cells_pass(self): + result = tier_one_gates([cell('p001'), cell('p002')]) + + self.assertTrue(result['passed']) + self.assertEqual(result['failures'], {}) + + def test_persisted_assembly_violations_fail_the_gate(self): + result = tier_one_gates([ + cell('p001', violations=['3 courses from P1 exceeds the cap of 2']), + ]) + + self.assertFalse(result['passed']) + self.assertIn('assembly_violations', result['failures']) + + def test_a_complete_pathway_of_the_wrong_length_fails(self): + short = cell('p001') + short['course_keys'] = ['A+1', 'B+2'] + + result = tier_one_gates([short]) + + self.assertFalse(result['passed']) + self.assertIn('wrong_length', result['failures']) + + def test_an_incomplete_pathway_returning_courses_fails(self): + """ + A partial set handed to a client would render as a pathway nobody claimed was one. + """ + result = tier_one_gates([cell('p001', keys=('A+1',), complete=False)]) + + self.assertFalse(result['passed']) + self.assertIn('partial_pathways_returned', result['failures']) + + def test_an_incomplete_pathway_with_no_courses_is_fine(self): + result = tier_one_gates([cell('p001', keys=(), complete=False)]) + + self.assertTrue(result['passed']) + + def test_skipped_cells_are_not_gated(self): + result = tier_one_gates([cell('p001', skipped='dry run', violations=['x'])]) + + self.assertTrue(result['passed']) + self.assertEqual(result['cells_checked'], 0) + + +class TestTierTwoBar(TestCase): + """ + Tests for ``tier_two_bar``. + """ + + def _scores(self, spec): + """``spec`` maps persona id to (split, passed).""" + return [ + {'persona_id': pid, 'domain': split, 'split': split, 'scoreable': True, + 'expect_no_coverage': False, 'expected_course_count': 1, 'cells_planned': 1, + 'cells_ran': 1, 'passed': passed, 'best_recall': 1.0 if passed else 0.0, + 'cells': []} + for pid, (split, passed) in spec.items() + ] + + def test_the_count_bar_is_applied(self): + scores = self._scores({ + 'p1': (SPLIT_TECHNOLOGY, True), 'p2': (SPLIT_NON_TECHNOLOGY, True), + }) + + self.assertTrue(tier_two_bar(scores, min_passing=2)['count_met']) + self.assertFalse(tier_two_bar(scores, min_passing=3)['count_met']) + + def test_a_zero_split_fails_even_when_the_count_is_met(self): + """ + The rule an aggregate cannot express: concentrated failure is not shippable + regardless of the total. + """ + scores = self._scores({ + 'p1': (SPLIT_NON_TECHNOLOGY, True), + 'p2': (SPLIT_NON_TECHNOLOGY, True), + 'p3': (SPLIT_TECHNOLOGY, False), + }) + + result = tier_two_bar(scores, min_passing=2) + + self.assertTrue(result['count_met']) + self.assertFalse(result['no_zero_split']) + self.assertFalse(result['passed']) + self.assertEqual(result['zero_splits'], [SPLIT_TECHNOLOGY]) + + def test_a_split_with_no_scoreable_personas_does_not_fail_the_rule(self): + """ + Silence is not failure. Treating an unpopulated split as zero would block the bar + on ground-truth authoring rather than on pipeline quality. + """ + scores = self._scores({ + 'p1': (SPLIT_NON_TECHNOLOGY, True), 'p2': (SPLIT_NON_TECHNOLOGY, True), + }) + + result = tier_two_bar(scores, min_passing=2) + + self.assertTrue(result['passed']) + self.assertEqual(result['per_split'][SPLIT_TECHNOLOGY]['scoreable'], 0) + + def test_unscoreable_personas_are_excluded_from_both_sides_of_the_ratio(self): + scores = self._scores({'p1': (SPLIT_TECHNOLOGY, True)}) + scores.append({ + 'persona_id': 'p2', 'domain': 'technology', 'split': SPLIT_TECHNOLOGY, + 'scoreable': False, 'expect_no_coverage': False, 'expected_course_count': 0, + 'cells_planned': 1, 'cells_ran': 1, 'passed': None, 'best_recall': None, + 'cells': [], + }) + + result = tier_two_bar(scores, min_passing=1) + + self.assertEqual(result['scoreable'], 1) + self.assertEqual(result['passing'], 1) + + def test_passing_and_failing_ids_are_listed(self): + scores = self._scores({ + 'p1': (SPLIT_TECHNOLOGY, True), 'p2': (SPLIT_NON_TECHNOLOGY, False), + }) + + result = tier_two_bar(scores, min_passing=1) + + self.assertEqual(result['passing_persona_ids'], ['p1']) + self.assertEqual(result['failing_persona_ids'], ['p2']) + + +class TestTierThreeMetrics(TestCase): + """ + Tests for ``tier_three_metrics``. + """ + + def test_the_technology_gap_is_quantified(self): + """Scenario: The technology gap is quantified.""" + personas = [ + make_persona('p1', domain='technology'), + make_persona('p2', domain='healthcare'), + ] + cells = [ + cell('p1', keys=five_keys('Other+1')), + cell('p2', keys=('IBM+DA0101EN',)), + ] + scores = [score_persona(p, cells) for p in personas] + + metrics = tier_three_metrics(scores, cells) + + self.assertEqual(metrics['splits'][SPLIT_TECHNOLOGY]['mean_recall'], 0.0) + self.assertEqual(metrics['splits'][SPLIT_NON_TECHNOLOGY]['mean_recall'], 1.0) + self.assertEqual(metrics['technology_delta'], -1.0) + + def test_consistency_is_measured_across_runs(self): + """Scenario: Consistency is measured across runs.""" + personas = [make_persona('p1')] + cells = [ + cell('p1', run=1, keys=five_keys('A+1', 'B+2')), + cell('p1', run=2, keys=five_keys('A+1', 'C+3')), + cell('p1', run=3, keys=five_keys('A+1', 'B+2')), + ] + scores = [score_persona(p, cells) for p in personas] + + consistency = tier_three_metrics(scores, cells)['cross_run_consistency'] + + self.assertEqual(consistency['pairs_compared'], 3) + self.assertIsNotNone(consistency['mean_jaccard']) + + def test_a_single_run_reports_no_consistency_rather_than_perfect(self): + """Reporting 1.0 would claim perfect stability from no evidence.""" + personas = [make_persona('p1')] + cells = [cell('p1')] + scores = [score_persona(p, cells) for p in personas] + + consistency = tier_three_metrics(scores, cells)['cross_run_consistency'] + + self.assertEqual(consistency['pairs_compared'], 0) + self.assertIsNone(consistency['mean_jaccard']) + + def test_the_career_mode_delta_is_reported(self): + personas = [make_persona('p1')] + cells = [ + cell('p1', mode='auto', keys=five_keys('Other+1')), + cell('p1', mode='oracle', keys=five_keys('IBM+DA0101EN')), + ] + scores = [score_persona(p, cells) for p in personas] + + modes = tier_three_metrics(scores, cells)['career_mode_delta'] + + self.assertEqual(modes['auto'], 0) + self.assertEqual(modes['oracle'], 1) + self.assertEqual(modes['delta'], 1) + + def test_zero_hit_and_completion_rates_are_reported(self): + personas = [make_persona('p1'), make_persona('p2')] + cells = [cell('p1'), cell('p2', keys=(), complete=False)] + scores = [score_persona(p, cells) for p in personas] + + metrics = tier_three_metrics(scores, cells) + + self.assertEqual(metrics['completion_rate'], 0.5) + self.assertEqual(metrics['zero_hit_rate'], 0.5) + + def test_the_unexplained_pathway_rate_is_tracked(self): + """A pathway that shipped without rationales is milder than one that failed.""" + personas = [make_persona('p1'), make_persona('p2')] + cells = [cell('p1'), cell('p2', rationale_count=0)] + scores = [score_persona(p, cells) for p in personas] + + metrics = tier_three_metrics(scores, cells) + + self.assertEqual(metrics['unexplained_pathway_rate'], 0.5) + + def test_unfilled_rung_rates_are_reported_per_level(self): + personas = [make_persona('p1')] + cells = [cell('p1', unfilled=['Advanced'])] + scores = [score_persona(p, cells) for p in personas] + + rates = tier_three_metrics(scores, cells)['unfilled_rung_rate'] + + self.assertEqual(rates['Advanced'], 1.0) + self.assertEqual(rates['Introductory'], 0.0) + + +class TestScoreRun(TestCase): + """ + Tests for ``score_run``. + """ + + def test_shippable_requires_tier_one_and_tier_two(self): + personas = [make_persona('p1'), make_persona('p2', domain='healthcare')] + cells = [cell('p1'), cell('p2')] + + report = score_run(personas, cells, min_passing=2) + + self.assertTrue(report['tier_one']['passed']) + self.assertTrue(report['tier_two']['passed']) + self.assertTrue(report['shippable']) + + def test_a_tier_one_failure_makes_the_run_unshippable_regardless_of_quality(self): + """Tier 1 failing invalidates the quality numbers rather than sitting beside them.""" + personas = [make_persona('p1'), make_persona('p2', domain='healthcare')] + cells = [cell('p1', violations=['bad']), cell('p2')] + + report = score_run(personas, cells, min_passing=2) + + self.assertTrue(report['tier_two']['passed']) + self.assertFalse(report['tier_one']['passed']) + self.assertFalse(report['shippable']) + + def test_tier_three_never_gates(self): + personas = [make_persona('p1'), make_persona('p2', domain='healthcare')] + cells = [cell('p1'), cell('p2')] + + report = score_run(personas, cells, min_passing=2) + + self.assertTrue(report['shippable']) + self.assertIsNotNone(report['tier_three']['technology_delta']) + + +class TestRegressionVerdict(TestCase): + """ + Tests for ``regression_verdict``. + """ + + def _report(self, passing, tech_recall): + return { + 'tier_two': {'passing': passing}, + 'tier_three': { + 'completion_rate': 1.0, + 'splits': { + SPLIT_TECHNOLOGY: {'mean_recall': tech_recall}, + SPLIT_NON_TECHNOLOGY: {'mean_recall': 0.5}, + }, + }, + } + + def test_no_previous_run_yields_no_verdict_rather_than_a_pass(self): + """Inventing a baseline would report a pass from no evidence.""" + self.assertIsNone(regression_verdict(self._report(3, 0.2), None)) + + def test_a_decrease_is_a_regression(self): + verdict = regression_verdict(self._report(2, 0.2), self._report(3, 0.2)) + + self.assertFalse(verdict['passed']) + self.assertEqual(verdict['regressions'][0]['metric'], 'tier_two.passing') + + def test_an_increase_is_reported_as_an_improvement(self): + verdict = regression_verdict(self._report(4, 0.2), self._report(3, 0.2)) + + self.assertTrue(verdict['passed']) + self.assertEqual(verdict['improvements'][0]['metric'], 'tier_two.passing') + + def test_a_split_recall_drop_is_caught(self): + verdict = regression_verdict(self._report(3, 0.1), self._report(3, 0.3)) + + self.assertFalse(verdict['passed']) + self.assertIn('technology', verdict['regressions'][0]['metric']) + + def test_an_unmeasurable_metric_is_skipped_rather_than_treated_as_zero(self): + verdict = regression_verdict(self._report(3, None), self._report(3, 0.3)) + + self.assertTrue(verdict['passed']) + + +class TestReportPathwayHarnessCommand(TestCase): + """ + Tests for the ``report_pathway_harness`` command. + """ + + def setUp(self): + super().setUp() + self._tmpdir = tempfile.TemporaryDirectory() + self.addCleanup(self._tmpdir.cleanup) + self.root = Path(self._tmpdir.name) + + self.persona_dir = self.root / 'personas' + self.persona_dir.mkdir() + for pid, domain in (('p001-tech', 'technology'), ('p002-health', 'healthcare')): + (self.persona_dir / f'{pid}.yaml').write_text( + yaml.safe_dump(persona_dict(pid, domain=domain), sort_keys=False), + ) + + def write_traces(self, name, cells, **config): + """Write a traces file in the shape run_pathway_harness exports.""" + path = self.root / name + path.write_text(json.dumps({ + 'run_config': {'customer_uuid': '417306cb-b24a-4d06-b83c-fb2a61d7fb96', + 'model_backend': 'xpert', 'rerank_enabled': True, 'runs': 1, + **config}, + 'cells': cells, + })) + return path + + def call(self, traces, **kwargs): + """Invoke the report command and capture stdout.""" + stdout = StringIO() + kwargs.setdefault('persona_dir', str(self.persona_dir)) + call_command('report_pathway_harness', str(traces), stdout=stdout, **kwargs) + return stdout.getvalue() + + def test_a_passing_run_reports_that_it_meets_the_bar(self): + traces = self.write_traces('t.json', [cell('p001-tech'), cell('p002-health')]) + + output = self.call(traces, min_passing=2) + + self.assertIn('MEETS THE BAR', output) + self.assertIn('TIER 1', output) + self.assertIn('TIER 2', output) + self.assertIn('TIER 3', output) + + def test_a_zero_split_is_called_out_explicitly(self): + traces = self.write_traces('t.json', [ + cell('p001-tech', keys=five_keys('Other+1')), cell('p002-health'), + ]) + + output = self.call(traces, min_passing=1) + + self.assertIn('ZERO', output) + self.assertIn('DOES NOT MEET THE BAR', output) + + def test_an_unscoped_run_is_flagged(self): + traces = self.write_traces('t.json', [cell('p001-tech')], customer_uuid='') + + output = self.call(traces, min_passing=1) + + self.assertIn('upper bound', output) + + def test_the_regression_bar_compares_against_a_previous_run(self): + current = self.write_traces('cur.json', [ + cell('p001-tech', keys=five_keys('Other+1')), cell('p002-health'), + ]) + previous = self.write_traces('prev.json', [ + cell('p001-tech'), cell('p002-health'), + ]) + + output = self.call(current, previous=str(previous), min_passing=1) + + self.assertIn('REGRESSION BAR', output) + self.assertIn('tier_two.passing', output) + + def test_no_previous_run_says_so_rather_than_claiming_a_pass(self): + traces = self.write_traces('t.json', [cell('p001-tech')]) + + output = self.call(traces, min_passing=1) + + self.assertIn('nothing to compare', output) + + def test_the_report_is_exportable(self): + traces = self.write_traces('t.json', [cell('p001-tech'), cell('p002-health')]) + out = self.root / 'nested' / 'report.json' + + self.call(traces, min_passing=2, output_json=str(out)) + + report = json.loads(out.read_text()) + self.assertIn('tier_one', report) + self.assertIn('tier_two', report) + self.assertTrue(report['shippable']) + + def test_a_missing_traces_file_is_a_command_error(self): + with self.assertRaisesRegex(CommandError, 'does not exist'): + self.call(self.root / 'nope.json') + + def test_a_non_traces_json_file_is_a_command_error(self): + path = self.root / 'other.json' + path.write_text(json.dumps({'something': 'else'})) + + with self.assertRaisesRegex(CommandError, 'not a harness traces file'): + self.call(path) + + def test_malformed_json_is_a_command_error(self): + path = self.root / 'bad.json' + path.write_text('{not json') + + with self.assertRaisesRegex(CommandError, 'not valid JSON'): + self.call(path) diff --git a/enterprise_access/apps/pathways/__init__.py b/enterprise_access/apps/pathways/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/enterprise_access/apps/pathways/api.py b/enterprise_access/apps/pathways/api.py new file mode 100644 index 00000000..8187e91a --- /dev/null +++ b/enterprise_access/apps/pathways/api.py @@ -0,0 +1,418 @@ +""" +Domain-layer API for career discovery. + +Two stages, each with one external dependency and its own failure mode: + +* ``derive_learning_intent`` asks Xpert what a learner's intake means, in terms of skills + and a search query. It reuses the ``prompts`` app's domain functions so the prompt stays + admin-editable and versioned instead of being hard-coded here. +* ``retrieve_careers`` turns that intent into exactly one search against the Lightcast + jobs index. + +The query shape is a port of the learner portal MFE's ``careerRetrieval.ts``. The +asymmetry in it is the load-bearing part: **skills are optional filters (boosts), never +hard filters.** A hard filter on a skill name that is not a facet value returns zero hits +and says nothing about why, which is exactly the silent-drop failure the Chunk 3 +diagnostic measured on the catalog side. Only industries and job sources -- values a +caller is expected to have grounded against the index already -- become hard filters. + +No HTTP or DRF machinery here, so both stages are testable without a request. +""" +import logging +from typing import Any + +from django.conf import settings + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchClient +from enterprise_access.apps.prompts import api as prompts_api +from enterprise_access.apps.prompts.models import PromptType, XpertLearnerPathwaysSystemPrompt + +logger = logging.getLogger(__name__) + +# Matches the MFE's CAREER_RETRIEVAL_LIMIT. Ten is a card-list length, not a corpus size: +# this is a "which career do you want?" prompt, not a search results page. +CAREER_HITS_PER_PAGE = 10 + +# Optional-filter budgets, per the MFE. Past a handful of boosts the ranking signal is +# diluted rather than sharpened, and preferred skills are deliberately weaker than +# required ones. +MAX_REQUIRED_SKILL_FILTERS = 4 +MAX_PREFERRED_SKILL_FILTERS = 2 +PREFERRED_SKILL_FILTER_SCORE = 1 + +SKILLS_FACET = 'skills.name' +INDUSTRY_NAMES_FACET = 'industry_names' +JOB_SOURCES_FACET = 'job_sources' + +# The jobs index is multilingual and holds translated *duplicates* of the same role: the +# Spanish record's identifier is the English one plus a "-es" suffix, so both surface +# together for the same query. Measured against `prod_taxonomy` on 2026-09-09: +# 87,026 records total, 43,513 with `metadata_language:en`. A "biomedical engineer" query +# returns 37 hits unfiltered and 23 filtered, and the unfiltered set contains pairs like +# `ET609056574BB0BB43` / `ET609056574BB0BB43-es` ("Clinical Specialist" / +# "Especialista Clinico"). +# +# That is exactly the defect persona 2's author recorded: "3 careers in spanish, which +# are identical the ones above them in english. When you select one, both are +# selected/highlighted" -- the shared identifier prefix is why selection affects both. +# The learner portal MFE applies the same restriction via `filterByMetadataLanguage`. +# +# So this filter is not optional polish. Omitting it reproduces a known, reported bug. +METADATA_LANGUAGE_FACET = 'metadata_language' +SUPPORTED_METADATA_LANGUAGE = 'en' + +# The jobs index ANDs every query word and has no `removeWordsIfNoResults` configured, +# exactly as the catalog index does -- but the effect is harsher here, because a job +# record is short. Measured on `prod_taxonomy` on 2026-09-09 with the intake sentence +# "Become a biomedical engineer for a major pharma company": **every** prefix of it +# returns 0 hits, including the single word "Become", since no job name contains it. The +# same query with `allOptional` returns 1,358. +# +# `build_career_query` prefers Xpert's `condensed_algolia_query`, which is free text and +# may well be a phrase. Without this parameter, one unmatched word anywhere in it takes +# career retrieval to zero and dead-ends the whole pipeline before course retrieval is +# even reached. There is no safe query-length cap to apply instead -- one word is already +# enough to fail. +# +# This buys result *volume*, not relevance, so `retrieve_careers` persists the query and +# hit count for the harness to score. See the Chunk 3 gate result. +REMOVE_WORDS_IF_NO_RESULTS = 'allOptional' + +# The only jobs-index attributes this pipeline consumes. ``external_id`` is the Lightcast +# job identifier and is the career's identity everywhere downstream -- names are neither +# unique nor stable in the taxonomy. +CAREER_ATTRIBUTES = ['external_id', 'name', 'skills', 'industry_names'] + +# Separators that only ever appear in a parsing artifact ("SQL & Python", "Excel + +# Tableau"), never in a Lightcast skill name. Filtering on one boosts nothing and spends +# a filter slot, so they are dropped on both the career and course paths in the MFE. +_COMPOUND_SEPARATORS = (' & ', ' + ') + + +def dedupe_names(values: Any) -> list[str]: + """ + Strip, drop empties, and de-duplicate a sequence of names, preserving order. + + Order is preserved because it carries relevance: the first required skill is the + fallback query, and the filter budgets take a prefix. + """ + stripped = (value.strip() for value in values or [] if isinstance(value, str)) + return list(dict.fromkeys(name for name in stripped if name)) + + +def coerce_name_list(value: Any) -> list[str]: + """ + Coerce one untrusted field of a model response into a list of names. + + A model asked for a list will sometimes return a bare string; accepting that is + cheaper than failing the whole run over it. Anything else yields an empty list, so a + malformed field drops a filter rather than raising mid-pipeline. + """ + if isinstance(value, str): + value = [value] + if not isinstance(value, list): + return [] + return dedupe_names(value) + + +def is_malformed_compound(name: str) -> bool: + """Whether ``name`` is a compound parsing artifact rather than a single skill.""" + return any(separator in name for separator in _COMPOUND_SEPARATORS) + + +def _quote_facet_value(value: str) -> str: + """Quote and escape a value for interpolation into an Algolia filter expression.""" + escaped = value.replace('"', '\\"') + return f'"{escaped}"' + + +def _or_clause(facet_name: str, values: list[str]) -> str: + """Build a parenthesised ``OR`` group over one facet's values.""" + joined = ' OR '.join(f'{facet_name}:{_quote_facet_value(value)}' for value in values) + return f'({joined})' + + +def build_career_query(*, condensed_query: str, skills_required: list[str]) -> str: + """ + Derive the text query for the jobs search. + + Prefers the model's condensed query, falling back to the first required skill. The + fallback matters: an empty Algolia query matches everything, so without it a run + whose intent extraction produced no query would be ranked by filters alone. + """ + query = (condensed_query or '').strip() + if query: + return query + + fallback_terms = dedupe_names(skills_required) + return fallback_terms[0] if fallback_terms else '' + + +def build_career_filters(*, industries: list[str], job_sources: list[str]) -> str | None: + """ + Build the hard (must-match) filter expression. + + Always includes the language restriction; industries and job sources are added only + when supplied. Never returns ``None`` in practice, since the language clause always + applies. + """ + clauses = [f'{METADATA_LANGUAGE_FACET}:{SUPPORTED_METADATA_LANGUAGE}'] + clauses += [ + _or_clause(facet_name, values) + for facet_name, values in ( + (INDUSTRY_NAMES_FACET, dedupe_names(industries)), + (JOB_SOURCES_FACET, dedupe_names(job_sources)), + ) + if values + ] + return ' AND '.join(clauses) if clauses else None + + +def build_optional_skill_filters(*, skills_required: list[str], skills_preferred: list[str]) -> list[str]: + """ + Build Algolia ``optionalFilters`` from the derived skills. + + Required skills are unscored, so they carry Algolia's default optional-filter weight; + preferred skills are added at a lower explicit score. Malformed compounds are dropped + and each list is capped, so a model that returns thirty skills cannot flatten the + ranking. + """ + required = [ + name for name in dedupe_names(skills_required) if not is_malformed_compound(name) + ][:MAX_REQUIRED_SKILL_FILTERS] + preferred = [ + name for name in dedupe_names(skills_preferred) if not is_malformed_compound(name) + ][:MAX_PREFERRED_SKILL_FILTERS] + + return [ + f'{SKILLS_FACET}:{_quote_facet_value(name)}' for name in required + ] + [ + f'{SKILLS_FACET}:{_quote_facet_value(name)}' + for name in preferred + ] + + +def career_candidate_from_hit(hit: dict[str, Any]) -> dict[str, Any] | None: + """ + Map one jobs-index hit to a career candidate, or ``None`` if it cannot be identified. + + A hit missing either its Lightcast ``external_id`` or its name is dropped rather than + given a placeholder: a career the learner cannot be sent back to us by identifier is + not a usable candidate, and a fabricated id would corrupt the harness's ground-truth + comparison. + + Carries no match percentage. The POC hardcoded ``0.95`` on every card; the MFE has + since removed it, because no verified compatible domain value exists. + """ + external_id = (hit.get('external_id') or '').strip() + name = (hit.get('name') or '').strip() + if not external_id or not name: + logger.warning( + 'Dropping jobs-index hit with objectID=%r: external_id and name are both required.', + hit.get('objectID'), + ) + return None + + raw_skills = hit.get('skills') or [] + return { + 'external_id': external_id, + 'name': name, + 'skills': dedupe_names( + skill.get('name') for skill in raw_skills if isinstance(skill, dict) + ), + 'industries': coerce_name_list(hit.get('industry_names')), + } + + +def derive_learning_intent(*, intake: dict[str, Any], conversation_id: str) -> dict[str, Any]: + """ + Ask Xpert to derive skills and a search query from a learner's intake. + + Args: + intake: The four validated intake fields, passed through to Xpert verbatim. + conversation_id: Tracing identifier for the Xpert request. + + Raises: + PromptError: If no prompt is configured or the Xpert call fails. + XpertAPIResponseError: If the response body is not JSON. + + Returns: + A dict of ``skills_required``, ``skills_preferred`` and ``condensed_algolia_query``. + """ + prompt = prompts_api.get_current_prompt( + prompt_model=XpertLearnerPathwaysSystemPrompt, + prompt_type=PromptType.LEARNER_INTENT, + ) + xpert_response = prompts_api.send_xpert_message( + prompt=prompt, + messages=prompts_api.build_messages(intake), + conversation_id=conversation_id, + tags=settings.XPERT_LEARNER_PATHWAYS_RAG_TAGS, + prompt_type=PromptType.LEARNER_INTENT, + ) + + payload = xpert_response.as_json() + if not isinstance(payload, dict): + raise prompts_api.PromptError( + f'Expected a JSON object from prompt_type={PromptType.LEARNER_INTENT!r}, ' + f'got {type(payload).__name__}.' + ) + + condensed_query = payload.get('condensed_algolia_query') + return { + 'skills_required': coerce_name_list(payload.get('skills_required')), + 'skills_preferred': coerce_name_list(payload.get('skills_preferred')), + 'condensed_algolia_query': condensed_query.strip() if isinstance(condensed_query, str) else '', + } + + +def retrieve_careers( + *, + intent: dict[str, Any], + industries: list[str] | None = None, + job_sources: list[str] | None = None, +) -> dict[str, Any]: + """ + Search the Lightcast jobs index for careers matching a derived intent. + + Issues exactly one search. The jobs index takes the plain search key -- there is no + secured-key variant, and ``search_jobs_index`` has no parameter for one. + + Args: + intent: A ``derive_learning_intent`` result. + industries: Hard-filter values for ``industry_names``. Expected to be real facet + values; unlike a skill boost, an unmatched hard filter returns nothing. + job_sources: Hard-filter values for ``job_sources``. + + Raises: + AlgoliaClientError: On a misconfigured credential or a failed search. + + Returns: + A dict of ``query``, ``hit_count`` and mapped ``careers``. The query and hit count + are recorded because a full result set is not evidence that retrieval worked -- + relaxing a query buys volume, not relevance -- so the harness needs both to tell + those apart without re-running anything. + """ + query = build_career_query( + condensed_query=intent.get('condensed_algolia_query', ''), + skills_required=intent.get('skills_required', []), + ) + search_params: dict[str, Any] = { + 'hitsPerPage': CAREER_HITS_PER_PAGE, + 'attributesToRetrieve': CAREER_ATTRIBUTES, + 'removeWordsIfNoResults': REMOVE_WORDS_IF_NO_RESULTS, + } + + filters = build_career_filters( + industries=industries or [], + job_sources=job_sources or [], + ) + if filters: + search_params['filters'] = filters + + optional_filters = build_optional_skill_filters( + skills_required=intent.get('skills_required', []), + skills_preferred=intent.get('skills_preferred', []), + ) + if optional_filters: + search_params['optionalFilters'] = optional_filters + + response = AlgoliaSearchClient().search_jobs_index(query, **search_params) + hits = response.get('hits') or [] + candidates = [career_candidate_from_hit(hit) for hit in hits if isinstance(hit, dict)] + + return { + 'query': query, + 'hit_count': len(hits), + 'careers': [candidate for candidate in candidates if candidate], + } + + +def enrich_rationales(*, selected_career: str, course_keys: list[str], + learner_profile: dict[str, Any], conversation_id: str) -> dict[str, Any]: + """ + Ask Xpert why each delivered course fits the selected career. + + Reuses the existing ``recommendations_feedback`` prompt read-only, exactly as + ``derive_learning_intent`` reuses ``learner_intent``. That is the point of doing this + as a separate step rather than taking rationales off the re-rank response: the prompt + stays admin-editable and versioned, and the wording a learner sees here cannot drift + from the wording the live MFE endpoint produces. + + It also runs on the **delivered five**, not the candidate twenty, so four fifths of + the explanation work is not paid for and thrown away. + + Args: + selected_career: The career the learner chose. + course_keys: The keys of the assembled pathway, in order. + learner_profile: The learner's intake, passed through to the prompt. + conversation_id: Tracing identifier for the Xpert request. + + Raises: + PromptError: If no prompt is configured or the Xpert call fails. + XpertAPIResponseError: If the response body is not JSON. + + Returns: + A dict of ``reasons`` (course key to rationale) and ``prompt_revision``. Only keys + that were actually asked about are returned -- a rationale for a course not in the + pathway is a fabrication, and the same untrusted-output rule applies here as in + re-ranking. + """ + prompt = prompts_api.get_current_prompt( + prompt_model=XpertLearnerPathwaysSystemPrompt, + prompt_type=PromptType.RECOMMENDATIONS_FEEDBACK, + ) + xpert_response = prompts_api.send_xpert_message( + prompt=prompt, + messages=prompts_api.build_messages({ + 'selected_career': selected_career, + 'course_keys': list(course_keys), + 'learner_profile': dict(learner_profile or {}), + }), + conversation_id=conversation_id, + tags=settings.XPERT_LEARNER_PATHWAYS_RAG_TAGS, + prompt_type=PromptType.RECOMMENDATIONS_FEEDBACK, + ) + + payload = xpert_response.as_json() + if not isinstance(payload, dict): + raise prompts_api.PromptError( + f'Expected a JSON object from prompt_type={PromptType.RECOMMENDATIONS_FEEDBACK!r}, ' + f'got {type(payload).__name__}.' + ) + + raw_reasons = payload.get('reasons') + requested = set(course_keys) + reasons = {} + if isinstance(raw_reasons, dict): + reasons = { + key: value for key, value in raw_reasons.items() + if isinstance(key, str) and key in requested and isinstance(value, str) and value.strip() + } + + missing = requested - set(reasons) + if missing: + # Not an error. A course with no rationale renders without one, which is better + # than failing the pathway or inventing an explanation. + logger.info( + 'Rationale enrichment returned no reason for %d of %d course(s).', + len(missing), len(requested), + ) + + return {'reasons': reasons, 'prompt_revision': prompt_revision(prompt)} + + +def prompt_revision(prompt) -> str: + """ + Identify which stored prompt revision produced a result. + + django-simple-history captures every edit, so a rationale generated last week may have + come from wording that is no longer in the row. Without this, a change in tone between + runs cannot be attributed. + """ + history = getattr(prompt, 'history', None) + latest = history.first() if history is not None else None + if latest is not None and getattr(latest, 'history_id', None) is not None: + return str(latest.history_id) + modified = getattr(prompt, 'modified', None) + return modified.isoformat() if modified else '' diff --git a/enterprise_access/apps/pathways/apps.py b/enterprise_access/apps/pathways/apps.py new file mode 100644 index 00000000..47e364b9 --- /dev/null +++ b/enterprise_access/apps/pathways/apps.py @@ -0,0 +1,17 @@ +""" +App configuration for the learner pathways pipeline. +""" +from django.apps import AppConfig + + +class PathwaysConfig(AppConfig): + """ + Server-side learner pathway generation. + + Owns the workflows, steps and endpoints that turn a learner's intake into an + ordered course pathway. Pipeline logic lives here, as production code; the + evaluation harness in ``apps.pathway_eval`` only calls it and scores the result. + """ + default_auto_field = 'django.db.models.BigAutoField' + name = 'enterprise_access.apps.pathways' + verbose_name = 'Learner Pathways' diff --git a/enterprise_access/apps/pathways/catalog_translation.py b/enterprise_access/apps/pathways/catalog_translation.py new file mode 100644 index 00000000..a3ae45ec --- /dev/null +++ b/enterprise_access/apps/pathways/catalog_translation.py @@ -0,0 +1,266 @@ +""" +Domain-layer API for translating a career's vocabulary into the catalog's. + +Careers come from the Lightcast jobs index; courses come from the catalog index. This is +the join between them, and it is where the pipeline's measured quality problem lives. + +Three stages, deliberately separated because each has a different cost and failure mode: + +1. ``snapshot_catalog_facets`` — one zero-hit search that reads the scoped catalog's skill + facet vocabulary. Cheap, and the *only* authority on what will actually match. +2. ``translate_skills`` — pure resolution against that snapshot, no network. Recovers the + canonical form of a skill (``Python`` → ``Python (Programming Language)``) without a + model call or a maintained alias map. See ``skill_vocabulary``. +3. ``refine_unmatched_skills`` — a *conditional* second pass over Algolia's facet-search + endpoint, for terms the snapshot could not serve. + +Why stage 3 exists, and why it is conditional +--------------------------------------------- +A facet snapshot is capped at ``maxValuesPerFacet`` (1,000), and the live ``skill_names`` +facet returns exactly 1,000 values — i.e. it is truncated. Measured consequence: the +resolver dropped ``Welding`` and ``AWS`` even though ``skill_names:Welding`` matches a +real course, because they fall outside the top 1,000 by count. So a snapshot-only design +silently loses the long tail, which is disproportionately the non-technology vocabulary. + +Facet search has no such cap, but it costs one request per unresolved term. Most careers +resolve fully from the snapshot, so running it unconditionally would pay for nothing on +the common path — hence ``TranslateToCatalogStep`` gates it with ``should_execute``. + +Facet-search results are treated as *candidates only* and re-validated, because that +endpoint is not filtered to the enterprise catalog or to ``content_type:course``, and its +counts are inflated by per-customer record duplication. +""" +import logging +from typing import Any + +from enterprise_access.apps.api_client.algolia_client import AlgoliaClientError, AlgoliaSearchClient +from enterprise_access.apps.pathways.skill_vocabulary import ( + SKILL_FACET_FIELDS, + MatchType, + SkillMatch, + VocabularyIndex, + normalize_term, + resolve_skill_terms +) + +logger = logging.getLogger(__name__) + +# Facet fields read from the catalog. ``subjects`` is snapshotted for later use by the +# re-ranker but is never a skill-filter candidate. +CATALOG_FACET_FIELDS = (*SKILL_FACET_FIELDS, 'subjects') + +# Algolia's ceiling. Requesting it makes the truncation visible rather than silent: a +# facet that comes back with exactly this many values is almost certainly incomplete. +MAX_VALUES_PER_FACET = 1000 + +COURSE_SCOPE_FILTER = 'content_type:course' + +# Filter budgets, per the plan. Strict values become hard facet filters, so the cap is +# tighter -- each one can only narrow the result set, and an over-constrained query is the +# failure mode the retrieval ladder was invented to paper over. +MAX_STRICT_SKILLS = 8 +MAX_BOOST_SKILLS = 12 + +# How many facet-search candidates to consider per unresolved term. +FACET_SEARCH_HITS_PER_TERM = 20 + + +def snapshot_catalog_facets( + *, + secured_key=None, + allow_unscoped: bool = False, + algolia_client: AlgoliaSearchClient | None = None, +) -> dict[str, Any]: + """ + Read the scoped catalog's facet vocabulary with a single zero-hit search. + + ``hitsPerPage: 0`` because only the facets are wanted; the hits would be wasted + payload. Scoped to ``content_type:course`` so the vocabulary matches what course + retrieval will actually search — if the snapshot and the search disagree on scope, a + skill can be "grounded" against a value no course in scope carries. + + Returns a dict of facet field to values, plus a ``truncated`` list naming any facet + that came back at the cap and is therefore incomplete. + + Raises: + AlgoliaClientError: On a misconfigured credential or a failed search. + """ + client = algolia_client or AlgoliaSearchClient() + response = client.search_catalog_index( + '', + secured_key=secured_key, + allow_unscoped=allow_unscoped, + filters=COURSE_SCOPE_FILTER, + hitsPerPage=0, + facets=list(CATALOG_FACET_FIELDS), + maxValuesPerFacet=MAX_VALUES_PER_FACET, + ) + + facets = response.get('facets') or {} + snapshot: dict[str, Any] = {} + truncated = [] + for facet_field in CATALOG_FACET_FIELDS: + # Algolia omits a facet entirely when it has no values in scope. + values = list((facets.get(facet_field) or {}).keys()) + snapshot[facet_field] = values + if len(values) >= MAX_VALUES_PER_FACET: + truncated.append(facet_field) + + snapshot['truncated'] = truncated + if truncated: + logger.info( + 'Catalog facet snapshot is truncated at %d values for %s; ' + 'unresolved terms will need facet search to reach the long tail.', + MAX_VALUES_PER_FACET, truncated, + ) + return snapshot + + +def translate_skills(*, terms: list[str], facet_snapshot: dict[str, Any]) -> dict[str, Any]: + """ + Resolve skill terms against the snapshot and split them into strict and boost sets. + + High-confidence matches (exact, or a canonical ``term (qualifier)`` form) become + strict hard filters; weaker containment matches become boosts, because containment can + drift to a neighbouring concept and a wrong hard filter returns nothing. + + Returns a dict carrying ``strict``, ``boost``, ``unresolved`` and ``resolution_rate``, + shaped for direct persistence on a step record. + """ + result = resolve_skill_terms(terms, facet_snapshot) + + strict = [match for match in result.matches if match.is_high_confidence][:MAX_STRICT_SKILLS] + strict_values = {match.catalog_value for match in strict} + boost = [ + match for match in result.matches + if not match.is_high_confidence and match.catalog_value not in strict_values + ][:MAX_BOOST_SKILLS] + + return { + 'strict': [_match_to_dict(match) for match in strict], + 'boost': [_match_to_dict(match) for match in boost], + 'unresolved': result.unresolved, + 'resolution_rate': result.resolution_rate, + } + + +def _match_to_dict(match: SkillMatch) -> dict[str, str]: + return { + 'term': match.term, + 'catalog_value': match.catalog_value, + 'catalog_field': match.catalog_field, + 'match_type': match.match_type.value, + } + + +def refine_unmatched_skills( + *, + unresolved: list[str], + facet_snapshot: dict[str, Any], + secured_key=None, + allow_unscoped: bool = False, + algolia_client: AlgoliaSearchClient | None = None, +) -> dict[str, Any]: + """ + Recover unresolved terms via Algolia's facet-search endpoint. + + One request per term, so this is worth running only when the snapshot has already + failed — see the module docstring. + + A candidate is accepted only if it also resolves under the same rules the snapshot + path uses, applied to the candidate list. That keeps one definition of "this term + means that catalog value" rather than a looser second one, and it is what stops + ``AWS`` resolving to ``AWS Certified Solutions Architect Associate`` merely because + that is the highest-count candidate. + + A term that cannot be recovered stays unresolved. Failures of individual facet + searches are recorded, not raised: losing one term is better than failing the step. + """ + client = algolia_client or AlgoliaSearchClient() + known = {normalize_term(value) for values in + (facet_snapshot.get(field) or [] for field in SKILL_FACET_FIELDS) + for value in values} + + recovered = [] + still_unresolved = [] + errors = [] + + for term in unresolved: + candidates, error = _facet_search_candidates( + client, term, secured_key, allow_unscoped, + ) + if error: + errors.append(f'{term}: {error}') + still_unresolved.append(term) + continue + + # Candidates already in the snapshot were considered and rejected on the first + # pass; re-offering them would change the answer for no new information. + novel = [value for value in candidates if normalize_term(value) not in known] + match = VocabularyIndex({SKILL_FACET_FIELDS[0]: novel}).resolve(term) + if match is None: + still_unresolved.append(term) + continue + recovered.append(_match_to_dict(match)) + + if recovered: + logger.info( + 'Facet search recovered %d of %d term(s) missing from the capped snapshot.', + len(recovered), len(unresolved), + ) + + return { + 'recovered': recovered, + 'unresolved': still_unresolved, + 'errors': errors, + } + + +def _facet_search_candidates(client, term, secured_key, allow_unscoped): + """Return ``(candidate_values, error)`` for one term, never raising.""" + try: + hits = client.search_facet_values( + SKILL_FACET_FIELDS[0], + term, + secured_key=secured_key, + allow_unscoped=allow_unscoped, + max_facet_hits=FACET_SEARCH_HITS_PER_TERM, + ) + except AlgoliaClientError as exc: + logger.warning('Facet search failed for term %r: %s', term, exc) + return [], str(exc) + return [hit['value'] for hit in hits if hit.get('value')], None + + +def merge_refinement(translation: dict[str, Any], refinement: dict[str, Any]) -> dict[str, Any]: + """ + Fold recovered terms into a translation, respecting the original filter budgets. + + Recovered matches are appended rather than interleaved, so a snapshot match always + outranks a facet-search match for the same budget slot: the snapshot is the only + source that is definitely in scope. + """ + merged = dict(translation) + strict_values = {entry['catalog_value'] for entry in translation['strict']} + boost_values = {entry['catalog_value'] for entry in translation['boost']} + + strict_additions = [ + entry for entry in refinement['recovered'] + if entry['match_type'] in (MatchType.EXACT.value, MatchType.QUALIFIED.value) and + entry['catalog_value'] not in strict_values + ] + boost_additions = [ + entry for entry in refinement['recovered'] + if entry['match_type'] == MatchType.CONTAINED.value and + entry['catalog_value'] not in boost_values and + entry['catalog_value'] not in strict_values + ] + + merged['strict'] = (translation['strict'] + strict_additions)[:MAX_STRICT_SKILLS] + merged['boost'] = (translation['boost'] + boost_additions)[:MAX_BOOST_SKILLS] + merged['unresolved'] = refinement['unresolved'] + + resolved_count = len(merged['strict']) + len(merged['boost']) + total = resolved_count + len(merged['unresolved']) + merged['resolution_rate'] = (resolved_count / total) if total else None + return merged diff --git a/enterprise_access/apps/pathways/content_keys.py b/enterprise_access/apps/pathways/content_keys.py new file mode 100644 index 00000000..0aecc5cc --- /dev/null +++ b/enterprise_access/apps/pathways/content_keys.py @@ -0,0 +1,27 @@ +""" +The catalog's content-key vocabulary. + +Domain knowledge, not harness knowledge: what a course key looks like, and why a course +*run* key is never one. Lives here so the evaluation harness can depend on the domain +rather than keeping a second copy of the rule -- see ``docs/architecture-patterns.md`` +pattern 16, "Evaluation harnesses own no domain logic". +""" +import re + +# Course keys as they appear in the Algolia catalog index's ``key`` field: +# "+", e.g. "HarvardX+ER22.1x", "IBM+DA0101EN", "CodeSignal+34". +COURSE_KEY_PATTERN = re.compile(r'^[\w.\-]+\+[\w.\-]+$') + +# A course *run* key. Valid elsewhere in the platform, and wrong wherever a course key +# belongs: the catalog index keys courses, so a run key can never match a hit. +COURSE_RUN_KEY_PREFIX = 'course-v1:' + + +def is_course_run_key(key: str) -> bool: + """Whether ``key`` is a course *run* key rather than a course key.""" + return bool(key) and key.startswith(COURSE_RUN_KEY_PREFIX) + + +def is_valid_course_key(key: str) -> bool: + """Whether ``key`` is a well-formed catalog course key.""" + return bool(key) and not is_course_run_key(key) and bool(COURSE_KEY_PATTERN.match(key)) diff --git a/enterprise_access/apps/pathways/course_retrieval.py b/enterprise_access/apps/pathways/course_retrieval.py new file mode 100644 index 00000000..779044e1 --- /dev/null +++ b/enterprise_access/apps/pathways/course_retrieval.py @@ -0,0 +1,265 @@ +""" +Domain-layer API for retrieving course candidates from the catalog index. + +One broad query, then curate. This replaces the POC's four-step retrieval ladder, which +existed to compensate for a query too narrow to return anything -- the ladder was a +symptom, not a design. Widening once and selecting afterwards is the alternative the +Chunk 3 gate prescribed, and ``pathway_assembly`` is the "afterwards". + +Four things here are measured decisions rather than defaults, all against the pinned 2U +catalog on 2026-09-10: + +1. **Retrieve 20, not 5.** Relevance ranking is heavily introductory at rank 5 and + recovers by rank 20 -- ``data analyst`` returns 5/0/0 by level in its top 5 and + 16/3/1 in its top 20. Assembling a spanning set of 5 needs the wider window. +2. **``removeWordsIfNoResults: allOptional``.** The index ANDs every query word and has + no fallback configured, so an 8-word query returns *zero* hits rather than poor ones. + This buys volume, not relevance, so ``hit_count`` is persisted for scoring. +3. **A hard ``language`` filter.** 26.2% of the 2U catalog is taught in a language other + than English, and it appears from rank 6 -- exactly where (1) looks. Note this is + ``language`` (instruction), not ``metadata_language`` (record translation). +4. **Strict skill filters narrow, so a set that cannot form a ladder is broadened.** A + hard filter on a facet value buys precision, and it also shrinks the window *before* + ``pathway_assembly`` can span the difficulty rungs -- measured, a strict filter turned + ``Data Analyst`` from 2/2/1 into 5/0/0. So when the strict set is too thin *or* sits + entirely on one rung, a second unfiltered search runs and its hits are **appended** + rather than substituted: the precise courses keep their rank, and assembly gets the + width it needs. See ``MIN_CANDIDATES_FOR_ASSEMBLY`` and ``MIN_RUNGS_SPANNED``. +""" +import logging +from typing import Any + +from django.conf import settings + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchClient +from enterprise_access.apps.pathways.api import dedupe_names, is_malformed_compound +from enterprise_access.apps.pathways.pathway_assembly import LEVEL_ORDER, PATHWAY_SIZE, SUPPORTED_LANGUAGE + +logger = logging.getLogger(__name__) + +# The window ``pathway_assembly`` selects five courses out of. See (1) above. +CANDIDATE_HITS_PER_PAGE = 20 + +# Everything the re-ranker and the assembler need, and nothing else. Descriptions are the +# bulk of a course record, so they are requested here and nowhere upstream. +COURSE_ATTRIBUTES = [ + 'key', + 'title', + 'short_description', + 'full_description', + 'level_type', + 'partners', + 'language', + 'skill_names', + 'subjects', +] + +CONTENT_TYPE_FACET = 'content_type' +COURSE_CONTENT_TYPE = 'course' +LANGUAGE_FACET = 'language' +CUSTOMER_FACET = 'enterprise_customer_uuids' +SKILL_NAMES_FACET = 'skill_names' + +REMOVE_WORDS_IF_NO_RESULTS = 'allOptional' + +# Strict skill values become hard facet filters, so the budget is tight -- each one can +# only narrow. Boosts are optional filters and cost nothing but ranking signal. +MAX_STRICT_FILTERS = 4 +MAX_BOOST_FILTERS = 8 + +# Query length is capped rather than left to the intake. The index ANDs every word, and +# even with ``allOptional`` a very long query is mostly noise competing for ranking. +MAX_QUERY_WORDS = 12 + +# Below this many candidates, the strict-filtered set is broadened. Measured 2026-09-10 +# against the pinned 2U catalog: a strict skill filter narrows the window *before* +# assembly can span it, so precision bought at retrieval time costs the level ladder. +# +# career strict hits / mix loose hits / mix +# Data Analyst 17 5/0/0 20 2/2/1 +# Project Manager 15 3/2/0 20 2/2/1 +# Financial Analyst 2 1/1/0 12 2/2/1 +# Machine Learning Engineer 20 2/2/1 20 2/2/1 +# +MIN_CANDIDATES_FOR_ASSEMBLY = PATHWAY_SIZE * 3 + +# ...but a hit count alone is the wrong signal, and ``Data Analyst`` is the proof: 17 +# strict hits cleared the count above and *still* assembled to 5/0/0, because every one of +# them sat on the Introductory rung. What matters is whether the set can form a ladder at +# all, so the number of distinct populated rungs is checked as well. +MIN_RUNGS_SPANNED = 2 + + +def build_course_query(*, career_name: str, boost_terms: list[str]) -> str: + """ + Build the text query for course retrieval. + + The career name leads because it is the one phrase a learner would recognise; skill + terms follow to broaden it. Truncated at ``MAX_QUERY_WORDS`` on a word boundary. + """ + words: list[str] = [] + for part in [career_name, *boost_terms]: + for word in (part or '').split(): + if len(words) >= MAX_QUERY_WORDS: + return ' '.join(words) + words.append(word) + return ' '.join(words) + + +def build_course_filters(*, strict_skills: list[str], customer_uuid: str = '') -> str: + """ + Build the Algolia ``filters`` expression. + + ``content_type`` and ``language`` are unconditional; the customer scope is applied + when one is supplied. Strict skills are ``OR``-ed together rather than ``AND``-ed -- + a course rarely carries every skill of a career, and ``AND`` would routinely return + nothing. + """ + clauses = [ + f'{CONTENT_TYPE_FACET}:{COURSE_CONTENT_TYPE}', + f'{LANGUAGE_FACET}:"{SUPPORTED_LANGUAGE}"', + ] + if customer_uuid: + clauses.append(f'{CUSTOMER_FACET}:"{customer_uuid}"') + if strict_skills: + joined = ' OR '.join(f'{SKILL_NAMES_FACET}:"{value}"' for value in strict_skills) + clauses.append(f'({joined})') + return ' AND '.join(clauses) + + +def build_optional_skill_filters(boost_terms: list[str]) -> list[str]: + """Turn boost terms into Algolia ``optionalFilters`` on the skill facet.""" + return [ + f'{SKILL_NAMES_FACET}:{value}' + for value in boost_terms[:MAX_BOOST_FILTERS] + ] + + +def skill_values(translation: dict[str, Any], bucket: str, limit: int) -> list[str]: + """ + Read the catalog values out of one bucket of a ``translate_skills`` result. + + Compound artifacts ("SQL & Python") are dropped: they match no facet value, so they + spend a filter slot to boost nothing. + """ + entries = translation.get(bucket) or [] + values = [ + entry.get('catalog_value', '') if isinstance(entry, dict) else str(entry) + for entry in entries + ] + return [value for value in dedupe_names(values) if not is_malformed_compound(value)][:limit] + + +def rungs_spanned(hits) -> int: + """ + How many distinct difficulty rungs a candidate set populates. + + Counted rather than assumed from the hit count, because a large single-rung set is + exactly the case that produces five introductory courses. + """ + return len({ + hit.get('level_type') for hit in hits + if hit.get('level_type') in LEVEL_ORDER + }) + + +def retrieve_candidate_courses( + *, + career_name: str, + translation: dict[str, Any], + customer_uuid: str = '', + secured_key=None, + allow_unscoped: bool = False, + algolia_client: AlgoliaSearchClient | None = None, +) -> dict[str, Any]: + """ + Retrieve up to ``CANDIDATE_HITS_PER_PAGE`` course candidates for one career. + + Issues one search with the strict skill filters applied. When that set is too thin, or + sits on too few difficulty rungs to form a ladder, a second unfiltered search runs and + its hits are appended to the first -- not substituted. Appending keeps the + precisely-matched courses ahead of the broadly-matched ones, so assembly still prefers + them while having the width it needs. + + Whether the broadening fired is returned, because "this career has no courses in this + catalog" and "these skill values over-constrained a set that does exist" are different + diagnoses that lead to different work. + + Args: + career_name: The selected career's display name; leads the text query. + translation: A ``catalog_translation.translate_skills`` result. + customer_uuid: Enterprise customer to scope to. Empty means unscoped. + secured_key: Optional secured Algolia key for request-scoped traffic. + allow_unscoped: Permit the plain search key. Required for offline runs. + + Raises: + AlgoliaClientError: On a misconfigured credential or a failed search. + + Returns: + ``query``, ``hit_count``, ``courses``, ``strict_filters_applied``, + ``strict_hit_count``, ``strict_rungs_spanned``, ``broadened`` and ``zero_hits``. + """ + client = algolia_client or AlgoliaSearchClient() + strict = skill_values(translation, 'strict', MAX_STRICT_FILTERS) + boosts = skill_values(translation, 'boost', MAX_BOOST_FILTERS) + + query = build_course_query(career_name=career_name, boost_terms=boosts) + optional_filters = build_optional_skill_filters(boosts) + + def search(strict_skills): + params: dict[str, Any] = { + 'hitsPerPage': CANDIDATE_HITS_PER_PAGE, + 'attributesToRetrieve': COURSE_ATTRIBUTES, + 'removeWordsIfNoResults': REMOVE_WORDS_IF_NO_RESULTS, + 'filters': build_course_filters( + strict_skills=strict_skills, customer_uuid=customer_uuid, + ), + } + if optional_filters: + params['optionalFilters'] = optional_filters + response = client.search_catalog_index( + query, secured_key=secured_key, allow_unscoped=allow_unscoped, **params, + ) + return [hit for hit in (response.get('hits') or []) if isinstance(hit, dict)] + + hits = search(strict) + strict_hit_count = len(hits) + broadened = False + + strict_rungs = rungs_spanned(hits) + too_thin = len(hits) < MIN_CANDIDATES_FOR_ASSEMBLY + too_flat = strict_rungs < MIN_RUNGS_SPANNED + + if strict and (too_thin or too_flat): + logger.info( + 'Course retrieval returned %d hit(s) across %d rung(s) with %d strict skill ' + 'filter(s) (thin=%s, flat=%s); broadening.', + strict_hit_count, strict_rungs, len(strict), too_thin, too_flat, + ) + broadened = True + seen = {hit.get('key') for hit in hits} + hits = hits + [hit for hit in search([]) if hit.get('key') not in seen] + + if not hits: + # Not an error. "No courses for this career in this catalog" is a real answer, and + # the caller turns it into an explicit no-pathway rather than a padded one. + logger.info( + 'Course retrieval found no candidates for career %r (query=%r, broadened=%s).', + career_name, query, broadened, + ) + + return { + 'query': query, + 'hit_count': len(hits), + 'courses': hits, + 'strict_filters_applied': strict, + 'strict_hit_count': strict_hit_count, + 'strict_rungs_spanned': strict_rungs, + 'broadened': broadened, + 'zero_hits': not hits, + } + + +def eval_customer_uuid() -> str: + """The customer the offline harness scopes to, or empty when none is pinned.""" + return (getattr(settings, 'PATHWAYS_EVAL_CUSTOMER_UUID', '') or '').strip() diff --git a/enterprise_access/apps/pathways/migrations/0001_initial.py b/enterprise_access/apps/pathways/migrations/0001_initial.py new file mode 100644 index 00000000..cf375430 --- /dev/null +++ b/enterprise_access/apps/pathways/migrations/0001_initial.py @@ -0,0 +1,73 @@ +# Generated by Django 5.2.17 on 2026-09-09 23:31 + +import django.utils.timezone +import jsonfield.fields +import model_utils.fields +import uuid +from django.db import migrations, models + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ] + + operations = [ + migrations.CreateModel( + name='CareerDiscoveryWorkflow', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ], + options={ + 'abstract': False, + }, + ), + migrations.CreateModel( + name='ExtractIntentStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + migrations.CreateModel( + name='RetrieveCareersStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + ] diff --git a/enterprise_access/apps/pathways/migrations/0002_snapshotcatalogfacetsstep_translatetocatalogstep.py b/enterprise_access/apps/pathways/migrations/0002_snapshotcatalogfacetsstep_translatetocatalogstep.py new file mode 100644 index 00000000..5cbfc502 --- /dev/null +++ b/enterprise_access/apps/pathways/migrations/0002_snapshotcatalogfacetsstep_translatetocatalogstep.py @@ -0,0 +1,55 @@ +# Generated by Django 5.2.17 on 2026-09-10 00:05 + +import django.utils.timezone +import jsonfield.fields +import model_utils.fields +import uuid +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pathways', '0001_initial'), + ] + + operations = [ + migrations.CreateModel( + name='SnapshotCatalogFacetsStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + migrations.CreateModel( + name='TranslateToCatalogStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + ] diff --git a/enterprise_access/apps/pathways/migrations/0003_assemblepathwaystep_pathwayassemblyworkflow_and_more.py b/enterprise_access/apps/pathways/migrations/0003_assemblepathwaystep_pathwayassemblyworkflow_and_more.py new file mode 100644 index 00000000..bcd92261 --- /dev/null +++ b/enterprise_access/apps/pathways/migrations/0003_assemblepathwaystep_pathwayassemblyworkflow_and_more.py @@ -0,0 +1,91 @@ +# Generated by Django 5.2.17 on 2026-09-10 01:17 + +import django.utils.timezone +import jsonfield.fields +import model_utils.fields +import uuid +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pathways', '0002_snapshotcatalogfacetsstep_translatetocatalogstep'), + ] + + operations = [ + migrations.CreateModel( + name='AssemblePathwayStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + migrations.CreateModel( + name='PathwayAssemblyWorkflow', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ], + options={ + 'abstract': False, + }, + ), + migrations.CreateModel( + name='RerankCandidatesStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + migrations.CreateModel( + name='RetrieveCandidatesStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + ] diff --git a/enterprise_access/apps/pathways/migrations/0004_enrichrationalestep.py b/enterprise_access/apps/pathways/migrations/0004_enrichrationalestep.py new file mode 100644 index 00000000..f76fbbce --- /dev/null +++ b/enterprise_access/apps/pathways/migrations/0004_enrichrationalestep.py @@ -0,0 +1,36 @@ +# Generated by Django 5.2.17 on 2026-09-10 11:56 + +import django.utils.timezone +import jsonfield.fields +import model_utils.fields +import uuid +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pathways', '0003_assemblepathwaystep_pathwayassemblyworkflow_and_more'), + ] + + operations = [ + migrations.CreateModel( + name='EnrichRationaleStep', + fields=[ + ('created', model_utils.fields.AutoCreatedField(default=django.utils.timezone.now, editable=False, verbose_name='created')), + ('modified', model_utils.fields.AutoLastModifiedField(default=django.utils.timezone.now, editable=False, verbose_name='modified')), + ('is_removed', models.BooleanField(default=False)), + ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False, unique=True)), + ('input_data', jsonfield.fields.JSONField(blank=True, default=None)), + ('output_data', jsonfield.fields.JSONField(blank=True, default=None, null=True)), + ('succeeded_at', models.DateTimeField(blank=True, null=True)), + ('failed_at', models.DateTimeField(blank=True, null=True)), + ('exception_message', models.TextField(blank=True, null=True)), + ('workflow_record_uuid', models.UUIDField(help_text='UUID of the workflow record')), + ('preceding_step_uuid', models.UUIDField(help_text='UUID of the preceding workflow step record, if any', null=True)), + ], + options={ + 'abstract': False, + }, + ), + ] diff --git a/enterprise_access/apps/pathways/migrations/__init__.py b/enterprise_access/apps/pathways/migrations/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/enterprise_access/apps/pathways/model_backends/__init__.py b/enterprise_access/apps/pathways/model_backends/__init__.py new file mode 100644 index 00000000..9d5843d9 --- /dev/null +++ b/enterprise_access/apps/pathways/model_backends/__init__.py @@ -0,0 +1,76 @@ +""" +Model backends for the pathway pipeline, and the registry that selects one. + +Callers ask for a backend by configuration, never by import: + + backend = get_model_backend(prompt_type=PromptType.CANDIDATE_RERANK) + response = backend.complete(system_prompt=..., user_content=..., trace_id=...) + +That indirection is the point of the chunk. A step that imported ``ClaudeBackend`` +directly would make "which model produced this?" a code question rather than a +configuration one, and switching backends for a comparison run would need a deploy. +""" +from django.conf import settings + +from enterprise_access.apps.pathways.model_backends.base import ( + ModelBackend, + ModelBackendConfigurationError, + ModelBackendError, + ModelBackendRequestError, + ModelResponse, + ModelResponseParseError +) +from enterprise_access.apps.pathways.model_backends.claude import BACKEND_NAME as CLAUDE_BACKEND +from enterprise_access.apps.pathways.model_backends.claude import ClaudeBackend +from enterprise_access.apps.pathways.model_backends.openai import BACKEND_NAME as OPENAI_BACKEND +from enterprise_access.apps.pathways.model_backends.openai import OpenAIBackend +from enterprise_access.apps.pathways.model_backends.xpert import BACKEND_NAME as XPERT_BACKEND +from enterprise_access.apps.pathways.model_backends.xpert import XpertBackend + +__all__ = [ + 'CLAUDE_BACKEND', + 'OPENAI_BACKEND', + 'XPERT_BACKEND', + 'ClaudeBackend', + 'ModelBackend', + 'ModelBackendConfigurationError', + 'ModelBackendError', + 'ModelBackendRequestError', + 'ModelResponse', + 'ModelResponseParseError', + 'OpenAIBackend', + 'XpertBackend', + 'get_model_backend', +] + +BACKEND_NAMES = (XPERT_BACKEND, CLAUDE_BACKEND, OPENAI_BACKEND) + + +def get_model_backend(*, prompt_type: str, backend_name: str | None = None) -> ModelBackend: + """ + Return the configured model backend. + + Args: + prompt_type: Which stored prompt the Xpert backend should use. Required even when + the Claude backend is selected, so that flipping the setting does not change + the call signature at every call site. + backend_name: Overrides ``settings.PATHWAYS_MODEL_BACKEND``. Intended for a + comparison run that wants both backends in one process. + + Raises: + ModelBackendConfigurationError: If the name does not match a known backend. An + unknown name is a typo in configuration, and falling back to a default would + hide it -- while silently sending traffic to a *paid* backend nobody chose. + """ + name = (backend_name or settings.PATHWAYS_MODEL_BACKEND or '').strip().lower() + + if name == XPERT_BACKEND: + return XpertBackend(prompt_type=prompt_type) + if name == CLAUDE_BACKEND: + return ClaudeBackend() + if name == OPENAI_BACKEND: + return OpenAIBackend() + + raise ModelBackendConfigurationError( + f'{name!r} is not a known model backend. Expected one of {", ".join(BACKEND_NAMES)}.' + ) diff --git a/enterprise_access/apps/pathways/model_backends/base.py b/enterprise_access/apps/pathways/model_backends/base.py new file mode 100644 index 00000000..afcaf0f8 --- /dev/null +++ b/enterprise_access/apps/pathways/model_backends/base.py @@ -0,0 +1,168 @@ +""" +The model-backend contract: one interface over Xpert and a direct reasoning model. + +Production code, not harness code. The point is that "which model produced this pathway, +how many tokens did it cost, and how long did it take" becomes a query over persisted step +records rather than something a bespoke comparison rig has to instrument. So every backend +returns the same ``ModelResponse``, and the step that calls one records it verbatim. + +Nothing here logs prompt text or response bodies. The learner's intake is user-authored +content and a response can quote it back, so both stay out of logs at this layer -- the +same rule ``prompts/api.py`` already follows. Token counts and elapsed time are safe to +log and are the only things worth logging anyway. +""" +import logging +import time +from abc import ABC, abstractmethod +from dataclasses import dataclass, field + +logger = logging.getLogger(__name__) + + +class ModelBackendError(Exception): + """ + Base class for model-backend failures. + + Catch this to handle any backend error without caring which backend produced it, + which is the whole point of the adapter. + """ + + +class ModelBackendConfigurationError(ModelBackendError): + """ + Raised when a backend cannot be used as configured. + + Never transient: no request was sent, and retrying changes nothing. Kept distinct + from ``ModelBackendRequestError`` so a caller does not retry a missing API key. + """ + + +class ModelBackendRequestError(ModelBackendError): + """Raised when a backend was reachable but the request failed.""" + + +class ModelResponseParseError(ModelBackendError): + """Raised when a response could not be parsed into the shape the caller asked for.""" + + +@dataclass(frozen=True) +class ModelResponse: + """ + One completion, normalised across backends. + + ``input_tokens`` and ``output_tokens`` are ``None`` rather than ``0`` when a backend + does not report them -- Xpert does not -- because zero is a measurement and ``None`` + is the absence of one, and a cost report that silently treats unknown as free is worse + than one that says it does not know. + """ + + content: str + backend: str + model: str = '' + input_tokens: int | None = None + output_tokens: int | None = None + elapsed_ms: int = 0 + metadata: dict = field(default_factory=dict) + + @property + def total_tokens(self) -> int | None: + """Combined token count, or ``None`` if either side is unreported.""" + if self.input_tokens is None or self.output_tokens is None: + return None + return self.input_tokens + self.output_tokens + + def as_json(self): + """ + Parse ``content`` as JSON. + + Raises: + ModelResponseParseError: If the content is not JSON. The offending text is + deliberately not included -- it can quote the learner's own intake back. + """ + import json # pylint: disable=import-outside-toplevel + + try: + return json.loads(self.content.strip()) + except json.JSONDecodeError as exc: + raise ModelResponseParseError( + f'{self.backend} response was not valid JSON: {exc.msg} ' + f'(at position {exc.pos} of {len(self.content)} characters).' + ) from exc + + def to_trace_dict(self) -> dict: + """ + The subset safe to persist on a step record and to log. + + Excludes ``content``: a step stores its own parsed output, and keeping the raw + body out of the trace keeps user-authored text out of a second place. + """ + return { + 'backend': self.backend, + 'model': self.model, + 'input_tokens': self.input_tokens, + 'output_tokens': self.output_tokens, + 'elapsed_ms': self.elapsed_ms, + } + + +class ModelBackend(ABC): + """ + One way of issuing a completion. + + Subclasses implement ``_complete``; ``complete`` wraps it with timing so elapsed + milliseconds are measured identically for every backend rather than each one being + trusted to do its own arithmetic. + """ + + #: Short stable name, used in settings and persisted on traces. + name: str = '' + + @abstractmethod + def _complete(self, *, system_prompt: str, user_content: str, trace_id: str) -> ModelResponse: + """Issue the request. Timing is applied by ``complete``.""" + + def complete(self, *, system_prompt: str, user_content: str, trace_id: str) -> ModelResponse: + """ + Issue one completion and return a normalised response. + + Args: + system_prompt: The system instruction. + user_content: The user message. May contain learner-authored text, so it is + never logged. + trace_id: Identifier tying this call to a persisted step record. + + Raises: + ModelBackendConfigurationError: The backend is misconfigured; no request sent. + ModelBackendRequestError: The request was attempted and failed. + """ + started = time.monotonic() + try: + response = self._complete( + system_prompt=system_prompt, + user_content=user_content, + trace_id=trace_id, + ) + except ModelBackendError: + # Already typed by the backend. Logged with the trace id only -- no prompt, + # no response body, and no exception message that might embed either. + logger.warning( + 'Model backend %r failed for trace_id=%s.', self.name, trace_id, + ) + raise + + elapsed_ms = int((time.monotonic() - started) * 1000) + logger.info( + 'Model backend %r completed trace_id=%s in %dms (tokens in/out: %s/%s).', + self.name, trace_id, elapsed_ms, response.input_tokens, response.output_tokens, + ) + # Backends do not set their own timing; ``complete`` owns it so the number means + # the same thing everywhere. + return ModelResponse( + content=response.content, + backend=response.backend, + model=response.model, + input_tokens=response.input_tokens, + output_tokens=response.output_tokens, + elapsed_ms=elapsed_ms, + metadata=response.metadata, + ) diff --git a/enterprise_access/apps/pathways/model_backends/claude.py b/enterprise_access/apps/pathways/model_backends/claude.py new file mode 100644 index 00000000..8f6e9dd9 --- /dev/null +++ b/enterprise_access/apps/pathways/model_backends/claude.py @@ -0,0 +1,124 @@ +""" +The Claude backend: a direct reasoning-model call, metered. + +Exists so re-ranking can be evaluated against a model that reports token counts and takes +a caller-supplied system prompt. Xpert does neither, which makes cost comparison and +prompt iteration awkward on that path alone. + +``anthropic`` is imported lazily, inside the method. Two reasons, and the second is the +load-bearing one: the package is an optional paid dependency that a deployment which only +uses Xpert should not need installed, and importing at module scope would make every +import of this app fail when it is absent -- including the tests for the Xpert path. +""" +import logging + +from django.conf import settings + +from enterprise_access.apps.pathways.model_backends.base import ( + ModelBackend, + ModelBackendConfigurationError, + ModelBackendRequestError, + ModelResponse +) + +logger = logging.getLogger(__name__) + +BACKEND_NAME = 'claude' + +# Generous but bounded. A re-rank returns an ordered list of at most 20 keys with short +# rationales, so a run that wants more than this has gone wrong rather than gone long. +DEFAULT_MAX_TOKENS = 4096 + + +class ClaudeBackend(ModelBackend): + """ + Issues completions directly against the Anthropic Messages API. + + The system prompt is the caller's, not a stored row -- this backend is for evaluating + prompt variants that are not yet worth persisting as admin-editable configuration. + """ + + name = BACKEND_NAME + + def __init__(self, *, model: str | None = None, api_key: str | None = None, + max_tokens: int = DEFAULT_MAX_TOKENS, client=None): + self.model = model or settings.PATHWAYS_CLAUDE_MODEL + self.max_tokens = max_tokens + self._api_key = api_key or settings.ANTHROPIC_API_KEY + # Injected in tests so nothing here needs the package or the network. + self._client = client + + def _get_client(self): + """Build the Anthropic client, or explain precisely what is missing.""" + if self._client is not None: + return self._client + if not self._api_key: + raise ModelBackendConfigurationError( + 'ANTHROPIC_API_KEY is not configured, so the claude backend cannot be used. ' + 'Set it, or select the xpert backend via PATHWAYS_MODEL_BACKEND.' + ) + try: + import anthropic # pylint: disable=import-outside-toplevel + except ImportError as exc: + raise ModelBackendConfigurationError( + 'The anthropic package is not installed, so the claude backend cannot be ' + 'used. Install it, or select the xpert backend via PATHWAYS_MODEL_BACKEND.' + ) from exc + + self._client = anthropic.Anthropic(api_key=self._api_key) + return self._client + + def _complete(self, *, system_prompt: str, user_content: str, trace_id: str) -> ModelResponse: + """Issue one Messages API request.""" + client = self._get_client() + + try: + message = client.messages.create( + model=self.model, + max_tokens=self.max_tokens, + system=system_prompt, + messages=[{'role': 'user', 'content': user_content}], + ) + except ModelBackendConfigurationError: + raise + except Exception as exc: + # Deliberately broad: the SDK's exception hierarchy is not a dependency this + # module should take, and every failure here means the same thing to a caller. + # The message is the exception *type*, not its text, which can echo the + # request body back. + raise ModelBackendRequestError( + f'Anthropic request failed ({type(exc).__name__}) for model {self.model!r}.' + ) from exc + + return ModelResponse( + content=_extract_text(message), + backend=self.name, + model=getattr(message, 'model', self.model) or self.model, + input_tokens=_usage_value(message, 'input_tokens'), + output_tokens=_usage_value(message, 'output_tokens'), + metadata={'stop_reason': getattr(message, 'stop_reason', None)}, + ) + + +def _extract_text(message) -> str: + """ + Concatenate the text blocks of a Messages API response. + + Content is a list of typed blocks; non-text blocks are skipped rather than + stringified, so a future block type cannot silently corrupt a JSON payload. + """ + parts = [] + for block in getattr(message, 'content', None) or []: + text = getattr(block, 'text', None) + if text: + parts.append(text) + return ''.join(parts) + + +def _usage_value(message, attribute: str) -> int | None: + """Read one token count, returning ``None`` when the response omits usage.""" + usage = getattr(message, 'usage', None) + if usage is None: + return None + value = getattr(usage, attribute, None) + return value if isinstance(value, int) else None diff --git a/enterprise_access/apps/pathways/model_backends/openai.py b/enterprise_access/apps/pathways/model_backends/openai.py new file mode 100644 index 00000000..377cdc22 --- /dev/null +++ b/enterprise_access/apps/pathways/model_backends/openai.py @@ -0,0 +1,148 @@ +""" +The OpenAI backend: a direct metered call, for evaluating prompt and model variants. + +Same role as ``claude.py`` — a caller-supplied system prompt, reported token counts, and no +database row — so the two are interchangeable and comparable. Whether a model-class +difference exists is a question the harness should answer from persisted traces rather than +one anybody should assert, and that needs at least two metered backends to compare. + +``openai`` is imported lazily, inside the method, for the same two reasons ``anthropic`` +is: it is an optional paid dependency a deployment using only Xpert should not need +installed, and a module-scope import would make every import of this app fail when it is +absent — including the tests for the other backends. + +One deliberate difference from the Claude backend: this one asks for +``response_format={'type': 'json_object'}``. Every prompt in this pipeline requires JSON, +and OpenAI can enforce that server-side, which removes an entire failure mode rather than +handling it. Note it requires the word "JSON" to appear in the prompt; ours do. +""" +import logging + +from django.conf import settings + +from enterprise_access.apps.pathways.model_backends.base import ( + ModelBackend, + ModelBackendConfigurationError, + ModelBackendRequestError, + ModelResponse +) + +logger = logging.getLogger(__name__) + +BACKEND_NAME = 'openai' + +# Generous but bounded, matching the Claude backend. A re-rank returns an ordered list of +# at most 20 keys with short rationales, so a run wanting more has gone wrong not long. +DEFAULT_MAX_TOKENS = 4096 + + +class OpenAIBackend(ModelBackend): + """ + Issues completions directly against OpenAI's chat completions API. + + The system prompt is the caller's, not a stored row — this backend is for evaluating + variants that are not yet worth persisting as admin-editable configuration. + """ + + name = BACKEND_NAME + + def __init__(self, *, model: str | None = None, api_key: str | None = None, + max_tokens: int = DEFAULT_MAX_TOKENS, client=None): + self.model = model or settings.PATHWAYS_OPENAI_MODEL + self.max_tokens = max_tokens + self._api_key = api_key or settings.OPENAI_API_KEY + # Injected in tests so nothing here needs the package or the network. + self._client = client + + def _get_client(self): + """Build the OpenAI client, or explain precisely what is missing.""" + if self._client is not None: + return self._client + if not self._api_key: + raise ModelBackendConfigurationError( + 'OPENAI_API_KEY is not configured, so the openai backend cannot be used. ' + 'Set it, or select another backend via PATHWAYS_MODEL_BACKEND.' + ) + try: + import openai # pylint: disable=import-outside-toplevel + except ImportError as exc: + raise ModelBackendConfigurationError( + 'The openai package is not installed, so the openai backend cannot be ' + 'used. Install it, or select another backend via PATHWAYS_MODEL_BACKEND.' + ) from exc + + self._client = openai.OpenAI(api_key=self._api_key) + return self._client + + def _complete(self, *, system_prompt: str, user_content: str, trace_id: str) -> ModelResponse: + """Issue one chat-completions request.""" + client = self._get_client() + + try: + completion = client.chat.completions.create( + model=self.model, + max_tokens=self.max_tokens, + response_format={'type': 'json_object'}, + messages=[ + {'role': 'system', 'content': system_prompt}, + {'role': 'user', 'content': user_content}, + ], + ) + except ModelBackendConfigurationError: + raise + except Exception as exc: + # Deliberately broad, and reporting the exception *type* rather than its text: + # the SDK's message can echo the request body, which holds the learner's own + # intake. Taking the SDK's exception hierarchy as a dependency would also make + # this module care about a detail no caller distinguishes. + raise ModelBackendRequestError( + f'OpenAI request failed ({type(exc).__name__}) for model {self.model!r}.' + ) from exc + + return ModelResponse( + content=_first_message_content(completion), + backend=self.name, + model=getattr(completion, 'model', self.model) or self.model, + input_tokens=_usage_value(completion, 'prompt_tokens'), + output_tokens=_usage_value(completion, 'completion_tokens'), + metadata={'finish_reason': _first_finish_reason(completion)}, + ) + + +def _first_choice(completion): + """The first choice, or ``None`` when the response carries none.""" + choices = getattr(completion, 'choices', None) or [] + return choices[0] if choices else None + + +def _first_message_content(completion) -> str: + """ + The first choice's message content, or an empty string. + + Empty rather than raising: the caller already degrades an unusable response to + retrieval order, and a missing choice is that same case rather than a new one. + """ + choice = _first_choice(completion) + message = getattr(choice, 'message', None) if choice else None + return getattr(message, 'content', None) or '' + + +def _first_finish_reason(completion): + """Why generation stopped — ``length`` here means the response was truncated.""" + choice = _first_choice(completion) + return getattr(choice, 'finish_reason', None) if choice else None + + +def _usage_value(completion, attribute: str) -> int | None: + """ + Read one token count, returning ``None`` when the response omits usage. + + ``None`` rather than ``0``, per the ``ModelResponse`` contract: zero is a measurement, + and a cost report that silently treats unknown as free is worse than one that says it + does not know. + """ + usage = getattr(completion, 'usage', None) + if usage is None: + return None + value = getattr(usage, attribute, None) + return value if isinstance(value, int) else None diff --git a/enterprise_access/apps/pathways/model_backends/xpert.py b/enterprise_access/apps/pathways/model_backends/xpert.py new file mode 100644 index 00000000..8afb6aa2 --- /dev/null +++ b/enterprise_access/apps/pathways/model_backends/xpert.py @@ -0,0 +1,82 @@ +""" +The Xpert backend: the same path the live prompt endpoints already use. + +Wraps ``prompts/api.py`` rather than reimplementing it, so a prompt stays admin-editable +and versioned, and this backend cannot drift from the endpoints that share its prompts. + +Xpert reports no token counts, so ``input_tokens`` and ``output_tokens`` are ``None``. +That is a real limitation of comparing this backend against a metered one on cost, and it +is recorded as absence rather than papered over with zeros. +""" +import logging + +from django.conf import settings + +from enterprise_access.apps.pathways.api import prompt_revision +from enterprise_access.apps.pathways.model_backends.base import ( + ModelBackend, + ModelBackendConfigurationError, + ModelBackendRequestError, + ModelResponse +) +from enterprise_access.apps.prompts import api as prompts_api +from enterprise_access.apps.prompts.api_client import XpertAPIError +from enterprise_access.apps.prompts.models import XpertLearnerPathwaysSystemPrompt + +logger = logging.getLogger(__name__) + +BACKEND_NAME = 'xpert' + + +class XpertBackend(ModelBackend): + """ + Issues completions through Xpert, using a stored prompt. + + Unlike a direct model backend, the system prompt here is *not* supplied by the caller + -- it comes from the ``prompts`` app row for ``prompt_type``. A caller that passes + ``system_prompt`` gets it appended as context rather than replacing the stored prompt, + because silently overriding an admin-editable prompt would make the admin UI a lie. + """ + + name = BACKEND_NAME + + def __init__(self, prompt_type: str, *, tags=None): + self.prompt_type = prompt_type + self.tags = tags if tags is not None else settings.XPERT_LEARNER_PATHWAYS_RAG_TAGS + + def _complete(self, *, system_prompt: str, user_content: str, trace_id: str) -> ModelResponse: + """Send one message through Xpert with the stored prompt for ``prompt_type``.""" + try: + prompt = prompts_api.get_current_prompt( + prompt_model=XpertLearnerPathwaysSystemPrompt, + prompt_type=self.prompt_type, + ) + except prompts_api.PromptError as exc: + # No prompt row configured. Not transient -- no request was sent. + raise ModelBackendConfigurationError( + f'No active Xpert prompt for prompt_type={self.prompt_type!r}.' + ) from exc + + messages = [ + prompts_api.XpertRequestMessage(role='user', content=user_content), + ] + + try: + response = prompts_api.send_xpert_message( + prompt=prompt, + messages=messages, + conversation_id=trace_id, + tags=self.tags, + prompt_type=self.prompt_type, + ) + except (prompts_api.PromptError, XpertAPIError) as exc: + raise ModelBackendRequestError( + f'Xpert request failed for prompt_type={self.prompt_type!r}.' + ) from exc + + return ModelResponse( + content=response.content, + backend=self.name, + model=self.prompt_type, + metadata={'prompt_revision': prompt_revision(prompt)}, + ) diff --git a/enterprise_access/apps/pathways/models.py b/enterprise_access/apps/pathways/models.py new file mode 100644 index 00000000..b0eaae70 --- /dev/null +++ b/enterprise_access/apps/pathways/models.py @@ -0,0 +1,1123 @@ +""" +Models for the learner pathways pipeline: the conditional workflow base, and the +concrete career-discovery workflow built on it. + +The pathway pipeline needs one thing the provisioning workflows do not: a step that can +decide at run time that it has nothing to do. Provisioning steps are unconditional — every +step of provisioning an enterprise has to happen — so ``AbstractWorkflow`` has no notion +of skipping. + +``AbstractConditionalWorkflow`` adds exactly that, and nothing else. It is a subclass +rather than a change to ``apps/workflow/``: provisioning is live, and it must be +impossible for this feature to alter its behaviour. If the pattern proves out, upstreaming +it into ``apps/workflow/`` is a later conversation with that code's owner. +""" +# Subclassing costs us a copy of the parent's step loop: ``AbstractWorkflow.process_input`` +# runs it in one monolithic method, so there is no seam ``process_input`` below could reuse +# without editing that method -- which is exactly what this module declines to do. pylint +# only honours a duplicate-code disable at module scope, so it has to sit here rather than +# on the method. Drop it if the pattern is ever upstreamed. +# pylint: disable=duplicate-code +import logging +from typing import Optional + +from attrs import define, field, make_class, validators +from django.utils.functional import cached_property + +from enterprise_access.apps.pathways import api as pathways_api +from enterprise_access.apps.pathways import catalog_translation, course_retrieval, pathway_assembly, reranking +from enterprise_access.apps.prompts import api as prompts_api +from enterprise_access.apps.prompts.api_client import XpertAPIError +from enterprise_access.apps.workflow.exceptions import UnitOfWorkException +from enterprise_access.apps.workflow.models import AbstractWorkflow, AbstractWorkflowStep +from enterprise_access.apps.workflow.serialization import BaseInputOutput +from enterprise_access.toggles import learner_pathways_candidate_rerank_enabled + +logger = logging.getLogger(__name__) + + +class AbstractConditionalWorkflow(AbstractWorkflow): + """ + An ``AbstractWorkflow`` whose steps may opt out of executing. + + A step class may define:: + + @classmethod + def should_execute(cls, accumulated_output, workflow): + return ... + + Returning ``False`` skips the step: **no step record is created**, so a skipped step + is distinguishable from one that ran and produced nothing. Subsequent steps still + execute and still receive the accumulated output of the steps that did run, with the + skipped step's output key left unset. + + A step that does not define ``should_execute`` always executes, so a workflow whose + steps all omit it behaves exactly as ``AbstractWorkflow`` does. + + .. no_pii: This model has no PII + """ + + class Meta: + abstract = True + + @cached_property + def input_class(self): + """ + As ``AbstractWorkflow.input_class``, but with each field typed ``Optional``. + + See ``output_class`` for why. + """ + return self._make_io_class('Input', 'input_class') + + @cached_property + def output_class(self): + """ + As ``AbstractWorkflow.output_class``, but with each field typed ``Optional``. + + The parent builds this class with ``field(type=step_class.output_class, + default=None)`` -- the default is ``None`` but the declared type is not optional. + cattrs generates its (un)structure functions from the declared type, so it emits + code that dereferences every field unconditionally. Under ``AbstractWorkflow`` + that is safe, because every step always runs and therefore every field is always + populated. Once a step can be skipped, its field stays ``None`` and + ``to_dict()`` raises ``AttributeError: 'NoneType' object has no attribute ...`` + while serialising the workflow's own output. + + Declaring the fields ``Optional`` makes cattrs emit None-tolerant code, so a + skipped step round-trips as a ``null`` rather than crashing the run. + """ + return self._make_io_class('Output', 'output_class') + + def _make_io_class(self, suffix, step_attribute_name): + """ + Build the dynamic workflow input/output class for this workflow's step list. + + Mirrors the parent's use of ``attrs.make_class`` over the steps' ``KEY`` fields, + differing only in declaring each field ``Optional``. + """ + class_name = self.__class__.__name__ + suffix + attributes = {} + for step_class in self.steps: + step_io_class = getattr(step_class, step_attribute_name) + attributes[step_io_class.KEY] = field( + type=Optional[step_io_class], + default=None, + ) + return make_class(class_name, attributes, bases=(BaseInputOutput,)) + + @staticmethod + def step_should_execute(workflow_step_class, accumulated_output, workflow): + """ + Whether ``workflow_step_class`` should run, defaulting to ``True``. + + Kept as a separate method so the default-on behaviour is testable directly and so + a subclass can change the convention without reimplementing the loop. + """ + should_execute = getattr(workflow_step_class, 'should_execute', None) + if should_execute is None: + return True + return bool(should_execute(accumulated_output, workflow)) + + def process_input(self, accumulated_output=None, **kwargs): + """ + Execute each step in order, skipping any whose ``should_execute`` returns ``False``. + + Mirrors ``AbstractWorkflow.process_input`` -- including get-or-create of step + records, skipping steps that already succeeded, and accumulating output -- with + the conditional check added before a step record is created. + + Returns: + An instance of ``self.output_class`` accumulating each executed step's output. + """ + if self.succeeded_at: + logger.info( + '%s (uuid=%s) already succeeded at %s, skipping re-execution', + self.__class__.__name__, self.uuid, self.succeeded_at, + ) + return None + + accumulated_output = accumulated_output or self.output_class() + + logger.info( + 'Starting conditional workflow %s (uuid=%s) with steps=%s', + self.__class__.__name__, self.uuid, + [step_class.__name__ for step_class in self.steps], + ) + + preceding_step_record = None + for workflow_step_class in self.steps: + if not self.step_should_execute(workflow_step_class, accumulated_output, self): + logger.info( + 'Workflow %s (uuid=%s): step %s opted out, no step record created', + self.__class__.__name__, self.uuid, workflow_step_class.__name__, + ) + continue + + input_object = self.get_input_object_for_step_type(workflow_step_class) + input_data = input_object.to_dict() if input_object else {} + step_record_kwargs = { + 'workflow_record_uuid': self.uuid, + 'defaults': { + 'input_data': input_data, + } + } + if preceding_step_record: + step_record_kwargs['defaults']['preceding_step_uuid'] = preceding_step_record.uuid + + step_record, created = workflow_step_class.objects.get_or_create(**step_record_kwargs) + logger.info( + 'Workflow %s (uuid=%s): step %s record %s (created=%s, step_uuid=%s)', + self.__class__.__name__, self.uuid, workflow_step_class.__name__, + 'created' if created else 'reused', created, step_record.uuid, + ) + preceding_step_record = step_record + + if step_record.succeeded_at: + logger.info( + 'Workflow %s (uuid=%s): step %s (step_uuid=%s) already succeeded at %s, skipping', + self.__class__.__name__, self.uuid, workflow_step_class.__name__, + step_record.uuid, step_record.succeeded_at, + ) + setattr( + accumulated_output, + workflow_step_class.output_class.KEY, + step_record.output_object, + ) + continue + + step_output = step_record.execute(accumulated_output=accumulated_output) + setattr( + accumulated_output, + workflow_step_class.output_class.KEY, + step_output, + ) + + logger.info( + 'Completed conditional workflow %s (uuid=%s)', + self.__class__.__name__, self.uuid, + ) + return accumulated_output + + +############################################################################# +# Career discovery: the concrete workflow behind POST learner-pathways/careers/ +############################################################################# + +# Xpert conversation IDs are keyed on the *step record* rather than the request ID the +# prompt endpoints use. A step can be re-executed outside the request that created it +# (the runbook's remediation is re-running a workflow), and the step UUID is the one +# identifier that ties an Xpert conversation back to a persisted trace either way. +CONVERSATION_ID_PREFIX = 'enterprise-access:career-discovery' + +# Course descriptions are truncated before persistence. A full description can run to +# several kilobytes of marketing HTML, and five of them in one step record turns a +# trace into a blob; the re-ranker only needs enough to judge topical fit. +CANDIDATE_DESCRIPTION_CHARS = 1200 + +_is_str = validators.instance_of(str) +_is_int = validators.instance_of(int) +_is_str_list = validators.deep_iterable( + member_validator=validators.instance_of(str), + iterable_validator=validators.instance_of(list), +) + + +@define +class ExtractIntentInput(BaseInputOutput): + """ + The learner's intake, verbatim. Identical to ``LearningIntentRequestSerializer``'s + four fields, because the same prompt consumes both. + """ + KEY = 'extract_intent_input' + + selected_goals: str = field(validator=_is_str) + free_text: str = field(validator=_is_str) + known_context: str = field(validator=_is_str) + interested_industries: str = field(validator=_is_str) + + +@define +class ExtractIntentOutput(BaseInputOutput): + """ + What the learner's intake means, in the vocabulary the jobs index can be searched in. + """ + KEY = 'extract_intent_output' + + skills_required: list[str] = field(factory=list, validator=_is_str_list) + skills_preferred: list[str] = field(factory=list, validator=_is_str_list) + condensed_algolia_query: str = field(default='', validator=_is_str) + + +@define +class RetrieveCareersInput(BaseInputOutput): + """ + Hard-filter values for the jobs search. + + Both default to empty, and the careers endpoint leaves them that way. The intake's + ``interested_industries`` is learner free text ("healthcare, technology"), and a hard + filter on a value that is not a facet value returns zero hits *silently* -- the + failure mode the Chunk 3 diagnostic measured. Free text belongs in the text query, + where partial matching applies; these fields exist for a caller that has grounded + real facet values first. + """ + KEY = 'retrieve_careers_input' + + industries: list[str] = field(factory=list, validator=_is_str_list) + job_sources: list[str] = field(factory=list, validator=_is_str_list) + + +@define +class CareerCandidate(BaseInputOutput): + """ + One career from the Lightcast taxonomy. + + Identified by ``external_id``: taxonomy names are neither unique nor stable, so a + name cannot be a key. Carries no match percentage -- see + ``pathways.api.career_candidate_from_hit``. + """ + KEY = 'career_candidate' + + external_id: str = field(validator=_is_str) + name: str = field(validator=_is_str) + skills: list[str] = field(factory=list, validator=_is_str_list) + industries: list[str] = field(factory=list, validator=_is_str_list) + + +@define +class RetrieveCareersOutput(BaseInputOutput): + """ + The careers retrieved, plus what was actually asked of the index. + + ``query`` and ``hit_count`` are persisted because a full result set is not evidence + that retrieval worked: relaxing a query buys volume, not relevance. Recording both + lets a report tell a good retrieval from a padded one without re-running the search. + """ + KEY = 'retrieve_careers_output' + + careers: list[CareerCandidate] = field(factory=list) + query: str = field(default='', validator=_is_str) + hit_count: int = field(default=0, validator=_is_int) + + +class ExtractIntentStepException(UnitOfWorkException): + """Raised when learning intent could not be derived from the learner's intake.""" + + +class RetrieveCareersStepException(UnitOfWorkException): + """Raised when the jobs-index search for careers could not be completed.""" + + +class ExtractIntentStep(AbstractWorkflowStep): + """ + Derives skills and a search query from the learner's intake, via Xpert. + + Reuses the existing ``learner_intent`` prompt read-only, so this step and the live + ``learning-intent`` endpoint cannot drift apart. + + .. no_pii: Stores no user identifier. ``input_data`` holds the learner-authored intake + text submitted with the request, which is not linked to a user record. + """ + exception_class = ExtractIntentStepException + input_class = ExtractIntentInput + output_class = ExtractIntentOutput + + def process_input(self, accumulated_output=None, **kwargs): + result_dict = pathways_api.derive_learning_intent( + intake=self.input_object.to_dict(), + conversation_id=f'{CONVERSATION_ID_PREFIX}:{self.uuid}', + ) + return self.output_class.from_dict(result_dict) + + +class RetrieveCareersStep(AbstractWorkflowStep): + """ + Searches the Lightcast jobs index for careers matching the derived intent. + + .. no_pii: This model has no PII + """ + exception_class = RetrieveCareersStepException + input_class = RetrieveCareersInput + output_class = RetrieveCareersOutput + + def process_input(self, accumulated_output=None, **kwargs): + intent_output = getattr(accumulated_output, ExtractIntentOutput.KEY, None) + if intent_output is None: + raise self.exception_class( + f'{self.__class__.__name__} requires {ExtractIntentOutput.KEY} in the accumulated output.' + ) + + input_object = self.input_object + result_dict = pathways_api.retrieve_careers( + intent=intent_output.to_dict(), + industries=input_object.industries, + job_sources=input_object.job_sources, + ) + return self.output_class.from_dict(result_dict) + + +class CareerDiscoveryWorkflow(AbstractConditionalWorkflow): + """ + Intake in, career candidates out. + + Subclasses the conditional base rather than ``AbstractWorkflow`` because the pathway + workflows that extend this pipeline (facet translation onward) do have steps that opt + out. Neither step here defines ``should_execute``, so execution is identical to + ``AbstractWorkflow``'s today. + + .. no_pii: Stores no user identifier. ``input_data`` holds the learner-authored intake + text submitted with the request, which is not linked to a user record. + """ + steps = [ + ExtractIntentStep, + RetrieveCareersStep, + ] + + @classmethod + def generate_input_dict(cls, intake_data): + """ + Build ``input_data`` for a workflow record from validated intake fields. + + ``RetrieveCareersInput`` is deliberately left empty; see its docstring for why the + intake's free-text industries are not piped into a hard filter. + """ + return { + ExtractIntentInput.KEY: { + field_name: intake_data[field_name] + for field_name in ('selected_goals', 'free_text', 'known_context', 'interested_industries') + }, + RetrieveCareersInput.KEY: {}, + } + + def career_candidates(self): + """ + The retrieved careers, or an empty list if the retrieval step never succeeded. + + Reads the persisted output rather than an in-memory result so a completed run can + be re-serialized later without re-executing anything. + """ + careers_output = (self.output_data or {}).get(RetrieveCareersOutput.KEY) or {} + return careers_output.get('careers') or [] + + +# --------------------------------------------------------------------------------------- +# Chunk 7: catalog translation. These steps belong to the pathway-assembly workflow, which +# is built in Chunk 10; they are defined here so their tables and behaviour land with the +# translation work rather than with the workflow that composes them. +# --------------------------------------------------------------------------------------- + + +@define +class SnapshotCatalogFacetsInput(BaseInputOutput): + """ + Nothing is needed to take a snapshot beyond the credential the step already has. + + Kept as an explicit empty class rather than reusing ``Empty`` so the step has its own + ``KEY`` in the workflow's input/output classes. + """ + KEY = 'snapshot_catalog_facets_input' + + allow_unscoped: bool = field(default=False, validator=validators.instance_of(bool)) + + +@define +class SnapshotCatalogFacetsOutput(BaseInputOutput): + """ + The catalog's skill and subject vocabulary, as it exists in the searched scope. + + ``truncated`` names any facet that came back at Algolia's 1,000-value ceiling and is + therefore incomplete. It is persisted because it changes how the next step's result + should be read: a term unresolved against a truncated snapshot may still exist in the + catalog, and that is the difference between "not in this catalog" and "not in the + first thousand values". + """ + KEY = 'snapshot_catalog_facets_output' + + skill_names: list[str] = field(factory=list, validator=_is_str_list) + subjects: list[str] = field(factory=list, validator=_is_str_list) + truncated: list[str] = field(factory=list, validator=_is_str_list) + + def as_facet_snapshot(self): + """ + Rebuild the mapping the translation functions expect. + + ``skills.name`` is folded into ``skill_names`` on the way in, so this class has one + list rather than two; the resolver only needs to know which values exist, and + keeping a dotted attribute name off an attrs field avoids a serialization edge. + """ + return {'skill_names': list(self.skill_names), 'skills.name': []} + + +@define +class TranslateToCatalogInput(BaseInputOutput): + """ + The skill terms to translate: the selected career's skills plus the derived intent's. + + Both sources are passed in rather than read from the accumulated output, so this step + is usable by a caller that already knows which career was chosen -- the learner's + choice sits between career discovery and pathway assembly. + """ + KEY = 'translate_to_catalog_input' + + career_skills: list[str] = field(factory=list, validator=_is_str_list) + skills_required: list[str] = field(factory=list, validator=_is_str_list) + skills_preferred: list[str] = field(factory=list, validator=_is_str_list) + allow_unscoped: bool = field(default=False, validator=validators.instance_of(bool)) + + +@define +class SkillFilter(BaseInputOutput): + """One resolved skill, and how confidently it was resolved.""" + KEY = 'skill_filter' + + term: str = field(validator=_is_str) + catalog_value: str = field(validator=_is_str) + catalog_field: str = field(validator=_is_str) + match_type: str = field(validator=_is_str) + + +@define +class TranslateToCatalogOutput(BaseInputOutput): + """ + Catalog-valid facet values, split by how they may be used. + + ``unresolved`` and ``resolution_rate`` are first-class output, not log lines: a + dropped skill was previously invisible, and "how much of this career could the catalog + even express?" is the question the retrieval diagnostic showed we most need answered. + """ + KEY = 'translate_to_catalog_output' + + strict: list[SkillFilter] = field(factory=list) + boost: list[SkillFilter] = field(factory=list) + unresolved: list[str] = field(factory=list, validator=_is_str_list) + resolution_rate: float | None = field(default=None) + refined: bool = field(default=False, validator=validators.instance_of(bool)) + + +class SnapshotCatalogFacetsStepException(UnitOfWorkException): + """Raised when the catalog facet vocabulary could not be read.""" + + +class TranslateToCatalogStepException(UnitOfWorkException): + """Raised when career skills could not be translated into catalog facet values.""" + + +class SnapshotCatalogFacetsStep(AbstractWorkflowStep): + """ + Reads the scoped catalog's facet vocabulary. + + Separate from translation because it is one cheap request whose result is reusable, + while translation is pure computation. Splitting them means a re-run of a failed + translation does not re-fetch the snapshot. + + .. no_pii: This model has no PII + """ + exception_class = SnapshotCatalogFacetsStepException + input_class = SnapshotCatalogFacetsInput + output_class = SnapshotCatalogFacetsOutput + + def process_input(self, accumulated_output=None, **kwargs): + snapshot = catalog_translation.snapshot_catalog_facets( + allow_unscoped=self.input_object.allow_unscoped, + ) + # The two skill facets are merged: the resolver only cares which values exist, and + # `skill_names` already takes precedence on a collision. + skill_names = list(dict.fromkeys( + (snapshot.get('skill_names') or []) + (snapshot.get('skills.name') or []) + )) + return self.output_class( + skill_names=skill_names, + subjects=snapshot.get('subjects') or [], + truncated=snapshot.get('truncated') or [], + ) + + +class TranslateToCatalogStep(AbstractWorkflowStep): + """ + Resolves career and intent skills onto real catalog facet values. + + Runs the cheap pure-resolution pass first, then -- **only if terms remain + unresolved** -- a facet-search refinement that costs one request per term. That + condition is why this pipeline needs ``AbstractConditionalWorkflow``: on the common + path the refinement is skipped entirely, and whether it fired is recorded on the + output so the harness can count how often the snapshot was insufficient. + + .. no_pii: This model has no PII + """ + exception_class = TranslateToCatalogStepException + input_class = TranslateToCatalogInput + output_class = TranslateToCatalogOutput + + def process_input(self, accumulated_output=None, **kwargs): + snapshot_output = getattr(accumulated_output, SnapshotCatalogFacetsOutput.KEY, None) + if snapshot_output is None: + raise self.exception_class( + 'Cannot translate skills without a catalog facet snapshot; ' + f'{SnapshotCatalogFacetsStep.__name__} must run first.' + ) + + facet_snapshot = snapshot_output.as_facet_snapshot() + terms = list(dict.fromkeys( + self.input_object.career_skills + + self.input_object.skills_required + + self.input_object.skills_preferred + )) + + translation = catalog_translation.translate_skills( + terms=terms, + facet_snapshot=facet_snapshot, + ) + + refined = False + if translation['unresolved']: + refinement = catalog_translation.refine_unmatched_skills( + unresolved=translation['unresolved'], + facet_snapshot=facet_snapshot, + allow_unscoped=self.input_object.allow_unscoped, + ) + translation = catalog_translation.merge_refinement(translation, refinement) + refined = True + + return self.output_class( + strict=[SkillFilter.from_dict(entry) for entry in translation['strict']], + boost=[SkillFilter.from_dict(entry) for entry in translation['boost']], + unresolved=translation['unresolved'], + resolution_rate=translation['resolution_rate'], + refined=refined, + ) + + +# --------------------------------------------------------------------------------------- +# Chunks 8-10: course retrieval, re-ranking, and pathway assembly. +# --------------------------------------------------------------------------------------- + +# Trace prefix for the re-rank model call. Keyed on the step record for the same reason +# career discovery's is -- a step can be re-executed outside the request that created it. +RERANK_TRACE_PREFIX = 'enterprise-access:pathway-rerank' +ENRICH_TRACE_PREFIX = 'enterprise-access:pathway-rationale' + + +@define +class RetrieveCandidatesInput(BaseInputOutput): + """ + Which career to retrieve courses for, and the scope to retrieve them in. + + ``career_name`` rather than ``external_id``: the catalog index knows nothing about + Lightcast identifiers, so the name is what can actually be searched. The id stays on + the career-discovery output for attribution. + """ + KEY = 'retrieve_candidates_input' + + career_name: str = field(default='', validator=_is_str) + customer_uuid: str = field(default='', validator=_is_str) + allow_unscoped: bool = field(default=False, validator=validators.instance_of(bool)) + + +@define +class CourseCandidate(BaseInputOutput): + """ + One retrieved course, carrying what the re-ranker and assembler need. + + Both descriptions are held because the re-ranker judges topical fit and a title alone + is often ambiguous ("Foundations of Client Care 2" says little about its subject). + """ + KEY = 'course_candidate' + + key: str = field(validator=_is_str) + title: str = field(default='', validator=_is_str) + short_description: str = field(default='', validator=_is_str) + full_description: str = field(default='', validator=_is_str) + level_type: str = field(default='', validator=_is_str) + partner: str = field(default='', validator=_is_str) + language: str = field(default='', validator=_is_str) + + @classmethod + def from_hit(cls, hit): + """Build a candidate from a raw catalog hit.""" + candidate = pathway_assembly.Candidate.from_hit(hit) + return cls( + key=candidate.key, + title=candidate.title, + short_description=(hit.get('short_description') or '')[:CANDIDATE_DESCRIPTION_CHARS], + full_description=(hit.get('full_description') or '')[:CANDIDATE_DESCRIPTION_CHARS], + level_type=candidate.level_type, + partner=candidate.partner, + language=candidate.language, + ) + + def to_assembly_hit(self): + """Render back into the hit shape ``pathway_assembly`` consumes.""" + return { + 'key': self.key, + 'title': self.title, + 'level_type': self.level_type, + 'partners': [{'name': self.partner}] if self.partner else [], + 'language': self.language, + } + + +@define +class RetrieveCandidatesOutput(BaseInputOutput): + """ + The retrieved candidate window, plus what was asked of the index. + + ``zero_hits`` and ``broadened`` are first-class output because they are different + diagnoses: nothing in this catalog for this career, versus skill filters that + over-constrained a set that does exist. The original plan recorded a "scope-only + fallback" here instead, which measures a ladder step this design removed. + + ``strict_hit_count`` is kept alongside ``hit_count`` so a report can say how much of + the window was precisely matched rather than merely broadly matched. + """ + KEY = 'retrieve_candidates_output' + + courses: list[CourseCandidate] = field(factory=list) + query: str = field(default='', validator=_is_str) + hit_count: int = field(default=0, validator=_is_int) + strict_filters_applied: list[str] = field(factory=list, validator=_is_str_list) + strict_hit_count: int = field(default=0, validator=_is_int) + strict_rungs_spanned: int = field(default=0, validator=_is_int) + broadened: bool = field(default=False, validator=validators.instance_of(bool)) + zero_hits: bool = field(default=True, validator=validators.instance_of(bool)) + + +@define +class RerankCandidatesInput(BaseInputOutput): + """ + What the re-ranker needs beyond the candidate set it reads from accumulated output. + + ``enabled`` exists so the model call can be turned off per run without a code change. + Chunk 9a's deterministic assembly produces a valid pathway on its own, so a run with + the model disabled is a meaningful baseline rather than a broken one -- and it is the + A/B that says what the model is worth. + """ + KEY = 'rerank_candidates_input' + + career_name: str = field(default='', validator=_is_str) + enabled: bool = field(default=True, validator=validators.instance_of(bool)) + + +@define +class RerankCandidatesOutput(BaseInputOutput): + """ + The re-ranked key order, and what had to be discarded to trust it. + + ``fabricated_keys`` turns the platform's known key-invention defect into a counted + metric rather than an anecdote. ``backend``/``model``/token counts come straight from + ``ModelResponse.to_trace_dict``, so model comparison is a query over these records. + """ + KEY = 'rerank_candidates_output' + + ordered_keys: list[str] = field(factory=list, validator=_is_str_list) + rationales: dict = field(factory=dict) + fabricated_keys: list[str] = field(factory=list, validator=_is_str_list) + executed: bool = field(default=False, validator=validators.instance_of(bool)) + backend: str = field(default='', validator=_is_str) + model: str = field(default='', validator=_is_str) + prompt_revision: str = field(default='', validator=_is_str) + input_tokens: int | None = field(default=None) + output_tokens: int | None = field(default=None) + elapsed_ms: int = field(default=0, validator=_is_int) + + +@define +class AssemblePathwayInput(BaseInputOutput): + """Nothing beyond what the preceding steps produced.""" + KEY = 'assemble_pathway_input' + + +@define +class PathwayCourse(BaseInputOutput): + """One course in a delivered pathway, in its taught order.""" + KEY = 'pathway_course' + + key: str = field(validator=_is_str) + title: str = field(default='', validator=_is_str) + level_type: str = field(default='', validator=_is_str) + partner: str = field(default='', validator=_is_str) + rationale: str = field(default='', validator=_is_str) + + +@define +class AssemblePathwayOutput(BaseInputOutput): + """ + The delivered pathway, or an explicit absence of one. + + ``violations`` carries the Tier 1 gate results. They are persisted rather than raised + because a pathway that fails a correctness gate is a bug worth *seeing* in a harness + run -- raising would hide it behind a failed workflow with no comparable trace. + """ + KEY = 'assemble_pathway_output' + + courses: list[PathwayCourse] = field(factory=list) + complete: bool = field(default=False, validator=validators.instance_of(bool)) + unfilled_rungs: list[str] = field(factory=list, validator=_is_str_list) + level_mix: dict = field(factory=dict) + ineligible: dict = field(factory=dict) + violations: list[str] = field(factory=list, validator=_is_str_list) + + +@define +class EnrichRationaleInput(BaseInputOutput): + """ + What the rationale prompt needs beyond the assembled pathway. + + ``learner_profile`` is the learner's intake, passed through to the existing + ``recommendations_feedback`` prompt in the shape that endpoint already sends. + """ + KEY = 'enrich_rationale_input' + + selected_career: str = field(default='', validator=_is_str) + learner_profile: dict = field(factory=dict) + enabled: bool = field(default=True, validator=validators.instance_of(bool)) + + +@define +class EnrichRationaleOutput(BaseInputOutput): + """ + One rationale per delivered course, plus the prompt revision that produced them. + + A course with no rationale is normal, not a failure: it renders without one, which is + better than failing the pathway or inventing an explanation. + """ + KEY = 'enrich_rationale_output' + + reasons: dict = field(factory=dict) + executed: bool = field(default=False, validator=validators.instance_of(bool)) + prompt_revision: str = field(default='', validator=_is_str) + error: str = field(default='', validator=_is_str) + + +class RetrieveCandidatesStepException(UnitOfWorkException): + """Raised when course candidates could not be retrieved.""" + + +class RerankCandidatesStepException(UnitOfWorkException): + """Raised when the candidate set could not be re-ranked.""" + + +class AssemblePathwayStepException(UnitOfWorkException): + """Raised when a pathway could not be assembled from the candidate set.""" + + +class EnrichRationaleStepException(UnitOfWorkException): + """Raised when per-course rationales could not be generated.""" + + +class RetrieveCandidatesStep(AbstractWorkflowStep): + """ + Retrieves the candidate window for the selected career. + + One broad query, per the Chunk 3 gate, rather than the POC's four-step ladder. + + .. no_pii: This model has no PII + """ + exception_class = RetrieveCandidatesStepException + input_class = RetrieveCandidatesInput + output_class = RetrieveCandidatesOutput + + def process_input(self, accumulated_output=None, **kwargs): + translation_output = getattr(accumulated_output, TranslateToCatalogOutput.KEY, None) + if translation_output is None: + raise self.exception_class( + 'Cannot retrieve candidates without a catalog translation; ' + f'{TranslateToCatalogStep.__name__} must run first.' + ) + + result = course_retrieval.retrieve_candidate_courses( + career_name=self.input_object.career_name, + translation=translation_output.to_dict(), + customer_uuid=self.input_object.customer_uuid, + allow_unscoped=self.input_object.allow_unscoped, + ) + return self.output_class( + courses=[CourseCandidate.from_hit(hit) for hit in result['courses']], + query=result['query'], + hit_count=result['hit_count'], + strict_filters_applied=result['strict_filters_applied'], + strict_hit_count=result['strict_hit_count'], + strict_rungs_spanned=result['strict_rungs_spanned'], + broadened=result['broadened'], + zero_hits=result['zero_hits'], + ) + + +class RerankCandidatesStep(AbstractWorkflowStep): + """ + Orders the candidate set for topical fit, via the configured model backend. + + Skipped when disabled or when there is nothing to re-rank -- which is what + ``AbstractConditionalWorkflow`` is for. A skipped re-rank is not a failure: Chunk 9a's + deterministic assembly produces a valid pathway from the unordered candidate set, so + the model's contribution is measurable as a delta rather than assumed. + + Chunk 9a handles the structural guarantees (level spread, provider cap, duplicates) + deterministically, so nothing here asks the model for them. What is asked is the one + thing assembly demonstrably cannot do: keep topically unrelated courses out. + + .. no_pii: This model has no PII + """ + exception_class = RerankCandidatesStepException + input_class = RerankCandidatesInput + output_class = RerankCandidatesOutput + + @classmethod + def should_execute(cls, accumulated_output, workflow): + """ + Run only when the switch is on, the caller asked for it, and there is something + to order. + + The administrator switch is checked *first* and independently of the workflow's + own ``enabled`` input, so turning it off stops paid model calls for every caller + at once -- including harness runs, which supply their own input and would + otherwise ignore it. + """ + if not learner_pathways_candidate_rerank_enabled(): + return False + rerank_input = (workflow.input_data or {}).get(RerankCandidatesInput.KEY) or {} + if not rerank_input.get('enabled', True): + return False + candidates_output = getattr(accumulated_output, RetrieveCandidatesOutput.KEY, None) + return bool(candidates_output and candidates_output.courses) + + def process_input(self, accumulated_output=None, **kwargs): + candidates_output = getattr(accumulated_output, RetrieveCandidatesOutput.KEY, None) + if candidates_output is None: + raise self.exception_class( + 'Cannot re-rank without a candidate set; ' + f'{RetrieveCandidatesStep.__name__} must run first.' + ) + + result = reranking.rerank_candidates( + career_name=self.input_object.career_name, + candidates=[candidate.to_dict() for candidate in candidates_output.courses], + trace_id=f'{RERANK_TRACE_PREFIX}:{self.uuid}', + ) + return self.output_class( + ordered_keys=result['ordered_keys'], + rationales=result['rationales'], + fabricated_keys=result['fabricated_keys'], + executed=True, + backend=result['trace'].get('backend', ''), + model=result['trace'].get('model', ''), + prompt_revision=result.get('prompt_revision', ''), + input_tokens=result['trace'].get('input_tokens'), + output_tokens=result['trace'].get('output_tokens'), + elapsed_ms=result['trace'].get('elapsed_ms', 0), + ) + + +class AssemblePathwayStep(AbstractWorkflowStep): + """ + Selects the delivered five courses and records the Tier 1 gate results. + + Reads the re-rank order when it ran and falls back to retrieval order when it did not, + so a disabled or skipped model call still yields a pathway. + + .. no_pii: This model has no PII + """ + exception_class = AssemblePathwayStepException + input_class = AssemblePathwayInput + output_class = AssemblePathwayOutput + + def process_input(self, accumulated_output=None, **kwargs): + candidates_output = getattr(accumulated_output, RetrieveCandidatesOutput.KEY, None) + if candidates_output is None: + raise self.exception_class( + 'Cannot assemble a pathway without a candidate set; ' + f'{RetrieveCandidatesStep.__name__} must run first.' + ) + + rerank_output = getattr(accumulated_output, RerankCandidatesOutput.KEY, None) + ordered = self.order_candidates(candidates_output.courses, rerank_output) + rationales = dict(rerank_output.rationales) if rerank_output else {} + + assembly = pathway_assembly.assemble_pathway( + [candidate.to_assembly_hit() for candidate in ordered] + ) + violations = ( + pathway_assembly.validate_pathway(assembly.courses) + if assembly.is_complete else [] + ) + + return self.output_class( + courses=[ + PathwayCourse( + key=course.key, + title=course.title, + level_type=course.level_type, + partner=course.partner, + rationale=rationales.get(course.key, ''), + ) + for course in assembly.courses + ], + complete=assembly.is_complete, + unfilled_rungs=assembly.unfilled_rungs, + level_mix=assembly.realised_level_mix, + ineligible=assembly.ineligible, + violations=violations, + ) + + @staticmethod + def order_candidates(candidates, rerank_output): + """ + Apply the re-rank order when one exists, keeping unranked candidates behind it. + + Unranked candidates are appended rather than dropped: the model may return fewer + keys than it was given, and discarding the remainder would shrink the window that + assembly needs to span the rungs. + """ + if not rerank_output or not rerank_output.ordered_keys: + return list(candidates) + + by_key = {candidate.key: candidate for candidate in candidates} + ordered = [by_key[key] for key in rerank_output.ordered_keys if key in by_key] + ranked_keys = {candidate.key for candidate in ordered} + return ordered + [ + candidate for candidate in candidates if candidate.key not in ranked_keys + ] + + +class EnrichRationaleStep(AbstractWorkflowStep): + """ + Generates one rationale per delivered course, via the stored feedback prompt. + + Runs **after** assembly and on the delivered five, not the candidate twenty. Two + reasons, and both are why this is a separate step rather than a field on the re-rank + response: + + * It reuses the existing ``recommendations_feedback`` prompt read-only, so the wording + a learner sees cannot drift from the live MFE endpoint's, and it stays + admin-editable and versioned. A rationale taken off the re-rank response would come + from a prompt chosen for ordering, and under the Claude backend from one that is not + in the database at all. + * Explaining and ordering are different jobs with different failure modes. A bad + rationale must not be able to reorder a pathway. + + Skipped when disabled or when there is no pathway to explain -- a third genuine + consumer of ``AbstractConditionalWorkflow``. A failure is recorded on the output + rather than raised: a pathway with no rationales is still a pathway, and losing the + explanations is a much smaller loss than losing the recommendation. + + .. no_pii: Stores no user identifier. ``input_data`` holds the learner-authored intake + text submitted with the request, which is not linked to a user record. + """ + exception_class = EnrichRationaleStepException + input_class = EnrichRationaleInput + output_class = EnrichRationaleOutput + + @classmethod + def should_execute(cls, accumulated_output, workflow): + """Run only when enabled and a complete pathway exists to explain.""" + enrich_input = (workflow.input_data or {}).get(EnrichRationaleInput.KEY) or {} + if not enrich_input.get('enabled', True): + return False + assembly_output = getattr(accumulated_output, AssemblePathwayOutput.KEY, None) + return bool(assembly_output and assembly_output.complete) + + def process_input(self, accumulated_output=None, **kwargs): + assembly_output = getattr(accumulated_output, AssemblePathwayOutput.KEY, None) + if assembly_output is None: + raise self.exception_class( + 'Cannot enrich rationales without an assembled pathway; ' + f'{AssemblePathwayStep.__name__} must run first.' + ) + + course_keys = [course.key for course in assembly_output.courses] + try: + result = pathways_api.enrich_rationales( + selected_career=self.input_object.selected_career, + course_keys=course_keys, + learner_profile=self.input_object.learner_profile, + conversation_id=f'{ENRICH_TRACE_PREFIX}:{self.uuid}', + ) + except (prompts_api.PromptError, XpertAPIError) as exc: + # Recorded, not raised. The pathway is already assembled and valid. + logger.warning('Rationale enrichment failed (%s); pathway ships unexplained.', + type(exc).__name__) + return self.output_class(executed=True, error=f'{type(exc).__name__}: {exc}') + + return self.output_class( + reasons=result['reasons'], + executed=True, + prompt_revision=result['prompt_revision'], + ) + + +class PathwayAssemblyWorkflow(AbstractConditionalWorkflow): + """ + Selected career in, five ordered courses out. + + The full pathway pipeline: read the catalog's vocabulary, translate the career's + skills into it, retrieve a candidate window, optionally re-rank, then assemble. + + Two steps here are conditional, which is what this workflow needs the conditional + base for: the facet-search refinement inside ``TranslateToCatalogStep``, and + ``RerankCandidatesStep`` as a whole. + + .. no_pii: Stores no user identifier. ``input_data`` holds a career name and skill + terms, which are not linked to a user record. + """ + steps = [ + SnapshotCatalogFacetsStep, + TranslateToCatalogStep, + RetrieveCandidatesStep, + RerankCandidatesStep, + AssemblePathwayStep, + EnrichRationaleStep, + ] + + @classmethod + def generate_input_dict(cls, *, career_name, career_skills=None, skills_required=None, + skills_preferred=None, customer_uuid='', allow_unscoped=False, + rerank_enabled=True, enrich_enabled=True, learner_profile=None): + """Build ``input_data`` for a pathway run.""" + return { + SnapshotCatalogFacetsInput.KEY: {'allow_unscoped': allow_unscoped}, + TranslateToCatalogInput.KEY: { + 'career_skills': list(career_skills or []), + 'skills_required': list(skills_required or []), + 'skills_preferred': list(skills_preferred or []), + 'allow_unscoped': allow_unscoped, + }, + RetrieveCandidatesInput.KEY: { + 'career_name': career_name, + 'customer_uuid': customer_uuid, + 'allow_unscoped': allow_unscoped, + }, + RerankCandidatesInput.KEY: { + 'career_name': career_name, + 'enabled': rerank_enabled, + }, + AssemblePathwayInput.KEY: {}, + EnrichRationaleInput.KEY: { + 'selected_career': career_name, + 'learner_profile': dict(learner_profile or {}), + 'enabled': enrich_enabled, + }, + } + + def pathway(self): + """ + The assembled pathway as a plain dict, or ``None`` if assembly never succeeded. + + Reads persisted output so a completed run can be re-serialized without + re-executing anything, and folds in the rationales the enrichment step produced. + Merging here rather than in ``AssemblePathwayStep`` keeps the two steps + independent: assembly cannot depend on a step that runs after it, and a skipped or + failed enrichment leaves the pathway intact with empty rationales. + """ + output = (self.output_data or {}).get(AssemblePathwayOutput.KEY) + if not output or not output.get('complete'): + return None + + enrichment = (self.output_data or {}).get(EnrichRationaleOutput.KEY) or {} + reasons = enrichment.get('reasons') or {} + if not reasons: + return output + + merged = dict(output) + merged['courses'] = [ + {**course, 'rationale': reasons.get(course.get('key'), course.get('rationale', ''))} + for course in output.get('courses') or [] + ] + return merged diff --git a/enterprise_access/apps/pathways/pathway_assembly.py b/enterprise_access/apps/pathways/pathway_assembly.py new file mode 100644 index 00000000..42b2d799 --- /dev/null +++ b/enterprise_access/apps/pathways/pathway_assembly.py @@ -0,0 +1,334 @@ +""" +Assembling five retrieved candidates into a pathway, and checking the result. + +Pure: no network, no database, no model call. Takes Algolia hits, returns a selection. +That is deliberate -- everything here is a constraint-satisfaction rule that can be +reasoned about and tested directly, so none of it should be delegated to a model. + +Why constrained assembly rather than "take the top 5" +----------------------------------------------------- +Product reported the defect that motivated this: *"you'll get 2 intro courses from +different providers, so 2 101 courses but zero 102 courses."* Measured against the pinned +2U catalog on 2026-09-10, that is exactly what relevance ranking does at rank 5: + +=========================== =============== ================ +Query top 5 (I/M/A) top 20 (I/M/A) +=========================== =============== ================ +``data analyst`` 5 / 0 / 0 16 / 3 / 1 +``project manager`` 5 / 0 / 0 14 / 4 / 2 +``machine learning engineer`` 1 / 4 / 0 7 / 10 / 3 +``biomedical engineer`` 4 / 1 / 0 7 / 7 / 0 +=========================== =============== ================ + +Three things follow, and they shape this module: + +1. **The rungs exist.** 12 of 14 probe skills have courses at all three ``level_type`` + values in the 2U catalog. The intermediate courses are not missing; they are just + below the rank-5 cut. So this is a selection defect, not a content gap. +2. **"Duplicate" means a repeated rung, not a repeated key.** The collision product + reported is two *different* courses from *different* providers at the same level. + De-duplicating on ``key`` is necessary and catches none of it. +3. **Reaching to rank 20 for level diversity drags in other languages.** 26.2% of the 2U + catalog (767 of 2,927 courses) is taught in a language other than English, and it + shows from rank 6 onward -- ``python programming`` returns 6 Spanish courses in its + top 20. So the fix for (1) requires the language filter in ``eligible_candidates``. + The two defects are coupled: solving the first exposes the second. + +Ordering is best-effort, and says so +------------------------------------ +``level_type`` is a noisy difficulty signal. Across the full 4,094-course census, 27% of +intro-titled courses are not tagged ``Introductory``, and 19% of advanced-titled ones are +tagged ``Introductory`` -- ``Advanced Project Management``, ``Data Science: Capstone`` and +``Python Programming: Intermediate Concepts`` are all tagged ``Introductory``. It is +reliable enough in aggregate to drive the quota (the table above works), and not reliable +enough to order five specific courses, so ``TITLE_LEVEL_CUES`` breaks ties within a rung +and the result is not claimed to be a guaranteed difficulty ordering. +""" +import logging +import re +from dataclasses import dataclass, field + +from enterprise_access.apps.pathways.content_keys import is_valid_course_key + +logger = logging.getLogger(__name__) + +# Decision 3: a pathway is exactly five courses, or explicitly no pathway. Never a short +# set -- four courses returned silently reads as success to a client. +PATHWAY_SIZE = 5 + +# Catalog ``level_type`` values, easiest first. +LEVEL_INTRODUCTORY = 'Introductory' +LEVEL_INTERMEDIATE = 'Intermediate' +LEVEL_ADVANCED = 'Advanced' +LEVEL_ORDER = (LEVEL_INTRODUCTORY, LEVEL_INTERMEDIATE, LEVEL_ADVANCED) + +# The target shape of a pathway. Sums to ``PATHWAY_SIZE``. Advanced gets one slot rather +# than a fair share because it is only 7% of the pinned catalog -- asking for two would +# fail the quota on most skills and fall through to backfill every time. +DEFAULT_LEVEL_QUOTA = { + LEVEL_INTRODUCTORY: 2, + LEVEL_INTERMEDIATE: 2, + LEVEL_ADVANCED: 1, +} + +# No provider may supply more than this. ``biomedical engineer`` returned 5 of 5 from one +# partner, which is a pathway a learner would read as an advertisement. +MAX_PER_PARTNER = 2 + +# The only language of instruction currently supported. This is the ``language`` +# attribute (what the course is taught in), *not* ``metadata_language`` (which +# translation of the record is served) -- see ``docs/references/algolia_search.md``. +SUPPORTED_LANGUAGE = 'English' + +# Title cues used only to break ties inside a rung, never to override ``level_type``. +TITLE_LEVEL_CUES = ( + (re.compile(r'\b(introduction|introductory|intro|basics?|fundamentals?|foundations?|' + r'beginner|getting started|101)\b', re.IGNORECASE), -1), + (re.compile(r'\b(advanced|expert|mastering|masterclass|capstone|deep dive)\b', + re.IGNORECASE), 1), +) + + +@dataclass(frozen=True) +class Candidate: + """One retrieved course, normalised out of an Algolia hit.""" + + key: str + title: str = '' + level_type: str = '' + partner: str = '' + language: str = '' + + @classmethod + def from_hit(cls, hit: dict) -> 'Candidate': + """ + Build a candidate from a raw catalog hit. + + ``partners`` is a list of objects on the hit; the first is treated as the owning + provider, which is how the learner portal attributes a course. + """ + partners = hit.get('partners') or [] + first_partner = partners[0] if partners and isinstance(partners[0], dict) else {} + return cls( + key=(hit.get('key') or '').strip(), + title=(hit.get('title') or '').strip(), + level_type=(hit.get('level_type') or '').strip(), + partner=(first_partner.get('name') or '').strip(), + language=(hit.get('language') or '').strip(), + ) + + @property + def title_cue(self) -> int: + """-1 if the title reads as introductory, 1 if advanced, 0 if it says nothing.""" + for pattern, weight in TITLE_LEVEL_CUES: + if pattern.search(self.title): + return weight + return 0 + + @property + def difficulty_rank(self) -> tuple: + """ + Sort key for "roughly easiest first". + + ``level_type`` dominates because it is the only signal that is populated for every + course; the title cue only orders courses that share a level. + """ + level_index = LEVEL_ORDER.index(self.level_type) if self.level_type in LEVEL_ORDER else len(LEVEL_ORDER) + return (level_index, self.title_cue) + + +@dataclass +class PathwayAssembly: + """The result of assembling a pathway, including what could not be satisfied.""" + + courses: list = field(default_factory=list) + unfilled_rungs: list = field(default_factory=list) + ineligible: dict = field(default_factory=dict) + + @property + def is_complete(self) -> bool: + """Whether a full pathway was built.""" + return len(self.courses) == PATHWAY_SIZE + + @property + def realised_level_mix(self) -> dict: + """How many courses landed on each rung. Tracked rather than gated.""" + mix = {level: 0 for level in LEVEL_ORDER} + for course in self.courses: + if course.level_type in mix: + mix[course.level_type] += 1 + return mix + + +def eligible_candidates(hits, *, supported_language: str = SUPPORTED_LANGUAGE): + """ + Filter raw hits down to the courses a pathway may contain. + + Returns ``(candidates, ineligible_counts)``. The counts are returned rather than + logged away because "the candidate set was large but mostly unusable" and "retrieval + found little" are different diagnoses that a bare pathway cannot distinguish. + """ + candidates = [] + ineligible: dict = {} + seen = set() + + def reject(reason): + ineligible[reason] = ineligible.get(reason, 0) + 1 + + for hit in hits: + candidate = Candidate.from_hit(hit) + if not is_valid_course_key(candidate.key): + reject('invalid_course_key') + continue + if candidate.key in seen: + reject('duplicate_key') + continue + if supported_language and candidate.language and candidate.language != supported_language: + reject('unsupported_language') + continue + seen.add(candidate.key) + candidates.append(candidate) + + return candidates, ineligible + + +def assemble_pathway(hits, *, level_quota=None, max_per_partner: int = MAX_PER_PARTNER): + """ + Select ``PATHWAY_SIZE`` courses spanning the level quota, capped per provider. + + Two passes. The first fills the quota **rung by rung, scarcest rung first**; within a + rung, candidates are taken in relevance order, so relevance still decides *which* + course is chosen and the quota only decides how many. The second backfills from + whatever is left when a rung is genuinely empty, because a skill can legitimately have + no advanced course (``Nursing`` has none in the pinned catalog) and failing the pathway + for that would be indistinguishable from a retrieval failure. + + Why scarcest-first, which is not obvious + ---------------------------------------- + The level quota and the provider cap compete for the same candidates, and a single + relevance-ordered pass lets the most plentiful rung spend the scarce resource. Measured + against the pinned 2U catalog: ``Data Analyst`` returned 17 candidates spanning all + three rungs and still assembled to 5/0/0, because the two Introductory picks used up + IBM's entire provider allowance and every Intermediate candidate was also IBM's. + + Processing the rung with the fewest available candidates first gives the scarce rung + first claim on provider capacity. Advanced is only 7% of the catalog, so under a + relevance-ordered pass it loses that competition almost every time. + + Returns a ``PathwayAssembly``. When fewer than ``PATHWAY_SIZE`` eligible candidates + exist it returns an incomplete assembly rather than padding -- the caller reports no + pathway, per Decision 3. + """ + quota = dict(level_quota or DEFAULT_LEVEL_QUOTA) + candidates, ineligible = eligible_candidates(hits) + + chosen: list = [] + per_partner: dict = {} + chosen_keys = set() + + def take(candidate): + chosen.append(candidate) + chosen_keys.add(candidate.key) + per_partner[candidate.partner] = per_partner.get(candidate.partner, 0) + 1 + + def partner_is_full(candidate): + # An unattributed course cannot be attributed to a provider, so it cannot be + # counted against one either. + return bool(candidate.partner) and per_partner.get(candidate.partner, 0) >= max_per_partner + + by_rung = { + level: [c for c in candidates if c.level_type == level] + for level in quota + } + # Scarcest rung first. Ties break on LEVEL_ORDER so the result stays deterministic. + rung_order = sorted( + quota, + key=lambda level: ( + len(by_rung[level]), + LEVEL_ORDER.index(level) if level in LEVEL_ORDER else len(LEVEL_ORDER), + ), + ) + + for level in rung_order: + for candidate in by_rung[level]: + if len(chosen) >= PATHWAY_SIZE or quota[level] <= 0: + break + if partner_is_full(candidate): + continue + quota[level] -= 1 + take(candidate) + + unfilled = [level for level, remaining in quota.items() if remaining > 0] + + for candidate in candidates: + if len(chosen) >= PATHWAY_SIZE: + break + if candidate.key in chosen_keys or partner_is_full(candidate): + continue + take(candidate) + + assembly = PathwayAssembly( + courses=sorted(chosen, key=lambda candidate: candidate.difficulty_rank), + unfilled_rungs=unfilled, + ineligible=ineligible, + ) + if not assembly.is_complete: + logger.info( + 'Assembled %d of %d courses from %d eligible candidates (%d hits, ineligible: %s).', + len(assembly.courses), PATHWAY_SIZE, len(candidates), len(hits), ineligible or {}, + ) + return assembly + + +def validate_pathway(courses, *, customer_catalog_keys=None, + max_per_partner: int = MAX_PER_PARTNER, + supported_language: str = SUPPORTED_LANGUAGE): + """ + Apply the Tier 1 correctness gates and return a list of violation strings. + + Tier 1 is the set of checks that are bugs rather than quality judgements, so an empty + list is a precondition for reporting any quality metric at all -- a pathway that fails + these should not be scored, it should fail the build. + + ``customer_catalog_keys`` is optional because proving catalog membership needs a + browse-scoped key (Open Decision 6); when it is not supplied that gate is skipped + rather than assumed to pass. + """ + violations = [] + + if len(courses) != PATHWAY_SIZE: + violations.append( + f'a pathway must contain exactly {PATHWAY_SIZE} courses, got {len(courses)}' + ) + + keys = [course.key for course in courses] + for key in keys: + if not is_valid_course_key(key): + violations.append(f'{key!r} is not a valid catalog course key') + + duplicates = {key for key in keys if keys.count(key) > 1} + for key in sorted(duplicates): + violations.append(f'{key!r} appears more than once') + + for course in courses: + if course.language and course.language != supported_language: + violations.append( + f'{course.key!r} is taught in {course.language!r}, not {supported_language!r}' + ) + + per_partner: dict = {} + for course in courses: + if course.partner: + per_partner[course.partner] = per_partner.get(course.partner, 0) + 1 + for partner, count in sorted(per_partner.items()): + if count > max_per_partner: + violations.append( + f'{count} courses from {partner!r} exceeds the cap of {max_per_partner}' + ) + + if customer_catalog_keys is not None: + for key in keys: + if key not in customer_catalog_keys: + violations.append(f'{key!r} is not in the pinned customer catalog') + + return violations diff --git a/enterprise_access/apps/pathways/prompts.py b/enterprise_access/apps/pathways/prompts.py new file mode 100644 index 00000000..71ae0d79 --- /dev/null +++ b/enterprise_access/apps/pathways/prompts.py @@ -0,0 +1,94 @@ +""" +The canonical default text for pathway prompts that this app owns. + +The ``prompts`` app owns the *model*; the wording is pathway domain knowledge, so it lives +here. Two consumers read from this module: + +* The Xpert backend reads the **database row**, not this constant — so an admin edit takes + effect without a deploy, which is the whole point of the prompts app. This module is the + text the row is *seeded* with (see ``prompts/migrations/0003_seed_candidate_rerank_prompt``). +* The Claude and OpenAI backends take a caller-supplied system prompt and have no database + row, so they use this constant directly. + +That means an admin edit changes Xpert's behaviour and not the direct backends'. It is a +real asymmetry rather than an oversight: those backends exist to evaluate prompt variants +that are not yet worth persisting as admin-editable configuration. Once a variant wins, it +belongs in the row. +""" + +# What this prompt is *not* asked to do is as load-bearing as what it is. Chunk 9a's +# ``pathway_assembly`` already guarantees five courses, a spread across difficulty rungs, +# no duplicates and no more than two courses from one provider -- deterministically and +# under test. Asking a model for those as well would be asking it to reproduce arithmetic, +# and any disagreement would then have to be adjudicated. +# +# So the model is asked for exactly one thing: topical relevance, which is the thing +# assembly demonstrably cannot do. Measured against the pinned 2U catalog on 2026-09-10, +# with ``removeWordsIfNoResults: allOptional`` the candidate window is wide enough to hold +# the right rungs and loose enough to hold the wrong subjects -- a ``python programming`` +# query returned "AI in Architectural Design: Introduction", and ``biomedical engineer`` +# returned "Water and Wastewater Treatment Engineering". +# +# Ranking those last is what keeps them out of the delivered five, because assembly fills +# each rung from the front of the window. +CANDIDATE_RERANK_SYSTEM_PROMPT = """\ +You rank candidate courses by how well each one prepares a learner for a named career. + +You will receive a career name and a list of candidate courses, each with a key, a title +and a short description. Return a ranking of those courses by topical relevance to that +career, plus a one-sentence reason for each. + +Judge topical relevance ONLY. Do not consider, and do not try to balance: +- difficulty or course level +- which provider or university offers the course +- whether two courses cover similar ground +- how many courses to recommend + +Those are all decided after you, by code, and optimising for them here makes that harder +rather than easier. + +How to rank: +- Rank EVERY key you were given, exactly once, most relevant first. +- A course that has little or nothing to do with the career goes at the end. Do not drop + it -- its position is how you tell us it is a poor fit. +- Use ONLY keys that appear in the input. Never invent, correct or reformat a key. If you + are unsure about a key, leave it out entirely rather than guessing at it. +- Judge the course, not the title. A description that clearly addresses the career's work + outranks a title that merely shares a word with the career name. +- Where a career name is broad ("Analyst", "Engineer"), prefer courses that teach the + concrete skills that career is practised with over courses that only discuss the field. + +How to write each reason: +- One sentence, under 30 words, addressed to the learner. +- Say how the course connects to that career's actual work. +- No marketing language, no superlatives, and no claims about outcomes, salary or + employability. +- If a course is a poor fit, say so plainly. "Covers water treatment rather than the data + work this role involves" is more useful than a stretch. + +Return JSON only, with no prose before or after it.""" + +# Appended to the system prompt at runtime by ``prompts_api.build_system_prompt``. +CANDIDATE_RERANK_OUTPUT_SCHEMA = { + 'type': 'object', + 'required': ['ordered_keys'], + 'additionalProperties': False, + 'properties': { + 'ordered_keys': { + 'type': 'array', + 'description': ( + 'Every candidate key, exactly once, most topically relevant to the career ' + 'first. Keys must be copied verbatim from the input.' + ), + 'items': {'type': 'string'}, + }, + 'rationales': { + 'type': 'object', + 'description': ( + 'One sentence per course key explaining how it connects to the career. ' + 'Keys must appear in ordered_keys.' + ), + 'additionalProperties': {'type': 'string'}, + }, + }, +} diff --git a/enterprise_access/apps/pathways/reranking.py b/enterprise_access/apps/pathways/reranking.py new file mode 100644 index 00000000..a4baaeba --- /dev/null +++ b/enterprise_access/apps/pathways/reranking.py @@ -0,0 +1,182 @@ +""" +Domain-layer API for re-ranking a candidate course set with a model. + +Deliberately narrow. Chunk 9a's ``pathway_assembly`` already guarantees the *structural* +properties of a pathway — five courses, spread across levels, no duplicates, no more than +two from one provider — deterministically and testably. Asking a model for those as well +would be asking it to reproduce arithmetic, and any disagreement would then have to be +adjudicated. So nothing here requests a level mix or a de-duplication. + +What is asked for is the one thing measurement showed assembly cannot do: **topical +relevance.** With ``removeWordsIfNoResults: allOptional`` the candidate window is wide +enough to contain the right rungs and loose enough to contain the wrong subjects — a live +``python programming`` query returned ``AI in Architectural Design: Introduction``, and +``biomedical engineer`` returned ``Water and Wastewater Treatment Engineering``. Ordering +by topical fit is what moves those to the back of the window, where assembly will not +reach them. + +The model's output is treated as untrusted +------------------------------------------ +It returns *keys*, never course records, and every key is checked against the candidate +set it was given. A key that was not in the input is dropped and counted in +``fabricated_keys`` — the platform has a known key-invention defect, and counting it makes +it a metric rather than an anecdote. A response that is unusable in full degrades to the +retrieval order rather than failing the pathway, because a worse ordering is a better +outcome than no pathway at all. +""" +import json +import logging + +from enterprise_access.apps.pathways.model_backends import ModelBackendError, get_model_backend +from enterprise_access.apps.pathways.prompts import CANDIDATE_RERANK_OUTPUT_SCHEMA, CANDIDATE_RERANK_SYSTEM_PROMPT +from enterprise_access.apps.prompts.api import compose_system_prompt +from enterprise_access.apps.prompts.models import PromptType + +logger = logging.getLogger(__name__) + +# The direct backends (claude, openai) take a caller-supplied system prompt and have no +# database row, so they use the module constant. The Xpert backend ignores this and reads +# its admin-editable row, seeded by ``prompts/migrations/0003_seed_candidate_rerank_prompt``. +# See ``apps/pathways/prompts.py`` for why that asymmetry is deliberate. +# +# The schema is appended here, exactly as ``build_system_prompt`` appends it on the Xpert +# path. Without it the prompt asks for JSON but never names ``ordered_keys``, so the model +# invents its own field names and ``parse_rerank_response`` reads none of them. A live +# gpt-4o run on 2026-09-10 did precisely that: a valid, well-reasoned ranking, returned +# under different keys, discarded in full and logged only as "no ordered_keys list". +FALLBACK_SYSTEM_PROMPT = compose_system_prompt( + CANDIDATE_RERANK_SYSTEM_PROMPT, CANDIDATE_RERANK_OUTPUT_SCHEMA, +) + +# Descriptions are truncated before they reach the model. The full text is marketing copy +# whose tail rarely changes a relevance judgement, and 20 untruncated descriptions is a +# large, mostly wasted prompt. +DESCRIPTION_CHARS_FOR_MODEL = 400 + + +def build_user_content(*, career_name: str, candidates: list[dict]) -> str: + """ + Render the request the model sees. + + Only the fields a relevance judgement needs are included. ``level_type`` and + ``partner`` are deliberately withheld: they are what assembly uses, and offering them + invites the model to optimise for constraints it is not being asked about. + """ + return json.dumps( + { + 'career': career_name, + 'candidates': [ + { + 'key': candidate.get('key', ''), + 'title': candidate.get('title', ''), + 'description': ( + candidate.get('short_description') or + candidate.get('full_description') or + '' + )[:DESCRIPTION_CHARS_FOR_MODEL], + } + for candidate in candidates + ], + }, + separators=(',', ':'), + ) + + +def parse_rerank_response(payload, allowed_keys) -> dict: + """ + Validate a model response against the candidate set it was given. + + Returns ``ordered_keys``, ``rationales`` and ``fabricated_keys``. A malformed payload + yields empty lists rather than raising: the caller degrades to retrieval order, and a + bad response should cost the ordering, not the pathway. + """ + if not isinstance(payload, dict): + logger.warning('Re-rank response was not a JSON object; ignoring the ordering.') + return {'ordered_keys': [], 'rationales': {}, 'fabricated_keys': []} + + raw_keys = payload.get('ordered_keys') + if not isinstance(raw_keys, list): + logger.warning('Re-rank response had no ordered_keys list; ignoring the ordering.') + return {'ordered_keys': [], 'rationales': {}, 'fabricated_keys': []} + + allowed = set(allowed_keys) + ordered_keys: list[str] = [] + fabricated: list[str] = [] + for key in raw_keys: + if not isinstance(key, str) or not key: + continue + if key in ordered_keys: + # A repeated key is not a fabrication, just noise -- the first wins. + continue + if key in allowed: + ordered_keys.append(key) + else: + fabricated.append(key) + + if fabricated: + logger.warning( + 'Re-rank returned %d key(s) absent from the candidate set; dropped.', + len(fabricated), + ) + + raw_rationales = payload.get('rationales') + rationales = {} + if isinstance(raw_rationales, dict): + rationales = { + key: value for key, value in raw_rationales.items() + if isinstance(key, str) and key in allowed and isinstance(value, str) + } + + return { + 'ordered_keys': ordered_keys, + 'rationales': rationales, + 'fabricated_keys': fabricated, + } + + +def rerank_candidates(*, career_name: str, candidates: list[dict], trace_id: str, + backend=None) -> dict: + """ + Order a candidate set by topical relevance to a career. + + Args: + career_name: The selected career's display name. + candidates: ``CourseCandidate`` dicts, in retrieval order. + trace_id: Ties the model call to the persisted step record. + backend: Overrides the configured backend. For tests and comparison runs. + + Returns: + ``ordered_keys``, ``rationales``, ``fabricated_keys``, ``prompt_revision`` and a + ``trace`` dict from ``ModelResponse.to_trace_dict``. A backend failure or an + unparseable response returns empty ordering rather than raising, so the caller + degrades to retrieval order. + """ + allowed_keys = [candidate.get('key', '') for candidate in candidates] + model_backend = backend or get_model_backend(prompt_type=PromptType.CANDIDATE_RERANK) + + empty = { + 'ordered_keys': [], 'rationales': {}, 'fabricated_keys': [], + 'prompt_revision': '', 'trace': {}, + } + + try: + response = model_backend.complete( + system_prompt=FALLBACK_SYSTEM_PROMPT, + user_content=build_user_content(career_name=career_name, candidates=candidates), + trace_id=trace_id, + ) + except ModelBackendError as exc: + # Logged without the prompt or the response body, per the backend contract. + logger.warning('Re-rank skipped: model backend failed (%s).', type(exc).__name__) + return empty + + try: + payload = response.as_json() + except ModelBackendError: + logger.warning('Re-rank skipped: response was not JSON.') + return {**empty, 'trace': response.to_trace_dict()} + + result = parse_rerank_response(payload, allowed_keys) + result['prompt_revision'] = str(response.metadata.get('prompt_revision', '') or '') + result['trace'] = response.to_trace_dict() + return result diff --git a/enterprise_access/apps/pathways/skill_vocabulary.py b/enterprise_access/apps/pathways/skill_vocabulary.py new file mode 100644 index 00000000..affacb4a --- /dev/null +++ b/enterprise_access/apps/pathways/skill_vocabulary.py @@ -0,0 +1,314 @@ +""" +Resolving skill names onto the catalog's actual facet vocabulary. + +The problem this solves, measured against the production catalog index on 2026-09-09: + +=========================== ===== ================================ ===== +Term a learner/model writes Hits What the catalog actually holds Hits +=========================== ===== ================================ ===== +``Python`` 0 ``Python (Programming Language)`` 95 +``SQL`` 0 ``SQL (Programming Language)`` 42 +``Java`` 0 ``Java (Programming Language)`` 27 +``Excel`` 0 ``Microsoft Excel`` 26 +=========================== ===== ================================ ===== + +The catalog's ``skill_names`` are Lightcast-canonical, so the short name anyone would +actually write is frequently **absent from the vocabulary entirely**. Exact-match +grounding therefore drops the most in-demand technical skills silently — they do not +produce bad results, they produce a dropped filter. + +The retrieval diagnostic established this is the binding constraint: against +product-authored ground truth, recall@20 was 12% by default and 23% even after widening +the query with ``removeWordsIfNoResults=allOptional`` -- and 0% for every technology +persona under both settings. Query construction is worth real recall, but it does not +reach the courses whose skills were never expressible in the first place. + +Two things this deliberately is not +----------------------------------- +**Not a hand-maintained alias map.** Only 96 of 1,492 sampled vocabulary values carry a +parenthetical qualifier, so there is no small rule set to encode, and the taxonomy took +thousands of skill updates in a single month — a static map would rot. + +**Not a model call.** Resolving ``Python`` to ``Python (Programming Language)`` is a +vocabulary lookup, not a judgement. The vocabulary is queryable, so ask it. + +Instead: match candidate terms against the real vocabulary, exactly where possible and by +*whole-word* containment otherwise, with the match type recorded so a caller can decide +how much to trust each one. Candidates can be widened via Algolia's facet-search endpoint +(``AlgoliaSearchClient.search_facet_values``), but every candidate is still validated +against the scoped vocabulary before use — facet search is unfiltered, so its counts and +membership say nothing about a particular enterprise's catalog. +""" +import logging +import re +from dataclasses import dataclass +from enum import Enum + +logger = logging.getLogger(__name__) + +# Catalog fields that hold skill facet values, in priority order. ``skill_names`` wins a +# collision, matching the MFE's own precedence in `catalogSkillTranslation.ts`. +SKILL_FACET_FIELDS = ('skill_names', 'skills.name') + +# Terms shorter than this are not resolved by containment: two- and three-letter strings +# match far too much (``JS`` matches ``JSON``, ``ML`` matches nothing useful) and an +# over-eager expansion is worse than a dropped filter. +MIN_CONTAINMENT_TERM_LENGTH = 4 + + +class MatchType(Enum): + """ + How a term was matched, in descending order of confidence. + + Recorded rather than discarded because the caller's tolerance differs: a strict facet + filter should only use high-confidence matches, while a soft boost can afford a + weaker one. + """ + + EXACT = 'exact' + #: The vocabulary value is the term plus a parenthetical qualifier -- + #: ``Python`` -> ``Python (Programming Language)``. Very high confidence. + QUALIFIED = 'qualified' + #: The term appears as a whole word inside the value -- ``Excel`` -> + #: ``Microsoft Excel``. Good, but capable of drifting (``Azure`` -> + #: ``Azure Machine Learning``), so ranked by how much extra the value carries. + CONTAINED = 'contained' + + @property + def is_high_confidence(self): + """Whether a match of this type is safe to use as a hard facet filter.""" + return self in (MatchType.EXACT, MatchType.QUALIFIED) + + +@dataclass(frozen=True) +class SkillMatch: + """One resolved skill: the term asked for, and the catalog value to actually query.""" + + term: str + catalog_value: str + catalog_field: str + match_type: MatchType + + @property + def is_high_confidence(self): + return self.match_type.is_high_confidence + + +def normalize_term(value): + """Casefold and collapse whitespace for comparison. Never used as a query value.""" + return ' '.join((value or '').split()).casefold() + + +def _qualified_pattern(term): + """Match `` ()`` — the canonical Lightcast disambiguation shape.""" + return re.compile(rf'^{re.escape(term)}\s*\([^)]+\)$', re.IGNORECASE) + + +def _whole_word_pattern(term): + r""" + Match ``term`` as a whole word anywhere in a value. + + Whole-word is what keeps ``JS`` from matching ``JSON`` and ``Java`` from matching + ``JavaScript``. ``\b`` is unreliable next to ``+``, ``#`` and ``.`` (``C++``, ``C#``, + ``Node.js``), so the boundaries are spelled out as "not a word character". + """ + escaped = re.escape(term) + return re.compile(rf'(? 3 -> doubled to 6 -> 36, but (2 + 2) -> 4 -> not doubled -> 16. + """ + + steps = [ + AddStep, + ConditionalDoubleStep, + SquareStep, + ] + + +class UnconditionalTestWorkflow(AbstractConditionalWorkflow): + """ + The same shape, but with no step defining ``should_execute``. + + Used to prove the conditional base is behaviour-preserving by default. + """ + + steps = [ + AddStep, + SquareStep, + ] diff --git a/enterprise_access/apps/pathways/tests/test_api.py b/enterprise_access/apps/pathways/tests/test_api.py new file mode 100644 index 00000000..f5690a41 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_api.py @@ -0,0 +1,334 @@ +""" +Tests for the career discovery domain layer. + +Every test mocks Xpert and Algolia; nothing here issues a network call. +""" +from unittest import mock + +import ddt +from django.test import TestCase + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.pathways import api as pathways_api +from enterprise_access.apps.prompts.api import PromptError +from enterprise_access.apps.prompts.api_client import XpertAPIResponseError, XpertResponseMessage +from enterprise_access.apps.prompts.models import PromptType +from enterprise_access.apps.prompts.tests.factories import XpertLearnerPathwaysSystemPromptFactory + +PATCH_XPERT_CLIENT = 'enterprise_access.apps.prompts.api.XpertAPIClient' +PATCH_ALGOLIA_CLIENT = 'enterprise_access.apps.pathways.api.AlgoliaSearchClient' + +INTAKE = { + 'selected_goals': 'move into data analysis', + 'free_text': 'I report on spreadsheets all day and want to automate it', + 'known_context': 'operations analyst, five years', + 'interested_industries': 'healthcare, technology', +} + +JOBS_HIT = { + 'external_id': 'ETE78CD2CDFFFAC66B', + 'name': 'Data Analyst', + 'skills': [{'name': 'SQL (Programming Language)'}, {'name': 'Data Analysis'}], + 'industry_names': ['Health Care', 'Information'], +} + + +@ddt.ddt +class TestNameHelpers(TestCase): + """Tests for the name-normalisation helpers.""" + + @ddt.data( + ([' SQL ', 'SQL', ''], ['SQL']), + (['Python', 'Excel', 'Python'], ['Python', 'Excel']), + ([None, 3, 'Nursing'], ['Nursing']), + (None, []), + ) + @ddt.unpack + def test_dedupe_names(self, values, expected): + assert pathways_api.dedupe_names(values) == expected + + def test_dedupe_names_preserves_order(self): + # Order carries relevance: the first required skill becomes the fallback query. + assert pathways_api.dedupe_names(['C', 'A', 'B', 'A']) == ['C', 'A', 'B'] + + @ddt.data( + ('SQL', ['SQL']), + (['SQL', 'SQL'], ['SQL']), + (None, []), + ({'skills': ['SQL']}, []), + (7, []), + ) + @ddt.unpack + def test_coerce_name_list(self, value, expected): + assert pathways_api.coerce_name_list(value) == expected + + @ddt.data( + ('SQL & Python', True), + ('Excel + Tableau', True), + ('Research & Development Management', True), + ('SQL (Programming Language)', False), + ('C++', False), + ) + @ddt.unpack + def test_is_malformed_compound(self, name, expected): + assert pathways_api.is_malformed_compound(name) is expected + + +@ddt.ddt +class TestQueryConstruction(TestCase): + """Tests for the Algolia query, filters and optional filters.""" + + def test_condensed_query_is_preferred(self): + query = pathways_api.build_career_query( + condensed_query=' data analyst ', + skills_required=['SQL'], + ) + assert query == 'data analyst' + + def test_falls_back_to_first_required_skill(self): + query = pathways_api.build_career_query(condensed_query='', skills_required=['', 'SQL', 'Python']) + assert query == 'SQL' + + def test_query_is_empty_when_nothing_is_available(self): + assert pathways_api.build_career_query(condensed_query=None, skills_required=[]) == '' + + def test_language_is_always_filtered_even_with_no_other_criteria(self): + """ + The jobs index holds translated duplicates of the same role -- the Spanish + record's identifier is the English one plus "-es" -- so both surface for the same + query, which is the defect persona 2's author reported. The language clause is + therefore unconditional, not a refinement. + """ + assert pathways_api.build_career_filters( + industries=[], job_sources=[], + ) == 'metadata_language:en' + + @ddt.data( + (['Health Care'], [], 'metadata_language:en AND (industry_names:"Health Care")'), + ([], ['lightcast'], 'metadata_language:en AND (job_sources:"lightcast")'), + ( + ['Health Care', 'Information'], + ['lightcast'], + 'metadata_language:en AND (industry_names:"Health Care" OR industry_names:"Information")' + ' AND (job_sources:"lightcast")', + ), + ) + @ddt.unpack + def test_hard_filters(self, industries, job_sources, expected): + assert pathways_api.build_career_filters(industries=industries, job_sources=job_sources) == expected + + def test_filter_values_are_quote_escaped(self): + built = pathways_api.build_career_filters(industries=['Say "Yes"'], job_sources=[]) + assert built == 'metadata_language:en AND (industry_names:"Say \\"Yes\\"")' + + def test_required_skills_are_unscored_and_preferred_are_scored(self): + optional_filters = pathways_api.build_optional_skill_filters( + skills_required=['SQL'], + skills_preferred=['Tableau'], + ) + assert optional_filters == [ + 'skills.name:"SQL"', + 'skills.name:"Tableau"', + ] + + def test_optional_filters_are_capped(self): + optional_filters = pathways_api.build_optional_skill_filters( + skills_required=[f'required-{index}' for index in range(9)], + skills_preferred=[f'preferred-{index}' for index in range(9)], + ) + + assert len(optional_filters) == ( + pathways_api.MAX_REQUIRED_SKILL_FILTERS + pathways_api.MAX_PREFERRED_SKILL_FILTERS + ) + assert optional_filters[0] == 'skills.name:"required-0"' + assert optional_filters[-1] == 'skills.name:"preferred-1"' + + def test_malformed_compounds_are_dropped(self): + optional_filters = pathways_api.build_optional_skill_filters( + skills_required=['SQL & Python', 'SQL'], + skills_preferred=['Excel + Tableau'], + ) + assert optional_filters == ['skills.name:"SQL"'] + + def test_no_skills_yields_no_optional_filters(self): + assert pathways_api.build_optional_skill_filters(skills_required=[], skills_preferred=[]) == [] + + +@ddt.ddt +class TestCareerCandidateMapping(TestCase): + """Tests for mapping jobs-index hits onto career candidates.""" + + def test_hit_is_mapped(self): + assert pathways_api.career_candidate_from_hit(JOBS_HIT) == { + 'external_id': 'ETE78CD2CDFFFAC66B', + 'name': 'Data Analyst', + 'skills': ['SQL (Programming Language)', 'Data Analysis'], + 'industries': ['Health Care', 'Information'], + } + + @ddt.data( + {'name': 'Data Analyst'}, + {'external_id': 'ETE78CD2CDFFFAC66B'}, + {'external_id': ' ', 'name': 'Data Analyst'}, + {}, + ) + def test_unidentifiable_hits_are_dropped(self, hit): + # Dropped rather than given a placeholder id: a fabricated identifier would + # corrupt the harness's ground-truth comparison. + assert pathways_api.career_candidate_from_hit(hit) is None + + def test_malformed_skill_entries_are_ignored(self): + hit = {**JOBS_HIT, 'skills': ['SQL', None, {'name': 'SQL'}, {'name': ''}, {}]} + assert pathways_api.career_candidate_from_hit(hit)['skills'] == ['SQL'] + + def test_no_match_percentage_is_fabricated(self): + candidate = pathways_api.career_candidate_from_hit(JOBS_HIT) + assert 'match_percentage' not in candidate + assert not any('match' in key for key in candidate) + + +class TestDeriveLearningIntent(TestCase): + """Tests for the Xpert-backed intent extraction, with the Xpert client mocked.""" + + @classmethod + def setUpTestData(cls): + super().setUpTestData() + cls.prompt = XpertLearnerPathwaysSystemPromptFactory(prompt_type=PromptType.LEARNER_INTENT) + + def _mock_xpert(self, mock_client_class, content): + """Point the patched Xpert client at a canned response body.""" + mock_client_class.return_value.send_message.return_value = XpertResponseMessage( + role='assistant', + content=content, + ) + return mock_client_class.return_value.send_message + + @mock.patch(PATCH_XPERT_CLIENT) + def test_intent_is_normalized(self, mock_client_class): + self._mock_xpert(mock_client_class, ( + '{"skills_required": [" SQL ", "SQL"], "skills_preferred": ["Tableau"], ' + '"condensed_algolia_query": " data analyst "}' + )) + + intent = pathways_api.derive_learning_intent(intake=INTAKE, conversation_id='conv-1') + + assert intent == { + 'skills_required': ['SQL'], + 'skills_preferred': ['Tableau'], + 'condensed_algolia_query': 'data analyst', + } + + @mock.patch(PATCH_XPERT_CLIENT) + def test_learner_intent_prompt_and_rag_tags_are_used(self, mock_client_class): + send_message = self._mock_xpert(mock_client_class, '{"skills_required": []}') + + pathways_api.derive_learning_intent(intake=INTAKE, conversation_id='conv-1') + + _, kwargs = send_message.call_args + assert kwargs['conversation_id'] == 'conv-1' + assert kwargs['tags'] == ['discovery', 'edx-available-course'] + assert self.prompt.system_prompt in kwargs['system_prompt'] + + @mock.patch(PATCH_XPERT_CLIENT) + def test_bare_string_skill_list_is_tolerated(self, mock_client_class): + self._mock_xpert(mock_client_class, '{"skills_required": "SQL", "skills_preferred": 4}') + + intent = pathways_api.derive_learning_intent(intake=INTAKE, conversation_id='conv-1') + + assert intent['skills_required'] == ['SQL'] + assert not intent['skills_preferred'] + + @mock.patch(PATCH_XPERT_CLIENT) + def test_non_object_response_raises(self, mock_client_class): + self._mock_xpert(mock_client_class, '["SQL"]') + + with self.assertRaises(PromptError): + pathways_api.derive_learning_intent(intake=INTAKE, conversation_id='conv-1') + + @mock.patch(PATCH_XPERT_CLIENT) + def test_unparseable_response_raises(self, mock_client_class): + self._mock_xpert(mock_client_class, 'not json') + + with self.assertRaises(XpertAPIResponseError): + pathways_api.derive_learning_intent(intake=INTAKE, conversation_id='conv-1') + + def test_missing_prompt_raises(self): + self.prompt.delete() + + with self.assertRaises(PromptError): + pathways_api.derive_learning_intent(intake=INTAKE, conversation_id='conv-1') + + +class TestRetrieveCareers(TestCase): + """Tests for the jobs-index search, with the Algolia client mocked.""" + + def _mock_search(self, mock_client_class, response): + mock_client_class.return_value.search_jobs_index.return_value = response + return mock_client_class.return_value.search_jobs_index + + @mock.patch(PATCH_ALGOLIA_CLIENT) + def test_search_params_and_result(self, mock_client_class): + search = self._mock_search(mock_client_class, {'hits': [JOBS_HIT], 'nbHits': 1}) + + result = pathways_api.retrieve_careers( + intent={ + 'condensed_algolia_query': 'data analyst', + 'skills_required': ['SQL'], + 'skills_preferred': ['Tableau'], + }, + industries=['Health Care'], + ) + + args, kwargs = search.call_args + assert args == ('data analyst',) + assert kwargs['hitsPerPage'] == pathways_api.CAREER_HITS_PER_PAGE == 10 + assert kwargs['attributesToRetrieve'] == ['external_id', 'name', 'skills', 'industry_names'] + assert kwargs['filters'] == 'metadata_language:en AND (industry_names:"Health Care")' + assert kwargs['optionalFilters'] == ['skills.name:"SQL"', 'skills.name:"Tableau"'] + assert result['query'] == 'data analyst' + assert result['hit_count'] == 1 + assert result['careers'] == [pathways_api.career_candidate_from_hit(JOBS_HIT)] + + @mock.patch(PATCH_ALGOLIA_CLIENT) + def test_query_words_are_made_optional(self, mock_client_class): + """ + The jobs index ANDs every query word with no fallback configured, and a job record + is short: measured against the live index, *every* prefix of a normal intake + sentence returns 0 hits, including a single common word. Since the query prefers + Xpert's free-text `condensed_algolia_query`, one unmatched word would otherwise + take career retrieval to zero and dead-end the pipeline. There is no safe + query-length cap to use instead -- one word already fails. + """ + search = self._mock_search(mock_client_class, {'hits': []}) + + pathways_api.retrieve_careers(intent={'condensed_algolia_query': 'nurse practitioner'}) + + _, kwargs = search.call_args + assert kwargs['removeWordsIfNoResults'] == 'allOptional' + + @mock.patch(PATCH_ALGOLIA_CLIENT) + def test_language_filter_is_sent_even_with_no_other_criteria(self, mock_client_class): + search = self._mock_search(mock_client_class, {'hits': []}) + + pathways_api.retrieve_careers(intent={'condensed_algolia_query': 'nurse'}) + + _, kwargs = search.call_args + assert kwargs['filters'] == 'metadata_language:en' + assert 'optionalFilters' not in kwargs + + @mock.patch(PATCH_ALGOLIA_CLIENT) + def test_hit_count_counts_hits_not_candidates(self, mock_client_class): + # A full result set is not evidence retrieval worked, so both numbers are kept. + self._mock_search(mock_client_class, {'hits': [JOBS_HIT, {'name': 'No id'}, 'garbage']}) + + result = pathways_api.retrieve_careers(intent={'condensed_algolia_query': 'data analyst'}) + + assert result['hit_count'] == 3 + assert len(result['careers']) == 1 + + @mock.patch(PATCH_ALGOLIA_CLIENT) + def test_search_failure_propagates(self, mock_client_class): + mock_client_class.return_value.search_jobs_index.side_effect = AlgoliaSearchError('boom') + + with self.assertRaises(AlgoliaSearchError): + pathways_api.retrieve_careers(intent={'condensed_algolia_query': 'data analyst'}) diff --git a/enterprise_access/apps/pathways/tests/test_career_discovery.py b/enterprise_access/apps/pathways/tests/test_career_discovery.py new file mode 100644 index 00000000..9dfd8581 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_career_discovery.py @@ -0,0 +1,223 @@ +""" +Tests for the career discovery workflow, its steps, and the trace they leave behind. + +Every test mocks Xpert and Algolia; nothing here issues a network call. +""" +from unittest import mock + +from django.test import TestCase + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.pathways.models import ( + CareerDiscoveryWorkflow, + ExtractIntentInput, + ExtractIntentOutput, + ExtractIntentStep, + RetrieveCareersInput, + RetrieveCareersOutput, + RetrieveCareersStep, + RetrieveCareersStepException +) +from enterprise_access.apps.prompts.api_client import XpertAPIRequestError, XpertResponseMessage +from enterprise_access.apps.prompts.models import PromptType +from enterprise_access.apps.prompts.tests.factories import XpertLearnerPathwaysSystemPromptFactory +from enterprise_access.apps.workflow.exceptions import UnitOfWorkException + +PATCH_XPERT_CLIENT = 'enterprise_access.apps.prompts.api.XpertAPIClient' +PATCH_ALGOLIA_CLIENT = 'enterprise_access.apps.pathways.api.AlgoliaSearchClient' + +INTAKE = { + 'selected_goals': 'move into data analysis', + 'free_text': 'I report on spreadsheets all day and want to automate it', + 'known_context': 'operations analyst, five years', + 'interested_industries': 'healthcare, technology', +} + +XPERT_CONTENT = ( + '{"skills_required": ["SQL"], "skills_preferred": ["Tableau"], ' + '"condensed_algolia_query": "data analyst"}' +) + +JOBS_RESPONSE = { + 'hits': [{ + 'external_id': 'ETE78CD2CDFFFAC66B', + 'name': 'Data Analyst', + 'skills': [{'name': 'SQL (Programming Language)'}], + 'industry_names': ['Health Care'], + }], + 'nbHits': 1, +} + + +class CareerDiscoveryWorkflowTestMixin: + """Shared set-up for workflow executions with Xpert and Algolia mocked.""" + + @classmethod + def setUpTestData(cls): + super().setUpTestData() + XpertLearnerPathwaysSystemPromptFactory(prompt_type=PromptType.LEARNER_INTENT) + + def setUp(self): + super().setUp() + self.xpert_patcher = mock.patch(PATCH_XPERT_CLIENT) + self.mock_xpert = self.xpert_patcher.start().return_value + self.mock_xpert.send_message.return_value = XpertResponseMessage( + role='assistant', + content=XPERT_CONTENT, + ) + self.addCleanup(self.xpert_patcher.stop) + + self.algolia_patcher = mock.patch(PATCH_ALGOLIA_CLIENT) + self.mock_algolia = self.algolia_patcher.start().return_value + self.mock_algolia.search_jobs_index.return_value = JOBS_RESPONSE + self.addCleanup(self.algolia_patcher.stop) + + def create_workflow(self, intake=None): + return CareerDiscoveryWorkflow.objects.create( + input_data=CareerDiscoveryWorkflow.generate_input_dict(intake or INTAKE), + ) + + +class TestCareerDiscoveryWorkflowInput(TestCase): + """Tests for how a workflow record's input is built.""" + + def test_generate_input_dict(self): + input_dict = CareerDiscoveryWorkflow.generate_input_dict(INTAKE) + + assert input_dict[ExtractIntentInput.KEY] == INTAKE + # Empty on purpose: the intake's free-text industries are not facet values, and a + # hard filter on a non-facet value returns zero hits with no signal that it did. + assert input_dict[RetrieveCareersInput.KEY] == {} + + def test_steps_are_composed_in_order(self): + assert CareerDiscoveryWorkflow.steps == [ExtractIntentStep, RetrieveCareersStep] + + +class TestCareerDiscoveryWorkflowExecution(CareerDiscoveryWorkflowTestMixin, TestCase): + """Tests for a successful end-to-end execution.""" + + def test_workflow_returns_career_candidates(self): + workflow = self.create_workflow() + + workflow.execute() + + assert workflow.career_candidates() == [{ + 'external_id': 'ETE78CD2CDFFFAC66B', + 'name': 'Data Analyst', + 'skills': ['SQL (Programming Language)'], + 'industries': ['Health Care'], + }] + + def test_derived_intent_drives_the_jobs_search(self): + self.create_workflow().execute() + + args, kwargs = self.mock_algolia.search_jobs_index.call_args + assert args == ('data analyst',) + assert kwargs['optionalFilters'] == ['skills.name:"SQL"', 'skills.name:"Tableau"'] + + def test_every_executed_step_leaves_a_trace(self): + workflow = self.create_workflow() + + workflow.execute() + + intent_step = ExtractIntentStep.objects.get(workflow_record_uuid=workflow.uuid) + careers_step = RetrieveCareersStep.objects.get(workflow_record_uuid=workflow.uuid) + + for step_record in (intent_step, careers_step): + assert step_record.input_data is not None + assert step_record.output_data + assert step_record.succeeded_at is not None + assert step_record.failed_at is None + assert step_record.created is not None + + assert intent_step.input_data == INTAKE + assert intent_step.output_data['skills_required'] == ['SQL'] + assert careers_step.output_data['query'] == 'data analyst' + assert careers_step.output_data['hit_count'] == 1 + # The soft UUID linkage is what makes the executed order reconstructable. + assert careers_step.preceding_step_uuid == intent_step.uuid + + def test_workflow_record_persists_both_step_outputs(self): + workflow = self.create_workflow() + + workflow.execute() + + assert workflow.succeeded_at is not None + assert set(workflow.output_data) == {ExtractIntentOutput.KEY, RetrieveCareersOutput.KEY} + + def test_no_match_percentage_is_persisted(self): + workflow = self.create_workflow() + + workflow.execute() + + assert 'match' not in str(workflow.output_data) + + def test_conversation_id_identifies_the_step_record(self): + # A step can be re-executed outside the request that created it, so the trace + # handle -- not the request id -- is what ties an Xpert call to a record. + workflow = self.create_workflow() + + workflow.execute() + + intent_step = ExtractIntentStep.objects.get(workflow_record_uuid=workflow.uuid) + _, kwargs = self.mock_xpert.send_message.call_args + assert str(intent_step.uuid) in kwargs['conversation_id'] + + def test_output_round_trips_through_json(self): + workflow = self.create_workflow() + workflow.execute() + + reloaded = CareerDiscoveryWorkflow.objects.get(uuid=workflow.uuid) + careers_output = RetrieveCareersOutput.from_dict(reloaded.output_data[RetrieveCareersOutput.KEY]) + + assert careers_output.careers[0].external_id == 'ETE78CD2CDFFFAC66B' + assert careers_output.to_dict() == reloaded.output_data[RetrieveCareersOutput.KEY] + + +class TestCareerDiscoveryWorkflowFailures(CareerDiscoveryWorkflowTestMixin, TestCase): + """Tests that a failing step is recorded rather than swallowed.""" + + def test_xpert_failure_is_recorded_and_stops_the_workflow(self): + self.mock_xpert.send_message.side_effect = XpertAPIRequestError('xpert exploded') + workflow = self.create_workflow() + + with self.assertRaises(UnitOfWorkException): + workflow.execute() + + intent_step = ExtractIntentStep.objects.get(workflow_record_uuid=workflow.uuid) + assert intent_step.failed_at is not None + assert 'xpert exploded' in intent_step.exception_message + assert intent_step.output_data is None + # No partial results: the second step never ran, so it has no record at all. + assert RetrieveCareersStep.objects.count() == 0 + assert workflow.failed_at is not None + assert workflow.output_data is None + self.mock_algolia.search_jobs_index.assert_not_called() + + def test_algolia_failure_is_recorded_on_its_own_step(self): + self.mock_algolia.search_jobs_index.side_effect = AlgoliaSearchError('algolia exploded') + workflow = self.create_workflow() + + with self.assertRaises(UnitOfWorkException): + workflow.execute() + + careers_step = RetrieveCareersStep.objects.get(workflow_record_uuid=workflow.uuid) + assert careers_step.failed_at is not None + assert 'algolia exploded' in careers_step.exception_message + # The step that did succeed keeps its record, so a re-run skips it. + assert ExtractIntentStep.objects.get(workflow_record_uuid=workflow.uuid).succeeded_at is not None + assert workflow.career_candidates() == [] + + def test_careers_step_requires_the_intent_output(self): + # Executed outside its workflow, the step must refuse rather than search on nothing. + step = RetrieveCareersStep.objects.create( + workflow_record_uuid=self.create_workflow().uuid, + input_data={}, + ) + + with self.assertRaises(RetrieveCareersStepException): + step.execute(accumulated_output=object()) + + step.refresh_from_db() + assert step.failed_at is not None + self.mock_algolia.search_jobs_index.assert_not_called() diff --git a/enterprise_access/apps/pathways/tests/test_catalog_translation.py b/enterprise_access/apps/pathways/tests/test_catalog_translation.py new file mode 100644 index 00000000..2bfb6886 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_catalog_translation.py @@ -0,0 +1,371 @@ +""" +Tests for translating career vocabulary into catalog vocabulary. +""" +from unittest import mock + +import ddt +from django.test import TestCase + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.pathways import catalog_translation as translation + +PATCH_ALGOLIA_CLIENT = 'enterprise_access.apps.pathways.catalog_translation.AlgoliaSearchClient' + +# Verbatim `skill_names` values observed in the production catalog index on 2026-09-09. +SNAPSHOT = { + 'skill_names': [ + 'Python (Programming Language)', + 'SQL (Programming Language)', + 'Microsoft Excel', + 'Excel Macros', + 'Data Analysis', + 'Machine Learning', + 'Nursing', + ], + 'skills.name': ['Communication'], + 'subjects': ['Computer Science', 'Business & Management'], +} + + +class FakeAlgoliaClient: + """Records catalog searches and facet searches, replaying scripted responses.""" + + def __init__(self, search_response=None, facet_hits=None, facet_error=None): + self.search_response = search_response or {} + self.facet_hits = facet_hits or {} + self.facet_error = facet_error + self.search_calls = [] + self.facet_calls = [] + + def search_catalog_index(self, query, **kwargs): + """Stand in for ``AlgoliaSearchClient.search_catalog_index``.""" + self.search_calls.append({'query': query, **kwargs}) + return self.search_response + + def search_facet_values(self, facet_name, facet_query, **kwargs): + """Stand in for ``AlgoliaSearchClient.search_facet_values``.""" + self.facet_calls.append({'facet': facet_name, 'query': facet_query, **kwargs}) + if self.facet_error: + raise self.facet_error + values = self.facet_hits.get(facet_query, []) + return [{'value': value, 'count': 1} for value in values] + + +@ddt.ddt +class TestSnapshotCatalogFacets(TestCase): + """ + Tests for ``snapshot_catalog_facets``. + """ + + def test_snapshot_reads_the_skill_and_subject_facets(self): + client = FakeAlgoliaClient(search_response={'facets': { + 'skill_names': {'Python (Programming Language)': 95, 'Data Analysis': 141}, + 'skills.name': {'Communication': 10}, + 'subjects': {'Computer Science': 500}, + }}) + + snapshot = translation.snapshot_catalog_facets(allow_unscoped=True, algolia_client=client) + + self.assertEqual(snapshot['skill_names'], ['Python (Programming Language)', 'Data Analysis']) + self.assertEqual(snapshot['skills.name'], ['Communication']) + self.assertEqual(snapshot['subjects'], ['Computer Science']) + self.assertEqual(snapshot['truncated'], []) + + def test_snapshot_is_scoped_to_courses_and_asks_for_no_hits(self): + """ + The snapshot must search the same scope course retrieval will, or a skill can be + grounded against a value no in-scope course carries. + """ + client = FakeAlgoliaClient(search_response={'facets': {}}) + + translation.snapshot_catalog_facets(allow_unscoped=True, algolia_client=client) + + call = client.search_calls[0] + self.assertEqual(call['query'], '') + self.assertEqual(call['filters'], 'content_type:course') + self.assertEqual(call['hitsPerPage'], 0) + self.assertEqual(call['maxValuesPerFacet'], translation.MAX_VALUES_PER_FACET) + self.assertEqual(call['facets'], list(translation.CATALOG_FACET_FIELDS)) + + def test_truncation_is_detected_and_named(self): + """ + A facet at the cap is almost certainly incomplete. Measured on the live index, + ``skill_names`` returns exactly 1,000 values, which is why the refinement pass + exists at all -- so this must be visible, not silent. + """ + at_cap = {f'skill-{index}': 1 for index in range(translation.MAX_VALUES_PER_FACET)} + client = FakeAlgoliaClient(search_response={'facets': {'skill_names': at_cap}}) + + snapshot = translation.snapshot_catalog_facets(allow_unscoped=True, algolia_client=client) + + self.assertEqual(snapshot['truncated'], ['skill_names']) + + def test_absent_facets_become_empty_lists(self): + """Algolia omits a facet entirely when it has no values in scope.""" + client = FakeAlgoliaClient(search_response={}) + + snapshot = translation.snapshot_catalog_facets(allow_unscoped=True, algolia_client=client) + + for facet_field in translation.CATALOG_FACET_FIELDS: + self.assertEqual(snapshot[facet_field], []) + + @mock.patch(PATCH_ALGOLIA_CLIENT) + def test_client_is_constructed_when_not_injected(self, mock_client_class): + mock_client_class.return_value.search_catalog_index.return_value = {'facets': {}} + + translation.snapshot_catalog_facets(allow_unscoped=True) + + mock_client_class.assert_called_once() + + +@ddt.ddt +class TestTranslateSkills(TestCase): + """ + Tests for ``translate_skills``. + """ + + def test_high_confidence_matches_become_strict_and_weak_ones_boost(self): + result = translation.translate_skills(terms=['Python', 'Excel'], facet_snapshot=SNAPSHOT) + + self.assertEqual( + [entry['catalog_value'] for entry in result['strict']], + ['Python (Programming Language)'], + ) + self.assertEqual( + [entry['catalog_value'] for entry in result['boost']], + ['Microsoft Excel'], + ) + + def test_only_real_facet_values_survive(self): + """Scenario: Only real facet values survive.""" + result = translation.translate_skills( + terms=['Python', 'Underwater Basket Weaving'], + facet_snapshot=SNAPSHOT, + ) + + emitted = [entry['catalog_value'] for entry in result['strict'] + result['boost']] + self.assertNotIn('Underwater Basket Weaving', emitted) + self.assertEqual(result['unresolved'], ['Underwater Basket Weaving']) + + def test_skill_counts_are_capped(self): + """Scenario: Skill counts are capped.""" + strict_values = [f'Skill {index}' for index in range(20)] + boost_values = [f'Prefixed Boost {index}' for index in range(20)] + snapshot = {'skill_names': strict_values + boost_values, 'skills.name': []} + terms = strict_values + [f'Boost {index}' for index in range(20)] + + result = translation.translate_skills(terms=terms, facet_snapshot=snapshot) + + self.assertEqual(len(result['strict']), translation.MAX_STRICT_SKILLS) + self.assertEqual(len(result['boost']), translation.MAX_BOOST_SKILLS) + + def test_a_value_is_never_both_strict_and_boost(self): + """A repeated facet value narrows nothing and only costs query length.""" + result = translation.translate_skills( + terms=['Python', 'Python (Programming Language)'], + facet_snapshot=SNAPSHOT, + ) + + strict = {entry['catalog_value'] for entry in result['strict']} + boost = {entry['catalog_value'] for entry in result['boost']} + self.assertFalse(strict & boost) + + def test_match_type_is_recorded(self): + result = translation.translate_skills( + terms=['Data Analysis', 'Python', 'Excel'], + facet_snapshot=SNAPSHOT, + ) + + by_term = {entry['term']: entry['match_type'] + for entry in result['strict'] + result['boost']} + self.assertEqual(by_term['Data Analysis'], 'exact') + self.assertEqual(by_term['Python'], 'qualified') + self.assertEqual(by_term['Excel'], 'contained') + + @ddt.data( + (['Python', 'Data Analysis'], 1.0), + (['Python', 'Nonsense'], 0.5), + (['Nonsense One', 'Nonsense Two'], 0.0), + ) + @ddt.unpack + def test_resolution_rate_is_reported(self, terms, expected): + result = translation.translate_skills(terms=terms, facet_snapshot=SNAPSHOT) + + self.assertEqual(result['resolution_rate'], expected) + + def test_empty_terms_produce_an_empty_translation(self): + result = translation.translate_skills(terms=[], facet_snapshot=SNAPSHOT) + + self.assertEqual(result['strict'], []) + self.assertEqual(result['boost'], []) + self.assertIsNone(result['resolution_rate']) + + +class TestRefineUnmatchedSkills(TestCase): + """ + Tests for the conditional facet-search refinement pass. + """ + + def test_facet_search_recovers_a_term_missing_from_the_capped_snapshot(self): + """ + The reason this pass exists. Measured on the live index, ``Welding`` matches a + real course but falls outside the top 1,000 facet values, so the snapshot cannot + serve it. + """ + client = FakeAlgoliaClient(facet_hits={'Welding': ['Welding', 'Welding Equipment']}) + + result = translation.refine_unmatched_skills( + unresolved=['Welding'], facet_snapshot=SNAPSHOT, + allow_unscoped=True, algolia_client=client, + ) + + self.assertEqual( + [entry['catalog_value'] for entry in result['recovered']], ['Welding'], + ) + self.assertEqual(result['unresolved'], []) + + def test_one_request_is_issued_per_unresolved_term(self): + client = FakeAlgoliaClient(facet_hits={'Welding': ['Welding']}) + + translation.refine_unmatched_skills( + unresolved=['Welding', 'Brazing'], facet_snapshot=SNAPSHOT, + allow_unscoped=True, algolia_client=client, + ) + + self.assertEqual([call['query'] for call in client.facet_calls], ['Welding', 'Brazing']) + + def test_candidates_already_in_the_snapshot_are_not_reconsidered(self): + """ + They were considered and rejected on the first pass; re-offering them would + change the answer for no new information. + """ + client = FakeAlgoliaClient(facet_hits={'Spreadsheet': ['Microsoft Excel', 'Excel Macros']}) + + result = translation.refine_unmatched_skills( + unresolved=['Spreadsheet'], facet_snapshot=SNAPSHOT, + allow_unscoped=True, algolia_client=client, + ) + + self.assertEqual(result['recovered'], []) + self.assertEqual(result['unresolved'], ['Spreadsheet']) + + def test_a_term_that_only_matches_loosely_is_not_recovered(self): + """ + ``AWS`` should not become ``AWS Certified Solutions Architect Associate`` merely + because that is the top facet-search candidate -- a certification is not the + platform. The same resolution rules apply here as to the snapshot. + """ + client = FakeAlgoliaClient(facet_hits={'AWS': [ + 'AWS Certified Solutions Architect Associate', + 'AWS Serverless', + ]}) + + result = translation.refine_unmatched_skills( + unresolved=['AWS'], facet_snapshot=SNAPSHOT, + allow_unscoped=True, algolia_client=client, + ) + + recovered = [entry['catalog_value'] for entry in result['recovered']] + # Containment still applies, so *something* may match -- but never the longest, + # most-specific certification name. + self.assertNotIn('AWS Certified Solutions Architect Associate', recovered) + + def test_an_unrecoverable_term_stays_unresolved(self): + client = FakeAlgoliaClient(facet_hits={}) + + result = translation.refine_unmatched_skills( + unresolved=['Underwater Basket Weaving'], facet_snapshot=SNAPSHOT, + allow_unscoped=True, algolia_client=client, + ) + + self.assertEqual(result['recovered'], []) + self.assertEqual(result['unresolved'], ['Underwater Basket Weaving']) + + def test_a_failed_facet_search_is_recorded_not_raised(self): + """Losing one term is better than failing the step.""" + client = FakeAlgoliaClient(facet_error=AlgoliaSearchError('boom')) + + result = translation.refine_unmatched_skills( + unresolved=['Welding'], facet_snapshot=SNAPSHOT, + allow_unscoped=True, algolia_client=client, + ) + + self.assertEqual(result['recovered'], []) + self.assertEqual(result['unresolved'], ['Welding']) + self.assertEqual(len(result['errors']), 1) + self.assertIn('boom', result['errors'][0]) + + +class TestMergeRefinement(TestCase): + """ + Tests for folding recovered terms back into a translation. + """ + + def test_recovered_matches_are_appended_by_confidence(self): + base = translation.translate_skills(terms=['Python'], facet_snapshot=SNAPSHOT) + refinement = { + 'recovered': [ + {'term': 'Welding', 'catalog_value': 'Welding', + 'catalog_field': 'skill_names', 'match_type': 'exact'}, + {'term': 'Cloud', 'catalog_value': 'Cloud Computing', + 'catalog_field': 'skill_names', 'match_type': 'contained'}, + ], + 'unresolved': [], + 'errors': [], + } + + merged = translation.merge_refinement(base, refinement) + + self.assertEqual( + [entry['catalog_value'] for entry in merged['strict']], + ['Python (Programming Language)', 'Welding'], + ) + self.assertEqual( + [entry['catalog_value'] for entry in merged['boost']], ['Cloud Computing'], + ) + self.assertEqual(merged['resolution_rate'], 1.0) + + def test_snapshot_matches_keep_their_budget_slots(self): + """ + The snapshot is the only source known to be in scope, so it outranks facet search + when the budget is tight. + """ + snapshot = {'skill_names': [f'Skill {index}' for index in range(20)], 'skills.name': []} + base = translation.translate_skills( + terms=[f'Skill {index}' for index in range(20)], facet_snapshot=snapshot, + ) + refinement = { + 'recovered': [{'term': 'Welding', 'catalog_value': 'Welding', + 'catalog_field': 'skill_names', 'match_type': 'exact'}], + 'unresolved': [], + 'errors': [], + } + + merged = translation.merge_refinement(base, refinement) + + self.assertEqual(len(merged['strict']), translation.MAX_STRICT_SKILLS) + self.assertNotIn('Welding', [entry['catalog_value'] for entry in merged['strict']]) + + def test_duplicate_recoveries_are_dropped(self): + base = translation.translate_skills(terms=['Python'], facet_snapshot=SNAPSHOT) + refinement = { + 'recovered': [{'term': 'Python 3', 'catalog_value': 'Python (Programming Language)', + 'catalog_field': 'skill_names', 'match_type': 'qualified'}], + 'unresolved': [], + 'errors': [], + } + + merged = translation.merge_refinement(base, refinement) + + self.assertEqual(len(merged['strict']), 1) + + def test_unresolved_after_refinement_lowers_the_rate(self): + base = translation.translate_skills( + terms=['Python', 'Welding'], facet_snapshot=SNAPSHOT, + ) + refinement = {'recovered': [], 'unresolved': ['Welding'], 'errors': []} + + merged = translation.merge_refinement(base, refinement) + + self.assertEqual(merged['unresolved'], ['Welding']) + self.assertEqual(merged['resolution_rate'], 0.5) diff --git a/enterprise_access/apps/pathways/tests/test_course_retrieval.py b/enterprise_access/apps/pathways/tests/test_course_retrieval.py new file mode 100644 index 00000000..7246e1ea --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_course_retrieval.py @@ -0,0 +1,339 @@ +""" +Tests for course candidate retrieval. + +The broadening cases are worth the most attention. A strict skill filter buys precision +and narrows the window before ``pathway_assembly`` can span the difficulty rungs, so +broadening *appends* rather than substitutes -- and the tests pin that ordering, because +substituting would throw away the precision instead of supplementing it. +""" +import ddt +from django.test import TestCase, override_settings + +from enterprise_access.apps.api_client.algolia_client import AlgoliaSearchError +from enterprise_access.apps.pathways.course_retrieval import ( + CANDIDATE_HITS_PER_PAGE, + MAX_QUERY_WORDS, + MAX_STRICT_FILTERS, + MIN_CANDIDATES_FOR_ASSEMBLY, + build_course_filters, + build_course_query, + eval_customer_uuid, + retrieve_candidate_courses, + skill_values +) + +CUSTOMER_UUID = '417306cb-b24a-4d06-b83c-fb2a61d7fb96' + + +def translation(strict=(), boost=()): + """Build a ``translate_skills``-shaped result.""" + def entries(values): + return [ + {'term': v, 'catalog_value': v, 'catalog_field': 'skill_names', 'match_type': 'exact'} + for v in values + ] + return {'strict': entries(strict), 'boost': entries(boost), 'unresolved': []} + + +class FakeAlgoliaClient: + """Replays scripted catalog responses and records every search.""" + + def __init__(self, responses=None, error=None): + # A list, consumed in order, so the retry can return something different. + self.responses = list(responses or [{'hits': []}]) + self.error = error + self.calls = [] + + def search_catalog_index(self, query, **kwargs): + """Stand in for ``AlgoliaSearchClient.search_catalog_index``.""" + self.calls.append({'query': query, **kwargs}) + if self.error: + raise self.error + if len(self.responses) > 1: + return self.responses.pop(0) + return self.responses[0] + + +def hits(*keys, levels=None): + """Build a hit list. ``levels`` cycles through difficulty rungs when supplied.""" + cycle = list(levels or ['Introductory']) + return {'hits': [ + {'key': key, 'title': key, 'level_type': cycle[index % len(cycle)]} + for index, key in enumerate(keys) + ]} + + +def spanning(*keys): + """A hit list wide enough and laddered enough to need no broadening.""" + return hits(*keys, levels=['Introductory', 'Intermediate', 'Advanced']) + + +class TestBuildCourseQuery(TestCase): + """ + Tests for ``build_course_query``. + """ + + def test_the_career_name_leads(self): + query = build_course_query(career_name='Data Analyst', boost_terms=['Python', 'SQL']) + + self.assertTrue(query.startswith('Data Analyst')) + + def test_the_query_is_capped_on_a_word_boundary(self): + """ + The index ANDs every word; even relaxed, a very long query is mostly noise + competing for ranking signal. + """ + query = build_course_query( + career_name='A B C D E', boost_terms=['F G H I J K L M N O P'], + ) + + words = query.split() + self.assertEqual(len(words), MAX_QUERY_WORDS) + self.assertNotIn(' ', query) + + def test_an_empty_career_name_still_produces_a_query_from_boosts(self): + self.assertEqual(build_course_query(career_name='', boost_terms=['Welding']), 'Welding') + + +@ddt.ddt +class TestBuildCourseFilters(TestCase): + """ + Tests for ``build_course_filters``. + """ + + def test_content_type_and_language_are_unconditional(self): + filters = build_course_filters(strict_skills=[], customer_uuid='') + + self.assertIn('content_type:course', filters) + self.assertIn('language:"English"', filters) + + def test_the_customer_scope_is_applied_when_supplied(self): + filters = build_course_filters(strict_skills=[], customer_uuid=CUSTOMER_UUID) + + self.assertIn(f'enterprise_customer_uuids:"{CUSTOMER_UUID}"', filters) + + def test_strict_skills_are_ored_not_anded(self): + """ + A course rarely carries every skill of a career, so ANDing them would routinely + return nothing. + """ + filters = build_course_filters(strict_skills=['Welding', 'Blueprint Reading']) + + self.assertIn('(skill_names:"Welding" OR skill_names:"Blueprint Reading")', filters) + + def test_no_skill_clause_is_added_when_none_resolved(self): + self.assertNotIn('skill_names', build_course_filters(strict_skills=[])) + + +class TestSkillValues(TestCase): + """ + Tests for ``skill_values``. + """ + + def test_catalog_values_are_read_from_the_bucket(self): + values = skill_values(translation(strict=['Welding']), 'strict', 4) + + self.assertEqual(values, ['Welding']) + + def test_compound_artifacts_are_dropped(self): + """"SQL & Python" matches no facet value, so it spends a slot to boost nothing.""" + values = skill_values(translation(boost=['SQL & Python', 'Welding']), 'boost', 4) + + self.assertEqual(values, ['Welding']) + + def test_the_budget_is_a_prefix_because_order_carries_relevance(self): + values = skill_values(translation(strict=['A', 'B', 'C', 'D', 'E', 'F']), 'strict', 2) + + self.assertEqual(values, ['A', 'B']) + + def test_a_missing_bucket_is_empty_rather_than_an_error(self): + self.assertEqual(skill_values({}, 'strict', 4), []) + + +class TestRetrieveCandidateCourses(TestCase): + """ + Tests for ``retrieve_candidate_courses``. + """ + + def test_twenty_candidates_are_requested(self): + """ + Not five. Relevance ranking is intro-heavy at rank 5 and recovers by rank 20, and + assembly needs the wider window to span the rungs. + """ + client = FakeAlgoliaClient([hits('A+1')]) + + retrieve_candidate_courses( + career_name='Welder', translation=translation(), algolia_client=client, + ) + + self.assertEqual(client.calls[0]['hitsPerPage'], CANDIDATE_HITS_PER_PAGE) + + def test_the_query_is_relaxed_because_the_index_ands_every_word(self): + client = FakeAlgoliaClient([hits('A+1')]) + + retrieve_candidate_courses( + career_name='Welder', translation=translation(), algolia_client=client, + ) + + self.assertEqual(client.calls[0]['removeWordsIfNoResults'], 'allOptional') + + def test_candidates_carry_the_metadata_the_reranker_needs(self): + """Scenario: Candidates carry the metadata the re-ranker needs.""" + client = FakeAlgoliaClient([hits('A+1')]) + + retrieve_candidate_courses( + career_name='Welder', translation=translation(), algolia_client=client, + ) + + requested = client.calls[0]['attributesToRetrieve'] + for attribute in ('key', 'title', 'short_description', 'full_description', 'level_type'): + self.assertIn(attribute, requested) + + def test_boosts_become_optional_filters_not_hard_ones(self): + client = FakeAlgoliaClient([hits('A+1')]) + + retrieve_candidate_courses( + career_name='Welder', translation=translation(boost=['Welding']), + algolia_client=client, + ) + + self.assertEqual(client.calls[0]['optionalFilters'], ['skill_names:Welding']) + self.assertNotIn('skill_names:"Welding"', client.calls[0]['filters']) + + def test_the_strict_filter_budget_is_respected(self): + client = FakeAlgoliaClient([hits('A+1')]) + + result = retrieve_candidate_courses( + career_name='Welder', + translation=translation(strict=[f'S{i}' for i in range(10)]), + algolia_client=client, + ) + + self.assertEqual(len(result['strict_filters_applied']), MAX_STRICT_FILTERS) + + def test_a_thin_strict_set_is_broadened(self): + """ + Measured: a strict skill filter turned `Data Analyst` from 2/2/1 into 5/0/0 by + narrowing the window before assembly could span it. + """ + client = FakeAlgoliaClient([hits('A+1', 'B+2'), hits('C+3', 'D+4')]) + + result = retrieve_candidate_courses( + career_name='Welder', translation=translation(strict=['Welding']), + algolia_client=client, + ) + + self.assertEqual(len(client.calls), 2) + self.assertTrue(result['broadened']) + self.assertEqual(result['strict_hit_count'], 2) + self.assertNotIn('skill_names', client.calls[1]['filters']) + + def test_broadening_appends_rather_than_substitutes(self): + """ + The precisely-matched courses must keep their rank, or the precision the strict + filter bought is thrown away rather than supplemented. + """ + client = FakeAlgoliaClient([hits('A+1'), hits('Z+9', 'A+1', 'B+2')]) + + result = retrieve_candidate_courses( + career_name='Welder', translation=translation(strict=['Welding']), + algolia_client=client, + ) + + keys = [hit['key'] for hit in result['courses']] + self.assertEqual(keys[0], 'A+1') + # And the duplicate from the broad search is not re-added. + self.assertEqual(keys.count('A+1'), 1) + self.assertEqual(keys, ['A+1', 'Z+9', 'B+2']) + + def test_a_wide_laddered_strict_set_is_not_broadened(self): + """A second search would be pure cost when the window is already sufficient.""" + client = FakeAlgoliaClient([ + spanning(*[f'A+{i}' for i in range(MIN_CANDIDATES_FOR_ASSEMBLY)]), + ]) + + result = retrieve_candidate_courses( + career_name='Welder', translation=translation(strict=['Welding']), + algolia_client=client, + ) + + self.assertEqual(len(client.calls), 1) + self.assertFalse(result['broadened']) + self.assertEqual(result['strict_rungs_spanned'], 3) + + def test_a_wide_but_single_rung_strict_set_is_still_broadened(self): + """ + The measured `Data Analyst` case: 17 strict hits cleared the count threshold and + still assembled to 5/0/0, because every hit sat on the Introductory rung. A hit + count alone cannot detect that. + """ + flat = hits(*[f'A+{i}' for i in range(MIN_CANDIDATES_FOR_ASSEMBLY + 5)]) + client = FakeAlgoliaClient([flat, spanning('B+1', 'B+2', 'B+3')]) + + result = retrieve_candidate_courses( + career_name='Data Analyst', translation=translation(strict=['Data Analysis']), + algolia_client=client, + ) + + self.assertTrue(result['broadened']) + self.assertEqual(result['strict_rungs_spanned'], 1) + + def test_zero_hits_without_strict_filters_does_not_broaden(self): + """There is nothing left to relax, so a second identical search is pure cost.""" + client = FakeAlgoliaClient([{'hits': []}]) + + result = retrieve_candidate_courses( + career_name='Welder', translation=translation(), algolia_client=client, + ) + + self.assertEqual(len(client.calls), 1) + self.assertFalse(result['broadened']) + self.assertTrue(result['zero_hits']) + + def test_a_career_with_no_courses_is_reported_not_raised(self): + """ + Scenario (replacing the plan's scope-only fallback): "no courses for this career + in this catalog" is a real answer, and the caller turns it into a no-pathway. + """ + client = FakeAlgoliaClient([{'hits': []}, {'hits': []}]) + + result = retrieve_candidate_courses( + career_name='Underwater Basket Weaver', + translation=translation(strict=['Basket Weaving']), + algolia_client=client, + ) + + self.assertTrue(result['zero_hits']) + self.assertEqual(result['courses'], []) + + def test_a_search_failure_propagates(self): + """A transport failure is not a "no coverage" answer and must not look like one.""" + client = FakeAlgoliaClient(error=AlgoliaSearchError('boom')) + + with self.assertRaises(AlgoliaSearchError): + retrieve_candidate_courses( + career_name='Welder', translation=translation(), algolia_client=client, + ) + + def test_non_dict_hits_are_discarded(self): + client = FakeAlgoliaClient([{'hits': [{'key': 'A+1'}, 'garbage', None]}]) + + result = retrieve_candidate_courses( + career_name='Welder', translation=translation(), algolia_client=client, + ) + + self.assertEqual(result['courses'], [{'key': 'A+1'}]) + + +class TestEvalCustomerUuid(TestCase): + """ + Tests for ``eval_customer_uuid``. + """ + + @override_settings(PATHWAYS_EVAL_CUSTOMER_UUID=f' {CUSTOMER_UUID} ') + def test_the_configured_uuid_is_normalised(self): + self.assertEqual(eval_customer_uuid(), CUSTOMER_UUID) + + @override_settings(PATHWAYS_EVAL_CUSTOMER_UUID='') + def test_no_pinned_customer_is_an_empty_string(self): + self.assertEqual(eval_customer_uuid(), '') diff --git a/enterprise_access/apps/pathways/tests/test_model_backends.py b/enterprise_access/apps/pathways/tests/test_model_backends.py new file mode 100644 index 00000000..4d6313dc --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_model_backends.py @@ -0,0 +1,446 @@ +""" +Tests for the model-backend adapter. + +The interchangeability tests are the point of the chunk: they assert all three backends are +substitutable through the same call, which is what makes "compare the models" a query +rather than a rig. Nothing here touches the network -- Xpert is patched at the domain +function, and the two metered backends take an injected client. +""" +import json +from types import SimpleNamespace +from unittest import mock + +import ddt +from django.test import TestCase, override_settings + +from enterprise_access.apps.pathways.model_backends import ( + CLAUDE_BACKEND, + OPENAI_BACKEND, + XPERT_BACKEND, + ClaudeBackend, + ModelBackendConfigurationError, + ModelBackendRequestError, + ModelResponse, + ModelResponseParseError, + OpenAIBackend, + XpertBackend, + get_model_backend +) +from enterprise_access.apps.prompts.api import PromptError +from enterprise_access.apps.prompts.api_client import XpertAPIError +from enterprise_access.apps.prompts.models import PromptType + +PATCH_GET_PROMPT = 'enterprise_access.apps.pathways.model_backends.xpert.prompts_api.get_current_prompt' +PATCH_SEND = 'enterprise_access.apps.pathways.model_backends.xpert.prompts_api.send_xpert_message' + + +def fake_anthropic_message(text='{"ok": true}', *, input_tokens=120, output_tokens=45, + model='claude-sonnet-5', stop_reason='end_turn'): + """Build a stand-in for a Messages API response object.""" + return SimpleNamespace( + content=[SimpleNamespace(text=text, type='text')], + model=model, + stop_reason=stop_reason, + usage=SimpleNamespace(input_tokens=input_tokens, output_tokens=output_tokens), + ) + + +def fake_openai_completion(text='{"ok": true}', *, prompt_tokens=90, completion_tokens=30, + model='gpt-4o', finish_reason='stop'): + """Build a stand-in for a chat-completions response object.""" + return SimpleNamespace( + choices=[SimpleNamespace( + message=SimpleNamespace(content=text), finish_reason=finish_reason, + )], + model=model, + usage=SimpleNamespace(prompt_tokens=prompt_tokens, completion_tokens=completion_tokens), + ) + + +def fake_openai_client(completion=None, error=None): + """A client whose ``chat.completions.create`` returns a completion or raises.""" + create = (mock.Mock(side_effect=error) if error + else mock.Mock(return_value=completion or fake_openai_completion())) + return SimpleNamespace(chat=SimpleNamespace(completions=SimpleNamespace(create=create))) + + +def fake_client(message=None, error=None): + """A client whose ``messages.create`` returns ``message`` or raises ``error``.""" + create = mock.Mock(side_effect=error) if error else mock.Mock(return_value=message or fake_anthropic_message()) + return SimpleNamespace(messages=SimpleNamespace(create=create)) + + +class TestModelResponse(TestCase): + """ + Tests for ``ModelResponse``. + """ + + def test_total_tokens_sums_both_sides(self): + response = ModelResponse(content='', backend='x', input_tokens=10, output_tokens=5) + + self.assertEqual(response.total_tokens, 15) + + def test_total_tokens_is_none_when_either_side_is_unreported(self): + """ + Xpert reports no usage. Zero would read as "this call was free", which is a + measurement rather than the absence of one. + """ + self.assertIsNone(ModelResponse(content='', backend='x', output_tokens=5).total_tokens) + self.assertIsNone(ModelResponse(content='', backend='x', input_tokens=5).total_tokens) + + def test_as_json_parses_content(self): + response = ModelResponse(content=' {"a": 1} ', backend='x') + + self.assertEqual(response.as_json(), {'a': 1}) + + def test_as_json_raises_a_typed_error_without_echoing_the_body(self): + response = ModelResponse(content='sorry, I cannot do that', backend='xpert') + + with self.assertRaises(ModelResponseParseError) as ctx: + response.as_json() + + message = str(ctx.exception) + self.assertIn('xpert', message) + self.assertNotIn('sorry, I cannot do that', message) + + def test_the_trace_dict_excludes_the_response_body(self): + """ + A response can quote the learner's own intake back, so it stays out of the trace. + The step persists its own parsed output instead. + """ + response = ModelResponse( + content='learner said something private', backend='claude', + model='claude-sonnet-5', input_tokens=1, output_tokens=2, elapsed_ms=99, + ) + + trace = response.to_trace_dict() + + self.assertNotIn('content', trace) + self.assertEqual(trace, { + 'backend': 'claude', 'model': 'claude-sonnet-5', + 'input_tokens': 1, 'output_tokens': 2, 'elapsed_ms': 99, + }) + + +@ddt.ddt +class TestXpertBackend(TestCase): + """ + Tests for ``XpertBackend``. + """ + + def setUp(self): + super().setUp() + self.prompt = mock.Mock(modified=None) + self.prompt.history.first.return_value = SimpleNamespace(history_id=7) + + @mock.patch(PATCH_SEND) + @mock.patch(PATCH_GET_PROMPT) + def test_a_completion_returns_a_normalised_response(self, mock_prompt, mock_send): + mock_prompt.return_value = self.prompt + mock_send.return_value = SimpleNamespace(role='assistant', content='{"a": 1}') + + response = XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='ignored', user_content='{}', trace_id='trace-1', + ) + + self.assertEqual(response.content, '{"a": 1}') + self.assertEqual(response.backend, XPERT_BACKEND) + self.assertEqual(response.as_json(), {'a': 1}) + + @mock.patch(PATCH_SEND) + @mock.patch(PATCH_GET_PROMPT) + def test_the_prompt_revision_is_recorded(self, mock_prompt, mock_send): + """ + A pathway generated last week was generated by wording that may have changed + since, so a regression between runs has to be attributable to the prompt. + """ + mock_prompt.return_value = self.prompt + mock_send.return_value = SimpleNamespace(role='assistant', content='{}') + + response = XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='', user_content='{}', trace_id='trace-1', + ) + + self.assertEqual(response.metadata['prompt_revision'], '7') + + @mock.patch(PATCH_GET_PROMPT) + def test_a_missing_prompt_row_is_a_configuration_error(self, mock_prompt): + """Not transient: no request was sent, so a caller must not retry it.""" + mock_prompt.side_effect = PromptError('nope') + + with self.assertRaises(ModelBackendConfigurationError): + XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='', user_content='{}', trace_id='t', + ) + + @ddt.data(PromptError('boom'), XpertAPIError('boom')) + def test_a_failed_request_is_a_request_error(self, error): + with mock.patch(PATCH_GET_PROMPT) as mock_prompt, mock.patch(PATCH_SEND) as mock_send: + mock_prompt.return_value = self.prompt + mock_send.side_effect = error + + with self.assertRaises(ModelBackendRequestError): + XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='', user_content='{}', trace_id='t', + ) + + @mock.patch(PATCH_SEND) + @mock.patch(PATCH_GET_PROMPT) + def test_the_user_content_is_sent_verbatim(self, mock_prompt, mock_send): + mock_prompt.return_value = self.prompt + mock_send.return_value = SimpleNamespace(role='assistant', content='{}') + payload = json.dumps({'career': 'Welder'}) + + XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='', user_content=payload, trace_id='t', + ) + + sent = mock_send.call_args.kwargs['messages'] + self.assertEqual([message.content for message in sent], [payload]) + + +class TestClaudeBackend(TestCase): + """ + Tests for ``ClaudeBackend``. + """ + + def test_a_completion_returns_content_and_token_counts(self): + backend = ClaudeBackend(client=fake_client(), api_key='k') + + response = backend.complete(system_prompt='sys', user_content='hi', trace_id='t') + + self.assertEqual(response.content, '{"ok": true}') + self.assertEqual(response.backend, CLAUDE_BACKEND) + self.assertEqual(response.input_tokens, 120) + self.assertEqual(response.output_tokens, 45) + self.assertEqual(response.total_tokens, 165) + + def test_the_callers_system_prompt_is_used(self): + """Unlike Xpert, this backend takes the prompt from the caller.""" + client = fake_client() + backend = ClaudeBackend(client=client, api_key='k') + + backend.complete(system_prompt='be terse', user_content='hi', trace_id='t') + + self.assertEqual(client.messages.create.call_args.kwargs['system'], 'be terse') + + def test_only_text_blocks_are_concatenated(self): + """A future block type must not silently corrupt a JSON payload.""" + message = SimpleNamespace( + content=[ + SimpleNamespace(text='{"a":', type='text'), + SimpleNamespace(type='thinking'), + SimpleNamespace(text=' 1}', type='text'), + ], + model='m', stop_reason='end_turn', + usage=SimpleNamespace(input_tokens=1, output_tokens=1), + ) + backend = ClaudeBackend(client=fake_client(message), api_key='k') + + self.assertEqual(backend.complete( + system_prompt='', user_content='', trace_id='t', + ).as_json(), {'a': 1}) + + def test_a_response_without_usage_reports_none_rather_than_zero(self): + message = SimpleNamespace( + content=[SimpleNamespace(text='{}', type='text')], + model='m', stop_reason='end_turn', usage=None, + ) + backend = ClaudeBackend(client=fake_client(message), api_key='k') + + response = backend.complete(system_prompt='', user_content='', trace_id='t') + + self.assertIsNone(response.input_tokens) + self.assertIsNone(response.total_tokens) + + @override_settings(ANTHROPIC_API_KEY='') + def test_a_missing_api_key_is_a_configuration_error(self): + with self.assertRaisesRegex(ModelBackendConfigurationError, 'ANTHROPIC_API_KEY'): + ClaudeBackend().complete(system_prompt='', user_content='', trace_id='t') + + def test_a_provider_failure_is_a_request_error_naming_only_the_type(self): + """The SDK's message can echo the request body, so only its type is reported.""" + backend = ClaudeBackend( + client=fake_client(error=ValueError('learner said something private')), + api_key='k', + ) + + with self.assertRaises(ModelBackendRequestError) as ctx: + backend.complete(system_prompt='', user_content='', trace_id='t') + + message = str(ctx.exception) + self.assertIn('ValueError', message) + self.assertNotIn('learner said something private', message) + + +class TestOpenAIBackend(TestCase): + """ + Tests for ``OpenAIBackend``. + """ + + def test_a_completion_returns_content_and_token_counts(self): + backend = OpenAIBackend(client=fake_openai_client(), api_key='k') + + response = backend.complete(system_prompt='sys', user_content='hi', trace_id='t') + + self.assertEqual(response.content, '{"ok": true}') + self.assertEqual(response.backend, OPENAI_BACKEND) + self.assertEqual(response.input_tokens, 90) + self.assertEqual(response.output_tokens, 30) + self.assertEqual(response.total_tokens, 120) + + def test_json_mode_is_requested(self): + """ + Every prompt in this pipeline requires JSON, and OpenAI can enforce it + server-side -- which removes a failure mode rather than handling it. + """ + client = fake_openai_client() + + OpenAIBackend(client=client, api_key='k').complete( + system_prompt='sys', user_content='hi', trace_id='t', + ) + + self.assertEqual( + client.chat.completions.create.call_args.kwargs['response_format'], + {'type': 'json_object'}, + ) + + def test_the_system_and_user_messages_are_sent_separately(self): + client = fake_openai_client() + + OpenAIBackend(client=client, api_key='k').complete( + system_prompt='be terse', user_content='the payload', trace_id='t', + ) + + messages = client.chat.completions.create.call_args.kwargs['messages'] + self.assertEqual(messages[0], {'role': 'system', 'content': 'be terse'}) + self.assertEqual(messages[1], {'role': 'user', 'content': 'the payload'}) + + def test_a_truncated_response_is_visible_in_the_metadata(self): + """``length`` is how a silently-cut-off ordering becomes diagnosable.""" + completion = fake_openai_completion(finish_reason='length') + backend = OpenAIBackend(client=fake_openai_client(completion), api_key='k') + + response = backend.complete(system_prompt='', user_content='', trace_id='t') + + self.assertEqual(response.metadata['finish_reason'], 'length') + + def test_a_response_with_no_choices_yields_empty_content_rather_than_raising(self): + """The caller already degrades an unusable response; this is that same case.""" + completion = SimpleNamespace(choices=[], model='gpt-4o', usage=None) + backend = OpenAIBackend(client=fake_openai_client(completion), api_key='k') + + response = backend.complete(system_prompt='', user_content='', trace_id='t') + + self.assertEqual(response.content, '') + self.assertIsNone(response.input_tokens) + + @override_settings(OPENAI_API_KEY='') + def test_a_missing_api_key_is_a_configuration_error(self): + with self.assertRaisesRegex(ModelBackendConfigurationError, 'OPENAI_API_KEY'): + OpenAIBackend().complete(system_prompt='', user_content='', trace_id='t') + + def test_a_provider_failure_is_a_request_error_naming_only_the_type(self): + """The SDK's message can echo the request body, so only its type is reported.""" + backend = OpenAIBackend( + client=fake_openai_client(error=ValueError('learner said something private')), + api_key='k', + ) + + with self.assertRaises(ModelBackendRequestError) as ctx: + backend.complete(system_prompt='', user_content='', trace_id='t') + + message = str(ctx.exception) + self.assertIn('ValueError', message) + self.assertNotIn('learner said something private', message) + + +class TestBackendInterchangeability(TestCase): + """ + Scenario: Backends are interchangeable. + """ + + @mock.patch(PATCH_SEND) + @mock.patch(PATCH_GET_PROMPT) + def test_both_backends_expose_the_same_response_surface(self, mock_prompt, mock_send): + mock_prompt.return_value = mock.Mock(modified=None, **{'history.first.return_value': None}) + mock_send.return_value = SimpleNamespace(role='assistant', content='{"a": 1}') + + responses = [ + XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='s', user_content='u', trace_id='t'), + ClaudeBackend(client=fake_client(), api_key='k').complete( + system_prompt='s', user_content='u', trace_id='t'), + OpenAIBackend(client=fake_openai_client(), api_key='k').complete( + system_prompt='s', user_content='u', trace_id='t'), + ] + + for response in responses: + self.assertIsInstance(response.content, str) + self.assertIsInstance(response.elapsed_ms, int) + self.assertIn('elapsed_ms', response.to_trace_dict()) + self.assertIn(response.backend, (XPERT_BACKEND, CLAUDE_BACKEND, OPENAI_BACKEND)) + # Every backend's content must survive the same JSON parse. + self.assertIsInstance(response.as_json(), dict) + + @mock.patch(PATCH_SEND) + @mock.patch(PATCH_GET_PROMPT) + def test_elapsed_milliseconds_is_measured_by_the_base_class(self, mock_prompt, mock_send): + """ + A backend that reported its own timing could report it differently; the base owns + it so the number means the same thing everywhere. + """ + mock_prompt.return_value = mock.Mock(modified=None, **{'history.first.return_value': None}) + mock_send.return_value = SimpleNamespace(role='assistant', content='{}') + + response = XpertBackend(prompt_type=PromptType.LEARNER_INTENT).complete( + system_prompt='', user_content='', trace_id='t', + ) + + self.assertGreaterEqual(response.elapsed_ms, 0) + + +@ddt.ddt +class TestGetModelBackend(TestCase): + """ + Scenario: Backend selection is configuration. + """ + + @override_settings(PATHWAYS_MODEL_BACKEND='xpert') + def test_the_configured_backend_is_returned(self): + backend = get_model_backend(prompt_type=PromptType.LEARNER_INTENT) + + self.assertIsInstance(backend, XpertBackend) + + @override_settings(PATHWAYS_MODEL_BACKEND='claude') + def test_the_claude_backend_is_selectable_by_settings_alone(self): + self.assertIsInstance(get_model_backend(prompt_type=PromptType.LEARNER_INTENT), ClaudeBackend) + + @override_settings(PATHWAYS_MODEL_BACKEND='openai') + def test_the_openai_backend_is_selectable_by_settings_alone(self): + self.assertIsInstance(get_model_backend(prompt_type=PromptType.LEARNER_INTENT), OpenAIBackend) + + @override_settings(PATHWAYS_MODEL_BACKEND='xpert') + def test_an_explicit_name_overrides_the_setting(self): + """A comparison run needs both backends live in one process.""" + backend = get_model_backend( + prompt_type=PromptType.LEARNER_INTENT, backend_name='claude', + ) + + self.assertIsInstance(backend, ClaudeBackend) + + @ddt.data('gpt', '', 'XPERTT', None) + def test_an_unknown_backend_name_raises_rather_than_defaulting(self, name): + """ + Falling back to a default would hide a configuration typo -- and could silently + route traffic to a paid backend nobody selected. + """ + with override_settings(PATHWAYS_MODEL_BACKEND=name): + with self.assertRaises(ModelBackendConfigurationError): + get_model_backend(prompt_type=PromptType.LEARNER_INTENT) + + @override_settings(PATHWAYS_MODEL_BACKEND=' XPERT ') + def test_the_setting_is_normalised(self): + self.assertIsInstance( + get_model_backend(prompt_type=PromptType.LEARNER_INTENT), XpertBackend, + ) diff --git a/enterprise_access/apps/pathways/tests/test_models.py b/enterprise_access/apps/pathways/tests/test_models.py new file mode 100644 index 00000000..1d69cff3 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_models.py @@ -0,0 +1,194 @@ +""" +Tests for the conditional workflow base. +""" +import ddt +from django.test import TestCase + +from enterprise_access.apps.pathways.models import AbstractConditionalWorkflow +from enterprise_access.apps.pathways.tests.models import ( + AddStep, + ConditionalDoubleStep, + ConditionalTestWorkflow, + SquareStep, + UnconditionalTestWorkflow +) +from enterprise_access.apps.provisioning.models import ProvisionNewCustomerWorkflow +from enterprise_access.apps.workflow.models import AbstractWorkflow + + +def build_conditional_workflow(argument_1, argument_2): + """A ``ConditionalTestWorkflow`` whose input feeds ``AddStep``.""" + workflow = ConditionalTestWorkflow() + workflow.input_data = { + 'add_input': {'argument_1': argument_1, 'argument_2': argument_2}, + } + workflow.save() + return workflow + + +@ddt.ddt +class TestAbstractConditionalWorkflow(TestCase): + """ + Tests for ``AbstractConditionalWorkflow``. + """ + + def test_a_step_opts_out(self): + """Scenario: A step opts out.""" + # 2 + 2 = 4, which is even, so the doubling step declines to run. + workflow = build_conditional_workflow(2, 2) + + workflow.execute() + + # No step record at all for the skipped step -- distinguishable from a step that + # ran and produced nothing. + self.assertFalse( + ConditionalDoubleStep.objects.filter(workflow_record_uuid=workflow.uuid).exists() + ) + # The step after it still ran, and received the accumulated output of the step + # that did run: 4 ** 2, not (4 * 2) ** 2. + square_record = SquareStep.objects.get(workflow_record_uuid=workflow.uuid) + self.assertEqual(square_record.output_object.result, 16) + self.assertIsNotNone(square_record.succeeded_at) + + def test_a_step_opts_in(self): + """The same workflow, same code path, condition satisfied.""" + # 1 + 2 = 3, which is odd, so the doubling step runs: (3 * 2) ** 2 == 36. + workflow = build_conditional_workflow(1, 2) + + workflow.execute() + + double_record = ConditionalDoubleStep.objects.get(workflow_record_uuid=workflow.uuid) + self.assertEqual(double_record.output_object.result, 6) + square_record = SquareStep.objects.get(workflow_record_uuid=workflow.uuid) + self.assertEqual(square_record.output_object.result, 36) + + @ddt.data( + (2, 2, 16), # even -> skipped + (1, 2, 36), # odd -> executed + (3, 4, 196), # odd -> executed, (7*2)**2 + (4, 4, 64), # even -> skipped, 8**2 + ) + @ddt.unpack + def test_conditional_result_is_data_dependent(self, argument_1, argument_2, expected): + workflow = build_conditional_workflow(argument_1, argument_2) + + workflow.execute() + + self.assertEqual(workflow.output_object.square_output.result, expected) + + def test_skipped_step_leaves_its_output_key_unset(self): + """A skipped step must not fabricate output for downstream steps to read.""" + workflow = build_conditional_workflow(2, 2) + + workflow.execute() + + self.assertIsNone(workflow.output_object.double_output) + self.assertIsNotNone(workflow.output_object.add_output) + + def test_skipped_step_output_round_trips_as_null(self): + """ + Regression: the parent's dynamic output class declares each field with the step's + output type (default ``None``, but not ``Optional``), so cattrs emits + unstructure code that dereferences every field. That is safe only because + ``AbstractWorkflow`` steps always run. With a skipped step the field stays + ``None`` and ``to_dict()`` raised ``AttributeError`` mid-run -- which is why + ``AbstractConditionalWorkflow`` re-declares these fields as ``Optional``. + """ + workflow = build_conditional_workflow(2, 2) + + workflow.execute() + + # The persisted output is JSON with an explicit null for the skipped step. + self.assertIsNone(workflow.output_data['double_output']) + self.assertEqual(workflow.output_data['add_output']['result'], 4) + # And it structures back into an object without raising. + self.assertIsNone(workflow.output_object.double_output) + + def test_preceding_step_uuid_skips_over_the_skipped_step(self): + """ + The step chain must link to the step that actually ran before it, so a trace of + a run with a skipped step is still a connected chain. + """ + workflow = build_conditional_workflow(2, 2) + + workflow.execute() + + add_record = AddStep.objects.get(workflow_record_uuid=workflow.uuid) + square_record = SquareStep.objects.get(workflow_record_uuid=workflow.uuid) + self.assertEqual(square_record.preceding_step_uuid, add_record.uuid) + + def test_default_behaviour_is_unchanged(self): + """Scenario: Default behaviour is unchanged.""" + workflow = UnconditionalTestWorkflow() + workflow.input_data = {'add_input': {'argument_1': 3, 'argument_2': 4}} + workflow.save() + + workflow.execute() + + # Every step ran, in order, exactly as AbstractWorkflow would have run them. + self.assertTrue(AddStep.objects.filter(workflow_record_uuid=workflow.uuid).exists()) + self.assertEqual(workflow.output_object.square_output.result, 49) + + def test_step_should_execute_defaults_to_true(self): + """A step class with no ``should_execute`` always runs.""" + self.assertTrue( + AbstractConditionalWorkflow.step_should_execute(AddStep, None, None) + ) + + def test_step_should_execute_consults_the_step(self): + self.assertFalse( + AbstractConditionalWorkflow.step_should_execute(ConditionalDoubleStep, None, None) + ) + + def test_already_succeeded_workflow_is_not_re_executed(self): + """Idempotency is inherited, not lost, by the override.""" + workflow = build_conditional_workflow(1, 2) + workflow.execute() + original_output = workflow.output_data + + self.assertIsNone(workflow.process_input()) + self.assertEqual(workflow.output_data, original_output) + + def test_succeeded_steps_are_skipped_on_re_execution(self): + """ + Re-running a workflow reuses succeeded step records rather than duplicating them, + which is what makes the runbook's "just re-run it" advice safe. + """ + workflow = build_conditional_workflow(1, 2) + workflow.execute() + first_add_uuid = AddStep.objects.get(workflow_record_uuid=workflow.uuid).uuid + + # Clear the workflow's own success marker so process_input() runs the loop again. + workflow.succeeded_at = None + workflow.save() + workflow.execute() + + self.assertEqual( + AddStep.objects.filter(workflow_record_uuid=workflow.uuid).count(), 1 + ) + self.assertEqual( + AddStep.objects.get(workflow_record_uuid=workflow.uuid).uuid, first_add_uuid + ) + + +class TestProvisioningIsUnaffected(TestCase): + """ + Scenario: Provisioning is untouched. + + ``AbstractConditionalWorkflow`` subclasses ``AbstractWorkflow`` rather than modifying + it, so provisioning cannot be affected. This asserts the structural guarantee; the + behavioural guarantee is the provisioning suite itself, which runs unmodified. + """ + + def test_the_conditional_base_does_not_modify_abstract_workflow(self): + # AbstractWorkflow retains its own process_input, distinct from the override. + self.assertNotEqual( + AbstractWorkflow.process_input, + AbstractConditionalWorkflow.process_input, + ) + # And it has gained no conditional-execution surface. + self.assertFalse(hasattr(AbstractWorkflow, 'step_should_execute')) + + def test_provisioning_workflow_still_uses_the_unconditional_base(self): + self.assertTrue(issubclass(ProvisionNewCustomerWorkflow, AbstractWorkflow)) + self.assertFalse(issubclass(ProvisionNewCustomerWorkflow, AbstractConditionalWorkflow)) diff --git a/enterprise_access/apps/pathways/tests/test_pathway_assembly.py b/enterprise_access/apps/pathways/tests/test_pathway_assembly.py new file mode 100644 index 00000000..6c68dfbc --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_pathway_assembly.py @@ -0,0 +1,318 @@ +""" +Tests for pathway assembly and the Tier 1 correctness gates. + +The assembly cases are written against the *measured* defects rather than invented ones: +the "all five introductory" and "all five from one partner" fixtures reproduce what +``data analyst`` and ``biomedical engineer`` actually returned from the pinned 2U catalog. +""" +import ddt +from django.test import TestCase + +from enterprise_access.apps.pathways.pathway_assembly import ( + DEFAULT_LEVEL_QUOTA, + LEVEL_ADVANCED, + LEVEL_INTERMEDIATE, + LEVEL_INTRODUCTORY, + MAX_PER_PARTNER, + PATHWAY_SIZE, + Candidate, + assemble_pathway, + eligible_candidates, + validate_pathway +) + + +def hit(key, *, title='A Course', level=LEVEL_INTRODUCTORY, partner='edX', language='English'): + """Build a catalog hit in the shape Algolia returns.""" + return { + 'key': key, + 'title': title, + 'level_type': level, + 'partners': [{'name': partner}], + 'language': language, + } + + +def hits_all_introductory(count=8): + """Reproduces the reported defect: many intro courses, no higher rung.""" + return [ + hit(f'Org{i}+C{i}', title=f'Introduction to Thing {i}', partner=f'Partner{i % 3}') + for i in range(count) + ] + + +def hits_spanning_levels(): + """A candidate set where relevance puts the intro courses first.""" + return [ + hit('A+1', title='Introduction to Data', partner='Microsoft'), + hit('A+2', title='Data Basics', partner='Microsoft'), + hit('A+3', title='More Data', partner='Microsoft'), + hit('B+1', title='Applied Data', level=LEVEL_INTERMEDIATE, partner='IBM'), + hit('B+2', title='Data Engineering', level=LEVEL_INTERMEDIATE, partner='Delft'), + hit('C+1', title='Data Capstone', level=LEVEL_ADVANCED, partner='IBM'), + ] + + +@ddt.ddt +class TestEligibleCandidates(TestCase): + """ + Tests for ``eligible_candidates``. + """ + + def test_valid_hits_survive(self): + candidates, ineligible = eligible_candidates([hit('IBM+DA0101EN')]) + + self.assertEqual([candidate.key for candidate in candidates], ['IBM+DA0101EN']) + self.assertEqual(ineligible, {}) + + @ddt.data('course-v1:IBM+DA0101EN+1T2024', 'no-plus-sign', '', None) + def test_keys_that_are_not_course_keys_are_rejected(self, bad_key): + candidates, ineligible = eligible_candidates([hit(bad_key)]) + + self.assertEqual(candidates, []) + self.assertEqual(ineligible['invalid_course_key'], 1) + + def test_non_english_courses_are_rejected(self): + """ + 26% of the pinned catalog is taught in another language, and it appears from rank + 6 -- exactly where reaching for level diversity looks. + """ + candidates, ineligible = eligible_candidates([ + hit('A+1'), + hit('B+1', title='Programación en Python', language='Spanish'), + ]) + + self.assertEqual([candidate.key for candidate in candidates], ['A+1']) + self.assertEqual(ineligible['unsupported_language'], 1) + + def test_a_course_with_no_language_is_not_rejected(self): + """Absent metadata is not evidence of a non-English course.""" + candidates, _ = eligible_candidates([hit('A+1', language='')]) + + self.assertEqual(len(candidates), 1) + + def test_repeated_keys_collapse_to_one(self): + candidates, ineligible = eligible_candidates([hit('A+1'), hit('A+1')]) + + self.assertEqual(len(candidates), 1) + self.assertEqual(ineligible['duplicate_key'], 1) + + +@ddt.ddt +class TestAssemblePathway(TestCase): + """ + Tests for ``assemble_pathway``. + """ + + def test_the_quota_is_satisfied_when_the_rungs_are_populated(self): + assembly = assemble_pathway(hits_spanning_levels()) + + self.assertTrue(assembly.is_complete) + self.assertEqual(assembly.realised_level_mix, DEFAULT_LEVEL_QUOTA) + self.assertEqual(assembly.unfilled_rungs, []) + + def test_relevance_order_still_decides_which_course_fills_a_rung(self): + """The quota decides how many, not which -- so the first intro hit is kept.""" + assembly = assemble_pathway(hits_spanning_levels()) + + intro_keys = [c.key for c in assembly.courses if c.level_type == LEVEL_INTRODUCTORY] + self.assertEqual(intro_keys, ['A+1', 'A+2']) + + def test_an_all_introductory_candidate_set_still_returns_five(self): + """ + The measured `data analyst` case. There is no ladder to build, so the pathway + degrades to five intro courses and records which rungs went unfilled -- it must + not return two courses. + """ + assembly = assemble_pathway(hits_all_introductory()) + + self.assertTrue(assembly.is_complete) + self.assertEqual(assembly.realised_level_mix[LEVEL_INTRODUCTORY], PATHWAY_SIZE) + self.assertCountEqual(assembly.unfilled_rungs, [LEVEL_INTERMEDIATE, LEVEL_ADVANCED]) + + def test_no_provider_can_supply_more_than_the_cap(self): + """The measured `biomedical engineer` case: 5 of 5 from a single partner.""" + assembly = assemble_pathway([ + hit(f'Solo+{i}', title=f'Course {i}', partner='OnlyPartner') for i in range(8) + ] + [ + hit('Other+1', partner='Second'), hit('Other+2', partner='Third'), + hit('Other+3', partner='Fourth'), + ]) + + counts = {} + for course in assembly.courses: + counts[course.partner] = counts.get(course.partner, 0) + 1 + self.assertLessEqual(max(counts.values()), MAX_PER_PARTNER) + self.assertGreaterEqual(len(counts), 3) + + def test_the_scarcest_rung_claims_provider_capacity_first(self): + """ + The measured `Data Analyst` case: 17 candidates spanning all three rungs still + assembled to 5/0/0, because the Introductory picks used up one provider's entire + allowance and every Intermediate candidate belonged to that provider. + """ + assembly = assemble_pathway([ + # Plentiful introductory rung, all from one provider plus filler. + hit('A+1', title='Intro 1', partner='IBM'), + hit('A+2', title='Intro 2', partner='IBM'), + hit('A+3', title='Intro 3', partner='P2'), + hit('A+4', title='Intro 4', partner='P3'), + hit('A+5', title='Intro 5', partner='P4'), + # Scarce higher rungs, only available from IBM. + hit('B+1', title='Applied', level=LEVEL_INTERMEDIATE, partner='IBM'), + hit('C+1', title='Capstone', level=LEVEL_ADVANCED, partner='IBM'), + ]) + + mix = assembly.realised_level_mix + self.assertTrue(assembly.is_complete) + self.assertEqual(mix[LEVEL_ADVANCED], 1) + self.assertEqual(mix[LEVEL_INTERMEDIATE], 1) + self.assertLessEqual(max( + sum(1 for c in assembly.courses if c.partner == p) + for p in {c.partner for c in assembly.courses} + ), MAX_PER_PARTNER) + + def test_relevance_order_is_still_respected_inside_a_rung(self): + """Scarcest-first reorders the rungs, never the candidates within one.""" + assembly = assemble_pathway(hits_spanning_levels()) + + intro = [c.key for c in assembly.courses if c.level_type == LEVEL_INTRODUCTORY] + self.assertEqual(intro, ['A+1', 'A+2']) + + def test_courses_are_ordered_easiest_first(self): + assembly = assemble_pathway(hits_spanning_levels()) + + levels = [course.level_type for course in assembly.courses] + self.assertEqual(levels, [ + LEVEL_INTRODUCTORY, LEVEL_INTRODUCTORY, + LEVEL_INTERMEDIATE, LEVEL_INTERMEDIATE, + LEVEL_ADVANCED, + ]) + + def test_title_cues_break_ties_within_a_rung(self): + """ + `level_type` is 19-36% unreliable, so a title that clearly reads as introductory + sorts ahead of one that reads as advanced at the same tagged level. + """ + assembly = assemble_pathway([ + hit('A+1', title='Advanced Widgets', partner='P1'), + hit('A+2', title='Introduction to Widgets', partner='P2'), + hit('B+1', title='Applied Widgets', level=LEVEL_INTERMEDIATE, partner='P3'), + hit('B+2', title='Widget Systems', level=LEVEL_INTERMEDIATE, partner='P4'), + hit('C+1', title='Widget Capstone', level=LEVEL_ADVANCED, partner='P5'), + ]) + + self.assertEqual(assembly.courses[0].key, 'A+2') + + def test_too_few_eligible_candidates_yields_an_incomplete_assembly(self): + """Never pad. The caller turns this into an explicit no-pathway.""" + assembly = assemble_pathway([hit('A+1'), hit('B+1')]) + + self.assertFalse(assembly.is_complete) + self.assertEqual(len(assembly.courses), 2) + + def test_ineligible_reasons_survive_onto_the_assembly(self): + """ + "Retrieval found little" and "retrieval found plenty, all unusable" need different + fixes, so the assembly has to be able to tell them apart. + """ + assembly = assemble_pathway([ + hit('A+1'), + hit('course-v1:B+2+1T2024'), + hit('C+3', language='Spanish'), + ]) + + self.assertEqual(assembly.ineligible['invalid_course_key'], 1) + self.assertEqual(assembly.ineligible['unsupported_language'], 1) + + def test_unattributed_courses_are_not_counted_against_one_provider(self): + assembly = assemble_pathway([ + hit(f'A+{i}', title=f'Course {i}', partner='') for i in range(6) + ]) + + self.assertTrue(assembly.is_complete) + + def test_an_empty_candidate_set_is_not_an_error(self): + assembly = assemble_pathway([]) + + self.assertFalse(assembly.is_complete) + self.assertEqual(assembly.courses, []) + + +class TestValidatePathway(TestCase): + """ + Tests for the Tier 1 correctness gates. + """ + + def valid_courses(self): + return [ + Candidate(key='A+1', title='Intro', level_type=LEVEL_INTRODUCTORY, partner='P1', language='English'), + Candidate(key='A+2', title='Intro 2', level_type=LEVEL_INTRODUCTORY, partner='P2', language='English'), + Candidate(key='B+1', title='Applied', level_type=LEVEL_INTERMEDIATE, partner='P3', language='English'), + Candidate(key='B+2', title='More', level_type=LEVEL_INTERMEDIATE, partner='P4', language='English'), + Candidate(key='C+1', title='Capstone', level_type=LEVEL_ADVANCED, partner='P5', language='English'), + ] + + def test_a_well_formed_pathway_has_no_violations(self): + self.assertEqual(validate_pathway(self.valid_courses()), []) + + def test_a_short_pathway_is_a_violation(self): + violations = validate_pathway(self.valid_courses()[:4]) + + self.assertIn('exactly 5 courses, got 4', violations[0]) + + def test_a_run_key_is_a_violation(self): + courses = self.valid_courses() + courses[0] = Candidate(key='course-v1:A+1+1T2024', language='English') + + violations = validate_pathway(courses) + + self.assertTrue(any('not a valid catalog course key' in v for v in violations)) + + def test_a_repeated_key_is_a_violation(self): + courses = self.valid_courses() + courses[1] = courses[0] + + violations = validate_pathway(courses) + + self.assertTrue(any('appears more than once' in v for v in violations)) + + def test_a_non_english_course_is_a_violation(self): + courses = self.valid_courses() + courses[0] = Candidate(key='A+1', partner='P1', language='Spanish') + + violations = validate_pathway(courses) + + self.assertTrue(any("taught in 'Spanish'" in v for v in violations)) + + def test_provider_concentration_is_a_violation(self): + courses = [ + Candidate(key=f'A+{i}', partner='OnlyPartner', language='English') + for i in range(PATHWAY_SIZE) + ] + + violations = validate_pathway(courses) + + self.assertTrue(any('exceeds the cap' in v for v in violations)) + + def test_catalog_membership_is_checked_when_the_key_set_is_supplied(self): + courses = self.valid_courses() + + violations = validate_pathway(courses, customer_catalog_keys={'A+1', 'A+2', 'B+1', 'B+2'}) + + self.assertEqual( + violations, ["'C+1' is not in the pinned customer catalog"], + ) + + def test_catalog_membership_is_skipped_rather_than_assumed_when_unknown(self): + """ + Proving membership needs a browse-scoped key (Open Decision 6). Until then the + gate must not quietly pass. + """ + self.assertEqual(validate_pathway(self.valid_courses(), customer_catalog_keys=None), []) + + def test_an_assembled_pathway_passes_its_own_gates(self): + """The two halves of the module must agree; a caught regression here is real.""" + assembly = assemble_pathway(hits_spanning_levels()) + + self.assertEqual(validate_pathway(assembly.courses), []) diff --git a/enterprise_access/apps/pathways/tests/test_pathway_workflow.py b/enterprise_access/apps/pathways/tests/test_pathway_workflow.py new file mode 100644 index 00000000..00ec2ffa --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_pathway_workflow.py @@ -0,0 +1,607 @@ +""" +Tests for the pathway assembly steps and the workflow that composes them. + +The end-to-end tests here exercise the whole five-step chain against patched externals, +which is the only place the conditional-skip behaviour and the accumulated-output +plumbing are checked together. +""" +from unittest import mock +from uuid import uuid4 + +from django.test import TestCase +from edx_toggles.toggles.testutils import override_waffle_switch + +from enterprise_access.apps.pathways.models import ( + AssemblePathwayInput, + AssemblePathwayOutput, + AssemblePathwayStep, + AssemblePathwayStepException, + CourseCandidate, + EnrichRationaleInput, + EnrichRationaleStep, + EnrichRationaleStepException, + PathwayAssemblyWorkflow, + PathwayCourse, + RerankCandidatesInput, + RerankCandidatesOutput, + RerankCandidatesStep, + RetrieveCandidatesInput, + RetrieveCandidatesOutput, + RetrieveCandidatesStep, + RetrieveCandidatesStepException, + TranslateToCatalogOutput +) +from enterprise_access.apps.prompts.api import PromptError +from enterprise_access.apps.prompts.api_client import XpertAPIError +from enterprise_access.toggles import LEARNER_PATHWAYS_DISABLE_CANDIDATE_RERANK + +PATCH_RETRIEVE = 'enterprise_access.apps.pathways.course_retrieval.retrieve_candidate_courses' +PATCH_RERANK = 'enterprise_access.apps.pathways.reranking.rerank_candidates' +PATCH_SNAPSHOT = 'enterprise_access.apps.pathways.catalog_translation.snapshot_catalog_facets' +PATCH_ENRICH = 'enterprise_access.apps.pathways.models.pathways_api.enrich_rationales' + +CUSTOMER_UUID = '417306cb-b24a-4d06-b83c-fb2a61d7fb96' + + +class Accumulator: + """Stands in for the workflow's dynamically-built accumulated-output object.""" + + def __init__(self, **outputs): + for key, value in outputs.items(): + setattr(self, key, value) + + +def course_hit(key, *, title=None, level='Introductory', partner='edX', language='English'): + return { + 'key': key, + 'title': title or f'Course {key}', + 'short_description': 'short', + 'full_description': 'long', + 'level_type': level, + 'partners': [{'name': partner}], + 'language': language, + } + + +def spanning_hits(): + """Six candidates that can fill a 2/2/1 quota across four providers.""" + return [ + course_hit('A+1', partner='P1'), + course_hit('A+2', partner='P1'), + course_hit('A+3', partner='P2'), + course_hit('B+1', level='Intermediate', partner='P3'), + course_hit('B+2', level='Intermediate', partner='P4'), + course_hit('C+1', level='Advanced', partner='P4'), + ] + + +def retrieval_result(hits, **overrides): + return { + 'query': 'Welder Welding', + 'hit_count': len(hits), + 'courses': hits, + 'strict_filters_applied': ['Welding'], + 'strict_hit_count': len(hits), + 'strict_rungs_spanned': len({h['level_type'] for h in hits}), + 'broadened': False, + 'zero_hits': not hits, + **overrides, + } + + +def translation_output(): + return TranslateToCatalogOutput(strict=[], boost=[], unresolved=[], resolution_rate=1.0) + + +class TestCourseCandidate(TestCase): + """ + Tests for ``CourseCandidate``. + """ + + def test_a_hit_round_trips_through_the_assembly_shape(self): + candidate = CourseCandidate.from_hit(course_hit('A+1', partner='edX')) + + assembly_hit = candidate.to_assembly_hit() + + self.assertEqual(assembly_hit['key'], 'A+1') + self.assertEqual(assembly_hit['partners'], [{'name': 'edX'}]) + self.assertEqual(assembly_hit['language'], 'English') + + def test_an_unattributed_course_yields_no_partner_entry(self): + candidate = CourseCandidate.from_hit({'key': 'A+1', 'partners': []}) + + self.assertEqual(candidate.to_assembly_hit()['partners'], []) + + def test_descriptions_are_truncated_before_persistence(self): + """A trace holding five full marketing descriptions is a blob, not a trace.""" + candidate = CourseCandidate.from_hit( + {'key': 'A+1', 'full_description': 'x' * 10000, 'short_description': 'y' * 10000}, + ) + + self.assertLess(len(candidate.full_description), 10000) + self.assertLess(len(candidate.short_description), 10000) + + +class TestRetrieveCandidatesStep(TestCase): + """ + Tests for ``RetrieveCandidatesStep``. + """ + + def _step(self, **kwargs): + kwargs.setdefault('career_name', 'Welder') + return RetrieveCandidatesStep.objects.create( + workflow_record_uuid=uuid4(), + input_data=RetrieveCandidatesInput(**kwargs).to_dict(), + ) + + @mock.patch(PATCH_RETRIEVE) + def test_candidates_are_persisted_with_the_query_and_hit_count(self, mock_retrieve): + mock_retrieve.return_value = retrieval_result([course_hit('A+1')]) + + output = self._step().execute(accumulated_output=Accumulator( + translate_to_catalog_output=translation_output(), + )) + + self.assertEqual([c.key for c in output.courses], ['A+1']) + self.assertEqual(output.query, 'Welder Welding') + self.assertEqual(output.hit_count, 1) + self.assertFalse(output.zero_hits) + + @mock.patch(PATCH_RETRIEVE) + def test_the_customer_scope_is_passed_through(self, mock_retrieve): + mock_retrieve.return_value = retrieval_result([]) + + self._step(customer_uuid=CUSTOMER_UUID).execute(accumulated_output=Accumulator( + translate_to_catalog_output=translation_output(), + )) + + self.assertEqual(mock_retrieve.call_args.kwargs['customer_uuid'], CUSTOMER_UUID) + + @mock.patch(PATCH_RETRIEVE) + def test_zero_hits_is_recorded_rather_than_raised(self, mock_retrieve): + """ + Replaces the plan's "scope-only fallback is recorded" scenario, which measured a + ladder step this design removed. + """ + mock_retrieve.return_value = retrieval_result([], broadened=True) + + output = self._step().execute(accumulated_output=Accumulator( + translate_to_catalog_output=translation_output(), + )) + + self.assertTrue(output.zero_hits) + self.assertTrue(output.broadened) + self.assertEqual(output.courses, []) + + def test_a_missing_translation_fails_the_step_explicitly(self): + step = self._step() + + with self.assertRaises(RetrieveCandidatesStepException) as ctx: + step.execute(accumulated_output=Accumulator()) + + self.assertIn('catalog translation', str(ctx.exception)) + + +class TestRerankCandidatesStep(TestCase): + """ + Tests for ``RerankCandidatesStep``, including its skip conditions. + """ + + def _step(self, **kwargs): + kwargs.setdefault('career_name', 'Welder') + return RerankCandidatesStep.objects.create( + workflow_record_uuid=uuid4(), + input_data=RerankCandidatesInput(**kwargs).to_dict(), + ) + + def test_it_is_skipped_when_disabled(self): + """A disabled re-rank is the baseline arm of the A/B, not a broken run.""" + workflow = mock.Mock(input_data={RerankCandidatesInput.KEY: {'enabled': False}}) + accumulated = Accumulator(retrieve_candidates_output=RetrieveCandidatesOutput( + courses=[CourseCandidate(key='A+1')], + )) + + self.assertFalse(RerankCandidatesStep.should_execute(accumulated, workflow)) + + def test_it_is_skipped_when_there_are_no_candidates(self): + workflow = mock.Mock(input_data={RerankCandidatesInput.KEY: {'enabled': True}}) + accumulated = Accumulator(retrieve_candidates_output=RetrieveCandidatesOutput(courses=[])) + + self.assertFalse(RerankCandidatesStep.should_execute(accumulated, workflow)) + + def test_it_runs_when_enabled_with_candidates(self): + workflow = mock.Mock(input_data={RerankCandidatesInput.KEY: {'enabled': True}}) + accumulated = Accumulator(retrieve_candidates_output=RetrieveCandidatesOutput( + courses=[CourseCandidate(key='A+1')], + )) + + self.assertTrue(RerankCandidatesStep.should_execute(accumulated, workflow)) + + @override_waffle_switch(LEARNER_PATHWAYS_DISABLE_CANDIDATE_RERANK, True) + def test_the_admin_kill_switch_stops_it_even_when_the_caller_asked_for_it(self): + """ + The administrator switch overrides the workflow's own ``enabled`` input. + + That ordering is the point of the switch: the harness and any other offline + caller supply their own input, so a toggle that only narrowed the request path + would not actually stop paid model calls. + """ + workflow = mock.Mock(input_data={RerankCandidatesInput.KEY: {'enabled': True}}) + accumulated = Accumulator(retrieve_candidates_output=RetrieveCandidatesOutput( + courses=[CourseCandidate(key='A+1')], + )) + + self.assertFalse(RerankCandidatesStep.should_execute(accumulated, workflow)) + + @mock.patch(PATCH_RERANK) + def test_the_model_trace_is_persisted_on_the_output(self, mock_rerank): + mock_rerank.return_value = { + 'ordered_keys': ['B+2', 'A+1'], + 'rationales': {'B+2': 'closest fit'}, + 'fabricated_keys': ['Nope+1'], + 'prompt_revision': '7', + 'trace': {'backend': 'xpert', 'model': 'candidate_rerank', + 'input_tokens': 100, 'output_tokens': 20, 'elapsed_ms': 350}, + } + + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=RetrieveCandidatesOutput( + courses=[CourseCandidate(key='A+1'), CourseCandidate(key='B+2')], + ), + )) + + self.assertEqual(output.ordered_keys, ['B+2', 'A+1']) + self.assertEqual(output.fabricated_keys, ['Nope+1']) + self.assertEqual(output.backend, 'xpert') + self.assertEqual(output.prompt_revision, '7') + self.assertEqual(output.elapsed_ms, 350) + self.assertTrue(output.executed) + + +class TestAssemblePathwayStep(TestCase): + """ + Tests for ``AssemblePathwayStep``. + """ + + def _step(self): + return AssemblePathwayStep.objects.create( + workflow_record_uuid=uuid4(), + input_data=AssemblePathwayInput().to_dict(), + ) + + def _candidates(self, hits=None): + return RetrieveCandidatesOutput( + courses=[CourseCandidate.from_hit(hit) for hit in (hits or spanning_hits())], + ) + + def test_five_courses_are_selected_and_pass_the_tier_one_gates(self): + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates(), + )) + + self.assertTrue(output.complete) + self.assertEqual(len(output.courses), 5) + self.assertEqual(output.violations, []) + self.assertEqual(output.level_mix, {'Introductory': 2, 'Intermediate': 2, 'Advanced': 1}) + + def test_the_rerank_order_is_applied_when_it_ran(self): + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates(), + rerank_candidates_output=RerankCandidatesOutput( + ordered_keys=['A+3', 'A+1', 'A+2', 'B+1', 'B+2', 'C+1'], + rationales={'A+3': 'best starting point'}, + executed=True, + ), + )) + + intro = [c for c in output.courses if c.level_type == 'Introductory'] + self.assertEqual(intro[0].key, 'A+3') + self.assertEqual(intro[0].rationale, 'best starting point') + + def test_unranked_candidates_are_kept_behind_the_ranked_ones(self): + """ + The model may return fewer keys than it was given; dropping the remainder would + shrink the window assembly needs to span the rungs. + """ + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates(), + rerank_candidates_output=RerankCandidatesOutput( + ordered_keys=['C+1'], executed=True, + ), + )) + + self.assertTrue(output.complete) + self.assertEqual(len(output.courses), 5) + + def test_a_skipped_rerank_still_yields_a_pathway(self): + """Chunk 9a's assembly is sufficient on its own -- that is the baseline arm.""" + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates(), + )) + + self.assertTrue(output.complete) + self.assertTrue(all(course.rationale == '' for course in output.courses)) + + def test_too_few_candidates_yields_an_incomplete_pathway_not_a_padded_one(self): + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates([course_hit('A+1')]), + )) + + self.assertFalse(output.complete) + self.assertEqual(len(output.courses), 1) + + def test_ineligible_candidates_are_reported(self): + output = self._step().execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates( + spanning_hits() + [course_hit('D+1', language='Spanish')], + ), + )) + + self.assertEqual(output.ineligible.get('unsupported_language'), 1) + + def test_a_missing_candidate_set_fails_the_step_explicitly(self): + step = self._step() + + with self.assertRaises(AssemblePathwayStepException): + step.execute(accumulated_output=Accumulator()) + + def test_output_round_trips_through_the_database(self): + step = self._step() + + step.execute(accumulated_output=Accumulator( + retrieve_candidates_output=self._candidates(), + )) + step.refresh_from_db() + + self.assertEqual(len(step.output_object.courses), 5) + + +class TestEnrichRationaleStep(TestCase): + """ + Tests for ``EnrichRationaleStep``. + """ + + def _step(self, **kwargs): + kwargs.setdefault('selected_career', 'Welder') + return EnrichRationaleStep.objects.create( + workflow_record_uuid=uuid4(), + input_data=EnrichRationaleInput(**kwargs).to_dict(), + ) + + def _assembled(self, complete=True, keys=('A+1', 'B+1')): + return AssemblePathwayOutput( + courses=[PathwayCourse(key=key, title=key) for key in keys], + complete=complete, + ) + + def test_it_is_skipped_when_there_is_no_pathway_to_explain(self): + workflow = mock.Mock(input_data={EnrichRationaleInput.KEY: {'enabled': True}}) + accumulated = Accumulator(assemble_pathway_output=self._assembled(complete=False)) + + self.assertFalse(EnrichRationaleStep.should_execute(accumulated, workflow)) + + def test_it_is_skipped_when_disabled(self): + workflow = mock.Mock(input_data={EnrichRationaleInput.KEY: {'enabled': False}}) + accumulated = Accumulator(assemble_pathway_output=self._assembled()) + + self.assertFalse(EnrichRationaleStep.should_execute(accumulated, workflow)) + + def test_it_runs_when_a_complete_pathway_exists(self): + workflow = mock.Mock(input_data={EnrichRationaleInput.KEY: {'enabled': True}}) + accumulated = Accumulator(assemble_pathway_output=self._assembled()) + + self.assertTrue(EnrichRationaleStep.should_execute(accumulated, workflow)) + + @mock.patch(PATCH_ENRICH) + def test_only_the_delivered_courses_are_sent_for_explanation(self, mock_enrich): + """ + Not the candidate twenty. Four fifths of the explanation work would be paid for + and thrown away. + """ + mock_enrich.return_value = {'reasons': {}, 'prompt_revision': '3'} + + self._step().execute(accumulated_output=Accumulator( + assemble_pathway_output=self._assembled(keys=('A+1', 'B+1')), + )) + + self.assertEqual(mock_enrich.call_args.kwargs['course_keys'], ['A+1', 'B+1']) + + @mock.patch(PATCH_ENRICH) + def test_reasons_and_the_prompt_revision_are_persisted(self, mock_enrich): + mock_enrich.return_value = {'reasons': {'A+1': 'because'}, 'prompt_revision': '9'} + + step = self._step() + step.execute(accumulated_output=Accumulator( + assemble_pathway_output=self._assembled(), + )) + step.refresh_from_db() + + self.assertEqual(step.output_object.reasons, {'A+1': 'because'}) + self.assertEqual(step.output_object.prompt_revision, '9') + self.assertTrue(step.output_object.executed) + + @mock.patch(PATCH_ENRICH) + def test_a_prompt_failure_is_recorded_rather_than_raised(self, mock_enrich): + """ + A pathway with no rationales is still a pathway. Losing the explanations is a much + smaller loss than losing the recommendation. + """ + mock_enrich.side_effect = PromptError('no prompt configured') + + output = self._step().execute(accumulated_output=Accumulator( + assemble_pathway_output=self._assembled(), + )) + + self.assertIn('PromptError', output.error) + self.assertEqual(output.reasons, {}) + self.assertTrue(output.executed) + + @mock.patch(PATCH_ENRICH) + def test_an_xpert_failure_is_also_recorded_rather_than_raised(self, mock_enrich): + mock_enrich.side_effect = XpertAPIError('upstream down') + + output = self._step().execute(accumulated_output=Accumulator( + assemble_pathway_output=self._assembled(), + )) + + self.assertIn('XpertAPIError', output.error) + + def test_a_missing_assembly_fails_the_step_explicitly(self): + step = self._step() + + with self.assertRaises(EnrichRationaleStepException): + step.execute(accumulated_output=Accumulator()) + + +class TestPathwayAssemblyWorkflow(TestCase): + """ + End-to-end tests for the five-step workflow. + """ + + def _workflow(self, **kwargs): + kwargs.setdefault('career_name', 'Welder') + kwargs.setdefault('career_skills', ['Welding']) + return PathwayAssemblyWorkflow.objects.create( + input_data=PathwayAssemblyWorkflow.generate_input_dict(**kwargs), + ) + + def _patches(self, hits=None, rerank=None, enrich=None): + """Context managers patching the three external-touching functions.""" + snapshot = mock.patch(PATCH_SNAPSHOT, return_value={ + 'skill_names': ['Welding'], 'skills.name': [], 'subjects': [], 'truncated': [], + }) + retrieve = mock.patch(PATCH_RETRIEVE, return_value=retrieval_result( + hits if hits is not None else spanning_hits(), + )) + rerank_patch = mock.patch(PATCH_RERANK, return_value=rerank or { + 'ordered_keys': [], 'rationales': {}, 'fabricated_keys': [], + 'prompt_revision': '', 'trace': {}, + }) + enrich_patch = mock.patch(PATCH_ENRICH, return_value=enrich or { + 'reasons': {}, 'prompt_revision': '', + }) + return snapshot, retrieve, rerank_patch, enrich_patch + + def test_a_pathway_is_produced_end_to_end(self): + snapshot, retrieve, rerank, enrich = self._patches() + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich: + workflow.execute() + + pathway = workflow.pathway() + self.assertIsNotNone(pathway) + self.assertEqual(len(pathway['courses']), 5) + self.assertEqual(pathway['violations'], []) + + def test_every_step_is_inspectable_afterwards(self): + """Scenario: The composition is inspectable afterwards.""" + snapshot, retrieve, rerank, enrich = self._patches() + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich: + workflow.execute() + + for step_class in PathwayAssemblyWorkflow.steps: + record = step_class.objects.filter(workflow_record_uuid=workflow.uuid).first() + self.assertIsNotNone(record, f'{step_class.__name__} left no record') + self.assertIsNotNone(record.input_data) + + def test_a_disabled_rerank_skips_the_model_call_but_still_delivers(self): + snapshot, retrieve, rerank, enrich = self._patches() + workflow = self._workflow(rerank_enabled=False) + + with snapshot, retrieve, enrich, rerank as mock_rerank: + workflow.execute() + + mock_rerank.assert_not_called() + self.assertEqual(len(workflow.pathway()['courses']), 5) + + def test_a_career_with_no_courses_yields_no_pathway_rather_than_failing(self): + snapshot, retrieve, rerank, enrich = self._patches(hits=[]) + workflow = self._workflow(career_name='Underwater Basket Weaver') + + with snapshot, retrieve, enrich, rerank as mock_rerank: + workflow.execute() + + # Nothing to re-rank, so the model call is skipped rather than paid for. + mock_rerank.assert_not_called() + self.assertIsNone(workflow.pathway()) + + def test_a_skipped_step_serializes_as_null(self): + """ + Guards the cattrs defect that made ``Optional`` necessary on the generated IO + classes: a skipped step's output must round-trip as null, not crash. + """ + snapshot, retrieve, rerank, enrich = self._patches(hits=[]) + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich: + workflow.execute() + + workflow.refresh_from_db() + self.assertIsNone(workflow.output_data.get(RerankCandidatesOutput.KEY)) + + def test_rationales_from_enrichment_reach_the_delivered_pathway(self): + """ + Merged in ``pathway()`` rather than by the assembly step, which runs earlier and + must not depend on a later step. + """ + snapshot, retrieve, rerank, enrich = self._patches( + enrich={'reasons': {'A+1': 'a solid starting point'}, 'prompt_revision': '4'}, + ) + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich: + workflow.execute() + + rationales = {c['key']: c['rationale'] for c in workflow.pathway()['courses']} + self.assertEqual(rationales.get('A+1'), 'a solid starting point') + + def test_a_course_without_a_rationale_still_ships(self): + snapshot, retrieve, rerank, enrich = self._patches( + enrich={'reasons': {'A+1': 'only this one'}, 'prompt_revision': ''}, + ) + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich: + workflow.execute() + + courses = workflow.pathway()['courses'] + self.assertEqual(len(courses), 5) + self.assertEqual(len([c for c in courses if not c['rationale']]), 4) + + def test_enrichment_is_skipped_when_there_is_no_pathway(self): + """No pathway means nothing to explain, so the paid call is not made.""" + snapshot, retrieve, rerank, enrich = self._patches(hits=[]) + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich as mock_enrich: + workflow.execute() + + mock_enrich.assert_not_called() + + def test_a_disabled_enrichment_still_delivers_a_pathway(self): + snapshot, retrieve, rerank, enrich = self._patches() + workflow = self._workflow(enrich_enabled=False) + + with snapshot, retrieve, rerank, enrich as mock_enrich: + workflow.execute() + + mock_enrich.assert_not_called() + self.assertEqual(len(workflow.pathway()['courses']), 5) + + def test_the_rerank_order_reaches_assembly(self): + snapshot, retrieve, rerank, enrich = self._patches(rerank={ + 'ordered_keys': ['C+1', 'B+1', 'A+3', 'A+1', 'A+2', 'B+2'], + 'rationales': {'A+3': 'start here'}, + 'fabricated_keys': [], 'prompt_revision': '3', + 'trace': {'backend': 'xpert', 'model': 'candidate_rerank', 'elapsed_ms': 1}, + }) + workflow = self._workflow() + + with snapshot, retrieve, rerank, enrich: + workflow.execute() + + rationales = {c['key']: c['rationale'] for c in workflow.pathway()['courses']} + self.assertEqual(rationales.get('A+3'), 'start here') diff --git a/enterprise_access/apps/pathways/tests/test_prompts.py b/enterprise_access/apps/pathways/tests/test_prompts.py new file mode 100644 index 00000000..45719802 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_prompts.py @@ -0,0 +1,221 @@ +""" +Tests for the pathway prompt defaults and the row the seed migration creates. + +The contract tests here are the ones that were missing: they assert that what the parsers +accept and what the prompts *ask for* are the same thing, without a live model call. Every +other test in this app mocks the model response, which means those tests encode an +assumption about its shape -- and an assumption that is wrong fails identically in all of +them. +""" +import json + +from django.test import TestCase + +from enterprise_access.apps.pathways.prompts import CANDIDATE_RERANK_OUTPUT_SCHEMA, CANDIDATE_RERANK_SYSTEM_PROMPT +from enterprise_access.apps.pathways.reranking import FALLBACK_SYSTEM_PROMPT, parse_rerank_response +from enterprise_access.apps.prompts.api import build_system_prompt, compose_system_prompt +from enterprise_access.apps.prompts.models import PromptType, XpertLearnerPathwaysSystemPrompt + + +class TestCandidateRerankPromptDefaults(TestCase): + """ + Tests for the module-level prompt default. + """ + + def test_the_direct_backends_use_the_canonical_text(self): + """ + One definition, so a change to the wording reaches the claude and openai paths + automatically. The Xpert path reads its database row instead, which is the + deliberate asymmetry. + """ + self.assertTrue(FALLBACK_SYSTEM_PROMPT.startswith(CANDIDATE_RERANK_SYSTEM_PROMPT.strip())) + + def test_the_direct_backends_are_sent_the_output_schema(self): + """ + Regression test for a live defect: the fallback used to be the prompt text alone. + The text asks for JSON but never names ``ordered_keys`` -- that field name only + exists in the schema -- so gpt-4o returned a valid ranking under field names of + its own choosing and the parser discarded all of it. + + The Xpert path never had the bug, because ``build_system_prompt`` appends the + schema from the database row. The direct backends have no row, so they must + append it from the constant. + """ + self.assertIn('EXPECTED OUTPUT SCHEMA', FALLBACK_SYSTEM_PROMPT) + self.assertIn('ordered_keys', FALLBACK_SYSTEM_PROMPT) + self.assertIn('rationales', FALLBACK_SYSTEM_PROMPT) + + def test_both_prompt_paths_compose_identically(self): + """ + The direct backends and the Xpert row must produce the same string from the same + text and schema, or a prompt validated on one backend is not the prompt the other + sends. + """ + composed = compose_system_prompt( + CANDIDATE_RERANK_SYSTEM_PROMPT, CANDIDATE_RERANK_OUTPUT_SCHEMA, + ) + + self.assertEqual(FALLBACK_SYSTEM_PROMPT, composed) + + def test_the_prompt_tells_the_model_what_not_to_optimise_for(self): + """ + Assembly guarantees level spread, provider spread and de-duplication + deterministically. A prompt that also asked for them would create disagreements + somebody then has to adjudicate. + """ + text = CANDIDATE_RERANK_SYSTEM_PROMPT.lower() + self.assertIn('topical relevance only', text) + for excluded in ('difficulty', 'provider', 'similar ground', 'how many courses'): + self.assertIn(excluded, text) + + def test_the_prompt_forbids_inventing_keys(self): + """The platform has a known key-invention defect; the prompt names it explicitly.""" + text = CANDIDATE_RERANK_SYSTEM_PROMPT.lower() + self.assertIn('never invent', text) + self.assertIn('only keys that appear in the input', text) + + def test_the_prompt_forbids_outcome_claims_in_rationales(self): + """A rationale a learner reads must not promise employability or salary.""" + self.assertIn('no claims about outcomes', CANDIDATE_RERANK_SYSTEM_PROMPT.lower()) + + def test_the_prompt_says_json_which_openai_json_mode_requires(self): + """ + ``OpenAIBackend`` sends ``response_format={'type': 'json_object'}``, and OpenAI + rejects that unless the word appears in the prompt. Easy to break by rewording. + """ + self.assertIn('json', CANDIDATE_RERANK_SYSTEM_PROMPT.lower()) + + def test_the_output_schema_is_a_json_object_the_model_can_be_given(self): + """ + ``build_system_prompt`` appends it as formatted JSON, so it has to be a dict and + has to serialize. + """ + self.assertIsInstance(CANDIDATE_RERANK_OUTPUT_SCHEMA, dict) + self.assertEqual( + json.loads(json.dumps(CANDIDATE_RERANK_OUTPUT_SCHEMA)), + CANDIDATE_RERANK_OUTPUT_SCHEMA, + ) + + +class TestCandidateRerankSchemaMatchesTheParser(TestCase): + """ + Contract tests: a response conforming to the advertised schema must parse. + + This is the check that catches prompt-and-parser drift offline. If someone edits the + schema to advertise a different shape, these fail rather than the pipeline silently + dropping every ordering at runtime. + """ + + def test_the_schema_advertises_exactly_the_fields_the_parser_reads(self): + properties = set(CANDIDATE_RERANK_OUTPUT_SCHEMA['properties']) + + self.assertEqual(properties, {'ordered_keys', 'rationales'}) + self.assertEqual(CANDIDATE_RERANK_OUTPUT_SCHEMA['required'], ['ordered_keys']) + + def test_a_schema_conforming_response_parses_fully(self): + allowed = ['A+1', 'B+2', 'C+3'] + response = { + 'ordered_keys': ['B+2', 'A+1', 'C+3'], + 'rationales': { + 'B+2': 'Teaches the query skills this role uses daily.', + 'A+1': 'A grounding in the tools the role expects.', + 'C+3': 'Covers water treatment rather than the data work this role involves.', + }, + } + + result = parse_rerank_response(response, allowed) + + self.assertEqual(result['ordered_keys'], ['B+2', 'A+1', 'C+3']) + self.assertEqual(len(result['rationales']), 3) + self.assertEqual(result['fabricated_keys'], []) + + def test_rationales_are_optional_per_the_schema(self): + """``required`` lists only ordered_keys, so a response without them must parse.""" + result = parse_rerank_response({'ordered_keys': ['A+1']}, ['A+1']) + + self.assertEqual(result['ordered_keys'], ['A+1']) + self.assertEqual(result['rationales'], {}) + + def test_the_prompt_asks_for_every_key_which_the_parser_preserves_in_order(self): + """ + The prompt says rank every key rather than dropping the poor fits, because + assembly fills each rung from the front of the window -- so last place is how the + model says "poor fit" and dropping loses that signal. + """ + self.assertIn('Rank EVERY key', CANDIDATE_RERANK_SYSTEM_PROMPT) + + result = parse_rerank_response( + {'ordered_keys': ['C+3', 'B+2', 'A+1']}, ['A+1', 'B+2', 'C+3'], + ) + + self.assertEqual(result['ordered_keys'], ['C+3', 'B+2', 'A+1']) + + +class TestSeededCandidateRerankPrompt(TestCase): + """ + Tests for the row created by ``prompts/migrations/0003_seed_candidate_rerank_prompt``. + + Django runs migrations to build the test database, so the row exists here exactly as + it will in a freshly set-up environment. + """ + + def test_the_row_exists_after_migration(self): + """ + Without it, re-ranking degrades to retrieval order and logs a warning -- a silent + no-op that reads as "the model does not help". + """ + self.assertTrue( + XpertLearnerPathwaysSystemPrompt.objects.filter( + prompt_type=PromptType.CANDIDATE_RERANK, + ).exists() + ) + + def test_the_seeded_row_is_resolvable_as_the_current_prompt(self): + prompt = XpertLearnerPathwaysSystemPrompt.get_current( + prompt_type=PromptType.CANDIDATE_RERANK, + ) + + self.assertIsNotNone(prompt) + self.assertTrue(prompt.system_prompt.strip()) + + def test_the_seeded_row_carries_the_output_schema(self): + prompt = XpertLearnerPathwaysSystemPrompt.get_current( + prompt_type=PromptType.CANDIDATE_RERANK, + ) + + self.assertIsInstance(prompt.output_schema, dict) + self.assertEqual(set(prompt.output_schema['properties']), {'ordered_keys', 'rationales'}) + + def test_the_seeded_row_builds_a_system_prompt_with_its_schema_appended(self): + """End to end through the real prompt-assembly path the Xpert backend uses.""" + prompt = XpertLearnerPathwaysSystemPrompt.get_current( + prompt_type=PromptType.CANDIDATE_RERANK, + ) + + built = build_system_prompt(prompt) + + self.assertIn('topical relevance', built.lower()) + self.assertIn('EXPECTED OUTPUT SCHEMA', built) + self.assertIn('ordered_keys', built) + + def test_the_seeded_row_passes_the_models_own_validation(self): + """ + The migration writes through the historical model, which skips ``full_clean()``. + Saving it through the real model proves the seeded content is still valid. + """ + prompt = XpertLearnerPathwaysSystemPrompt.get_current( + prompt_type=PromptType.CANDIDATE_RERANK, + ) + + prompt.save() # full_clean() runs here; a ValidationError would fail this test + + def test_the_row_notes_that_it_is_editable(self): + """ + The prompts app exists so wording can change without a deploy. Someone finding a + migration-seeded row should be told they may edit it. + """ + prompt = XpertLearnerPathwaysSystemPrompt.get_current( + prompt_type=PromptType.CANDIDATE_RERANK, + ) + + self.assertIn('Edit freely', prompt.notes) diff --git a/enterprise_access/apps/pathways/tests/test_reranking.py b/enterprise_access/apps/pathways/tests/test_reranking.py new file mode 100644 index 00000000..70367c16 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_reranking.py @@ -0,0 +1,241 @@ +""" +Tests for model-backed candidate re-ranking. + +The validation tests carry most of the weight. A re-ranker's output is untrusted text +that names content keys, and the platform has a known key-invention defect — so "the +model returned a key we never gave it" has to be a counted metric, and a bad response has +to cost the ordering rather than the pathway. +""" +from unittest import mock + +import ddt +from django.test import TestCase + +from enterprise_access.apps.pathways.model_backends import ( + ModelBackendConfigurationError, + ModelBackendRequestError, + ModelResponse +) +from enterprise_access.apps.pathways.reranking import ( + DESCRIPTION_CHARS_FOR_MODEL, + build_user_content, + parse_rerank_response, + rerank_candidates +) + +CANDIDATES = [ + {'key': 'A+1', 'title': 'Intro to Welding', 'short_description': 'Basics.'}, + {'key': 'B+2', 'title': 'Applied Welding', 'short_description': 'More.'}, + {'key': 'C+3', 'title': 'Water Treatment', 'short_description': 'Unrelated.'}, +] + + +class FakeBackend: + """A backend returning a scripted response or raising a scripted error.""" + + name = 'fake' + + def __init__(self, content='{}', error=None, metadata=None): + self.content = content + self.error = error + self.metadata = metadata or {} + self.calls = [] + + def complete(self, **kwargs): + """Stand in for ``ModelBackend.complete``.""" + self.calls.append(kwargs) + if self.error: + raise self.error + return ModelResponse( + content=self.content, backend=self.name, model='fake-1', + input_tokens=10, output_tokens=5, elapsed_ms=12, metadata=self.metadata, + ) + + +class TestBuildUserContent(TestCase): + """ + Tests for ``build_user_content``. + """ + + def test_the_career_and_candidate_keys_are_included(self): + content = build_user_content(career_name='Welder', candidates=CANDIDATES) + + self.assertIn('Welder', content) + for candidate in CANDIDATES: + self.assertIn(candidate['key'], content) + + def test_level_and_partner_are_withheld_from_the_model(self): + """ + Those are what assembly uses. Offering them invites the model to optimise for + constraints it is not being asked about, which assembly then has to undo. + """ + content = build_user_content( + career_name='Welder', + candidates=[{**CANDIDATES[0], 'level_type': 'Introductory', 'partner': 'edX'}], + ) + + self.assertNotIn('Introductory', content) + self.assertNotIn('edX', content) + + def test_descriptions_are_truncated(self): + content = build_user_content( + career_name='Welder', + candidates=[{'key': 'A+1', 'title': 't', 'short_description': 'x' * 5000}], + ) + + self.assertNotIn('x' * (DESCRIPTION_CHARS_FOR_MODEL + 1), content) + + def test_the_full_description_is_used_when_the_short_one_is_missing(self): + content = build_user_content( + career_name='Welder', + candidates=[{'key': 'A+1', 'title': 't', 'full_description': 'the long one'}], + ) + + self.assertIn('the long one', content) + + +@ddt.ddt +class TestParseRerankResponse(TestCase): + """ + Tests for ``parse_rerank_response``. + """ + + allowed = ['A+1', 'B+2', 'C+3'] + + def test_a_valid_ordering_is_accepted(self): + result = parse_rerank_response( + {'ordered_keys': ['B+2', 'A+1'], 'rationales': {'B+2': 'because'}}, self.allowed, + ) + + self.assertEqual(result['ordered_keys'], ['B+2', 'A+1']) + self.assertEqual(result['rationales'], {'B+2': 'because'}) + self.assertEqual(result['fabricated_keys'], []) + + def test_fabricated_keys_are_dropped_and_counted(self): + """Scenario: Fabricated keys are rejected.""" + result = parse_rerank_response( + {'ordered_keys': ['A+1', 'Invented+999']}, self.allowed, + ) + + self.assertEqual(result['ordered_keys'], ['A+1']) + self.assertEqual(result['fabricated_keys'], ['Invented+999']) + + def test_repeated_keys_collapse_without_counting_as_fabrication(self): + result = parse_rerank_response({'ordered_keys': ['A+1', 'A+1']}, self.allowed) + + self.assertEqual(result['ordered_keys'], ['A+1']) + self.assertEqual(result['fabricated_keys'], []) + + def test_rationales_for_unknown_keys_are_discarded(self): + result = parse_rerank_response( + {'ordered_keys': ['A+1'], 'rationales': {'Invented+999': 'nope'}}, self.allowed, + ) + + self.assertEqual(result['rationales'], {}) + + def test_non_string_rationale_values_are_discarded(self): + result = parse_rerank_response( + {'ordered_keys': ['A+1'], 'rationales': {'A+1': {'nested': 1}}}, self.allowed, + ) + + self.assertEqual(result['rationales'], {}) + + @ddt.data( + None, [], 'a string', 42, + {'no_ordered_keys': 1}, + {'ordered_keys': 'not a list'}, + ) + def test_a_malformed_payload_yields_an_empty_ordering_rather_than_raising(self, payload): + """A bad response should cost the ordering, not the pathway.""" + result = parse_rerank_response(payload, self.allowed) + + self.assertEqual(result['ordered_keys'], []) + self.assertEqual(result['fabricated_keys'], []) + + def test_non_string_and_empty_keys_are_skipped(self): + result = parse_rerank_response( + {'ordered_keys': ['A+1', '', None, 7, 'B+2']}, self.allowed, + ) + + self.assertEqual(result['ordered_keys'], ['A+1', 'B+2']) + + +class TestRerankCandidates(TestCase): + """ + Tests for ``rerank_candidates``. + """ + + def test_an_ordering_and_a_trace_are_returned(self): + backend = FakeBackend(content='{"ordered_keys": ["B+2", "A+1"]}') + + result = rerank_candidates( + career_name='Welder', candidates=CANDIDATES, trace_id='t', backend=backend, + ) + + self.assertEqual(result['ordered_keys'], ['B+2', 'A+1']) + self.assertEqual(result['trace']['backend'], 'fake') + self.assertEqual(result['trace']['input_tokens'], 10) + self.assertEqual(result['trace']['elapsed_ms'], 12) + + def test_the_prompt_revision_is_stamped(self): + """Scenario: The prompt revision is stamped.""" + backend = FakeBackend(content='{"ordered_keys": []}', metadata={'prompt_revision': '42'}) + + result = rerank_candidates( + career_name='Welder', candidates=CANDIDATES, trace_id='t', backend=backend, + ) + + self.assertEqual(result['prompt_revision'], '42') + + def test_the_trace_id_reaches_the_backend(self): + backend = FakeBackend(content='{"ordered_keys": []}') + + rerank_candidates( + career_name='Welder', candidates=CANDIDATES, trace_id='trace-9', backend=backend, + ) + + self.assertEqual(backend.calls[0]['trace_id'], 'trace-9') + + def test_a_backend_failure_degrades_to_no_ordering(self): + backend = FakeBackend(error=ModelBackendRequestError('boom')) + + result = rerank_candidates( + career_name='Welder', candidates=CANDIDATES, trace_id='t', backend=backend, + ) + + self.assertEqual(result['ordered_keys'], []) + self.assertEqual(result['trace'], {}) + + def test_a_configuration_failure_also_degrades_rather_than_raising(self): + """ + A misconfigured backend must not take the pathway down: Chunk 9a produces a valid + pathway from the unordered set. + """ + backend = FakeBackend(error=ModelBackendConfigurationError('no key')) + + result = rerank_candidates( + career_name='Welder', candidates=CANDIDATES, trace_id='t', backend=backend, + ) + + self.assertEqual(result['ordered_keys'], []) + + def test_a_non_json_response_degrades_but_keeps_the_cost_trace(self): + """The call was still paid for, so its cost has to stay visible.""" + backend = FakeBackend(content='I cannot do that') + + result = rerank_candidates( + career_name='Welder', candidates=CANDIDATES, trace_id='t', backend=backend, + ) + + self.assertEqual(result['ordered_keys'], []) + self.assertEqual(result['trace']['input_tokens'], 10) + + @mock.patch('enterprise_access.apps.pathways.reranking.get_model_backend') + def test_the_configured_backend_is_used_when_none_is_injected(self, mock_get_backend): + mock_get_backend.return_value = FakeBackend(content='{"ordered_keys": []}') + + rerank_candidates(career_name='Welder', candidates=CANDIDATES, trace_id='t') + + self.assertEqual( + mock_get_backend.call_args.kwargs['prompt_type'], 'candidate_rerank', + ) diff --git a/enterprise_access/apps/pathways/tests/test_skill_vocabulary.py b/enterprise_access/apps/pathways/tests/test_skill_vocabulary.py new file mode 100644 index 00000000..7003d145 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_skill_vocabulary.py @@ -0,0 +1,282 @@ +""" +Tests for skill-name resolution against the catalog facet vocabulary. +""" +import ddt +from django.test import TestCase + +from enterprise_access.apps.pathways.skill_vocabulary import ( + MatchType, + VocabularyIndex, + normalize_term, + resolve_skill_terms +) + +# Verbatim `skill_names` facet values observed in the production catalog index on +# 2026-09-09. Using real values matters: the whole defect is that the vocabulary is not +# shaped the way a developer would guess. +REAL_VOCABULARY = { + 'skill_names': [ + 'Python (Programming Language)', + 'SQL (Programming Language)', + 'Java (Programming Language)', + 'JavaScript (Programming Language)', + 'Microsoft Excel', + 'Excel Macros', + 'Excel Formulas', + 'Tableau (Business Intelligence Software)', + 'Microsoft Azure', + 'Azure Machine Learning', + 'Docker (Software)', + 'Docker Container', + 'Pandas (Python Package)', + 'NumPy (Python Package)', + 'Data Analysis', + 'Machine Learning', + 'Project Management', + 'Nursing', + 'Welding', + 'Power BI', + 'JSON', + 'Kubernetes', + ], + 'skills.name': [ + 'Communication', + 'Data Analysis', + ], +} + + +@ddt.ddt +class TestVocabularyIndex(TestCase): + """ + Tests for ``VocabularyIndex``. + """ + + def setUp(self): + super().setUp() + self.index = VocabularyIndex(REAL_VOCABULARY) + + def test_index_dedupes_across_facet_fields_preferring_skill_names(self): + """``skill_names`` wins a collision, matching the MFE's precedence.""" + match = self.index.resolve('Data Analysis') + + self.assertEqual(match.catalog_field, 'skill_names') + # 'Data Analysis' appears in both fields but is indexed once. + self.assertEqual( + len(self.index), + len(set(v.casefold() for v in + REAL_VOCABULARY['skill_names'] + REAL_VOCABULARY['skills.name'])), + ) + + # -- the failures this module exists to fix --------------------------------------- + + @ddt.data( + ('Python', 'Python (Programming Language)'), + ('SQL', 'SQL (Programming Language)'), + ('Java', 'Java (Programming Language)'), + ('Tableau', 'Tableau (Business Intelligence Software)'), + ('Docker', 'Docker (Software)'), + ) + @ddt.unpack + def test_short_names_resolve_to_their_qualified_form(self, term, expected): + """These all returned zero hits under exact-match grounding.""" + match = self.index.resolve(term) + + self.assertEqual(match.catalog_value, expected) + self.assertEqual(match.match_type, MatchType.QUALIFIED) + self.assertTrue(match.is_high_confidence) + + def test_java_does_not_resolve_to_javascript(self): + """ + The dangerous near-miss: ``Java (Programming Language)`` and ``JavaScript + (Programming Language)`` are both present and, in the live index, have almost + identical counts. Whole-word matching is what separates them. + """ + self.assertEqual( + self.index.resolve('Java').catalog_value, + 'Java (Programming Language)', + ) + self.assertEqual( + self.index.resolve('JavaScript').catalog_value, + 'JavaScript (Programming Language)', + ) + + def test_python_does_not_resolve_to_a_python_package(self): + """``Pandas (Python Package)`` contains 'Python' but is a different concept.""" + match = self.index.resolve('Python') + + self.assertEqual(match.catalog_value, 'Python (Programming Language)') + + @ddt.data( + ('Excel', 'Microsoft Excel'), + ('Azure', 'Microsoft Azure'), + ) + @ddt.unpack + def test_vendor_prefixed_names_resolve_by_containment(self, term, expected): + """ + ``Excel`` is a whole-word *suffix* of ``Microsoft Excel``, not a prefix, so it + needs the containment rule. Shortest containing value wins, which is what picks + ``Microsoft Azure`` over ``Azure Machine Learning``. + """ + match = self.index.resolve(term) + + self.assertEqual(match.catalog_value, expected) + self.assertEqual(match.match_type, MatchType.CONTAINED) + # Containment is the loosest rule, so it must not be trusted as a hard filter. + self.assertFalse(match.is_high_confidence) + + # -- exact matches ---------------------------------------------------------------- + + @ddt.data('Data Analysis', 'Machine Learning', 'Project Management', 'Nursing', + 'Welding', 'Power BI', 'Kubernetes') + def test_already_canonical_names_match_exactly(self, term): + """Most of the vocabulary needs no resolution; those must not be perturbed.""" + match = self.index.resolve(term) + + self.assertEqual(match.catalog_value, term) + self.assertEqual(match.match_type, MatchType.EXACT) + + @ddt.data('python (programming language)', 'PYTHON (PROGRAMMING LANGUAGE)', + ' Python (Programming Language) ') + def test_exact_match_is_case_and_whitespace_insensitive(self, term): + match = self.index.resolve(term) + + self.assertEqual(match.catalog_value, 'Python (Programming Language)') + self.assertEqual(match.match_type, MatchType.EXACT) + + # -- guardrails ------------------------------------------------------------------- + + @ddt.data('JS', 'ML', 'AI', 'R') + def test_very_short_terms_are_not_expanded_by_containment(self, term): + """ + ``JS`` would otherwise match ``JSON``. An over-eager expansion is worse than a + dropped filter: it silently searches for the wrong thing. + """ + self.assertIsNone(self.index.resolve(term)) + + @ddt.data('Underwater Basket Weaving', 'Quantum Blockchain Synergy') + def test_absent_terms_resolve_to_nothing(self, term): + self.assertIsNone(self.index.resolve(term)) + + @ddt.data('', ' ', None) + def test_empty_terms_resolve_to_nothing(self, term): + self.assertIsNone(self.index.resolve(term)) + + def test_empty_vocabulary_resolves_nothing(self): + empty = VocabularyIndex({}) + + self.assertEqual(len(empty), 0) + self.assertIsNone(empty.resolve('Python')) + + def test_vocabulary_with_falsy_values_is_tolerated(self): + index = VocabularyIndex({'skill_names': ['Python (Programming Language)', '', None]}) + + self.assertEqual(len(index), 1) + + +@ddt.ddt +class TestResolveSkillTerms(TestCase): + """ + Tests for ``resolve_skill_terms``. + """ + + def test_resolution_reports_matches_and_what_it_dropped(self): + result = resolve_skill_terms( + ['Python', 'SQL', 'Underwater Basket Weaving'], + REAL_VOCABULARY, + ) + + self.assertEqual( + [match.catalog_value for match in result.matches], + ['Python (Programming Language)', 'SQL (Programming Language)'], + ) + # Dropped terms are part of the result, not a log line -- invisibility of dropped + # terms is the defect being fixed. + self.assertEqual(result.unresolved, ['Underwater Basket Weaving']) + self.assertAlmostEqual(result.resolution_rate, 2 / 3) + + def test_high_confidence_matches_exclude_containment(self): + result = resolve_skill_terms(['Python', 'Excel'], REAL_VOCABULARY) + + self.assertEqual(len(result.matches), 2) + self.assertEqual( + [match.catalog_value for match in result.high_confidence_matches], + ['Python (Programming Language)'], + ) + + def test_duplicate_terms_are_resolved_once(self): + result = resolve_skill_terms(['Python', 'python', 'PYTHON '], REAL_VOCABULARY) + + self.assertEqual(len(result.matches), 1) + + def test_two_terms_resolving_to_one_value_yield_one_match(self): + """A repeated facet filter narrows nothing and only costs query length.""" + result = resolve_skill_terms( + ['Python', 'Python (Programming Language)'], + REAL_VOCABULARY, + ) + + self.assertEqual(len(result.matches), 1) + + @ddt.data(None, [], ['', ' ']) + def test_no_usable_input_produces_an_empty_result(self, terms): + result = resolve_skill_terms(terms, REAL_VOCABULARY) + + self.assertEqual(result.matches, []) + self.assertEqual(result.unresolved, []) + self.assertIsNone(result.resolution_rate) + + def test_everything_unresolved_is_reported_as_a_zero_rate(self): + """ + A career whose whole skill set is absent must be distinguishable from one that + was never resolved -- rate 0.0, not None. + """ + result = resolve_skill_terms(['Nonsense Skill', 'Another Nonsense'], REAL_VOCABULARY) + + self.assertEqual(result.matches, []) + self.assertEqual(result.resolution_rate, 0.0) + + def test_result_serializes_for_a_step_record(self): + result = resolve_skill_terms(['Python', 'Excel', 'Nonsense'], REAL_VOCABULARY) + + payload = result.to_dict() + + self.assertEqual(payload['unresolved'], ['Nonsense']) + self.assertEqual(payload['matches'][0], { + 'term': 'Python', + 'catalog_value': 'Python (Programming Language)', + 'catalog_field': 'skill_names', + 'match_type': 'qualified', + }) + self.assertAlmostEqual(payload['resolution_rate'], 2 / 3) + + def test_a_real_career_skill_set_resolves(self): + """ + The skills carried by ``IBM+DA0101EN`` in the live index -- an end-to-end shape + check on realistic input rather than hand-picked terms. + """ + career_skills = [ + 'Machine Learning', 'Data Analysis', 'Basic Math', 'SciPy', + 'Data Visualization', 'Scikit-Learn (Python Package)', + 'NumPy (Python Package)', 'Pandas (Python Package)', + ] + + result = resolve_skill_terms(career_skills, REAL_VOCABULARY) + + self.assertIn('Machine Learning', [m.catalog_value for m in result.matches]) + self.assertIn('NumPy (Python Package)', [m.catalog_value for m in result.matches]) + # Skills genuinely absent from this vocabulary are reported, not silently dropped. + self.assertIn('SciPy', result.unresolved) + + +class TestNormalizeTerm(TestCase): + """ + Tests for ``normalize_term``. + """ + + def test_casefolds_and_collapses_whitespace(self): + self.assertEqual(normalize_term(' Python (Programming Language) '), + 'python (programming language)') + + def test_handles_none(self): + self.assertEqual(normalize_term(None), '') diff --git a/enterprise_access/apps/pathways/tests/test_translation_steps.py b/enterprise_access/apps/pathways/tests/test_translation_steps.py new file mode 100644 index 00000000..622abb72 --- /dev/null +++ b/enterprise_access/apps/pathways/tests/test_translation_steps.py @@ -0,0 +1,226 @@ +""" +Tests for the catalog-translation workflow steps. +""" +from unittest import mock +from uuid import uuid4 + +from django.test import TestCase + +from enterprise_access.apps.pathways.models import ( + SnapshotCatalogFacetsInput, + SnapshotCatalogFacetsOutput, + SnapshotCatalogFacetsStep, + TranslateToCatalogInput, + TranslateToCatalogStep, + TranslateToCatalogStepException +) + +PATCH_SNAPSHOT = 'enterprise_access.apps.pathways.catalog_translation.snapshot_catalog_facets' +PATCH_REFINE = 'enterprise_access.apps.pathways.catalog_translation.refine_unmatched_skills' + +SNAPSHOT_VALUES = [ + 'Python (Programming Language)', + 'Microsoft Excel', + 'Data Analysis', +] + + +def make_snapshot_output(skill_names=None, truncated=None): + return SnapshotCatalogFacetsOutput( + skill_names=skill_names if skill_names is not None else list(SNAPSHOT_VALUES), + subjects=['Computer Science'], + truncated=truncated or [], + ) + + +class Accumulator: + """Stands in for the workflow's dynamically-built accumulated-output object.""" + + def __init__(self, **outputs): + for key, value in outputs.items(): + setattr(self, key, value) + + +class TestSnapshotCatalogFacetsStep(TestCase): + """ + Tests for ``SnapshotCatalogFacetsStep``. + """ + + def _step(self, allow_unscoped=False): + return SnapshotCatalogFacetsStep.objects.create( + workflow_record_uuid=uuid4(), + input_data=SnapshotCatalogFacetsInput(allow_unscoped=allow_unscoped).to_dict(), + ) + + @mock.patch(PATCH_SNAPSHOT) + def test_both_skill_facets_are_merged_into_one_vocabulary(self, mock_snapshot): + """ + The resolver only needs to know which values exist, and ``skill_names`` already + wins a collision, so the two facets collapse to one list. + """ + mock_snapshot.return_value = { + 'skill_names': ['Python (Programming Language)', 'Data Analysis'], + 'skills.name': ['Data Analysis', 'Communication'], + 'subjects': ['Computer Science'], + 'truncated': [], + } + + output = self._step().execute() + + self.assertEqual( + output.skill_names, + ['Python (Programming Language)', 'Data Analysis', 'Communication'], + ) + self.assertEqual(output.subjects, ['Computer Science']) + + @mock.patch(PATCH_SNAPSHOT) + def test_truncation_is_persisted(self, mock_snapshot): + """ + Whether the snapshot was complete changes how an unresolved term should be read, + so it has to survive on the record. + """ + mock_snapshot.return_value = { + 'skill_names': SNAPSHOT_VALUES, 'skills.name': [], + 'subjects': [], 'truncated': ['skill_names'], + } + + step = self._step() + step.execute() + step.refresh_from_db() + + self.assertEqual(step.output_object.truncated, ['skill_names']) + + @mock.patch(PATCH_SNAPSHOT) + def test_allow_unscoped_is_passed_through(self, mock_snapshot): + mock_snapshot.return_value = {'skill_names': [], 'skills.name': [], + 'subjects': [], 'truncated': []} + + self._step(allow_unscoped=True).execute() + + self.assertTrue(mock_snapshot.call_args.kwargs['allow_unscoped']) + + @mock.patch(PATCH_SNAPSHOT) + def test_output_round_trips_through_the_facet_snapshot_shape(self, mock_snapshot): + mock_snapshot.return_value = {'skill_names': SNAPSHOT_VALUES, 'skills.name': [], + 'subjects': [], 'truncated': []} + + output = self._step().execute() + + self.assertEqual(output.as_facet_snapshot()['skill_names'], SNAPSHOT_VALUES) + + +class TestTranslateToCatalogStep(TestCase): + """ + Tests for ``TranslateToCatalogStep``. + """ + + def _step(self, **input_kwargs): + return TranslateToCatalogStep.objects.create( + workflow_record_uuid=uuid4(), + input_data=TranslateToCatalogInput(**input_kwargs).to_dict(), + ) + + def test_skills_resolve_to_catalog_values(self): + step = self._step(career_skills=['Python'], skills_required=['Data Analysis']) + + output = step.execute(accumulated_output=Accumulator( + snapshot_catalog_facets_output=make_snapshot_output(), + )) + + resolved = {entry.catalog_value for entry in output.strict} + self.assertIn('Python (Programming Language)', resolved) + self.assertIn('Data Analysis', resolved) + self.assertEqual(output.unresolved, []) + self.assertFalse(output.refined) + + def test_refinement_is_skipped_when_everything_resolves(self): + """Scenario: Refinement is skipped when unnecessary.""" + step = self._step(career_skills=['Python']) + + with mock.patch(PATCH_REFINE) as mock_refine: + output = step.execute(accumulated_output=Accumulator( + snapshot_catalog_facets_output=make_snapshot_output(), + )) + + mock_refine.assert_not_called() + self.assertFalse(output.refined) + + def test_refinement_runs_only_when_terms_remain_unresolved(self): + step = self._step(career_skills=['Python', 'Welding']) + + with mock.patch(PATCH_REFINE) as mock_refine: + mock_refine.return_value = { + 'recovered': [{'term': 'Welding', 'catalog_value': 'Welding', + 'catalog_field': 'skill_names', 'match_type': 'exact'}], + 'unresolved': [], + 'errors': [], + } + output = step.execute(accumulated_output=Accumulator( + snapshot_catalog_facets_output=make_snapshot_output(), + )) + + mock_refine.assert_called_once() + self.assertEqual(mock_refine.call_args.kwargs['unresolved'], ['Welding']) + self.assertIn('Welding', {entry.catalog_value for entry in output.strict}) + # Recorded so the harness can count how often the capped snapshot was insufficient. + self.assertTrue(output.refined) + + def test_unresolved_terms_survive_refinement_and_are_reported(self): + step = self._step(career_skills=['Underwater Basket Weaving']) + + with mock.patch(PATCH_REFINE) as mock_refine: + mock_refine.return_value = { + 'recovered': [], 'unresolved': ['Underwater Basket Weaving'], 'errors': [], + } + output = step.execute(accumulated_output=Accumulator( + snapshot_catalog_facets_output=make_snapshot_output(), + )) + + self.assertEqual(output.unresolved, ['Underwater Basket Weaving']) + self.assertEqual(output.resolution_rate, 0.0) + + def test_duplicate_terms_across_sources_are_resolved_once(self): + step = self._step( + career_skills=['Python'], + skills_required=['Python'], + skills_preferred=['python'], + ) + + output = step.execute(accumulated_output=Accumulator( + snapshot_catalog_facets_output=make_snapshot_output(), + )) + + self.assertEqual(len(output.strict), 1) + + def test_missing_snapshot_fails_the_step_explicitly(self): + """ + A missing snapshot must be a named failure, not an empty translation that reads + as "this career has no catalog coverage". + """ + step = self._step(career_skills=['Python']) + + with self.assertRaises(TranslateToCatalogStepException) as ctx: + step.execute(accumulated_output=Accumulator()) + + self.assertIn('facet snapshot', str(ctx.exception)) + step.refresh_from_db() + self.assertIsNotNone(step.failed_at) + self.assertIsNotNone(step.exception_message) + + def test_output_persists_and_round_trips(self): + step = self._step(career_skills=['Python', 'Excel']) + + step.execute(accumulated_output=Accumulator( + snapshot_catalog_facets_output=make_snapshot_output(), + )) + step.refresh_from_db() + + reloaded = step.output_object + self.assertEqual( + [entry.catalog_value for entry in reloaded.strict], + ['Python (Programming Language)'], + ) + self.assertEqual( + [entry.catalog_value for entry in reloaded.boost], ['Microsoft Excel'], + ) + self.assertEqual(reloaded.resolution_rate, 1.0) diff --git a/enterprise_access/apps/prompts/api.py b/enterprise_access/apps/prompts/api.py index 7f332d80..16fcda1f 100644 --- a/enterprise_access/apps/prompts/api.py +++ b/enterprise_access/apps/prompts/api.py @@ -53,24 +53,40 @@ def get_current_prompt( return prompt -def build_system_prompt(prompt: BaseSystemPrompt) -> str: +def compose_system_prompt(system_prompt: str, output_schema: Any = None) -> str: """ - Build the complete system prompt sent to Xpert. + Compose prompt text and an output schema into the string a model receives. - The configured prompt text is stripped of surrounding whitespace. - A non-empty output schema is appended as formatted JSON. + Split out from ``build_system_prompt`` so that callers holding a prompt *model* and + callers holding plain constants compose them the same way. The direct model backends + (claude, openai) are the second kind: they take a caller-supplied system prompt and + have no database row to read a schema from. + + That distinction is load-bearing rather than cosmetic. A prompt sent without its + schema never names the fields the response must use, so the model picks its own and + the parser finds nothing it recognises -- a silent degradation, because an unusable + re-rank response falls back to retrieval order by design. """ - system_prompt = prompt.system_prompt.strip() - output_schema = prompt.output_schema + composed = system_prompt.strip() if output_schema: - system_prompt += _SCHEMA_SEPARATOR + json.dumps( + composed += _SCHEMA_SEPARATOR + json.dumps( output_schema, indent=2, sort_keys=True, ) - return system_prompt + return composed + + +def build_system_prompt(prompt: BaseSystemPrompt) -> str: + """ + Build the complete system prompt sent to Xpert. + + The configured prompt text is stripped of surrounding whitespace. + A non-empty output schema is appended as formatted JSON. + """ + return compose_system_prompt(prompt.system_prompt, prompt.output_schema) def build_messages(validated_data: ValidatedData) -> list[XpertMessage]: diff --git a/enterprise_access/apps/prompts/migrations/0002_alter_historicalxpertlearnerpathwayssystemprompt_prompt_type_and_more.py b/enterprise_access/apps/prompts/migrations/0002_alter_historicalxpertlearnerpathwayssystemprompt_prompt_type_and_more.py new file mode 100644 index 00000000..bf1d2c9a --- /dev/null +++ b/enterprise_access/apps/prompts/migrations/0002_alter_historicalxpertlearnerpathwayssystemprompt_prompt_type_and_more.py @@ -0,0 +1,23 @@ +# Generated by Django 5.2.17 on 2026-09-10 01:17 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('prompts', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='historicalxpertlearnerpathwayssystemprompt', + name='prompt_type', + field=models.CharField(choices=[('learner_intent', 'Learner Intent'), ('recommendations_feedback', 'Recommendations Feedback'), ('candidate_rerank', 'Candidate Re-rank')], max_length=64), + ), + migrations.AlterField( + model_name='xpertlearnerpathwayssystemprompt', + name='prompt_type', + field=models.CharField(choices=[('learner_intent', 'Learner Intent'), ('recommendations_feedback', 'Recommendations Feedback'), ('candidate_rerank', 'Candidate Re-rank')], max_length=64), + ), + ] diff --git a/enterprise_access/apps/prompts/migrations/0003_seed_candidate_rerank_prompt.py b/enterprise_access/apps/prompts/migrations/0003_seed_candidate_rerank_prompt.py new file mode 100644 index 00000000..ad04c288 --- /dev/null +++ b/enterprise_access/apps/prompts/migrations/0003_seed_candidate_rerank_prompt.py @@ -0,0 +1,141 @@ +""" +Seed the ``candidate_rerank`` prompt row so a fresh environment has a working pipeline. + +Without a row, ``XpertBackend`` raises ``ModelBackendConfigurationError``, re-ranking +degrades to retrieval order, and the pathway pipeline runs *without* its model step while +still returning a valid pathway. That failure is silent by design -- losing the ordering is +better than losing the recommendation -- which is exactly why it needs seeding rather than +a setup instruction someone can miss. A run in that state reads as "the model does not +help" when the model was never called. + +Three properties this migration deliberately has: + +* **Idempotent.** ``get_or_create`` on ``prompt_type``, so an environment where someone + already authored the row by hand keeps their wording untouched. +* **Never clobbers an admin edit.** Same reason. The prompts app exists so wording can + change without a deploy; a migration that overwrote it would defeat that. +* **Self-contained.** The text is inlined rather than imported from + ``apps/pathways/prompts.py``. Migrations are frozen history and must keep working + against future code, so importing a constant that will legitimately change would make + this migration mean something different later. The live default lives in that module; + this is the snapshot it was seeded from. + +Because the historical model from ``apps.get_model`` carries neither the ``full_clean()`` +override nor django-simple-history's signals, no history row is written here. History for +this prompt therefore begins at the first admin edit, which is the correct reading: nobody +authored this revision, it shipped as a default. +""" +from django.db import migrations + +PROMPT_TYPE = 'candidate_rerank' + +SYSTEM_PROMPT = """\ +You rank candidate courses by how well each one prepares a learner for a named career. + +You will receive a career name and a list of candidate courses, each with a key, a title +and a short description. Return a ranking of those courses by topical relevance to that +career, plus a one-sentence reason for each. + +Judge topical relevance ONLY. Do not consider, and do not try to balance: +- difficulty or course level +- which provider or university offers the course +- whether two courses cover similar ground +- how many courses to recommend + +Those are all decided after you, by code, and optimising for them here makes that harder +rather than easier. + +How to rank: +- Rank EVERY key you were given, exactly once, most relevant first. +- A course that has little or nothing to do with the career goes at the end. Do not drop + it -- its position is how you tell us it is a poor fit. +- Use ONLY keys that appear in the input. Never invent, correct or reformat a key. If you + are unsure about a key, leave it out entirely rather than guessing at it. +- Judge the course, not the title. A description that clearly addresses the career's work + outranks a title that merely shares a word with the career name. +- Where a career name is broad ("Analyst", "Engineer"), prefer courses that teach the + concrete skills that career is practised with over courses that only discuss the field. + +How to write each reason: +- One sentence, under 30 words, addressed to the learner. +- Say how the course connects to that career's actual work. +- No marketing language, no superlatives, and no claims about outcomes, salary or + employability. +- If a course is a poor fit, say so plainly. "Covers water treatment rather than the data + work this role involves" is more useful than a stretch. + +Return JSON only, with no prose before or after it.""" + +OUTPUT_SCHEMA = { + 'type': 'object', + 'required': ['ordered_keys'], + 'additionalProperties': False, + 'properties': { + 'ordered_keys': { + 'type': 'array', + 'description': ( + 'Every candidate key, exactly once, most topically relevant to the career ' + 'first. Keys must be copied verbatim from the input.' + ), + 'items': {'type': 'string'}, + }, + 'rationales': { + 'type': 'object', + 'description': ( + 'One sentence per course key explaining how it connects to the career. ' + 'Keys must appear in ordered_keys.' + ), + 'additionalProperties': {'type': 'string'}, + }, + }, +} + +NOTES = ( + 'Seeded by migration 0003 as a working default. Edit freely -- this app exists so the ' + 'wording can change without a deploy, and every edit is preserved as a history row. ' + 'Note the direct model backends (claude, openai) do not read this row; they use the ' + 'constant in apps/pathways/prompts.py. See ADR 0037.' +) + + +def seed_candidate_rerank_prompt(apps, schema_editor): + """Create the row if it is absent, leaving any existing row untouched.""" + prompt_model = apps.get_model('prompts', 'XpertLearnerPathwaysSystemPrompt') + prompt_model.objects.get_or_create( + prompt_type=PROMPT_TYPE, + defaults={ + 'system_prompt': SYSTEM_PROMPT, + 'output_schema': OUTPUT_SCHEMA, + 'notes': NOTES, + }, + ) + + +def remove_candidate_rerank_prompt(apps, schema_editor): + """ + Remove the seeded row on reverse, but only if it is still the seeded text. + + A row someone has since edited is their content, not this migration's, so reversing + must not delete it. Reversing then leaves the row in place, which is the safe + asymmetry: an extra prompt row is harmless, a deleted one loses work. + """ + prompt_model = apps.get_model('prompts', 'XpertLearnerPathwaysSystemPrompt') + prompt_model.objects.filter( + prompt_type=PROMPT_TYPE, + system_prompt=SYSTEM_PROMPT, + ).delete() + + +class Migration(migrations.Migration): + """Data migration seeding the candidate re-rank prompt.""" + + dependencies = [ + ('prompts', '0002_alter_historicalxpertlearnerpathwayssystemprompt_prompt_type_and_more'), + ] + + operations = [ + migrations.RunPython( + seed_candidate_rerank_prompt, + remove_candidate_rerank_prompt, + ), + ] diff --git a/enterprise_access/apps/prompts/models.py b/enterprise_access/apps/prompts/models.py index 6e047524..4f0ea363 100644 --- a/enterprise_access/apps/prompts/models.py +++ b/enterprise_access/apps/prompts/models.py @@ -18,6 +18,7 @@ class PromptType(models.TextChoices): """Valid ``prompt_type`` values for ``XpertLearnerPathwaysSystemPrompt``.""" LEARNER_INTENT = 'learner_intent', 'Learner Intent' RECOMMENDATIONS_FEEDBACK = 'recommendations_feedback', 'Recommendations Feedback' + CANDIDATE_RERANK = 'candidate_rerank', 'Candidate Re-rank' class BaseSystemPrompt(TimeStampedModel): diff --git a/enterprise_access/apps/prompts/tests/test_admin.py b/enterprise_access/apps/prompts/tests/test_admin.py index d7527f8f..002d1128 100644 --- a/enterprise_access/apps/prompts/tests/test_admin.py +++ b/enterprise_access/apps/prompts/tests/test_admin.py @@ -171,7 +171,14 @@ def test_multiple_prompt_types_allowed(self): self.assertIsNotNone(prompt1.uuid) self.assertIsNotNone(prompt2.uuid) self.assertNotEqual(prompt1.uuid, prompt2.uuid) - self.assertEqual(XpertLearnerPathwaysSystemPrompt.objects.count(), 2) + # Counted per type rather than over the whole table: migration 0003 seeds a + # candidate_rerank row, and this test is about types coexisting, not table size. + self.assertEqual( + XpertLearnerPathwaysSystemPrompt.objects.filter( + prompt_type__in=[PromptType.LEARNER_INTENT, PromptType.RECOMMENDATIONS_FEEDBACK], + ).count(), + 2, + ) def test_form_uses_pretty_json_widget(self): """Test that the form uses PrettyJSONWidget for output_schema field.""" diff --git a/enterprise_access/apps/prompts/tests/test_models.py b/enterprise_access/apps/prompts/tests/test_models.py index f4f2e415..c2c1cb39 100644 --- a/enterprise_access/apps/prompts/tests/test_models.py +++ b/enterprise_access/apps/prompts/tests/test_models.py @@ -108,7 +108,14 @@ def test_unique_constraint_allows_one_row_per_prompt_type(self): XpertLearnerPathwaysSystemPromptFactory(prompt_type=PromptType.LEARNER_INTENT) XpertLearnerPathwaysSystemPromptFactory(prompt_type=PromptType.RECOMMENDATIONS_FEEDBACK) - self.assertEqual(XpertLearnerPathwaysSystemPrompt.objects.count(), 2) + # One row each for the two types created here. Asserted per type rather than as a + # table count, because migration 0003 seeds a candidate_rerank row and the + # constraint under test is per prompt_type. + for prompt_type in (PromptType.LEARNER_INTENT, PromptType.RECOMMENDATIONS_FEEDBACK): + self.assertEqual( + XpertLearnerPathwaysSystemPrompt.objects.filter(prompt_type=prompt_type).count(), + 1, + ) def test_edits_preserve_history_via_simple_history(self): prompt = XpertLearnerPathwaysSystemPromptFactory( diff --git a/enterprise_access/settings/base.py b/enterprise_access/settings/base.py index 07642d69..f7c1f32d 100644 --- a/enterprise_access/settings/base.py +++ b/enterprise_access/settings/base.py @@ -92,6 +92,8 @@ def root(*path_fragments): 'enterprise_access.apps.customer_billing', 'enterprise_access.apps.testimonials', 'enterprise_access.apps.prompts', + 'enterprise_access.apps.pathway_eval', + 'enterprise_access.apps.pathways', ) INSTALLED_APPS += THIRD_PARTY_APPS @@ -186,6 +188,11 @@ def root(*path_fragments): 'ssp_product': '120/hour', 'learner_pathways_learning_intent': '100/hour', 'learner_pathways_recommendation_feedback': '100/hour', + 'learner_pathways_careers': '100/hour', + # Tighter than the others on purpose: a pathway request runs a five-step workflow + # with several Algolia searches and, when the re-rank backend is enabled, a paid + # model call. The cost per request is an order of magnitude above the others here. + 'learner_pathways_pathway': '30/hour', }, } @@ -517,6 +524,48 @@ def root(*path_fragments): 'edx-available-course', ] +# Algolia search settings (apps/api_client/algolia_client.py). +# Search-only. The write-scoped ALGOLIA.API_KEY used by enterprise-catalog for indexing +# must never be configured here. +ALGOLIA_APP_ID = '' +# Plain, search-ACL-only key. Used for the jobs/taxonomy index, which secured keys cannot +# read. NOT for scoped catalog searches -- those use a secured key vended per enterprise +# by enterprise-catalog. +ALGOLIA_SEARCH_API_KEY = '' +ALGOLIA_CATALOG_INDEX_NAME = '' +ALGOLIA_JOBS_INDEX_NAME = '' +# Escape hatch for the offline retrieval diagnostic, which has no request and therefore +# no per-user secured key to vend. Enables searching the catalog index *unscoped* with the +# plain search key. Must stay False anywhere a learner response could be built from it. +ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH = False + +# The server-side learner pathways pipeline (apps/pathways) is gated by a waffle switch, +# `enterprise_access.learner_pathways_server_pipeline`, not by a setting -- so it can be +# turned on and off in Django admin without a deploy. See `enterprise_access/toggles.py` +# for the switch definitions and why a switch rather than a flag. + +# Which model backend the pathway pipeline issues completions through (apps/pathways/ +# model_backends). 'xpert' routes through the stored, admin-editable prompts and is the +# default because it is the path already in production. 'claude' is a direct metered call +# and needs ANTHROPIC_API_KEY plus the `anthropic` package -- selecting it without either +# raises rather than falling back, so a paid backend is never reached by accident. +PATHWAYS_MODEL_BACKEND = 'xpert' + +# Credentials for the two direct metered backends. Both stay empty here and are supplied +# per-environment (devstack local config, or edx-internal for deployed envs) -- never +# committed. Selecting a backend without its key raises rather than falling back, so a +# paid backend is never reached by accident. +ANTHROPIC_API_KEY = '' +PATHWAYS_CLAUDE_MODEL = 'claude-sonnet-5' +OPENAI_API_KEY = '' +PATHWAYS_OPENAI_MODEL = 'gpt-4o' + +# The enterprise customer the evaluation harness scopes to (Open Decision 1). Production +# requests take the customer from the request; this is only for offline runs, which have +# no request and therefore no secured Algolia key. Scoping by filter needs no secured key +# because `enterprise_customer_uuids` is facetable. +PATHWAYS_EVAL_CUSTOMER_UUID = '' + # Braze campaigns for learner credit browse and request(apps.subsidy_request) BRAZE_LEARNER_CREDIT_BNR_APPROVED_NOTIFICATION_CAMPAIGN = '' BRAZE_LEARNER_CREDIT_BNR_REMIND_NOTIFICATION_CAMPAIGN = '' diff --git a/enterprise_access/settings/test.py b/enterprise_access/settings/test.py index 3d7a40df..273eb31a 100644 --- a/enterprise_access/settings/test.py +++ b/enterprise_access/settings/test.py @@ -5,6 +5,7 @@ INSTALLED_APPS += ( 'enterprise_access.apps.workflow.tests', + 'enterprise_access.apps.pathways.tests', ) # IN-MEMORY TEST DATABASE diff --git a/enterprise_access/tests/test_toggles.py b/enterprise_access/tests/test_toggles.py index 1b07e631..a0f6ddc9 100644 --- a/enterprise_access/tests/test_toggles.py +++ b/enterprise_access/tests/test_toggles.py @@ -1,4 +1,9 @@ -import enterprise_access.toggles as toggles +"""Tests for the enterprise-access feature toggles.""" +import pytest +from edx_toggles.toggles import WaffleSwitch +from edx_toggles.toggles.testutils import override_waffle_switch + +from enterprise_access import toggles def test_enable_multi_license_entitlements_bff_enabled(monkeypatch): @@ -17,3 +22,52 @@ def test_enable_multi_license_entitlements_bff_disabled(monkeypatch): lambda: False ) assert toggles.enable_multi_license_entitlements_bff() is False + + +@pytest.mark.parametrize('switch', [ + toggles.LEARNER_PATHWAYS_SERVER_PIPELINE, + toggles.LEARNER_PATHWAYS_DISABLE_CANDIDATE_RERANK, +]) +def test_the_pathway_toggles_are_switches_not_flags(switch): + """ + Both pathway toggles must stay ``WaffleSwitch``. + + This is a product requirement, not a style preference: a switch is a single global + boolean an administrator sets in Django admin, and waffle honours ``?name=1`` + query-string overrides for *flags* only. Swapping either of these for a ``WaffleFlag`` + would make an unreleased, paid pipeline reachable by crafting a URL, and would also + break the offline callers -- the harness and management commands have no request to + evaluate a flag against. + """ + assert isinstance(switch, WaffleSwitch) + + +@pytest.mark.django_db +def test_the_pipeline_is_disabled_by_default(): + """A switch that has never been created reads as off, which is the safe default.""" + assert toggles.learner_pathways_server_pipeline_enabled() is False + + +@pytest.mark.django_db +def test_the_pipeline_can_be_enabled(): + with override_waffle_switch(toggles.LEARNER_PATHWAYS_SERVER_PIPELINE, True): + assert toggles.learner_pathways_server_pipeline_enabled() is True + + +@pytest.mark.django_db +def test_reranking_is_on_by_default_because_its_switch_is_a_kill_switch(): + """ + The re-rank toggle's polarity is inverted on purpose. + + An enable-style switch defaulting to off would mean enabling the pipeline yielded + pathways with no model input at all -- courses in retrieval order, no error, a + well-formed response. That silent degradation has already bitten this pipeline once, + so it must not be reachable by forgetting a second switch. + """ + assert toggles.learner_pathways_candidate_rerank_enabled() is True + + +@pytest.mark.django_db +def test_reranking_stops_when_the_kill_switch_is_on(): + with override_waffle_switch(toggles.LEARNER_PATHWAYS_DISABLE_CANDIDATE_RERANK, True): + assert toggles.learner_pathways_candidate_rerank_enabled() is False diff --git a/enterprise_access/toggles.py b/enterprise_access/toggles.py index 4b055a51..a0e74476 100644 --- a/enterprise_access/toggles.py +++ b/enterprise_access/toggles.py @@ -1,6 +1,6 @@ """Feature toggles for enterprise-access.""" -from edx_toggles.toggles import WaffleFlag +from edx_toggles.toggles import WaffleFlag, WaffleSwitch ENTERPRISE_ACCESS_NAMESPACE = 'enterprise_access' ENTERPRISE_ACCESS_LOG_PREFIX = '[enterprise_access] ' @@ -25,3 +25,76 @@ def enable_multi_license_entitlements_bff(): """Return whether multi-license BFF behavior is enabled.""" return ENABLE_MULTI_LICENSE_ENTITLEMENTS_BFF.is_enabled() + + +# A ``WaffleSwitch`` rather than a ``WaffleFlag``, deliberately. A switch is a single +# global boolean an administrator sets in Django admin; it takes no request, so it cannot +# be turned on per-user, by percentage, or by a ``?flag=1`` query string. Two consequences +# we want: +# +# * Nobody can enable an unreleased, paid pipeline for themselves by crafting a URL. Waffle +# only ever honours query-string overrides for *flags*, and never for switches. +# * It is readable where there is no request at all -- the evaluation harness and the +# management commands -- so one toggle governs the request path and offline runs alike. +# ``WaffleFlag.is_enabled()`` off-request cannot express that. + +# .. toggle_name: enterprise_access.learner_pathways_server_pipeline +# .. toggle_implementation: WaffleSwitch +# .. toggle_default: False +# .. toggle_description: Enables the server-side learner pathways pipeline +# (apps/pathways). When disabled, the careers and pathway endpoints return +# HTTP 404, indistinguishable from endpoints that do not exist. No other code +# path is affected, so rollback is turning this off rather than reverting +# behaviour. Off by default because the pipeline issues paid model calls. +# .. toggle_use_cases: open_edx +# .. toggle_creation_date: 2026-09-10 +LEARNER_PATHWAYS_SERVER_PIPELINE = WaffleSwitch( + f'{ENTERPRISE_ACCESS_NAMESPACE}.learner_pathways_server_pipeline', + __name__, +) + +# A kill switch, and named as one: ``False`` leaves re-ranking **on**. The polarity is +# deliberate and is the opposite of the switch above. +# +# An enable-style switch defaulting to off would mean that turning the pipeline on gives +# you pathways assembled with no model input at all -- courses in retrieval order, no +# error, and a perfectly well-formed five-course response. That is precisely the silent +# degradation this pipeline has already been bitten by once (see ADR 0037 and the +# ``ordered_keys`` defect), and it should not be reachable by forgetting to set a second +# switch. +# +# There is no cost exposure in defaulting it on: with the pipeline switch off, no endpoint +# runs, and the only other caller is the evaluation harness, which bounds its own spend +# with ``max_calls`` and ``--dry-run``. +# +# .. toggle_name: enterprise_access.learner_pathways_disable_candidate_rerank +# .. toggle_implementation: WaffleSwitch +# .. toggle_default: False +# .. toggle_description: Kill switch for the model-backed candidate re-ranking step of +# the learner pathways pipeline. Default False, meaning re-ranking runs. Turn it ON +# to stop the pipeline issuing paid model calls while leaving pathways working: +# re-ranking is the only step that calls an external model, and skipping it is a +# supported degradation rather than a failure, because deterministic assembly still +# produces a valid five-course pathway from the unordered candidate set. Use it as a +# cost or latency control, or if a model provider is failing. +# .. toggle_use_cases: open_edx +# .. toggle_creation_date: 2026-09-10 +LEARNER_PATHWAYS_DISABLE_CANDIDATE_RERANK = WaffleSwitch( + f'{ENTERPRISE_ACCESS_NAMESPACE}.learner_pathways_disable_candidate_rerank', + __name__, +) + + +def learner_pathways_server_pipeline_enabled(): + """Return whether the server-side learner pathways pipeline is enabled.""" + return LEARNER_PATHWAYS_SERVER_PIPELINE.is_enabled() + + +def learner_pathways_candidate_rerank_enabled(): + """ + Return whether model-backed candidate re-ranking is enabled. + + Reads the kill switch and inverts it, so callers ask the positive question and only + this module has to know about the polarity. + """ + return not LEARNER_PATHWAYS_DISABLE_CANDIDATE_RERANK.is_enabled() diff --git a/requirements/base.in b/requirements/base.in index e76fdc18..6f02bac9 100644 --- a/requirements/base.in +++ b/requirements/base.in @@ -1,7 +1,12 @@ # Core requirements for using this application -c constraints.txt +algoliasearch # Search-only client for the catalog and jobs indexes analytics-python +anthropic # Optional: the 'claude' pathway model backend. Imported lazily, + # so a deployment using only the 'xpert' backend never loads it. +openai # Optional: the 'openai' pathway model backend. Imported lazily, + # for the same reason. cattrs celery confluent-kafka[avro,schema-registry] @@ -38,6 +43,7 @@ openedx-events pygments pymemcache pytz +pyyaml # Persona fixture parsing (apps/pathway_eval) redis rules stripe diff --git a/requirements/base.txt b/requirements/base.txt index febdfd5b..16e0e070 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -6,6 +6,8 @@ # 2u-enterprise-subsidy-client==2.2.1 # via -r requirements/base.in +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via kombu analytics-python==1.4.post1 @@ -315,6 +317,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # code-annotations # drf-spectacular # drf-yasg @@ -327,6 +330,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # analytics-python # confluent-kafka # edx-drf-extensions diff --git a/requirements/dev.txt b/requirements/dev.txt index 22028b80..c7f92404 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -6,6 +6,8 @@ # 2u-enterprise-subsidy-client==2.2.1 # via -r requirements/validation.txt +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via # -r requirements/validation.txt @@ -616,6 +618,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # -r requirements/validation.txt # code-annotations # drf-spectacular @@ -635,6 +638,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # -r requirements/validation.txt # analytics-python # confluent-kafka diff --git a/requirements/doc.in b/requirements/doc.in index 9c89dfc6..d3d1f56f 100644 --- a/requirements/doc.in +++ b/requirements/doc.in @@ -7,3 +7,4 @@ doc8 # reStructuredText style checker sphinx-book-theme # Common theme for all Open edX projects readme_renderer # Validates README.rst for usage on PyPI Sphinx # Documentation builder +sphinxcontrib-mermaid # Renders the mermaid diagrams in docs/decisions/ diff --git a/requirements/doc.txt b/requirements/doc.txt index c63d0686..d72e2aa4 100644 --- a/requirements/doc.txt +++ b/requirements/doc.txt @@ -10,6 +10,8 @@ accessible-pygments==0.0.5 # via pydata-sphinx-theme alabaster==1.0.0 # via sphinx +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via # -r requirements/test.txt @@ -557,6 +559,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # -r requirements/test.txt # code-annotations # drf-spectacular @@ -573,6 +576,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # -r requirements/test.txt # analytics-python # confluent-kafka diff --git a/requirements/production.txt b/requirements/production.txt index 479e9291..bd775081 100644 --- a/requirements/production.txt +++ b/requirements/production.txt @@ -6,6 +6,8 @@ # 2u-enterprise-subsidy-client==2.2.1 # via -r requirements/base.txt +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via # -r requirements/base.txt @@ -403,6 +405,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # -r requirements/base.txt # -r requirements/production.in # code-annotations @@ -418,6 +421,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # -r requirements/base.txt # analytics-python # confluent-kafka diff --git a/requirements/quality.txt b/requirements/quality.txt index f94cb81c..f13d4ab7 100644 --- a/requirements/quality.txt +++ b/requirements/quality.txt @@ -6,6 +6,8 @@ # 2u-enterprise-subsidy-client==2.2.1 # via -r requirements/test.txt +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via # -r requirements/test.txt @@ -561,6 +563,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # -r requirements/test.txt # code-annotations # drf-spectacular @@ -577,6 +580,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # -r requirements/test.txt # analytics-python # confluent-kafka diff --git a/requirements/test.txt b/requirements/test.txt index f743ff8e..92c63a33 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -6,6 +6,8 @@ # 2u-enterprise-subsidy-client==2.2.1 # via -r requirements/base.txt +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via # -r requirements/base.txt @@ -490,6 +492,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # -r requirements/base.txt # code-annotations # drf-spectacular @@ -504,6 +507,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # -r requirements/base.txt # analytics-python # confluent-kafka diff --git a/requirements/validation.txt b/requirements/validation.txt index 1b209276..c164c568 100644 --- a/requirements/validation.txt +++ b/requirements/validation.txt @@ -8,6 +8,8 @@ # via # -r requirements/quality.txt # -r requirements/test.txt +algoliasearch==3.0.0 + # via -r requirements/base.in amqp==5.3.1 # via # -r requirements/quality.txt @@ -728,6 +730,7 @@ pytz==2026.3.post1 # snowflake-connector-python pyyaml==6.0.3 # via + # -r requirements/base.in # -r requirements/quality.txt # -r requirements/test.txt # code-annotations @@ -750,6 +753,7 @@ referencing==0.37.0 # jsonschema-specifications requests==2.34.2 # via + # algoliasearch # -r requirements/quality.txt # -r requirements/test.txt # analytics-python