Bbeggs/server side learner pathways

docs/architecture-patterns.md

@@ -90,6 +90,30 @@ Uses `edx-rbac` for fine-grained permissions with:
 ### 15. Use ddt to parameterize unit tests
 - Improve test DRYness by using the `ddt` packages `@data` and `@unpack` decorators.
 
+### 16. Evaluation harnesses own no domain logic
+- An eval app (`apps/pathway_eval`) loads fixtures, calls production code, scores results, reports
+- Pipeline logic — retrieval, translation, ranking — stays production code elsewhere; if the
+  harness reimplements any of it, the harness is what gets measured
+- Fixtures validate against the **real** request serializer, not a copy of its rules, so a
+  contract change fails at load time rather than mid-run
+- Ground-truth fixtures declare their own provenance in a machine-readable field
+  (`ground_truth_status: placeholder | expert_authored`). Authoring ground truth outlasts
+  building the harness, so "is this a real expectation?" must be queryable — a comment in a
+  fixture cannot keep a placeholder out of a headline metric
+- Incomplete ground truth **loads** and reports itself as unscoreable. It is neither a
+  validation error (it must be visible and chaseable) nor a scored failure
+- Diagnostics name their epistemic limits in the result. Where a credential can only probe
+  rather than enumerate, the outcome is `NOT_FOUND_IN_INDEX`, not "absent" — an over-claimed
+  negative drives the most expensive decisions
+
+### 17. Search clients bind a credential to an index, not to a service
+- Where two indexes need different credentials, make the wrong pairing unconstructable rather
+  than validated: separate methods per index, and no parameter for the credential that must
+  not be used there
+- Guard against a *misconfigured* credential too, not just a miscalled one
+- A degraded mode (unscoped search) requires both a settings flag and an explicit call-site
+  argument, and is never reached by fallback — missing or expired scoping raises
+
 ### Key Takeaways for Implementation:
 - Check permissions early using `@permission_required` decorator
 - Use separate serializers for request/response

docs/conf.py

@@ -59,6 +59,7 @@ def get_version(*file_paths):
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'sphinxcontrib.mermaid',
     'sphinx.ext.autodoc',
     'sphinx.ext.doctest',
     'sphinx.ext.intersphinx',

docs/decisions/README.rst

@@ -266,3 +266,15 @@ and related business logic.
 Accepted Sept 2025, this ADR describes the rational
 for identifying SSP Stripe products based on ``lookup_key``
 instead of the Stripe price id.
+
+`<0037-server-side-learner-pathway-pipeline.rst>`_
+***************************************************
+*Feature: Learner Pathways*
+
+In progress September 2026, this ADR describes moving learner pathway
+generation out of the learner-portal MFE into this service, built on the
+abstract workflow pattern as two workflows behind two endpoints. It records
+the measured index behaviour the design rests on -- both Algolia indexes AND
+every query word, the Lightcast-canonical skill vocabulary, and 31% of
+courses carrying no skill tags -- and is explicit that the measured quality
+bar is not yet met.

docs/references/algolia_search.md

@@ -0,0 +1,210 @@
+# Algolia search from enterprise-access
+
+How to query the catalog and jobs indexes, and the index behaviours that will bite you.
+Measured against the production indexes on 2026-09-09.
+
+Client: `enterprise_access/apps/api_client/algolia_client.py`.
+
+## Two indexes, two credentials, not interchangeable
+
+| Index | Credential | Why |
+| --- | --- | --- |
+| Catalog (`enterprise_catalog_incremental_prod`) | Enterprise-scoped **secured** key | Scoping is the point — it keeps results inside the learner's catalog |
+| Jobs / Lightcast taxonomy (`prod_taxonomy`) | Plain search key | Secured keys can't read it |
+
+A secured key sets `restrictIndices` to the catalog index and its replicas
+(enterprise-catalog's `generate_secured_api_key`), so sending one to the jobs index fails
+with an opaque Algolia error. The learner portal MFE encodes the same constraint as
+`unsupportedSecuredAlgoliaIndices = [ALGOLIA_INDEX_NAME_JOBS]`. `search_jobs_index()`
+refuses before issuing the request, including when the *configured* search key turns out
+to be a secured key (they base64-decode to a querystring containing `restrictIndices`).
+
+**The write key must never be configured here.** enterprise-catalog's `AlgoliaSearchClient`
+is an indexing/administration client built on `ALGOLIA.API_KEY` and has no `search()`.
+This client is search-only by construction.
+
+### Secured keys are vended per user, which constrains where you can use them
+
+`get_secured_algolia_api_key()` lives only on `EnterpriseCatalogUserV1ApiClient`, a
+`BaseUserApiClient` — it forwards user context, and the generated key carries a
+`userToken`. So a secured key is reachable from a request-backed code path and **not**
+from a management command or a Celery task. Offline tooling has to either accept unscoped
+results (`ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH`, off by default) or go through
+enterprise-catalog.
+
+Note also that `bffs.api.get_and_cache_secured_algolia_search_keys` caches for a fixed
+`SECURED_ALGOLIA_API_KEY_CACHE_TIMEOUT` and does **not** parse `valid_until`, despite its
+docstring saying so. `SecuredAlgoliaKey.is_expired()` checks it properly; treat a key with
+no `valid_until` as expired rather than assuming validity.
+
+## A search-only key cannot enumerate an index
+
+* No `browse` ACL — `/browse` returns 403.
+* Pagination is capped: with `hitsPerPage: 1000` the catalog index reports `nbPages: 1`
+  against `nbHits: 4094`. Page 1 and beyond return nothing.
+
+So you cannot build the set of course keys in an index with a search key. Establishing
+that a course is *absent* requires a browse-scoped key or enterprise-catalog's
+`contains_content_items`. Anything else is a probe, not a proof.
+
+`key` is also neither filterable (`filters: 'key:"IBM+DA0101EN"'` → 0 hits for a course
+that exists) nor searchable (`restrictSearchableAttributes: ['key']` → HTTP 400).
+
+## Scoping to an enterprise customer needs no secured key
+
+`enterprise_customer_uuids` is in the catalog index's `attributesForFaceting`
+(`enterprise-catalog`'s `apps/catalog/algolia_utils.py`), so this works with the plain
+search key:
+
+```
+filters: 'content_type:course AND enterprise_customer_uuids:<uuid>'
+```
+
+It is *also* in `unretrievableAttributes`, which means it never appears on a hit — but
+**that does not block faceting on it.** `facets: ['enterprise_customer_uuids']` returns
+values and counts, so customer UUIDs are enumerable (capped at 1,000 like any facet) and a
+candidate UUID can be verified by its hit count.
+
+Two consequences. Evaluation and diagnostic work can be scoped to a real customer without
+the secured-key machinery, which needs a request and a user token and so cannot run from a
+management command. And `unretrievableAttributes` should not be read as "private" — it
+hides the value from a *hit*, not from an aggregate.
+
+Measured 2026-09-10: the broadest customers see 4,057 of 4,094 courses (99.1%), so
+per-skill counts scoped to one differ from unscoped by about a single course. Academy
+customers are the exception at 13–16 courses.
+
+## Course keys are `<org>+<number>`, not run keys
+
+The catalog index's `key` field holds `HarvardX+ER22.1x`, `IBM+DA0101EN`,
+`CodeSignal+164`. A `course-v1:...` **course-run** key appears nowhere in the index. This
+is a silent failure mode: run keys look like course identifiers, so code or ground-truth
+data that uses them matches nothing and reads as a relevance problem.
+
+`aggregation_key` is `course:<key>`; `objectID` is `course-<uuid>-customer-uuids-<n>` and
+is not derivable from the course key.
+
+## Query semantics: every word is ANDed, and there is no fallback configured
+
+This is the single most surprising behaviour. Measured on one persona's goal text,
+filtered to `content_type:course`:
+
+| Query words | Hits |
+| --- | --- |
+| 1 (`Move`) | 533 |
+| 4 (`Move into a data`) | 90 |
+| 5 | 4 |
+| 6 | 1 |
+| 8 or more | **0** |
+
+`removeWordsIfNoResults` is not configured on the index, so a verbose query returns
+**zero hits, not poor hits**. A five-word career title (`Medical Surgical Registered
+Nurse Manager`) returns 0. Passing `removeWordsIfNoResults: 'allOptional'` turns that
+24-word query into 348 hits and the career title into 121. (`'lastWords'` does not
+rescue a long query.)
+
+Two consequences:
+
+1. Any natural-language query — a learner's free text, or a verbose model-generated
+   `condensed_algolia_query` — silently returns nothing. This is the mechanism behind a
+   retrieval ladder always descending to its widest step.
+2. **Relaxing the query is worth a lot, but it is not sufficient.** Against
+   product-authored ground truth, `removeWordsIfNoResults: 'allOptional'` moved
+   recall@20 from **12% to 23%** overall and **21% to 40%** for non-technology personas.
+   One search parameter is the cheapest available improvement.
+
+   It is still not evidence of success on its own: four personas went from 0 hits to 20
+   hits with recall unchanged at 0%. **Measure expected-key recall, never hit count** —
+   a full result set of the wrong courses is the same pathology as a scope-only fallback
+   wearing a different hat.
+
+Keep text queries to a few words. In the diagnostic, the *shortest* strategy (a bare
+career title) was the only one that retrieved anything at all without `allOptional`.
+
+## The skill facet vocabulary is Lightcast-canonical, and short names are absent
+
+`skill_names` holds disambiguated Lightcast forms. The short name a learner or a model
+would produce is usually **not a facet value at all**:
+
+| What you'd write | Hits | What the index actually holds | Hits |
+| --- | --- | --- | --- |
+| `Python` | 0 | `Python (Programming Language)` | 95 |
+| `SQL` | 0 | `SQL (Programming Language)` | 42 |
+| `Java` | 0 | `Java (Programming Language)` | 27 |
+| `Excel` | 0 | `Microsoft Excel` | 26 |
+
+Verified by direct `facetFilters` counts, not by reading the facet list — the facet
+vocabulary response is capped at `maxValuesPerFacet` (1000), so "missing from the list"
+is not evidence of absence.
+
+Some names *are* canonical as-is (`Data Analysis`, `Machine Learning`, `Project
+Management`, `Nursing`, `Leadership`). So the mismatch is systematic but not uniform, and
+it has a predictable shape: `X` → `X (Programming Language)` / `X (Python Package)` / a
+vendor-qualified form.
+
+Exact-match grounding against a facet snapshot therefore drops the most in-demand
+technical skills silently. That is a vocabulary-normalisation problem with a mechanical
+fix, not an LLM-paraphrasing problem.
+
+## Facet counts are inflated ~75-80x; `nbHits` under `facetFilters` is exact
+
+The catalog index de-duplicates at query time (`distinct` on `aggregation_key`) but
+**facet counts are computed before de-duplication**. The `subjects` facet reports 79,392
+for "Business & Management" against a real 1,015 courses. Any code or dashboard that
+displays a raw facet count is displaying a wrong number, roughly 75-80x too high.
+
+Use `nbHits` under `facetFilters` with `hitsPerPage: 0` instead — that is exact
+(verified against distinct keys returned on slices of 8, 35 and 1,015).
+
+The same caution applies to the facet-*search* endpoint's `count` field, which is
+additionally unfiltered by `content_type`. Treat those as candidate vocabulary only.
+
+## 31% of courses carry no skills at all
+
+A full census of all 4,094 courses found **1,272 (31.1%) with both `skill_names` and
+`skills` empty**; the median tagged course carries 5 skills.
+
+Do not sample by relevance rank to measure this. An earlier pass using the top 1,000
+hits reported 13.1%, less than half the true rate, because relevance rank is
+popularity-biased toward well-tagged content. Enumerate by slicing on a facet instead.
+
+Concentration matters more than the average: **130 of 141 "Artificial Intelligence"
+subject courses (92%) are untagged**, as are 115 of 122 Google Cloud courses and all 62
+CodeSignal courses. Those courses cannot be retrieved by any skill-facet query, whatever
+the vocabulary handling.
+
+## Two thirds of jobs have no skills, deterministically
+
+In a full census of 43,513 English-language jobs, **29,525 (67.9%) have an empty
+`skills` array**. The rule is exact, with no exceptions observed: a job carries skills
+**iff** it carries `job_sources: course_skill`. All 13,988 such jobs have skills; all
+29,525 industry-only jobs have none.
+
+So "the career resolved to a Lightcast entry" and "that entry has usable skills" are two
+separate gates, and for two thirds of careers skill-based course retrieval has nothing to
+work with. Check `job_sources` before relying on `skills.name`.
+
+`prod_taxonomy` also supports numeric filters on `id` (`filters: 'id >= X AND id <= Y'`),
+which makes full enumeration by bisection feasible there despite the pagination cap.
+
+## Settings
+
+```python
+ALGOLIA_APP_ID = ''
+ALGOLIA_SEARCH_API_KEY = ''            # plain, search ACL only
+ALGOLIA_CATALOG_INDEX_NAME = ''
+ALGOLIA_JOBS_INDEX_NAME = ''
+ALGOLIA_ALLOW_UNSCOPED_CATALOG_SEARCH = False   # diagnostics only
+```
+
+Index names and the plain search key are configured per environment in
+`edx-internal/frontends/frontend-app-learner-portal-enterprise/*_config.yml`; the search
+key is search-ACL-only and already ships in the MFE bundle.
+
+## Local development
+
+The devstack `enterprise-access` container has **no network egress** — DNS fails and even
+a raw-IP connection times out — so live Algolia calls cannot be made from inside it, and
+`algoliasearch` cannot be `pip install`ed there without staging the wheel via
+`docker cp`. Unit tests mock the client and are unaffected; anything that needs a real
+Algolia response has to run outside the container.

docs/references/career_discovery_workflow.md

@@ -0,0 +1,84 @@
+# Career discovery workflow
+
+`POST /api/v1/learner-pathways/careers/` — learner intake in, career candidates out, as a
+persisted two-step workflow. Code: `enterprise_access/apps/pathways/` (steps, workflow,
+domain API) and `enterprise_access/apps/api/v1/views/pathways.py` (endpoint).
+
+```
+intake (4 fields) -> ExtractIntentStep (Xpert, learner_intent prompt)
+                  -> RetrieveCareersStep (Algolia jobs index)
+                  -> [{external_id, name, skills, industries}]
+```
+
+Gated on `LEARNER_PATHWAYS_SERVER_PIPELINE_ENABLED` (default `False` → 404) and on the
+existing `LEARNER_PATHWAYS_LEARNER_ROLE` via a new
+`LEARNER_PATHWAYS_CAREER_DISCOVERY_PERMISSION`.
+
+## Why a workflow rather than a view function
+
+The step records *are* the trace. Each carries its own input, output, timing and failure,
+so diagnosing a bad run is a query rather than a reproduction, and re-executing a workflow
+skips the steps that already succeeded. That is also why nothing needed a bespoke tracing
+layer for the evaluation harness — the response returns `workflow_uuid` and the harness
+reads the records.
+
+`CareerDiscoveryWorkflow` subclasses `AbstractConditionalWorkflow`, not
+`AbstractWorkflow`. Neither of its steps defines `should_execute` today, so execution is
+identical; the base is there because the pathway workflows that extend this pipeline do
+have steps that opt out, and because its `Optional`-typed dynamic IO classes are what let
+a skipped step round-trip as `null`.
+
+## Skills are boosts, industries are hard filters
+
+Ported from the MFE's `careerRetrieval.ts`, and the asymmetry is the load-bearing part:
+
+| Signal | Algolia parameter | Why |
+| --- | --- | --- |
+| Required skills (max 4) | `optionalFilters`, unscored | An unmatched *hard* skill filter returns zero hits and says nothing about why |
+| Preferred skills (max 2) | `optionalFilters`, `<score=1>` | Weaker signal, so a weaker boost |
+| Industries, job sources | `filters` | Caller is expected to have grounded these against the index already |
+
+Compound artifacts (`"SQL & Python"`, `"Excel + Tableau"`) are dropped before filtering —
+they match nothing and spend a filter slot.
+
+**The intake's `interested_industries` is deliberately *not* piped into the hard filter.**
+It is learner free text ("healthcare, technology"), and a hard filter on a value that is
+not a facet value returns zero hits silently. Free text belongs in the text query, where
+partial matching applies. `RetrieveCareersInput.industries` exists for a caller that has
+grounded real facet values first, and the endpoint leaves it empty.
+
+## Careers are identified by `external_id`, and carry no match percentage
+
+`external_id` is the Lightcast job id (`ET` + 16 hex, e.g. `ETE78CD2CDFFFAC66B`). Taxonomy
+names are neither unique nor stable, so a name cannot be a key — and the evaluation
+personas record expected careers as `external_id`s for the same reason. A hit missing
+either its `external_id` or its name is dropped rather than given a placeholder: a
+fabricated id would corrupt the harness's ground-truth comparison.
+
+There is no match-percentage field anywhere in the pipeline or the response. The
+client-side POC hardcoded `0.95` on every card; the MFE removed it deliberately, because
+no verified compatible domain value exists.
+
+## What the step output records, and why
+
+`RetrieveCareersOutput` persists `query` and `hit_count` alongside the careers. A full
+result set is not evidence that retrieval worked — relaxing a query buys volume, not
+relevance — so a report needs both numbers to tell a real retrieval from a padded one
+without re-running the search.
+
+## Gotchas found while building
+
+* **A per-action `throttle_scope` needs a class-level sentinel.** DRF's `as_view()`
+  rejects any `@action` initkwarg that is not also an attribute on the viewset class, so
+  `throttle_scope: str | None = None` on the class is load-bearing, not decoration. Without
+  it the router raises `TypeError` at import time and every URL in the service fails to
+  resolve.
+* **Two viewsets can share a router prefix.** `learner-pathways` is registered twice, with
+  different basenames, so the careers endpoint sits beside the prompt endpoints without
+  touching them. Neither viewset has a `list` route, so nothing collides.
+* **The Xpert conversation ID is keyed on the step record, not the request ID.** A step can
+  be re-executed outside the request that created it, and the step UUID is the one
+  identifier that ties an Xpert conversation back to a persisted trace either way.
+* **Step tables persist learner-authored free text** (the intake), with no user
+  identifier. They are annotated `.. no_pii:` on that basis; if a user linkage is ever
+  added, the retirement pipeline has to be part of that change.

enterprise_access/apps/api/serializers/__init__.py

@@ -38,8 +38,14 @@
     TransactionsListResponseSerializer
 )
 from .learner_pathways import (
+    CareerCandidateSerializer,
+    CareerDiscoveryRequestSerializer,
+    CareerDiscoveryResponseSerializer,
     LearningIntentRequestSerializer,
     LearningIntentResponseSerializer,
+    PathwayCourseSerializer,
+    PathwayRequestSerializer,
+    PathwayResponseSerializer,
     RecommendationFeedbackRequestSerializer,
     RecommendationFeedbackResponseSerializer
 )