Proposal: a single canonical QuantEcon datasets repository, with a documented convention

[jstac]

Floating this for discussion. If there's rough agreement it should graduate to a QEP

Summary

Lecture data is currently hosted and referenced in at least four inconsistent ways across our lectures, with no guidance in the manual.

This causes frictions — see below.

I propose we adopt one canonical, general-purpose data repository, referenced by stable URLs, and document the convention in the manual.

Concretely: repurpose/rename QuantEcon/high_dim_data → QuantEcon/datasets rather than start a second repo.

Motivation

1. New-data files can't build on a branch (today's trigger).

While adding a population-by-age figure to prob_dist.md, the cell read the .xlsx via a https://github.com/QuantEcon/lecture-python-intro/raw/main/… URL — the pattern used by long_run_growth.md etc. Because the file was new and not yet on main, the local/PR build 404s and fails under CI's -nW. The only ways through are a two-step merge (land the data in one PR, then reference it in a second) or temporarily swapping the URL to a local path for the build. Both are fiddly and undocumented, and every contributor rediscovers this.

2. Four+ inconsistent data-hosting patterns, no rule. In lecture-python-intro alone:

• raw/main URL from lectures/datasets/ — mpd2020.xlsx, longprices.xls, chapter_3.xlsx
• raw/main URL from lectures/_static/lecture_specific/ — inequality Gini CSV, simple_linear_regression
• relative local path, no URL — msy_fishery.md: pd.read_csv('datasets/lingcod_msy_recovery.csv')
• external repo high_dim_data via two different hosts — plain raw.githubusercontent.com and media.githubusercontent.com (LFS)

There is no documented rule for which to use, so the choice is made ad hoc per lecture.

3. A lecture pinned to a data branch. mle.md reads from
…/high_dim_data/update_scf_noweights/SCF_plus/… — a feature branch, not main. If that branch is ever deleted or rebased, the lecture silently breaks. (Worth fixing regardless of this proposal.)

4. The data repo is mis-scoped. high_dim_data's README says "Data files for High Dimensional Economics." But its actual consumers are the intro lectures inequality, heavy_tails, mle (SCF wealth data, Forbes 2000/billionaires, US/Brazil city populations). It is already our de-facto general empirical-data repo — just misnamed, with a stale README.

5. Binary data bloats lecture repos. Committing .xlsx/.csv into the lecture repos (and into _static/) mixes binaries with narrative source. A dedicated data repo keeps lecture repos lean and text-focused.

6. No manual guidance. The style guide has zero coverage of data handling — the root cause of the inconsistency above.

7. Cross-series reuse. The same datasets (Maddison, SCF, …) are candidates for reuse across intro / programming / advanced / jax / julia. A shared data repo enables reuse instead of per-repo copies.

8. Colab / download self-containment. Relative paths break when a notebook is downloaded or opened in Colab; a stable external URL keeps notebooks self-contained. This is the legitimate reason to prefer URLs — and a dedicated data repo makes those URLs always resolvable regardless of the lecture branch, which is exactly what removes motivation #1.

Proposal

Adopt a single, dedicated, general-purpose data repository as the canonical home for lecture datasets, referenced by stable URLs, with the convention documented in the manual. Specifically, repurpose high_dim_data into QuantEcon/datasets, rather than create a second general repo — two general-purpose data repos would reintroduce the "which repo?" confusion.

Key design points:

• LFS is per-path (.gitattributes): large binaries tracked via LFS, small teaching files stay plain git in the same repo.
• Make it flat (just dump files) or organize by topic subfolders (demography/, inequality/, cross_section/, SCF_plus/, …).
• One documented URL form and a decision on main-vs-tag pinning (see open questions).

Alternatives considered

• Status quo (raw/main same-repo + two-step merges): ongoing friction, permanent per-lecture inconsistency.
• Relative local paths everywhere (the msy_fishery pattern): simplest single-PR build, but breaks Colab/download self-containment and still bloats lecture repos.
• Two repos split by "high-dim vs low-dim": the size boundary is fuzzy and technically unnecessary (LFS is per-path); the split is what causes confusion.
• New datasets repo alongside high_dim_data: two parallel general repos → "which repo?" ambiguity. Since high_dim_data isn't really book-specific, repurposing it is cleaner.

Rollout

• Confirm no external "High Dimensional Economics" project still depends on high_dim_data (nothing in the lecture series does — only intro's 3 files).
• Rename QuantEcon/high_dim_data → QuantEcon/datasets. GitHub redirects preserve existing URLs, so this is non-breaking; the sweep below can follow at leisure.
• Rewrite the README to state the real, general purpose.
• Set .gitattributes for per-path LFS (large binaries LFS; small teaching files plain git).
• Decide the layout (flat, or topic subfolders) and organize accordingly.
• Standardize the URL form and pinning policy (open question below).
• Sweep existing references to the new name (3 files in intro: inequality, heavy_tails, mle) and fix the mle.md branch-pin → main.
• Document in the manual — new styleguide/datasets.md: the routing rule ("data owned by a specific book/project → that repo; otherwise → datasets"), LFS-per-path, the URL form, and the pinning policy.
• Going forward: new lecture data → PR to datasets → reference by stable URL. A single lecture PR then builds green immediately.

Smaller issues to open regardless of the outcome

1. mle.md pins a data branch (update_scf_noweights) instead of main — latent breakage. (bug)
2. high_dim_data uses two URL hosts (raw vs media/LFS) for files in the same repo — standardize.

Open questions for the team

1. Repo name — datasets? data? Something else?
2. Layout — flat or by topic
3. Pinning — reference main (simple, mutable) or a tag/commit SHA (reproducible builds)?
4. URL form — settle plain raw.githubusercontent.com vs media.githubusercontent.com/LFS as a documented rule.

Recommendation on process

This touches every lecture series plus a data repo and establishes a working/tooling convention documented in the manual — per QEP-1 that is exactly what a QEP is for. Recommendation: use this issue to socialise and converge; once there's rough agreement, I'll graduate it to a QEP (type: standard) recording the datasets-repo convention, with styleguide/datasets.md as the implementation.

Context: this came out of a session adding a Japanese population age-distribution figure to prob_dist.md in lecture-python-intro.

[mmcky]

Thanks @jstac — I'm very much +1 on this, and the audit is spot on. The framing I'd add is that a dedicated data repo lets us cater to data needs directly, rather than piggy-backing on lecture infrastructure — data has its own lifecycle (provenance, refresh, licensing, size) and doesn't really belong in a narrative-source repo once it's used across multiple series.

Some history, one technical find that I think lets us sidestep your Q4 entirely, and thoughts on your open questions.

What exists today

I opened #129 for this ("DATA: Migrate high_dim_data to data") and we made a start: QuantEcon/data. The trigger was WASM/JupyterLite (#143) — pyodide can't fetch from most web APIs, so we needed snapshotted data at stable, CORS-friendly URLs. Worth adding to your motivation list, and it's why the repo has dynamic/ and scripts/ (periodic API snapshots) alongside static/.

A fair bit already landed in Feb 2025: intro's static data is already in QuantEcon/data today — mpd2020.xlsx, longprices.xls, chapter_3.xlsx, assignat.xlsx, dette.xlsx — via QuantEcon/data-lectures#5, plus a layout (QuantEcon/data-lectures#6) and a refresh script (QuantEcon/data-lectures#7). It then stalled on the last box of QuantEcon/data-lectures#4, "update all the paths in all the lectures", while we worked around the WASM issue. QuantEcon/data-lectures#1 (set up LFS) and QuantEcon/data-lectures#2 (fold high_dim_data in) are still open as well.

Q1 (name): data-lectures

I'd suggest we rename QuantEcon/data → data-lectures — data consumed by the lecture series.

It matches the taxonomy already documented in the manual's styleguide/repos.md — lecture-{name}, book-{name}, project-{name}, action-{name} are all {type}-{scope} — so your routing rule falls out of scope rather than size: consumed by lectures → data-lectures; owned by a specific book or package → that project's repo. It also leaves datasets free in case we ever want to host a curated QuantEcon dataset collection.

Q4 (URL form): a trap worth knowing about

The raw vs media split in high_dim_data isn't arbitrary — it's Git LFS. I tested all three hosts against the same LFS-tracked file and against a plain-git file:

URL form	LFS-tracked file	plain-git file
raw.githubusercontent.com/...	returns the LFS pointer text	works
github.com/{org}/{repo}/raw/{ref}/{path}	works	works
media.githubusercontent.com/media/...	works	404

This matters right now: QuantEcon/data has no LFS yet and holds mpd2020.xlsx (1.8MB) and dette.xlsx (632KB) as plain blobs. The moment we action QuantEcon/data-lectures#1, every raw.githubusercontent.com URL breaks silently by feeding pandas a pointer file — and that's the form intro uses today. (Also worth knowing: high_dim_data's .gitattributes is *.csv filter=lfs, i.e. every CSV regardless of size, so your per-path LFS point is a fix rather than just a setting.)

Which is why I'd propose data.quantecon.org/lectures

Rather than pick a GitHub host at all, let's serve the data ourselves via GitHub Pages on a custom domain — the same pattern as intro.quantecon.org. It gives the cleanest entry point for fetching data across every series, and sidesteps Q4:

• The URL decouples from storage. Lectures reference https://data.quantecon.org/lectures/..., and LFS-vs-plain-git becomes internal hygiene that's invisible to every lecture.
• CORS works — I verified Pages serves access-control-allow-origin: *, which is exactly what pyodide needs ([Plan/Idea] Supporting WASM based introductory series #143).
• It fixes the LFS bandwidth problem, which is metered and billed — monitoring bandwidth usage high_dim_data#4 is literally "monitoring bandwidth usage". Readers hit Pages bandwidth rather than our LFS quota.
• It's portable — a domain we own means we could move to R2/S3 later without touching a single lecture.

The /lectures prefix keeps the namespace open for /datasets or /books later without breaking any URL we ship now.

Two caveats for the QEP: the Pages deploy must checkout with lfs: true or we'd republish pointer files (our Pages builds are already build_type: workflow, so that's a one-line change); and Pages isn't a real CDN (~1GB site and 100GB/month soft limits), so worth monitoring.

data.quantecon.org already resolves — to an old NestJS box on AWS Sydney with no HTTPS listener — so I'll look into what's actually running there.

Q2 (layout): flat

I'd go flat, your first option. It's the one that really delivers your motivation #7: lecture-python and lecture-python-intro would both just fetch /lectures/mpd2020.xlsx, with no folder implying ownership. Topic folders would reintroduce a softer version of the consumer-keying problem the current data layout has.

The static/dynamic split then becomes metadata (provenance, refresh cadence) rather than directory structure — best surfaced in a generated index page at /lectures/, which doubles as the catalog. scripts/ stays in the repo, outside the published tree.

Q3 (pinning): main, with git history as the version record

Reference main. Git already gives us version control on the data, so we don't need a URL convention to reinvent it — a fix is just a commit, and if we ever need the old bytes they're a SHA away.

Two cases worth separating, though, because they look alike and want opposite handling. Corrections (bad parse, wrong units, corrupt rows) should be fixed in place: everyone should get the fix, and a new filename would leave the broken data live until we'd swept every reference. New vintages (Maddison 2020 → 2023, SCF 2019 → 2022) get a new filename, since that's genuinely different data rather than a new version of the same thing — the old vintage stays valid.

One consequence worth naming, since it couples back to Q4: serving from data.quantecon.org makes the published URL inherently "latest", so reproducing an old build becomes a git operation rather than a URL one. I think that's the right trade for continuously-rebuilt teaching material, but it's a deliberate one. The safety net is knowing what a correction touches — if the catalog records which lectures consume which dataset, a fix tells us exactly what to rebuild and review.

Process

Agree on the QEP, and the manual page is well overdue — it's the last unticked box on #129. One scope addition: styleguide/repos.md documents our repo-type taxonomy and has no data-repo type at all, so the QEP should add data-{scope} there, not just add styleguide/datasets.md.

Suggested consolidation: keep this issue as the design/QEP thread and let it absorb #129, with QuantEcon/data-lectures#1, QuantEcon/data-lectures#2 and QuantEcon/data-lectures#4 as a ready-made execution checklist.

+1 on both of your smaller issues regardless — the mle.md branch pin is live right now (update_scf_noweights is a real branch, and mle.md:93 still fetches from it, so it's latent rather than broken).

Happy to help pull the QEP together once we've converged.

[mmcky]

Following up with data: I ran a full audit of dataset usage across all 7 Python-family lecture repos (each repo's main, 2026-07-15 — method: git grep for every reader call, data-file extension, GitHub host, and API client, cross-checked against git ls-tree inventories and the actual contents of QuantEcon/data and high_dim_data). Digest below; the two sub-issues (#337, #338) carry the execution side.

Headline numbers

Metric	Count
Distinct static data files referenced by lectures	34 (intro 17, programming 5, python.myst 9, advanced 6)
Data files committed across the repos	50, of which 20 are dead weight (orphaned, superseded, or shadowed by %%file)
GitHub raw-URL spellings in production	6 (not 4 — including blob/…?raw=true and two refs/heads variants)
Live references into the pre-MyST lecture-python repo (last push Jan 2022, unarchived)	8
Lectures fetching live API data	11 (wbgapi, FRED via three different access methods, yfinance)
Lectures referencing QuantEcon/data	0 — despite it already holding 10 intro files since Feb 2025

Only 4 of the 7 repos touch data at all: dp, jax, and continuous_time_mcs are data-free apart from an embedded %%file toy graph (which is itself duplicated in 3 repos).

Findings that sharpen the proposal

1. The mle.md branch pin is worse than latent. SCF_plus_mini_no_weights.csv does not exist on high_dim_data's main at all — the update_scf_noweights branch is the file's only home, so branch deletion is an instant 404. Tracked in #337.

2. Eight URLs depend on the dormant legacy repo, while both consuming repos already commit their own unused copies of the very same files (ols ×3, pandas_panel ×5 across programming and python.myst). Tracked in #337.

3. Cross-series file reuse is exactly one trio today (the pandas_panel CSVs), but the underlying sources recur everywhere — FRED alone is consumed three different ways in 10 lectures, plus World Bank, SCF, Maddison, OECD, Forbes, Yahoo. The consolidation upside in motivation 7 is real; it just materializes at the source level rather than the file level.

4. Provenance is the second axis the convention needs, alongside static/dynamic. Of the 34 files: 7 constructed with a committed builder (the hansen_singleton make_data.py + README pattern is the gold standard), 5 constructed but the pipeline was never committed (documented in prose only — unreproducible today), 7 verbatim third-party files (citation is the provenance; note rehosting these raises a licensing question the QEP should answer), and 14 author-assembled with no pipeline at all. Suggests three documented classes: verbatim (metadata only), constructed (builder required in the data repo), dynamic snapshot (builder + refresh cadence).

5. On live APIs: only 3 of the 11 lectures use the API as pedagogy (programming's pandas lecture teaches wbgapi/yfinance access; intro's business_cycle and inequality teach series discovery). The other 8 just need a series and are snapshot-ready. Our own history shows the migration pressure: hansen_singleton went datareader → direct downloads → fully vendored with maintenance scripts in two consecutive PRs (QuantEcon/lecture-python.myst#925, QuantEcon/lecture-python.myst#926), programming evicted pandas_datareader twice (QuantEcon/lecture-python-programming#341, QuantEcon/lecture-python-programming#133), and the live business_cycle lecture still needed a manual data refresh (QuantEcon/lecture-python-intro#770). Proposed rule for the manual: live APIs are for teaching data access, not for getting data — and every live-API lecture gets a snapshot twin built by a scheduled, validated refresh pipeline (the dynamic/ + scripts/ split in QuantEcon/data already prototypes this; it just has no automation — the repo has no .github/ at all).

Process

This supports the QEP direction, with the pilot (#338) as the empirical step in between: migrate one dataset per hosting pattern (local-path → cross-series → LFS/external → dynamic twin), fold what we learn into a draft styleguide/datasets.md, then write the QEP recording a convention that has already worked.

The full registry behind this digest — every file with contents, consuming lectures, hosting pattern, and provenance class, plus the URL-spelling inventory, orphan list, API classification, and the draft dynamic-data framework — is browsable here: https://claude.ai/code/artifact/87174334-f9e0-4b5c-b4df-0739a5abcdc2

[mmcky]

The draft manual page is now up as the working design surface: QuantEcon/QuantEcon.manual#108 — styleguide/datasets.md (routing rule, URL form, provenance classes, live-API rule, dynamic refresh framework) plus the data-{scope} addition to styleguide/repos.md. Unsettled decisions are marked (open) inline; suggest we push design changes there as this thread converges, and merge it as the QEP's implementation.

[jstac]

Thanks @mmcky ! pls ping me if you need my input.