Repository navigation
quarto-core + xtask: extension-subtree infrastructure (julia epic Steps 3 + 4-infra) - #717
Merged
Merged
Conversation
Port of Q1's hidden `pull-git-subtree` dev command: fetches a subtree's remote and reconciles it into the repo via `git subtree add --squash` (no prior split) or `git subtree pull --squash` (new upstream commits), tracked via the `git-subtree-dir`/`git-subtree-split` trailers `git subtree` writes into its squash commit. SUBTREES starts empty; the first real row (julia) lands in the epic's Step 4. Covers the plan's five scenarios via inline unit tests against a real git fixture remote/consumer pair: add, no-op on immediate re-run, pull on a new upstream commit, unknown-name error listing available names, and omitted/`all` processing every table row. Also smoke-tested through the real `cargo run -p xtask -- pull-extension-subtree` binary. Plan: claude-notes/plans/2026-09-23-extension-subtree-infrastructure.md
Adds `builtin_extension_subtree_roots` + `all_builtin_extension_roots` to extension/mod.rs (env override -> per-subtree ResourceBundles, empty until a real subtree is vendored -> WASM VFS path), and pluralizes discover_extensions/discover_project_extensions from a single Option<&Path> builtin dir to &[&Path], scanned in order before user extensions. Updates all three production call sites and fixes a latent bug along the way: discover_extensions_only hardcoded no built-ins on WASM while stage/context.rs and ProjectConfig::parse_config correctly used builtin_extensions_path — both now go through the same all_builtin_extension_roots helper. Adds the WASM parallel (populate_extension_subtrees in wasm-quarto-hub-client, mirroring populate_builtin_extensions) and the resources/extension-subtrees/README.md placeholder both include_dir! call sites need to compile (git doesn't track empty directories). crates/wasm-quarto-hub-client/Cargo.lock picked up a large but legitimate diff from resolving that crate's dependencies fresh for the first time in this worktree; kept as-is. Plan: claude-notes/plans/2026-09-23-extension-subtree-infrastructure.md
Adds the synth-echo fake extension fixture (tests/fixtures/extension-subtrees/synth-echo/,
shaped like a whole vendored repo with the extension nested at
_extensions/synth-echo/) and two integration tests proving the Phase 1-2
infrastructure works end-to-end:
- discovers_synth_echo_via_subtree_root_env_var_with_zero_extensions_install:
discovery via QUARTO_EXTENSION_SUBTREES_DIR + all_builtin_extension_roots,
with no project _extensions/ install.
- e2e_single_file_render_executes_synthsub_via_subtree_root: a single-file
render (no _quarto.yml) of a {synthsub} cell through the real Deno engine
host, exercising discover_extensions_and_build_registry -- the other
caller of discover_extensions_only, closing the loop on Phase 2's
WASM-branch fix. Also verifies D4 (no Q1-style engine-name leak into
rendered metadata).
Both tests passed against the existing Phase 1-2 implementation with no
further code changes needed -- this phase is the proof, not new work.
HERMETIC_FIXTURES in engine_fixture_build.rs gained one name
("synth-echo"); ensure_bundle/build_bundle needed no path-aware variant,
since they already take an explicit ext_dir with no assumption about
where it lives.
Adds claude-notes/instructions/extension-subtrees.md, the runbook for
registering a real subtree (one SUBTREES row, one include_dir! payload
static) that the julia-engine epic step will follow.
Plan: claude-notes/plans/2026-09-23-extension-subtree-infrastructure.md
(all three phases complete)
…Step 3, F5)
Dynamic claiming lowercases the language (claimsLanguage: (language) =>
language.toLowerCase() === "julia"), but the static path never normalized
either side: parse_claims_map inserted claim keys verbatim and
lookup_static_claim did an exact claims.get(language). A {Julia} cell
therefore claimed dynamically yet silently missed the static/zero-load
resolution path that claims: exists to provide.
Fix (epic option 1, mirroring normalize_ext's file-extensions precedent):
- parse_claims_map lowercases claim keys at parse time, in both the
map-entry and top-level list-shorthand forms
- lookup_static_claim lowercases the language before lookup
TDD: three new tests RED before the fix, GREEN after —
parse_claims_map_lowercases_language_keys,
parse_claims_map_lowercases_list_shorthand_language (read.rs),
lookup_static_claim_is_case_insensitive (types.rs).
Gates: clippy -p quarto-core --all-targets -D warnings clean; cargo fmt
clean; cargo nextest run --workspace 14488 passed / 200 skipped (baseline
14485 + the 3 new tests, 0 failed); cargo xtask verify --skip-hub-build
--skip-hub-tests all steps passed. Epic plan updated (Step 3 DONE +
Developments entry).
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
This PR ports Quarto 1's extension-subtree machinery to q2 and proves it end-to-end — without vendoring any real engine. It is the infrastructure half of Step 4 of the julia-engine epic, deliberately split out so it never waited on external PRs. The original gate on the julia-specific remainder (a PumasAI-hosted branch, PumasAI/quarto-julia-engine#15) is gone as of 2026-09-24: the epic's Step 2c pivoted to using the
gordonwoodhull/quarto-julia-enginefork as q2's upstream of record until the Q1 schema change (quarto-dev/quarto-cli#14936) ships in a stable release. What still gates vendoring julia is only q2-internal sign-off (epic Q8/Q9). Once cleared, bundling the Julia engine is one row in a table and oneinclude_dir!registration — the runbook is included here (claude-notes/instructions/extension-subtrees.md).It also carries an independent q2 bug fix (epic Step 3 / finding F5): static engine-claim lookup was case-sensitive while dynamic claiming was not, so a
```{Julia}cell claimed dynamically but missed its own static declaration.The three pieces
1.
cargo xtask pull-extension-subtree— a port of Q1's hiddenpull-git-subtreedev command (src/command/dev-call/pull-git-subtree/cmd.ts). Maintains vendored extension repos as git subtrees underresources/extension-subtrees/: finds the last split via thegit-subtree-dir:trailer,subtree add --squashon first run, no-op when there's nothing new,subtree pull --squashotherwise. TheSUBTREEStable ships empty — adding the julia row is the gated Step 4 remainder. Test-only seams (QUARTO_SUBTREE_ROOT,--table <json>) let the tests run realgitagainst fixture remotes in tempdirs; this is xtask's first test module that shells out, and it's hermetic.2. Multi-root builtin discovery.
discover_extensions/discover_project_extensionsnow take&[&Path]builtin roots instead ofOption<&Path>; subtree payloads are scanned after the regular builtin dir and before user extensions, so last-match-wins still lets a user install override a bundled extension.builtin_extension_subtree_roots()resolves roots from (a) theQUARTO_EXTENSION_SUBTREES_DIRdev/test override, (b) per-subtree embeddedResourceBundlepayloads, (c) the WASM VFS. Along the way this fixes a real inconsistency:discover_extensions_onlyhardcodedNonefor builtins on WASM — WASM builds got no builtin extensions at all. That split is gone; native and WASM share one path.3. End-to-end proof with a fake extension.
synth-echo(a fixture shaped like a vendored repo, claiming asynthsublanguage) is discovered via the subtree-root mechanism with zero_extensions/install and executed by the real Deno engine host throughrender_to_file— the same entryquarto renderuses. The fake lives undertests/fixtures/and never ships: release binaries embed a placeholder README, and nothing is registered inEXTENSION_SUBTREE_PAYLOADSyet.Design decisions reviewers should know about
extension-subtreesroot). Each future subtree gets its owninclude_dir!static so we embed only its_extensions/payload (~68K for julia), never the whole vendored repo (~14M of tests and CI config).filterBundledSubtreeEnginesanalogue needed. Q1 strips bundled engines from pandoc metadata because it resolves engine objects into it; q2's extension-contributed engines are registry-only and never enter writer metadata. This is asserted by test (the engine's own name must not appear in rendered HTML), not merely surveyed.subtree add, the port returns instead of falling through to a same-invocationpullcheck — an apparent quirk in the JS, not reproduced.Test story
TDD throughout; every phase gated on clippy
-D warnings+ per-crate nextest, with fullcargo xtask verify(including hub-client build/tests) green at each boundary. Workspace totals at the final gate: 14485 passed, 0 failed, 200 skipped — +9 tests over the branch point, exactly the ones this PR adds:git subtreeagainst fixture remotes (add, no-op, pull, unknown-name, all-rows)Full details, including the empirical
git subtreebehaviors the tests encode, are in the plan.Explicitly out of scope
q2-static-declarationsbranch ofgordonwoodhull/quarto-julia-engine(epic Step 2c pivoted to the fork on 2026-09-24); gated only on the 14M git-history sign-off (epic Q8) and the no-Julia UX decision (Q9). Planned as a stacked PR on this one.