The Audacity of Cope: catching rustc's layout panics so std can fly#140
Draft
The Audacity of Cope: catching rustc's layout panics so std can fly#140
Conversation
Add a `make help` target that uses an awk script to parse structured
comments in the Makefile itself. `##` comments describe individual
targets; `###` comments act as section headers. The output is grouped
into four sections: Build, Test, Code quality, and Graph generation.
Internal helper targets (like `check-graphviz` and
`rustup-clear-toolchain`) are intentionally left without `##` comments
so they stay hidden from the help output.
Idiomatic fixes applied across the board:
- `.DEFAULT_GOAL := build` replaces the old `default: build` target.
The old approach created a real target called `default` that showed
up in tab-completion and could collide with other tooling;
`.DEFAULT_GOAL` is the built-in make mechanism for this.
- `$(MAKE)` replaces bare `make` in the `golden` target. When make is
invoked with flags (e.g., `make -j4`), a bare `make` in a recipe
starts a fresh make process that loses those flags and, critically,
the jobserver file descriptors. `$(MAKE)` preserves both.
- `.PHONY` is now declared immediately above every phony target. A
phony target is one that doesn't produce a file of the same name
(i.e., `make build` doesn't create a file called `build`). Without
the declaration, if a file named `build` or `clean` happened to
exist, make would see it as up-to-date and skip the target entirely.
Previously several targets (`build`, `clean`, `golden`, `format`,
`style-check`) were missing their declarations.
- Variable references normalized from `${VAR}` to `$(VAR)`. Both work
identically in GNU make, but `$(VAR)` is the conventional style.
`${VAR}` is visually ambiguous with shell variable expansion inside
recipes, so reserving curly braces for shell context (like `$${rust}`)
and parens for make context makes it easier to tell which layer is
doing the expansion.
- `TOOLCHAIN_NAME=''` fixed to `TOOLCHAIN_NAME=`. Unlike in shell,
make does not interpret quotes as delimiters: the old value was
literally the two-character string `''`, not empty. This happened to
be harmless (rustup would just fail to find a toolchain named `''`)
but was misleading.
- Removed the `ECHO_CMD=echo` variable, which was only used once and
added indirection with no benefit. Replaced with a direct `@echo`.
`\s` is not POSIX awk; BSD awk (macOS default) treats it as a literal `s`. The patterns worked anyway because the comments are always at column 0, but this was a latent portability bug. Use bare `^##` / `^###` anchors instead. Addresses review feedback on PR #139.
cds-amal
changed the title
cds-amal
force-pushed
the
cds/the-curious-case-of-stdlib
branch
from
63d36dc to
8220abc
Compare
Collaborator
Author
|
@dkcumming, perhaps #17 can be closed and we could add docs on this to address building |
ty.layout() returns Result<Layout, Error>; a deliberate API contract
that says "layout computation can fail, and we'll tell you about it
via Err." We were already handling that correctly with .ok(). Turns
out that's not enough: rustc's implementation panics before it ever
gets a chance to construct the Err.
The call chain (against nightly-2024-11-29, rustc commit a2545fd6fc):
1. layout_of_uncached needs to know whether a field type is Sized
2. calls type_known_to_meet_bound_modulo_regions(ty, Sized)
(compiler/rustc_trait_selection/src/traits/mod.rs:204)
3. which constructs TraitRef::new(tcx, sized_def_id, [ty]) and
passes it to pred_known_to_hold_modulo_regions as an
impl Upcast<Predicate>
4. the Upcast calls UpcastFrom<TraitRef> for Predicate
(compiler/rustc_middle/src/ty/predicate.rs:532)
5. which calls ty::Binder::dummy(from)
(compiler/rustc_middle/src/ty/predicate.rs:533)
6. Binder::dummy asserts !value.has_escaping_bound_vars()
(compiler/rustc_type_ir/src/binder.rs:106-109) and panics
The assert in Binder::dummy is legitimate: wrapping a value with
escaping bound vars in a dummy binder would be semantically wrong
(it would capture vars that belong to an outer binder). The bug is
upstream: type_known_to_meet_bound_modulo_regions blindly constructs
a TraitRef for <ty as Sized> without checking whether ty has
escaping bound vars. When layout_of_uncached asks "is this field
Sized?" for dyn fmt::Write (still under a lifetime binder from
Formatter<'a>), it feeds a type with escaping bound vars into a
code path that assumes they've already been substituted away.
The workaround has three parts:
1. try_layout_shape wraps ty.layout() in catch_unwind with
AssertUnwindSafe, returning Result<Option<LayoutShape>, String>.
On panic, the payload is downcast to extract the message.
2. The default panic hook is swapped out (take_hook/set_hook) for the
duration of the catch_unwind call, suppressing the noisy backtrace
that rustc's hook would otherwise print to stderr. The previous
hook is restored immediately after.
3. TyCollector gains a layout_panics: Vec<LayoutPanic> field.
layout_shape_or_record delegates to try_layout_shape and pushes
any Err into the vec; the type is still recorded with layout: None.
At the end of collect_and_analyze_items, accumulated panics are
reported as a single summary warning with the type and message.
This unblocks emitting smir.json for the standard library via
-Zbuild-std. Against nightly-2024-11-29, one layout panic is
observed:
warning: 1 type layout(s) could not be computed (rustc panicked):
type Ty { id: 20228, kind: RigidTy(Adt(AdtDef(DefId { id: 5070,
name: "core::fmt::Formatter" }), ...)) }:
`<dyn core::fmt::Write as core::marker::Sized>` has escaping
bound vars, so it cannot be wrapped in a dummy binder.
N.B. The fix on rustc's side would be for
type_known_to_meet_bound_modulo_regions to bail early (return false
or skip the check) when ty.has_escaping_bound_vars(), or for
layout_of_uncached to not query Sized for types it encounters under
a binder. Either way, the Result contract on ty.layout() should have
caught this before it became a panic at the stable_mir API boundary.
`make stdlib-smir` is a single command that builds stable-mir-json, installs the RUSTC wrapper, creates a throwaway crate in a temp dir, runs `cargo build -Zbuild-std` through our driver, and copies the resulting smir.json artifacts (hash suffixes stripped) into tests/stdlib-artifacts/. The temp crate is cleaned up automatically. The host target triple is detected at make-time via `rustc --print target-triple`, so no architecture is hardcoded. When you bump the nightly in rust-toolchain.toml, the same command regenerates everything against the new toolchain: one source of truth for the rustc version, one command for the artifacts. Also adds `make clean-stdlib-smir` to remove the output directory.
The fallback DYLD_LIBRARY_PATH / LD_LIBRARY_PATH in the generated wrapper script was hardcoded to nightly-2024-11-29-aarch64-apple-darwin. When rust-toolchain.toml pointed at a different nightly, the wrapper would link against the wrong toolchain's libraries. Replace the hardcoded path with `rustc --print sysroot` + "/lib", which resolves to whichever nightly rustup has active. The env var still takes precedence when set (existing behavior), but the fallback now stays in sync with rust-toolchain.toml automatically. Also deduplicates the macOS / Linux branches: the only difference is the env var name (DYLD_LIBRARY_PATH vs LD_LIBRARY_PATH), so a single code path handles both with a cfg-selected variable.
The stdlib-smir target previously went through the cargo_stable_mir_json install step to generate a wrapper script in ~/.stable-mir-json/, then used RUSTC=~/.stable-mir-json/debug.sh. This is unnecessary: the only thing the wrapper adds is DYLD_LIBRARY_PATH / LD_LIBRARY_PATH, which the Makefile can set directly via `rustc --print sysroot`. Now stdlib-smir points RUSTC at target/debug/stable_mir_json and sets the library path inline, the same way cargo does when it runs our binary via `cargo run`. No install step, no ~/.stable-mir-json/ indirection, no wrapper script in the critical path. The cargo_stable_mir_json binary still exists for external users who want to use stable-mir-json with their own cargo projects.
rust-toolchain.toml had a [metadata] rustc-commit field that manually
duplicated information the active toolchain already knows. If you
bumped the nightly channel and forgot to update rustc-commit, the UI
tests would silently run against the wrong compiler source.
Now the commit is derived from `rustc -vV | grep commit-hash` at the
point of use. rust-toolchain.toml's channel field is the single source
of truth; everything else follows from it:
channel --> rustup installs toolchain
--> rustc -vV gives the backing commit
--> ensure_rustc_commit.sh checks out that commit
--> rust-src component provides stdlib source for -Zbuild-std
Changes:
- ensure_rustc_commit.sh: replace yq + metadata read with rustc -vV
- rust-toolchain.toml: remove [metadata] section
- CI (test.yml): read channel via grep, install toolchain first, then
derive commit from rustc -vV; drops the yq install step entirely
- CHANGELOG.md: updated to reflect the new approach
cds-amal
force-pushed
the
cds/the-curious-case-of-stdlib
branch
from
8220abc to
2310378
Compare
The stable MIR public API evolves across nightlies: variants appear, disappear, and signatures change. Rather than supporting exactly one nightly, build.rs now parses `rustc -vV` to extract the compiler's commit-date and compares it against a table of known API breakpoints, emitting `cargo:rustc-cfg` flags (e.g. `smir_has_coroutine_closure`) that gate match arms on or off as needed. This keeps exhaustive matches correct on every supported nightly: without the flag the variant doesn't exist in the enum, so the arm is excluded; with the flag the arm is included to cover it. First breakpoint: `AggregateKind::CoroutineClosure`, added in nightlies >= 2024-12-14, gated behind `smir_has_coroutine_closure` in `src/mk_graph/util.rs`.
… target build.rs diagnostics (commit-date, enabled cfg flags) now use eprintln instead of cargo::warning, so normal builds are silent. The output is only visible with `cargo build -vv`. New `make build-info` target provides a convenient way to see which cfg flags build.rs detected; it touches build.rs to force a re-run (cargo caches build script results and suppresses stderr otherwise).
In nightlies >= 2025-01-28, stable MIR's Rvalue::AddressOf changed its first field from Mutability (Mut/Not) to RawPtrKind (Mut/Const/FakeForPtrMetadata). The old arms are gated behind #[cfg(not(smir_has_raw_ptr_kind))] and new arms behind #[cfg(smir_has_raw_ptr_kind)] in both mk_graph/util.rs and mk_graph/context.rs. Breakpoint date confirmed by binary search: nightly-2025-01-28 (commit-date 2025-01-27) is the last without the change; nightly-2025-01-29 (commit-date 2025-01-28) requires RawPtrKind.
…tions Two compat-layer breaks land between nightly-2025-01-25 and 2025-01-28: 1. RunCompiler struct removed from rustc_driver, replaced by a free function run_compiler() (commit-date 2025-01-24). Gated behind smir_has_run_compiler_fn in src/driver.rs. 2. MonoItemPartitions changed from a tuple (accessed via .1) to named fields (.codegen_units) (commit-date 2025-01-27). Gated behind smir_has_named_mono_item_partitions in src/compat/mono_collect.rs. Also gates the Mutability import in mk_graph/context.rs behind cfg(not(smir_has_raw_ptr_kind)) to avoid an unused-import warning on newer nightlies where AddressOf uses RawPtrKind instead. The codebase now builds cleanly on nightlies from 2024-11-29 through at least 2025-03-01.
The UI test lists (passing.tsv, failing.tsv) are tied to a specific
rustc commit. When the nightly changes, upstream test files get deleted,
renamed, or modified; the test runner dutifully tries to run them and
fails. The obvious fix (maintain a full copy of passing.tsv per nightly)
would mean 2888-line files that are 99.5% identical, with diffs that
tell you nothing useful.
So instead: base+delta.
The base lists stay as the canonical ground truth, generated against
nightly-2024-11-29 (now recorded in base-nightly.txt). A new script,
diff_test_lists.sh, diffs tests/ui/ in the rustc repo between the base
commit and a target commit, cross-references the results with the base
lists, and applies three kinds of mechanical changes:
1. Deletions: files removed upstream are dropped. Between nightly-2024-11-29
and nightly-2025-03-01, 9 passing tests disappeared (alias-uninit-value.rs,
artificial-block.rs, and friends).
2. Renames: files that moved get their paths updated. 5 tests were renamed
(e.g., assign-assign.rs -> codegen/assign-expr-unit-type.rs).
3. Manual overrides: behavior changes that git diffs can't detect go in
overrides/<nightly>.tsv. For nightly-2025-03-01, one test
(macro-metavar-expr-concat/repetitions.rs) needed a "skip" because
upstream rewrote it to use ${concat()} syntax that doesn't compile
through our driver.
The script has three modes: --report (human-readable summary), --chain
(incremental diffs between consecutive breakpoint nightlies, useful for
seeing exactly when each change landed), and --emit (writes effective
passing.tsv and failing.tsv to overrides/<nightly>/ for consumption by
run_ui_tests.sh).
run_ui_tests.sh now detects the active nightly via `rustup show
active-toolchain` and picks up the effective list from
overrides/<nightly>/ if it exists, falling back to the base list
otherwise. No flags, no env vars; it just works.
Both nightlies pass with zero failures:
nightly-2024-11-29: 2875/2875 (base list, 13 arch-skipped)
nightly-2025-03-01: 2865/2865 (effective list, 13 arch-skipped)
The old README predated the multi-nightly support; it described running scripts with cd and positional args that no longer matched the current interface. Rewritten to cover the directory layout, how the base+delta system works, the workflow for adding a new nightly, and the three modes of diff_test_lists.sh.
…ists Wraps diff_test_lists.sh --emit so you can generate effective UI test lists for a target nightly directly from make: RUST_DIR_ROOT=/path/to/rust make test-ui-emit NIGHTLY=nightly-2025-03-01
The build.rs breakpoint infrastructure now covers all the API changes
between nightly-2024-11-29 and nightly-2025-03-01, so we can finally
pin forward. This pulls in four months of rustc/stable-mir evolution:
- CoroutineClosure variant addition (2024-12-14)
- RunCompiler -> run_compiler() API change (2025-01-24)
- MonoItemPartitions tuple -> named fields (2025-01-27)
- AddressOf Mutability -> RawPtrKind (2025-01-28)
All of these are handled by cfg gates, so the codebase still compiles
against older nightlies too (verified against 2024-11-29).
Regenerated all 29 integration test golden files; the JSON output
changed substantially (mostly type representation and alloc layout
differences from upstream MIR changes). Also fixed 15 clippy
unneeded_struct_pattern warnings that the newer clippy now emits
for unit enum variant matches (e.g., Resume {} -> Resume).
…determinism
Three sources of cross-platform non-determinism in the jq normalisation
filter, surfaced by fn-ptr-in-arg failing on Linux CI:
1. Field projections embed a Ty index ({"Field": [field_idx, ty_id]})
that varies across platforms. The existing walk stripped .ty from
objects but missed these array-encoded IDs inside projection lists.
Now zeroed out during normalisation.
2. Items sorted with bare `sort` after hash truncation. The Rust-side
Ord (symbol_name!name) is deterministic on full hashes, but the jq
filter truncates hashes to strip non-deterministic suffixes; two
monomorphised Debug::fmt impls then share the same truncated
symbol_name and `sort` can't break the tie. Now using sort_by with
symbol_name + "|" + name, mirroring the Rust Ord key structure.
3. Interned `.id` fields (on MonoItemFn and const_ operands) vary
across platforms. Now stripped alongside `.def_id` in the global
walk pass.
cds-amal
force-pushed
the
cds/the-curious-case-of-stdlib
branch
from
ee2a477 to
72cf997
Compare
In nightly-2025-07-05, the `IndexedVal` trait (which provides `to_index()` and `to_val()` on opaque newtype wrappers like `Ty`, `Span`, `AllocId`, `VariantIdx`) became `pub(crate)`, making those methods inaccessible to external code. The fix is an adapter module (`src/compat/indexed_val.rs`) that provides free functions `to_index(&val)` and `to_val::<T>(idx)` with two cfg-gated implementations behind `smir_no_indexed_val`: Old nightlies: delegate straight to the trait methods (the trait is still public, so this is just a thin wrapper). New nightlies: the types are all `#[derive(Serialize)]` newtypes around `usize`, so `to_index` uses a minimal serde `Serializer` that intercepts the `serialize_newtype_struct -> serialize_u64` chain to extract the inner value. `to_val` goes the other way via `transmute_copy` with a compile-time size assertion (safe because the layout of a single-field newtype matches its field). Using rustc's vendored serde (not crates.io's) avoids version mismatch errors; the imports go through `super::serde::` which resolves to the `extern crate serde` in `compat/mod.rs`. Every call site across `mk_graph/` and `printer/` that previously called `.to_index()` or `Ty::to_val()` as trait methods now calls the free functions instead. The `types.rs` import is additionally gated on `#[cfg(feature = "debug_log")]` since all three uses live inside `debug_log_println!` macros that expand to nothing without the feature.
The `use super::stable_mir` import is only needed on old nightlies (where IndexedVal is public); gate it with `#[cfg(not(smir_no_indexed_val))]` to suppress the unused-import warning on nightly >= 2025-07-05.
Covers the build-time cfg detection strategy, the breakpoints matrix, three shim patterns (conditional match arm, mutually exclusive blocks, adapter module), and a step-by-step playbook for diagnosing new upstream breaks. Includes a section on the compounding payoff: the catch-up cost is front-loaded, steady-state maintenance is incremental, and backward compatibility across the supported range is a durable asset.
MIR output differs structurally across compiler versions (different span indices, reordered locals, changed lowering) in ways that the normalisation filter can't and shouldn't paper over. Every single golden file differs between nightly-2025-03-01 and nightly-2025-07-05. Move expected outputs from tests/integration/programs/*.expected to tests/integration/expected/<nightly>/*.expected. The Makefile detects the active nightly from rustc's commit-date and selects the matching golden directory, falling back to the pinned nightly's set when no specific directory exists. This means integration tests actually work across the supported nightly range, not just the pinned one: each nightly gets its own expected outputs, and `make golden` writes into the detected nightly's directory.
- Suppress unused ControlFlow from ty.visit() (return type changed from () to ControlFlow in newer nightlies) - Add explicit lifetime on TyCollector::new return type to satisfy mismatched_lifetime_syntaxes lint - cargo fmt import reordering
This is the first nightly where IndexedVal became pub(crate), exercising the serde/transmute adapter path in compat/indexed_val.rs. Golden files for this nightly are added alongside the existing nightly-2025-03-01 set. Both nightly-2025-03-01 and nightly-2025-07-05 pass all 29 integration tests with their respective golden files.
- nightly-compat.md: supported range diagram updated for 2025-07-05 as pinned nightly; added golden file workflow documentation and expanded quick reference table
…5-12-06) PointerCoercion::ReifyFnPointer changed from a unit variant to ReifyFnPointer(Safety) in commit-date 2025-12-05. The match in mir_visitor.rs needs cfg-gated arms for both shapes. New cfg flag: smir_has_reify_fn_pointer_safety (date: 2025-12-05).
Upstream changed the test source so it no longer compiles (rustc exit 101, not a driver bug). Add a skip override for both nightly-2025-11-19 and nightly-2025-12-06.
Nightly-2025-12-06 switched the default symbol mangling from legacy (_ZN...17h<hash>) to v0 (_R...Cs<hash>_). The v0 scheme embeds crate disambiguator hashes throughout the symbol (not just at the end), and these hashes differ between platforms (macOS vs Linux). The existing normalise filter only stripped the trailing 17 chars, which worked for legacy mangling but left embedded v0 hashes intact. Add a strip_hashes jq function that detects the mangling scheme and applies the appropriate normalization: gsub on C<base62>_ patterns for v0 symbols, trailing truncation for legacy.
…tly-2025-12-14) The display() method on FileName now takes RemapPathScopeComponents instead of FileNameDisplayPreference, which became module-private. Using RemapPathScopeComponents::all() as the scope to get the same remapped-path behavior as the old FileNameDisplayPreference::Remapped. New cfg flag: smir_no_filename_display_pref (date: 2025-12-13).
…25-12-23) Rvalue::NullaryOp and the NullOp enum were removed entirely in commit-date 2025-12-22. The runtime checks that were previously expressed as NullaryOp(NullOp::RuntimeChecks(_)) moved to a new Operand::RuntimeChecks(_) variant instead. New cfg flag: smir_no_nullary_op (date: 2025-12-22).
…ightly-2026-01-15
Span indices (like alloc_id, Ty, and def_id) are interned within a single rustc invocation but not stable across runs or platforms. This caused fn-ptr-in-arg and static-vtable-nonbuiltin-deref golden files to differ between macOS and Linux CI. Add del(.span) to the global walk in normalise-filter.jq, following the same pattern used for def_id and id. Regenerate all golden files across all 14 nightly directories.
The existing CI only builds and tests against the pinned nightly, which means a cfg-gating mistake (code that compiles on the pinned nightly but not on an older or newer one in the supported range) can slip through undetected until someone tries to use a different toolchain. This workflow discovers the nightly matrix dynamically from the golden-file directories under tests/integration/expected/; adding a new breakpoint nightly with its golden files is sufficient to include it in the matrix, no workflow edits needed. For each nightly it installs the toolchain, builds, and runs integration tests, with fail-fast disabled so one broken nightly doesn't mask failures elsewhere. Triggers: weekly schedule (Sunday 04:00 UTC), manual dispatch, and push to master. The current feature branch is temporarily included in the push trigger for testing; remove before merge.
The previous commit (a9ca4a3) regenerated golden files with span fields stripped, but the edit to normalise-filter.jq itself was lost during the rebase that squashed the two span-stripping commits together. The result: golden files had no spans, but the filter running on CI didn't strip them either, so every non-pinned nightly failed. This commit adds the missing del(.span) to the global walk and regenerates all 14 nightly golden file sets to match.
The normalise filter already stripped .ty (lowercase) fields and
replaced Field[1] Ty indices with placeholders, but missed two other
forms of interned Ty index that appear in the JSON output:
{"Type": <int>} - Ty index wrappers (e.g. inside Closure aggregates)
{"Array": <int>} - bare Ty index references (vs {"Array": {count,stride}})
These indices are consistent within a single rustc invocation but not
stable across platforms (macOS vs Linux), which caused fn-ptr-in-arg
and static-vtable-nonbuiltin-deref to fail on nightly-2025-12-06 and
nightly-2025-12-14 in the all-nightlies CI workflow.
Normalize both forms to 0 in the items walk (same pattern as Field[1]),
distinguishing the bare-integer Array case from the layout-descriptor
form. Regenerate all 14 nightly golden file sets.
…Adt arrays
Several Stable MIR constructs encode interned Ty or DefId indices as
bare integers at known array positions:
Cast[2] Ty index (target type)
Closure[0] DefId index
VTable[0] Ty index
Adt[0] AdtDef index
These are the same class of non-deterministic interned index that the
filter already handled for Field[1], {"Type": N}, and {"Array": N}, but
they survived normalization because they're positional integers in
arrays rather than keyed object fields. The result: fn-ptr-in-arg and
static-vtable-nonbuiltin-deref continued to fail on nightly-2025-12-06
and nightly-2025-12-14 where macOS and Linux interned these at
different indices.
Regenerate all 14 nightly golden file sets.
ADR-004 lays out the problem and the proposed fix. The normalise filter has been playing whack-a-mole with interned indices: every time upstream adds a new Ty or DefId field somewhere in the Body tree, the jq script needs a hand-written rule to strip it, and we only discover the gap when CI fails on the other platform. Three rounds of that in three commits was enough to motivate a structural solution. The receipts approach moves the schema knowledge (which values are interned) from the jq filter back to the Rust code, right where the types live. A spy serde Serializer observes the serialization and records which JSON paths carry interned indices; the normalise filter reads that receipt and applies it generically.
The printer now runs a spy serde Serializer over SmirJson before the real serialization pass. The spy tracks context (struct field names, enum variant names, tuple positions) and records every location where a known interned type (Ty, Span, AllocId, DefId, AdtDef, CrateNum, VariantIdx) appears. The result is written as a companion *.smir.receipts.json with three categories: interned_keys (struct fields like "span", "ty"), interned_newtypes (enum wrappers like "Type", "ClosureDef"), and interned_positions (tuple positions like Cast[2], Field[1]). The normalise filter doesn't consume these yet; that happens when this branch gets rebased onto the nightly-extension work where all the normalization rules already exist.
The normalise filter now reads the companion *.smir.receipts.json (emitted by the spy serializer) and applies three generic passes: interned_keys are deleted, interned_newtypes are zeroed, and interned_positions are zeroed. No more per-field jq rules for the body tree. All three receipt categories are pre-seeded with known interned paths, and the spy adds more dynamically as it discovers them. The seeding turns out to be critical: upstream's serialize_index_impl! macro provides a custom Serialize impl for interned types (Ty, Span, DefId, AllocId, etc.) that serializes them as bare integers via Serialize::serialize(&n, serializer), completely erasing the type name. The spy never sees serialize_newtype_struct or serialize_tuple_struct for these types; it just sees a u64. So the spy's dynamic discovery is blind to the most important interned types, and the seeded values carry all the weight. The spy still adds value for discovering interned paths in types that DO preserve their names through serde (e.g. GenericArgs, ClosureDef), but the baseline coverage comes from the seed lists.
…olden files VariantIdx is a structural index (variant 0 is always variant 0), not a compiler-session-interned value. Including it in INTERNED_TYPES caused the spy to record Adt[1] as an interned position on older nightlies (where serialize_index_impl! doesn't erase the type name), zeroing real variant discriminants in the normalized output. The global walk also now correctly normalizes interned indices inside allocs (e.g. VTable[0] in global_alloc entries), which the old items-only walk missed. Golden files for all nightlies regenerated to reflect both changes.
Each nightly job now also derives the rustc commit, shallow-clones rust-lang/rust at that commit, and runs `make test-ui`. This mirrors the existing UI test setup in test.yml but runs it across the full supported nightly range.
…ions The awk parser was missing a few cases that caused 22 UI test failures: - `needs-subprocess` tests fork/exec the compiled binary, but we run with `-Zno-codegen` so there is no binary to execute. These now skip unconditionally (accounts for 11 of the 22 failures). - `extern crate libc` tests fail with E0464 (multiple candidates for `libc`) because our sysroot ships both .rmeta and .rlib. Detected as a standalone pattern before the directive block so it catches the import regardless of surrounding directives. - Range edition syntax (e.g., `edition:2015..2021`) was being passed literally as `--edition 2015..2021`, which rustc rejects. The parser now extracts the earliest edition in the range, since every test in the range must compile with the earliest and later editions may reject deprecated syntax the test exercises. - Edition values now get validated: must be a 4-digit year or "future". Unrecognized values emit a WARNING to stderr (signal that something upstream may have changed). Unit tests cover all new cases including boundary notes.
…ate overrides Both `remake_ui_tests.sh` and `diff_test_lists.sh` now use the shared awk directive parser for skip/flag logic, so the filtering is consistently applied regardless of which tool generates or validates the test lists. `remake_ui_tests.sh` previously had its own inline `extract_test_flags()` function that only handled compile-flags, edition, and rustc-env (no skip logic at all). It now mirrors `run_ui_tests.sh`: host detection, foreign arch path filtering, and the full directive parser. Tests that would be skipped at runtime are skipped during list generation too. `diff_test_lists.sh --emit` now post-filters the effective passing list through the directive parser (reading each test file via `git show` at the target commit). This catches tests that exist in the diff but would be skipped at runtime (needs-subprocess, extern crate libc, etc.). All 12 nightly overrides regenerated with the updated filter: each loses ~87 tests that were previously included but would have been skipped or failed at runtime.
…empty lists Three CI issues from the last run, all in the UI test step: - nightly-2025-07-05 and nightly-2025-07-08 had integration test golden files but no UI test overrides. Without overrides, run_ui_tests.sh fell back to the base passing list (nightly-2024-11-29), which doesn't match the rust source tree at those commits: nearly every test failed with exit 101. Generated proper overrides via diff_test_lists.sh --emit. - nightly-2025-10-03's passing.tsv was zeroed by the earlier bulk regeneration run (the one that was interrupted). Regenerated: 2669 entries, matching what the directive-filtered base produces for that nightly. - run_ui_tests.sh now dies if an override exists but is empty (catches the 10-03 scenario), and warns when no override exists for a non-base nightly (catches the 07-05/07-08 scenario). Neither condition should silently proceed with a mismatched test list.
The directive parser now supports a `universal` flag that disables all platform-specific skip logic (only-<os>, only-<arch>, ignore-<os>, etc.) while still applying unconditional skips (needs-sanitizer, needs-subprocess, extern crate libc). diff_test_lists.sh --emit uses this mode so the generated overrides are correct regardless of which host produces them: generating on macOS/aarch64 now yields the same lists as generating on Linux/x86_64. Platform-specific filtering continues to happen at runtime in run_ui_tests.sh, which already had the full directive parser and arch-path filter. Since the diff between two git commits is immutable, --emit now caches: if a non-empty override already exists for a nightly, it skips generation entirely. Use --force to regenerate. The committed overrides themselves serve as the cache, so CI never recomputes them. All 14 nightly overrides regenerated. Counts are slightly higher than before (~30 more per nightly) because platform-specific tests are no longer pre-filtered; they get skipped at runtime instead.
cds-amal
force-pushed
the
cds/the-curious-case-of-stdlib
branch
from
3705a5b to
7a33f18
Compare
…y path
Nine tests fail on nightly-2025-07-{05,08} due to upstream source changes
between the base nightly and these dates (unsized_locals feature gate
removal, deref_patterns rework, remap-path-prefix-macro rewrite). These
are compile errors in the test source, not driver bugs. Added manual
override TSVs to skip them, matching the existing overrides for 07-11+.
Verified locally by building both nightlies in parallel (separate
--target-dir per toolchain) and running the full test suite with
pre-built binaries: 0 failures on both.
Also fixed run_ui_tests.sh to set DYLD_LIBRARY_PATH / LD_LIBRARY_PATH
when RUN_SMIR is provided. Without this, the pre-built binary can't
find rustc's dylibs and every test aborts with exit 134.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a pure spike branch .. it won't be merged.
Extending nightly support: 2024-11-29 through 2025-10-03
stable-mir-json can now generate
smir.jsonfor Rust's entire standard library.make stdlib-smirbuilds the driver, creates a throwaway crate, runscargo -Zbuild-stdthrough it, and collects the output: 21 artifacts (~100MB total) includingstd(44MB, ~12,000 items),core(16MB),alloc,proc_macro, and their transitive dependencies. This works against any nightly in the supported range; the stdlib's MIR is now a first-class output of this tool.Getting there required pushing the supported nightly range from a single pinned toolchain to an 11-month window spanning 11 breakpoints, including the
rustc_publiccrate rename. The nightly pin moves from2024-11-29to2025-10-03. Any nightly in the supported range compiles from this branch:RUSTUP_TOOLCHAIN=nightly-YYYY-MM-DD cargo build, done.The work falls into two halves: the compat layer itself (absorbing 11 upstream API changes via cfg-gated code), and the infrastructure that makes extending the range sustainable (admin scripts, per-nightly test artifacts, a base+delta UI test list system). The infrastructure existed in earlier commits on this branch; this PR extends and exercises it.
Nightly compatibility
The supported range now covers 11 breakpoints across 11 months of upstream stable MIR API evolution:
smir_has_coroutine_closureAggregateKind::CoroutineClosurevariant addedmk_graph/util.rssmir_has_run_compiler_fnRunCompilerstruct replaced byrun_compiler()free fndriver.rssmir_has_named_mono_item_partitionscollect_and_partition_mono_itemsreturn changed from tuple to named fieldscompat/mono_collect.rssmir_has_raw_ptr_kindRvalue::AddressOffirst field changed fromMutabilitytoRawPtrKindmk_graph/util.rs,mk_graph/context.rssmir_no_indexed_valIndexedValtrait becamepub(crate)compat/indexed_val.rs(adapter module); allmk_graph/andprinter/call sites use shimsmir_rustc_internal_movedrustc_internal::{internal,stable,run}moved fromrustc_smirtostable_mircompat/mod.rs(cfg-gated re-export),driver.rs(cfg-gated import)smir_has_global_alloc_typeidGlobalAlloc::TypeId { ty }variant addedmk_graph/index.rs,printer/collect.rs,printer/mir_visitor.rssmir_crate_renamedstable_mir->rustc_public,rustc_smir->rustc_public_bridgecompat/mod.rs,driver.rs(cfg-gatedextern cratealiases)smir_no_coroutine_movabilityMovabilityremoved fromCoroutinevariantsprinter/types.rs,mk_graph/util.rssmir_no_dyn_kindDynKindremoved fromTyKind::Dynamic(3 fields to 2)printer/types.rssmir_no_projection_subtypeProjectionElem::Subtypemoved toCastKind::Subtypemk_graph/util.rsThe
rustc_publiccrate rename (2025-07-14) turned out not to be an epoch boundary: cfg-gatedextern crate rustc_public as stable_miraliases handle it cleanly, with zero downstream code changes. The supported range extends well past the rename, through three additional breakpoints.Each breakpoint date was verified by fetching the nightly manifest from
static.rust-lang.org, extracting the backing commit, and runninggit merge-base --is-ancestoragainst the suspect change commit. The full playbook is documented indocs/nightly-compat.md.Nightly admin tooling
Adding support for a new nightly previously meant a manual multi-step dance: install the toolchain, build against it, generate integration test golden files, compute effective UI test lists from the base+delta system, create a manual override TSV from the nearest existing one, add the nightly to the DEFAULT_NIGHTLIES list in
diff_test_lists.sh, then updaterust-toolchain.tomlandREADME.md. Each step has its own incantation and it's easy to forget one (particularly the UI test list generation, which silently falls back to the base list if no effective list exists for the active nightly).scripts/nightly_admin.pycodifies this into three composable subcommands:make nightly-add NIGHTLY=... RUST_DIR_ROOT=...make golden), generate effective UI test lists (diff_test_lists.sh --emit), create override TSV from template, add to DEFAULT_NIGHTLIESmake nightly-check NIGHTLY=... RUST_DIR_ROOT=...make nightly-bump NIGHTLY=...rust-toolchain.tomlchannel andREADME.mdpinned nightly callout; warns if golden files or UI test lists are missing;--dry-runsupportedThe script delegates to existing Makefile targets, forcing the target nightly via the
RUSTUP_TOOLCHAINenv var so allrustc/cargoinvocations in the process tree use the right toolchain. The golden file generation runsmake golden(which invokes the driver against each test program and writes normalized output totests/integration/expected/<nightly>/). The UI test list generation runsdiff_test_lists.sh --emit, which diffstests/ui/in the rust repo between the base commit and the target nightly's backing commit, applies deletions and renames to the base passing/failing lists, layers on manual overrides from the override TSV, and writes the effective lists totests/ui/overrides/<nightly>/.Stdlib-only Python 3.9+; no external dependencies.
Per-nightly test artifacts
Integration test golden files are stored per-nightly under
tests/integration/expected/<nightly>/. Eight sets exist:The Makefile auto-detects the active nightly and selects the matching directory, falling back to the pinned nightly's set. Adding golden files for a new nightly is just
RUSTUP_TOOLCHAIN=nightly-YYYY-MM-DD make golden.UI test lists use a base+delta system:
diff_test_lists.shcomputes effective passing/failing lists from a base set (~2,889 tests) plus git diffs and manual overrides. Effective lists exist for six nightlies, with passing counts ranging from 2,879 (nightly-2025-03-01) down to 2,756 (nightly-2025-10-03) as upstream removes and rewrites tests.The
build.rscfg detection mechanismbuild.rsrunsrustc -vV, extracts thecommit-date, and compares it against the BREAKPOINTS table. For each breakpoint whose date is at or before the active compiler's commit-date, it emits acargo:rustc-cfgflag. Consumer code gates match arms with#[cfg(smir_has_*)]/#[cfg(not(smir_has_*))].Cfg names are declared via
cargo:rustc-check-cfg, so rustc never warns aboutunexpected_cfgson any nightly. Exhaustiveness is preserved on every supported nightly: without the flag, the gated arm is excluded and the match stays exhaustive; with it, the arm covers the new variant. No#[allow(unreachable_patterns)]needed.Three shim patterns recur across the breakpoints:
#[cfg]/#[cfg(not)]compat/indexed_val.rs)The full pattern catalogue with worked examples lives in
docs/nightly-compat.md.Other changes in this PR
The rustc layout panic (section 1 in prior PR versions):
ty.layout()panics in rustc when encountering types with escaping bound vars. Workaround:catch_unwindwith hook suppression, recording panics for summary reporting.make stdlib-smir: single command to generatesmir.jsonfor Rust's standard library via-Zbuild-std. Produces 21 artifacts (~100MB) includingstd,core,alloc.Single source of truth for toolchain version: everything derived from
rust-toolchain.toml'schannelfield; no manual commit caching.Integration test normalisation hardening: three fixes to
normalise-filter.jqfor cross-platform determinism (Field projection Ty IDs, item sort stability, interned.idfields).make fmtandmake clippytargets: independently callable, delegated frommake style-check.Tests
make integration-testpasses (29/29) with per-nightly golden files for eight nightliesmake test-uipasses (2,756/2,756 against nightly-2025-09-19)make test-directivespasses (46/46)make stdlib-smirproduces 21 valid artifactsRUSTUP_TOOLCHAIN=nightly-2024-11-29 cargo buildsucceedsRUSTUP_TOOLCHAIN=nightly-2025-10-03 cargo buildsucceedsTodo