Consolidated Raul comparison notebook: main vs #12, and the pole regression it found

[christianhbye]

Summary

One notebook, notebooks/sim_comparisons/raul_comparison.ipynb, replaces the three sim_comparison* notebooks. It compares Raul's reference simulator against mistsim under two environments: main before #12 (croissant 5.1.4) and #12 (croissant v5.3.0.dev2). The same cells run under two Jupyter kernels, each run caches its waterfalls under a tag derived from its croissant revision, and the comparison section plots every cached run it finds.

Impact: #12 introduced a large regression at latitude ±90, and it is now on main. Result (mean |mistsim − Raul|, and the arch ratio of mid-day to edge residual):

test	main (5.1.4)	#12 (v5.3.0.dev2)
MARS, with mountains	2.356 K, arch 4.01	1.303 K, arch 1.42
MARS, no mountains	2.260 K, arch 7.23	0.921 K, arch 1.15
North Pole, no mountains	4.389 K, arch 2.49	117.527 K, arch 1.00

• MARS improves on both metrics. The mean residual roughly halves, and the mid-day peak in the residual goes away, which is the time signature the pole-of-date change was meant to remove.
• The North Pole is 27× worse, and the offset is constant. A cyclic LST-lag scan brings the Move onto croissant v5.3.0.dev2 and its s2fft fork; put sky alm in the Earth frame of date #12 residual down to 1.65 K at a lag of about +66°, so the simulation is right except for a fixed azimuth rotation.
• Cause: croissant, filed as [BUG] ZYZ Euler extraction is ill-conditioned at beta->0: ~133 deg azimuth error at the geographic poles christianhbye/croissant#152. get_rot_mat returns a slightly non-orthogonal matrix (‖RRᵀ−I‖ ≈ 1e-4, from direction-dependent aberration). rotmat_to_eulerZYZ amplifies that by 1/sin β. Move onto croissant v5.3.0.dev2 and its s2fft fork; put sky alm in the Earth frame of date #12's CIRS frame puts the zenith within 20″ of the pole at ±90, so β collapses and the error becomes large. This was measured directly from croissant's rotation matrices, separately from the table. The table itself compares environments that also differ in s2fft and jax, so it cannot attribute anything to a single cause.
• Exposure on main. runs.yaml defines a southpole site at −90.0. No current run list uses its beams, so no stored mapmaking result is affected. data/synthetic_data/southpole-*.npz date from April, under croissant 5.1.x. That J2000 frame sat ~428″ off the zenith, which kept the error small. Measured at the runs.yaml start time and latitude −90, the error is 1.4° under croissant 5.1.4 and 38.5° under Move onto croissant v5.3.0.dev2 and its s2fft fork; put sky alm in the Earth frame of date #12. Do not regenerate southpole data until #152 is fixed.

The mapmaking results for the other sites are unaffected. Data and forward model share one Simulator, so frame changes cancel: a re-run of mars-lmax40 moved relative reconstruction error by 0.006%.

Changes

• notebooks/sim_comparisons/raul_comparison.ipynb (new): setup and environment tagging, Raul's reference, sky and beams, the three cached simulations, the comparison figures and summary table, a results cell stating the conclusion, and a disabled beam_az_rot=90 convention check.
• notebooks/sim_comparisons/results/.gitignore: cached waterfalls are regenerable and stay out of git.
• Deleted sim_comparison.ipynb, sim_comparison_azrot0.ipynb and sim_comparison_azrot90.ipynb. The new notebook covers them, since BEAM_AZ_ROT is a parameter. They remain in history.
• docs/superpowers/specs/…-design.md, docs/superpowers/plans/….md: the design and implementation plan.

Reproducing

Cell 0 contains the setup for the second kernel:

git worktree add ../mistsim-main daa5e7b   # last commit before #12
cd ../mistsim-main && uv sync --all-extras --dev
uv run python -m ipykernel install --user --name mistsim-cro514 \
    --display-name "mistsim (croissant 5.1.4)"

Run the notebook under that kernel first, then under the project kernel. Now that #12 is merged, main needs to be the pre-#12 commit (daa5e7b) for the old environment. The inputs under data/ are gitignored, as they already were in this repo.

Verification

• uv run pytest: 111 passed on this exact tree.
• The notebook runs end to end under both kernels, with distinct cache tags (5.1.4, 5.2.1+gitd972c5f). data/beam.npz and data/feko_beam.npz have the same md5 before and after.
• Each implementation task had a spec and quality review, plus a final whole-branch review.

🤖 Generated with Claude Code

https://claude.ai/code/session_012qDDdFg7x9CVkj4EGVT8AE