upgrade ACTS from v36 to v47

[bloodyyugo]

Upgrade ACTS versions for 36-->47. Closes #2100

---

Algorithm changes that may effect tracks:

• CKF branch handling
  The CKF received improvements to outlier rejection and branch pruning
  across these 11 major versions. This can shift the
  efficiency/fake-rate tradeoff even with identical inputs.

• GainMatrixSmoother → smoothTrack()
  The Kalman backward (RTS) smoother was rewritten as a free function.
  The math is identical in principle, but the implementation details
  (handling of degenerate states, track-end boundary conditions, numerical
  precision) may differ. Most likely source of small systematic shifts in
  momentum resolution or pull distributions.

• EigenStepper / RKN propagator
  Step-size control and Jacobian transport have had stability improvements.
  Effects appear mainly in track-to-truth residuals for long extrapolations
  (e.g., tagger perigee → ECAL scoring plane).

• GSF Bethe-Heitler handling
  AtlasBetheHeitlerApprox was renamed PolynomialBetheHeitlerApprox — the
  polynomial coefficients are the same. However, the GSF received active
  development between v36 and v47 in component reduction strategies, weight
  re-normalization, and handling of near-degenerate Gaussian mixtures. For
  LDMX, where the GSF drives electron momentum measurement, this is the
  highest-risk area for physics-level differences vs. trunk.

---

Interface changes etc:

• In v46, the calibrator and measurementSelector
  delegates moved out of CombinatorialKalmanFilterExtensions into a new
  Acts::TrackStateCreator object. The SourceLinkAccessorDelegate also moved there.

• DetectorElementBase -> Acts::SurfacePlacementBase.

• transform() override -> localToGlobalTransform() (matches new base class).

• surface->assignDetectorElement() -> surface->assignSurfacePlacement().

• thickness() is no longer a pure virtual in the base; override removed.

• isSensitive() added as a required override (returns true for sensor elements).

• Acts::ViewConfig now uses C++ designated initializers: {.color = {r, g, b}}.

• Added tagger_start_surface_ at ACTS x=-617mm (1mm inside the tagger
  volume outer boundary at x=-618mm, 1.5mm upstream of the L1 doublet at
  x=-615.5mm).

• Recoil tracking volume extended upstream so its low-x edge is at x=-1mm,
  placing the target plane (x=0) clearly inside the volume for the CKF Navigator
  (previously the target was on or outside the volume boundary, causing Navigator
  failures for recoil tracks).

• BetheHeitlerApprox changed from a local stack object to a shared_ptr:
  Old: BetheHeitlerApprox bh = Acts::makeDefaultBetheHeitlerApprox();
  New: auto bh = std::make_sharedActs::AtlasBetheHeitlerApprox(
  Acts::makeDefaultBetheHeitlerApprox());

• GsfOptions.propagatorPlainOptions now requires an explicit cast:
  gsf_options.propagatorPlainOptions =
  static_castActs::PropagatorPlainOptions(propagator_options);

• A separate GsfExtrapPropagator (EigenStepper + VoidNavigator) was added for
  post-fit track extrapolations (to target, ECAL). VoidNavigator is required
  because the ECAL scoring plane is outside the tracking geometry volume and the
  standard Navigator would stop propagation at the volume boundary.

• GainMatrixSmoother removed; Acts::smoothTrack() used instead, with its return
  value checked.

• Acts::FreeSquareMatrix renamed (TrackingUtils.cxx)
  Old: Acts::FreeSquareMatrix
  New: Acts::FreeMatrix
  (defined in Acts/Definitions/TrackParametrization.hpp, same as before)

• Acts::GeometryContext default constructor made private
  The no-arg Acts::GeometryContext() constructor is private in v47. Code that
  held Acts::GeometryContext as a member and relied on implicit default
  initialization would fail to compile.

[tomeichlersmith]

For LDMX, where the GSF drives electron momentum measurement, this is the
highest-risk area for physics-level differences vs. trunk.

Do you plan to do an initial study of these differences and post them here before the PR is merged? Or do you think it is too detailed (and the differences are likely improvements) so it will wait for future work?

[tomeichlersmith]

While I like this fallback in order to support older versions of the image, I think this PR should have a companion PR where the development image is updated to Acts v47. Is it backwards compatible? Could Acts v47 be used to compile what's on trunk right now?

[tomeichlersmith]

(These questions will just motivate how we document the development image and not necessarily whether we update the Acts within it.)

[bloodyyugo]

I've just been looking at the validation plots (and really just the signal sample). The biggest differences I see:

• tagger: the uncertainties for d0/z0/phi are smaller with v47, residuals the same, so the pulls are also broader. I don't think this is real...it's probably due to the change in the "origin" point which was moved closer so that it is in tracker volume. The pulls at the target X/Y pass the KS tests (though X vs n_hits fails but for some reason this plot only goes up to 12 hits, which must just be a bug).

• recoil: we get KS failurs for some dup and fake plots and this is expected (though honestly they look ok to me). Also see failures for target position X (both truth and reco?) but they are pretty close by eye. The locations projected to the ecal fail and look bad (even the gold plots look bad, now they look worse). We will have to figure this out.

• GSF: apart from the changes that feed down from the CKF tracks, the q/p error is much larger than before which effects the pulls (I think making them better).

• ECal Tracking! I forget to look at these earlier! There are quite a few failures in EcalSPTrackCompare (particularly the deltas) and EcalTrackAnalyzer. Most of the plots fail actually.

[bloodyyugo]

On your question on the image, no trunk as of now will not compile with v47...interface and class-name changes. What is the right thing to do here? I can't remember what we did when we updated to v36....

[tomeichlersmith]

It will just be nice to document it. If the answer is that the developer needs to either (a) merge their updates with the updated trunk or (b) pin their environment on an old image, that's okay I just wanted to know.

[github-actions]

Validation Results

Some validation samples failed! ❌

Sample	Status
cascade_history	✅ PASS
deep_ecal_gun	✅ PASS
hcal	✅ PASS
eat_signal	❌ FAIL (46 histograms failed KS test) (artifact)
ecal_pn	❌ FAIL (135 histograms failed KS test) (artifact)
inclusive	❌ FAIL (200 histograms failed KS test) (artifact)
it_pileup	❌ FAIL (271 histograms failed KS test) (artifact)
kaon_enhanced	❌ FAIL (115 histograms failed KS test) (artifact)
reduced_ldmx	❌ FAIL
signal	❌ FAIL (221 histograms failed KS test) (artifact)
signal_target_al	❌ FAIL (452 histograms failed KS test) (artifact)
target_genie	❌ FAIL (110 histograms failed KS test) (artifact)
target_pn_lyso	❌ FAIL (323 histograms failed KS test) (artifact)
target_ti_en	❌ FAIL (138 histograms failed KS test) (artifact)
wab_lhe	❌ FAIL (101 histograms failed KS test) (artifact)

eat_signal:

• 46 plots failed the KS test against gold.
• Text Differences Between Logs (9923 lines differ)
• Log character count differs by 61%: gold=1140607, new=1844041
• Timing for eat_signal: gold=1618s, new=1576s
• Timing within 2% of gold (tolerance 10%)

ecal_pn:

• 135 plots failed the KS test against gold.
• Text Differences Between Logs (55772 lines differ)
• Log character count differs by 1%: gold=24115214, new=24548294
• Timing for ecal_pn: gold=6718s, new=6702s
• Timing within 0% of gold (tolerance 10%)

inclusive:

• 200 plots failed the KS test against gold.
• Text Differences Between Logs (61584 lines differ)
• Log character count differs by 2%: gold=21025892, new=21578617
• Timing regression for inclusive: new=6643s vs gold=3573s (+85%, tolerance 10%)
• Timing for inclusive: gold=3573s, new=6643s

it_pileup:

• 271 plots failed the KS test against gold.
• Text Differences Between Logs (24971 lines differ)
• Log character count differs by 9%: gold=9752960, new=8841824
• Timing for it_pileup: gold=599s, new=639s
• Timing within 6% of gold (tolerance 10%)

kaon_enhanced:

• 115 plots failed the KS test against gold.
• Text Differences Between Logs (26747 lines differ)
• Log character count differs by 1%: gold=10113224, new=10302254
• Timing for kaon_enhanced: gold=3095s, new=3012s
• Timing within 2% of gold (tolerance 10%)

signal:

• 221 plots failed the KS test against gold.
• Text Differences Between Logs (55974 lines differ)
• Log character count differs by 2%: gold=16413683, new=16778912
• Timing for signal: gold=1189s, new=1205s
• Timing within 1% of gold (tolerance 10%)

signal_target_al:

• 452 plots failed the KS test against gold.
• Text Differences Between Logs (46203 lines differ)
• Log character count differs by 2%: gold=16496870, new=16986051
• Timing for signal_target_al: gold=936s, new=942s
• Timing within 0% of gold (tolerance 10%)

target_genie:

• 110 plots failed the KS test against gold.
• Text Differences Between Logs (1895 lines differ)
• Log character count differs by 7%: gold=1406660, new=1513713
• Timing regression for target_genie: new=9736s vs gold=5707s (+70%, tolerance 10%)
• Timing for target_genie: gold=5707s, new=9736s

target_pn_lyso:

• 323 plots failed the KS test against gold.
• Text Differences Between Logs (40675 lines differ)
• Log character count differs by 2%: gold=20168222, new=20743639
• Timing for target_pn_lyso: gold=6341s, new=6324s
• Timing within 0% of gold (tolerance 10%)

target_ti_en:

• 138 plots failed the KS test against gold.
• Text Differences Between Logs (877 lines differ)
• Log character count differs by 1%: gold=393174, new=398437
• Timing for target_ti_en: gold=2738s, new=2776s
• Timing within 1% of gold (tolerance 10%)

wab_lhe:

• 101 plots failed the KS test against gold.
• Text Differences Between Logs (19130 lines differ)
• Log character count differs by 4%: gold=13180440, new=12553398
• Timing for wab_lhe: gold=6754s, new=6830s
• Timing within 1% of gold (tolerance 10%)

cascade_history:

• Text Differences Between Logs (2 lines differ)
• Timing for cascade_history: gold=6217s, new=5981s
• Timing within 3% of gold (tolerance 10%)

deep_ecal_gun:

• Text Differences Between Logs (24 lines differ)
• Timing for deep_ecal_gun: gold=3763s, new=4129s
• Timing within 9% of gold (tolerance 10%)

hcal:

• Text Differences Between Logs (2 lines differ)
• Timing regression for hcal: new=853s vs gold=665s (+28%, tolerance 10%)
• Timing for hcal: gold=665s, new=853s

[bloodyyugo]

The most recent commit fixes the Ecal tracking. There were two issues, first the starting surface needed to be in the tracking volume (same thing we saw for recoil and tagger); also the surfaces given to ACTs didn't have a "sensitive" surface defined, so the tracking didn't know which surfaces the hits were supposed to be on (and what it should be swimming track candidates to).

So now we see tracks for ECal tracking...a few of the plots still fail the validation tests but looking at them they all look very close.

For the trackers, the only thing that sticks out to me is the ecal_pulls_of_loc0 (so, x position pull at the ecal). It's much broader for acts47 than the trunk. I've attached the plot RecoilGASDQM_ecal_Pulls_of_loc0.pdf.

Note, I think it's best to look at the "GAS" plots (i.e. after getting rid of highly redundant tracks). The raw ones, e.g. "RecoilDQM" I expect to be different because the changes to "outlier rejection and branch pruning" while there have been many changes to the GSF algorithm. For the recoil GAS tracks, only plots related to the loc0 pulls fail. For the tagger, the errors on the track parameters (not the parameter values themselves) change a bit so the _error and _pull plots fail the KS tests. It is a bit weird that this only shows up in the tagger and not the recoil though.