Tracker: pedestal subtraction and waveform building

[mgignac]

I am updating ldmx-sw, here are the details:

• Pedestal subtraction: dedicated class to perform subtraction. Currently relying on json inputs, but designed to be flexible enough to adapt in the future. Example script on how to generate the pedestals. Upon subtraction, data structure is the same as raw inputs, namely RawSiStripHits. In currently workflow, these outputs are not written to disk, but easily can be toggled.
• Waveform building: assemble all apv triggers into sequential, time-ordered waveform. Add helper scripts to plot waveforms. Simple minded cuts are applied to all waveforms before writing them to disk. This is something that will change once the waveform analysis is integrated into downstream digi+clustering steps. That is planned for a follow-up PR. A example sequence to decode ESA25 test-beam data to waveforms is added.
• Add header to EcalHelper -- had some compile time issues without.

Progress was discussed in July 16th tracker tracking meeting:
https://indico.fnal.gov/event/74596/#1-test-run-data-readout-and-pu

What are the issues that this addresses?

Check List

• I successfully compiled ldmx-sw with my developments.
• I read, understood and follow the coding rules.

• I ran my developments and the following shows that they are successful.

[tvami]

To fix the DarkBrem:

1. sync the submodule reference with trunk

git checkout trunk -- SimCore/G4DarkBreM

2. commit this change

git commit -m "checkout trunk -- SimCore/G4DarkBreM"

3. update the local clone of the submodule to be the trunk reference

git submodule update

4. push

git push

[bloodyyugo]

This works!

[tvami]

Thanks! Let's fix the darkbrem subpackage and then test and after that I'm happy to approve

[tvami]

/run-validation

[github-actions]

The validation workflow is running here: https://github.com/LDMX-Software/ldmx-sw/actions/runs/29848666303.

[github-actions]

Validation Results

Some validation samples failed! ❌

Sample	Status
cascade_history	✅ PASS
deep_ecal_gun	✅ PASS
ecal_pn	✅ PASS
hcal	✅ PASS
inclusive	✅ PASS
it_pileup	✅ PASS
kaon_enhanced	✅ PASS
reduced_ldmx	✅ PASS
signal	✅ PASS
signal_target_al	✅ PASS
target_genie	✅ PASS
target_pn_lyso	✅ PASS
target_ti_en	✅ PASS
wab_lhe	✅ PASS
eat_signal	❌ FAIL (64 histograms failed KS test) (artifact)

eat_signal:

• 64 plots failed the KS test against gold.
• Text Differences Between Logs (20008 lines differ)
• Log character count differs by 0% (gold=1140607, new=1131507); within tolerance
• Timing anomaly for eat_signal: new=1322s vs gold=1618s (-18%, tolerance 10%)
• Timing for eat_signal: gold=1618s, new=1322s

cascade_history:

• Text Differences Between Logs (2 lines differ)
• Timing for cascade_history: gold=6217s, new=6221s
• Timing within 0% of gold (tolerance 10%)

deep_ecal_gun:

• Text Differences Between Logs (22 lines differ)
• Timing for deep_ecal_gun: gold=3763s, new=4087s
• Timing within 8% of gold (tolerance 10%)

ecal_pn:

• Text Differences Between Logs (40 lines differ)
• Timing anomaly for ecal_pn: new=5902s vs gold=6718s (-12%, tolerance 10%)
• Timing for ecal_pn: gold=6718s, new=5902s

hcal:

• Text Differences Between Logs (2 lines differ)
• Timing regression for hcal: new=816s vs gold=665s (+22%, tolerance 10%)
• Timing for hcal: gold=665s, new=816s

inclusive:

• Text Differences Between Logs (52 lines differ)
• Timing regression for inclusive: new=6714s vs gold=3573s (+87%, tolerance 10%)
• Timing for inclusive: gold=3573s, new=6714s

it_pileup:

• Text Differences Between Logs (44 lines differ)
• Timing for it_pileup: gold=599s, new=558s
• Timing within 6% of gold (tolerance 10%)

kaon_enhanced:

• Text Differences Between Logs (30 lines differ)
• Timing for kaon_enhanced: gold=3095s, new=3050s
• Timing within 1% of gold (tolerance 10%)

reduced_ldmx:

• Text Differences Between Logs (22 lines differ)
• Timing for reduced_ldmx: gold=144s, new=150s
• Timing within 4% of gold (tolerance 10%)

signal:

• Text Differences Between Logs (48 lines differ)
• Log character count differs by 0% (gold=16413683, new=16413682); within tolerance
• Timing anomaly for signal: new=950s vs gold=1189s (-20%, tolerance 10%)
• Timing for signal: gold=1189s, new=950s

signal_target_al:

• Text Differences Between Logs (22 lines differ)
• Timing for signal_target_al: gold=936s, new=956s
• Timing within 2% of gold (tolerance 10%)

target_genie:

• Text Differences Between Logs (232 lines differ)
• Timing regression for target_genie: new=8871s vs gold=5707s (+55%, tolerance 10%)
• Timing for target_genie: gold=5707s, new=8871s

target_pn_lyso:

• Text Differences Between Logs (36 lines differ)
• Timing for target_pn_lyso: gold=6341s, new=6470s
• Timing within 2% of gold (tolerance 10%)

target_ti_en:

• Text Differences Between Logs (40 lines differ)
• Timing regression for target_ti_en: new=3103s vs gold=2738s (+13%, tolerance 10%)
• Timing for target_ti_en: gold=2738s, new=3103s

wab_lhe:

• Text Differences Between Logs (28 lines differ)
• Timing for wab_lhe: gold=6754s, new=7201s
• Timing within 6% of gold (tolerance 10%)

[github-actions]

Validation Results

All validation samples passed! ✅

Sample	Status
cascade_history	✅ PASS
deep_ecal_gun	✅ PASS
eat_signal	✅ PASS
ecal_pn	✅ PASS
hcal	✅ PASS
inclusive	✅ PASS
it_pileup	✅ PASS
kaon_enhanced	✅ PASS
reduced_ldmx	✅ PASS
signal	✅ PASS
signal_target_al	✅ PASS
target_genie	✅ PASS
target_pn_lyso	✅ PASS
target_ti_en	✅ PASS
wab_lhe	✅ PASS

cascade_history:

• Text Differences Between Logs (2 lines differ)
• Timing for cascade_history: gold=6217s, new=5983s
• Timing within 3% of gold (tolerance 10%)

deep_ecal_gun:

• Text Differences Between Logs (16 lines differ)
• Timing for deep_ecal_gun: gold=3763s, new=4155s
• Timing within 10% of gold (tolerance 10%)

eat_signal:

• Timing for eat_signal: gold=1618s, new=1596s
• Timing within 1% of gold (tolerance 10%)

ecal_pn:

• Text Differences Between Logs (30 lines differ)
• Timing for ecal_pn: gold=6718s, new=6706s
• Timing within 0% of gold (tolerance 10%)

hcal:

• Text Differences Between Logs (2 lines differ)
• Timing regression for hcal: new=821s vs gold=665s (+23%, tolerance 10%)
• Timing for hcal: gold=665s, new=821s

inclusive:

• Text Differences Between Logs (52 lines differ)
• Timing regression for inclusive: new=6600s vs gold=3573s (+84%, tolerance 10%)
• Timing for inclusive: gold=3573s, new=6600s

it_pileup:

• Text Differences Between Logs (40 lines differ)
• Timing for it_pileup: gold=599s, new=567s
• Timing within 5% of gold (tolerance 10%)

kaon_enhanced:

• Text Differences Between Logs (28 lines differ)
• Timing for kaon_enhanced: gold=3095s, new=3077s
• Timing within 0% of gold (tolerance 10%)

reduced_ldmx:

• Text Differences Between Logs (14 lines differ)
• Timing for reduced_ldmx: gold=144s, new=141s
• Timing within 2% of gold (tolerance 10%)

signal:

• Text Differences Between Logs (28 lines differ)
• Timing for signal: gold=1189s, new=1194s
• Timing within 0% of gold (tolerance 10%)

signal_target_al:

• Text Differences Between Logs (44 lines differ)
• Timing for signal_target_al: gold=936s, new=976s
• Timing within 4% of gold (tolerance 10%)

target_genie:

• Text Differences Between Logs (232 lines differ)
• Timing regression for target_genie: new=9693s vs gold=5707s (+69%, tolerance 10%)
• Timing for target_genie: gold=5707s, new=9693s

target_pn_lyso:

• Text Differences Between Logs (28 lines differ)
• Timing for target_pn_lyso: gold=6341s, new=6376s
• Timing within 0% of gold (tolerance 10%)

target_ti_en:

• Text Differences Between Logs (38 lines differ)
• Timing regression for target_ti_en: new=3181s vs gold=2738s (+16%, tolerance 10%)
• Timing for target_ti_en: gold=2738s, new=3181s

wab_lhe:

• Text Differences Between Logs (22 lines differ)
• Timing for wab_lhe: gold=6754s, new=6864s
• Timing within 1% of gold (tolerance 10%)

[mgignac]

Is there a policy on who should merge once approved? I'm starting to develop above this PR, but would be nice to have in trunk. Seems I have permissions to hit merge , but didn't want to short cut anything if someone else should officially sign off?

[tvami]

Seems I have permissions to hit merge

Yes please go ahead!

[tomeichlersmith]

its fun to click the button 😎