Add a test suite to detect backwards-incompatibility of new changes

.gitattributes

@@ -1,2 +1,3 @@
 * text=auto
 *.sh text eol=lf
+*.root binary

CONTRIBUTING.md

@@ -22,7 +22,8 @@ Thank you for your interest in contributing! As part of the SHiP Collaboration,
    pixi run test
    ```
    During iteration, `pixi run build` rebuilds incrementally without re-running ctest.
-8. **Submission**: Open a Pull Request against the `main` branch. Ensure the CI passes.
+8. **Schema changes**: The persistent schema is guarded by the backward-compatibility suite (see `tests/data/README.md`). If the `schema_snapshot` test fails after an intentional change, regenerate the gates in the same PR with `pixi run update-schema-snapshot` and `pixi run update-reference-head`; if a `compat_read_v*` test fails, your change breaks reading of existing data — make it compatible or mark the commit as a breaking change and update the reader expectations. The snapshot may also legitimately change on ROOT version bumps (type-name normalization); regenerate it in the lock-update PR after reviewing the diff. Never modify the frozen `tests/data/reference_v*.root` files.
+9. **Submission**: Open a Pull Request against the `main` branch. Ensure the CI passes.
 
 ## Licensing
 

README.md

@@ -39,6 +39,15 @@ find_package(SHiPDataModel REQUIRED)
 target_link_libraries(your_target PRIVATE SHiP::SHiPDataModel)
 ```
 
+## Backward compatibility
+
+Files written by released versions must stay readable: CI reads a frozen
+reference RNTuple file per release (`tests/data/reference_v*.root`) with the
+current code, and a committed schema snapshot fails CI on any schema change
+until it is deliberately regenerated. See
+[`tests/data/README.md`](tests/data/README.md) for the policy and what to do
+when these tests fail.
+
 ## Dependencies
 
 - ROOT 6.36+ (Core, RIO)

pixi.toml

@@ -57,9 +57,17 @@ root_cxx_standard = "==23"
 mp-units = ">=2.5"
 
 [tasks]
-configure = "cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo"
+# BUILD_TESTING is pinned on: this is the build directory the compat suite
+# and scripts/release.sh drive, and passing -D here also resets a cache left
+# at OFF by a packaging-style configure.
+configure = "cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_TESTING=ON"
 build = { cmd = "cmake --build build -j", depends-on = ["configure"] }
 test = { cmd = "ctest --test-dir build --output-on-failure", depends-on = ["build"] }
+# Deliberate regeneration of the schema-change gates; run these (and commit
+# the results) in the same PR as an intentional schema change. See
+# tests/data/README.md.
+update-schema-snapshot = { cmd = "build/tests/schema_snapshot --update tests/data/schema_snapshot.txt", depends-on = ["build"] }
+update-reference-head = { cmd = "build/tests/write_reference tests/data/reference_head.root", depends-on = ["build"] }
 
 [feature.lint.dependencies]
 prek = "*"

scripts/backfill_reference_files.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: LGPL-3.0-or-later
+# Copyright (C) CERN for the benefit of the SHiP Collaboration
+#
+# One-time backfill of the frozen compatibility reference files for releases
+# that predate the compatibility suite (see tests/data/README.md). For each
+# tag: build the tag's dictionary library, compile the CURRENT
+# tests/write_reference.cpp against the TAG's headers (the old headers define
+# the on-disk shape; reference_values.hpp adapts via __has_include and member
+# detection), and write tests/data/reference_<tag>.root.
+#
+# Run from the repo root inside the pixi environment:
+#   pixi run bash scripts/backfill_reference_files.sh
+#
+# Kept for provenance; releases from v0.5.0 on write their reference file at
+# release time via scripts/release.sh instead.
+
+set -euo pipefail
+
+REPO="$(git rev-parse --show-toplevel)"
+mkdir -p "${REPO}/tests/data"
+
+for tag in v0.1.0 v0.2.0 v0.3.0 v0.4.0; do
+    out="${REPO}/tests/data/reference_${tag}.root"
+    if [[ -e "${out}" ]]; then
+        echo "skip ${tag}: ${out} already exists (frozen files are never rewritten)"
+        continue
+    fi
+    wt="$(mktemp -d)/data-model-${tag}"
+    tmp="${out}.tmp"
+    # Never leave a partial file at the frozen path (the skip-check above
+    # would treat it as frozen) or a registered worktree behind on failure.
+    trap 'rm -f "${tmp}"; git worktree remove --force "${wt}" 2>/dev/null || true' EXIT
+    git worktree add "${wt}" "${tag}"
+    cmake -S "${wt}" -B "${wt}/build" -G Ninja \
+        -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_TESTING=OFF
+    cmake --build "${wt}/build" -j
+    # Tag headers first on the include path; reference_values.hpp resolves
+    # next to write_reference.cpp, i.e. to the current repo's copy.
+    "${CXX:-g++}" -std=c++23 -I "${wt}/include" $(root-config --cflags) \
+        "${REPO}/tests/write_reference.cpp" -o "${wt}/build/write_reference" \
+        $(root-config --libs) -lROOTNTuple \
+        -L "${wt}/build" -Wl,--no-as-needed -lSHiPDataModel -Wl,--as-needed \
+        -Wl,-rpath,"${wt}/build"
+    "${wt}/build/write_reference" "${tmp}"
+    mv "${tmp}" "${out}"
+    git worktree remove --force "${wt}"
+    trap - EXIT
+done

scripts/release.sh

@@ -17,9 +17,12 @@ The script must be run from a clean working tree. It will:
   1. bump the VERSION line in CMakeLists.txt
   2. bump version and date-released in CITATION.cff (if present)
   3. bump the [package] version in pixi.toml (if present)
-  4. regenerate CHANGELOG.md with `git cliff --tag v<version>`
-  5. create commit `chore(release): v<version>`
-  6. create annotated tag `v<version>`
+  4. write the frozen compatibility reference file
+     tests/data/reference_v<version>.root, refresh the schema snapshot and
+     run the test suite (requires pixi)
+  5. regenerate CHANGELOG.md with `git cliff --tag v<version>`
+  6. create commit `chore(release): v<version>`
+  7. create annotated tag `v<version>`
 
 Pushing is left to the operator:
   git push origin <branch> && git push origin v<version>
@@ -110,9 +113,47 @@ if [[ -f "${PIXI_FILE}" ]] && grep -qE '^version = "[0-9]+\.[0-9]+\.[0-9]+"' "${
     fi
 fi
 
+# Freeze the compatibility reference file for this release: written from
+# exactly the code being tagged, with the ROOT version pinned in pixi.lock at
+# this moment — capturing both the schema and the writing ROOT (see
+# tests/data/README.md). The snapshot regen should be a no-op if CI was
+# green; the test run gates the release on the compat suite, including the
+# new file (picked up via the CMake glob).
+if ! command -v pixi >/dev/null 2>&1; then
+    echo "error: 'pixi' not available; needed to write the compatibility reference file" >&2
+    exit 69
+fi
+REF_FILE="tests/data/reference_${TAG}.root"
+if [[ -e "${REF_FILE}" ]]; then
+    echo "error: ${REF_FILE} already exists; frozen reference files are never rewritten" >&2
+    echo "hint: a file left over from an interrupted release attempt is safe to remove" >&2
+    exit 65
+fi
+pixi run build
+# The compat suite is only built when build/ was configured with testing on;
+# fail here rather than on a bare "No such file" further down.
+if [[ ! -x ./build/tests/write_reference ]]; then
+    echo "error: ./build/tests/write_reference missing; reconfigure build/ with BUILD_TESTING=ON" >&2
+    exit 70
+fi
+# Write atomically: never leave a partial file at the frozen path (the
+# exists-check above would otherwise block a retry after a failed write).
+REF_TMP="${REF_FILE}.tmp"
+# The file stays provisional until the snapshot regen and the test suite have
+# both passed: on failure remove it, together with any partial temporary, so
+# no unvalidated file is left behind and the exists-check above does not block
+# a retry. It has to sit at its frozen path while the tests run, because the
+# compat suite discovers reference files through the CMake glob.
+trap 'rm -f "${REF_TMP}" "${REF_FILE}"' EXIT
+pixi run ./build/tests/write_reference "${REF_TMP}"
+mv "${REF_TMP}" "${REF_FILE}"
+pixi run update-schema-snapshot
+pixi run test
+trap - EXIT  # validated from here on: keep the frozen file
+
 git cliff --tag "${TAG}" -o CHANGELOG.md
 
-git add "${CMAKE_FILE}" CHANGELOG.md
+git add "${CMAKE_FILE}" CHANGELOG.md "${REF_FILE}" tests/data/schema_snapshot.txt
 if [[ -f "${CITATION_FILE}" ]]; then
     git add "${CITATION_FILE}"
 fi

tests/CMakeLists.txt

@@ -23,3 +23,67 @@ add_test(NAME rntuple_io COMMAND test_rntuple_io)
 add_executable(test_units test_units.cpp)
 target_link_libraries(test_units PRIVATE SHiP::SHiPUnits)
 add_test(NAME units COMMAND test_units)
+
+# --- Backward-compatibility suite (see data/README.md) ---
+
+add_executable(write_reference write_reference.cpp)
+target_link_libraries(
+    write_reference
+    PRIVATE ${link_dictionary} ROOT::Core ROOT::RIO ROOT::ROOTNTuple
+)
+
+add_executable(test_read_reference test_read_reference.cpp)
+target_link_libraries(
+    test_read_reference
+    PRIVATE ${link_dictionary} ROOT::Core ROOT::RIO ROOT::ROOTNTuple
+)
+
+# One ctest test per committed reference file, for clear failure attribution.
+# CONFIGURE_DEPENDS: a file added at release time is picked up on reconfigure.
+file(
+    GLOB reference_files
+    CONFIGURE_DEPENDS
+    "${CMAKE_CURRENT_SOURCE_DIR}/data/reference_*.root"
+)
+foreach(ref IN LISTS reference_files)
+    if(ref MATCHES "reference_(v[0-9]+\\.[0-9]+\\.[0-9]+|head)\\.root$")
+        add_test(
+            NAME compat_read_${CMAKE_MATCH_1}
+            COMMAND test_read_reference ${ref} ${CMAKE_MATCH_1}
+        )
+    endif()
+endforeach()
+
+# The snapshot tool needs the list of dictionary classes; derive it from
+# LinkDef.h so newly linked classes enter the snapshot automatically.
+file(
+    STRINGS ${PROJECT_SOURCE_DIR}/include/SHiP/LinkDef.h
+    linkdef_lines
+    REGEX "pragma link C\\+\\+ class SHiP::"
+)
+set(dictionary_classes "")
+foreach(line IN LISTS linkdef_lines)
+    if(NOT line MATCHES "std::vector")
+        string(REGEX MATCH "SHiP::[A-Za-z0-9_]+" cls "${line}")
+        string(APPEND dictionary_classes "\"${cls}\",\n")
+    endif()
+endforeach()
+file(
+    CONFIGURE
+    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/dictionary_classes.inc
+    CONTENT "${dictionary_classes}"
+    @ONLY
+)
+
+add_executable(schema_snapshot schema_snapshot.cpp)
+target_include_directories(schema_snapshot PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
+target_link_libraries(
+    schema_snapshot
+    PRIVATE ${link_dictionary} ROOT::Core ROOT::RIO ROOT::ROOTNTuple
+)
+add_test(
+    NAME schema_snapshot
+    COMMAND
+        schema_snapshot --check
+        ${CMAKE_CURRENT_SOURCE_DIR}/data/schema_snapshot.txt
+)

tests/data/README.md

@@ -0,0 +1,73 @@
+# Backward-compatibility reference data
+
+This directory holds the on-disk compatibility contract of the SHiP data
+model. Two independent safety nets are exercised by ctest (part of
+`pixi run test` and CI):
+
+1. **Compat read tests** (`compat_read_<version>`): every reference file
+   must be readable with the *current* library, materialized into the
+   current structs, with all values matching the canonical recipe. Members
+   that did not exist in the writing version must read back
+   default-initialized (ROOT RNTuple automatic schema evolution).
+2. **Schema snapshot** (`schema_snapshot`): `schema_snapshot.txt` is a text
+   dump of the persistent schema (TClass layout of every dictionary class +
+   the RNTuple field tree). CI fails on *any* schema change — even a
+   backward-compatible one — until the snapshot is deliberately regenerated
+   in the same PR, so every schema change is a conscious, reviewable
+   decision.
+
+## Files
+
+- `reference_v<X.Y.Z>.root` — frozen forever, **never modified or
+  regenerated**. Written at release time by `scripts/release.sh` from
+  exactly the tagged code with the ROOT version pinned in `pixi.lock` at
+  that moment, so it captures both the schema and the writing ROOT.
+  Exception: the files for v0.1.0–v0.4.0 predate this suite and were
+  backfilled with `scripts/backfill_reference_files.sh` — they were written
+  by each tag's *headers* (defining the on-disk schema) but by ROOT 6.40.02,
+  not the historical ROOT versions.
+- `reference_head.root` — tracks `main`; asserts "current code reads the
+  current schema". Regenerated in the same PR as any event-model change.
+- `schema_snapshot.txt` — committed schema dump, see above.
+
+Each file is an RNTuple named `events` with 2 entries and top-level fields
+`event_header` (v0.4.0+), `mcParticles`, `simHits`, `simParticles`,
+`recParticles`, `simResult`.
+
+## When a compat test fails in your PR
+
+- `schema_snapshot` fails, `compat_read_*` pass: you changed the persistent
+  schema in a backward-compatible way (e.g. added a member). If intentional,
+  run and commit in the same PR:
+
+  ```sh
+  pixi run update-schema-snapshot
+  pixi run update-reference-head
+  ```
+
+  This is at least a **minor** version bump.
+- `compat_read_v*` fails: your change breaks reading of existing files
+  (e.g. renaming a member silently drops its on-disk values — RNTuple
+  matches members by name). Either make the change compatible (e.g. an
+  [I/O customization rule](https://root.cern/doc/master/md_tree_2ntuple_2doc_2SchemaEvolution.html)
+  mapping the old name), or accept it as a **breaking change**: mark the
+  commit `!`/`BREAKING CHANGE` (major version bump) and adjust the
+  expectations in `tests/test_read_reference.cpp` (masking table) — never by
+  editing the frozen files.
+- `compat_read_head` fails but frozen versions pass: the current schema and
+  `reference_head.root` are out of sync — run `pixi run
+  update-reference-head` (plus the snapshot) in this PR.
+
+## Value recipe
+
+Expected values are defined in `tests/reference_values.hpp` and are part of
+the contract (a future non-C++ reader can check against the same formulas).
+For members that existed at v0.1.0 they equal the `SHiP::test::make*`
+generators in `tests/test_utils.hpp`; members added later have their own
+formulas there (e.g. `SimHit::geometryNodeId = 900 + 7*i + offset`,
+`RecParticle::hits` filled from `makeSimHits(offset + 2)`). Per entry
+`e` (0-based): field offsets are `e` for the top-level collections and
+`e + 5` inside `simResult`; each collection has 3 elements. A reference file
+written by version V contains values for exactly the members existing in V;
+newer members read back default-initialized and are masked accordingly in
+`test_read_reference.cpp`.