[TEST] Reproduce the source id collision with a real colliding pair

[noel-improv]

Status

The fix is now up as #541, and it carries these two commits, so the reproducer can land either way. Review this PR for the tests on their own, or review #541 for the tests and the fix together. Merging #541 merges these commits and closes this PR with them.

Nothing here has changed since it was opened. One thing does change in #541: the fix removes the xfail(strict=True) marker below and pins the graph characterisation tests to the legacy width, since a graph written before the fix still merges the colliding pair.

Description

A test that reproduces the source id collision described in #533, so the fix has something to turn green.

create_source_id truncates the text digest to 8 hex characters, and IdRewriter passes an empty string for the metadata component when a node carries no metadata, so a corpus loaded without metadata discriminates on 32 bits.

The pair below was found by hashing sequentially numbered documents until two shared a digest. It took 30,059 of them:

A = 'document 27347 body text'
B = 'document 30059 body text'
both -> aws::a4439cdb:d41d

No production code changes.

Changes

Three groups of tests, each doing a different job.

TestCollidingPair pins the arithmetic at width 8. These are facts about md5 rather than statements about configuration, so they hold whatever a run is set to. One covers the metadata component, which discriminates only when metadata actually differs between documents.

TestCollisionConsequences records what the collision costs. One prefix reads back as a single SourceDocument holding both documents' chunks, and nothing logs or raises, so a merged document cannot be told apart from a correct one.

TestSourceIdUniqueness is the assertion a fix has to satisfy, marked xfail(strict=True). It reads the configured width rather than a pinned one, so it asserts that the default is wide enough rather than anything about a particular width.

The download helper and the chunk node factory move into tests/unit/conftest.py. test_s3_based_docs.py already had a copy of both, so this removes one rather than adding a second.

Why xfail(strict=True) rather than a failing test

A plain failing test cannot be committed. xfail(strict=True) records the red result while leaving CI green, and fails the run once the assertions start passing, so the marker cannot outlive the defect.

default                       6 passed, 2 xfailed
--runxfail                    2 failed, with the real assertion output
SOURCE_ID_HASH_LENGTH=16      XPASS(strict) on both

Testing

• Unit tests added/updated
• Integration tests added (as appropriate)
• Existing tests pass (pytest)
• Tested manually

2141 pass, 2 xfailed across tests/unit. One pre-existing failure is unrelated and reproduces on a clean tree: test_integ_dependency_compatibility.py errors with No module named pip in this venv.

Checklist

• Code follows existing style and conventions
• License headers present on new files
• Documentation updated (if applicable)
• No breaking changes (or clearly documented)

Related issue: #533.

[noel-improv]

Closing in favour of #541, which carries these two commits plus the fix, so the reproducer and the change that turns it green land together as one PR.