feat: add the regression corpus for broad coverage

[tobocop2]

A broad corpus of ordinary documents, used to answer one question: does a change alter output anywhere it should not. Mathematics is incidental here. Breadth and volume are the point, which is what makes it a regression net rather than a set of chosen fixtures.

3,275 documents from five sources: PubMed Central open access, arXiv, Project Gutenberg, govdocs1 and Wikipedia.

Licensing, read rather than assumed

Every licence came from the document's own metadata or the publisher's stated terms:

• PMC, from each file's JATS ali:license_ref
• arXiv, per paper from its OAI record
• Gutenberg, govdocs1 and Wikipedia, from published policy, which is uniform for each

The result splits into the two lanes LICENSES.md already defines:

lane	count	terms
vendor	1,816	CC-BY-4.0, CC0, US public domain
reference	1,459	CC-BY-SA, CC-BY-NC*, arXiv nonexclusive-distrib

arXiv shows the split plainly. A paper released under CC-BY is vendored. A paper under the arXiv perpetual non-exclusive licence grants distribution rights to arXiv rather than to us, so the corpus records it and does not host it.

Nothing is excluded. Both lanes are in REGRESSION_PROVENANCE.md, which lists all 3,275 with source, licence and how the licence was determined.

The bytes stay out of git

The vendored set is about 1 GB. The corpus already keeps binaries in the bucket, and this follows that rule for bulk content of any format, so clones stay small.

The five regression directories join scripts/corpus-patterns.txt and .gitignore, which scripts/test_publish_corpus.py checks for divergence. The entries are path-scoped, so an ordinary .xml or .html fixture elsewhere in the tree is unaffected.

Getting the bytes in

python3 scripts/fetch_regression.py
python3 scripts/publish_corpus.py --bucket xberg-test-documents

scripts/regression-objects.json carries a source, sha256 and size for each of the 1,816. A download whose digest does not match is written to <path>.mismatch and reported, so a source that changed upstream cannot quietly become the fixture.

govdocs1 publishes 486 MB shards rather than individual files, so those 400 entries name the shard and the member inside it. A shard is downloaded once and every member taken from it before it is discarded.

Verified

Both fetch paths were run against the real sources: three PMC articles through the direct path, and four govdocs1 members extracted from a shard, all digests matching. scripts/test_publish_corpus.py passes, 31 tests.

[tobocop2]

Superseded by #7, which carries the same commit on a branch named for the corpus rather than the term it used to use.