feat(archaic): ghost lineages, Neanderthal tracts in named living haplotypes, and where the method breaks

[ruvnet]

Recovers two extinct hominin "ghost" lineages from present-day genomes on the rvDNA engine, sweeps the method until it fails to find its real limits, and then turns the same machinery on real 1000 Genomes and real archaic genomes to say something about people rather than about detectors. Ships an illustrated, scroll-driven report under artifacts/, titled rUv.

Responds to Zhang, Biddanda, Johnson, O'Dushlaine & Moorjani, Science, 30 July 2026, whose TRACE method found a ~800 ka lineage in every living human and a super-archaic lineage that reached us via Denisovans. Neither has ever been sequenced.

Two figures from the press coverage do not survive a read of the paper, and this PR does not repeat them: "Denisovans carry 3–5% super-archaic DNA" is a press-release number (TRACE's estimate is 0.3% of the Denisovan ancestry in Oceanians), and 1.8 Ma is a coalescence time — an upper bound on divergence — where the authors' conservative claim is "diverged over 500,000 years ago". Those figures are used here only as the simulation's planted splits, where truth is known by construction.

Detection is validated in simulation; everything claimed about real people is measured on real data, at the population level, and attributes no ancestry to any named person.

Headline results on real genomes

800 phased haplotypes (400 people, 80 per superpopulation) over chr22:20–21 Mb, 21,418 biallelic SNPs, plus Altai Neanderthal (~52×), Vindija 33.19 (~30×) and Denisova 3 (~30×) from the MPI-EVA snpAD call set over the same coordinates. Acquired without tabix, bcftools or pysam: parse the .tbi linear index in pure Python, HTTP Range into the bgzipped VCF, decode BGZF with zlib.

1 · African haplotypes coalesce exactly 2.00× deeper than every other superpopulation, which all land on the same median (46.4 ka vs 23.2 ka). All ten deepest segments in the region are African. 6,072,400 exact popcount comparisons in 0.4 s.

2 · Where the Neanderthal DNA actually is. The scan defines a Neanderthal-informative site: a position where a Neanderthal carries the non-reference allele and not one of the 160 African haplotypes does. Conditioning on absence in Africa makes it immune to the European weighting of GRCh37 — the test never references distance from the reference.

	non-African (test)	African (control)
Stretches found	847	0
Haplotypes carrying one	254 / 640	0 / 160
Share of sequence	1.83%	0%

The control returns zero, and 1.83% lands on the published ~2% figure for Neanderthal ancestry in non-Africans — from one megabase and a rule about what is missing from Africa.

Informative alleles per haplotype: EAS 24.8, SAS 9.0, AMR 5.8, EUR 2.5, AFR 0.0. East Asians carrying more Neanderthal ancestry than Europeans is the published (and counter-intuitive) ordering, recovered here. The 10× magnitude is not — genome-wide the excess is ~20% — and the report says so: one megabase is one locus.

3 · Dating the pulse, and a mistake the simulation saw coming. Mean surviving tract is 13,825 bp → 7,233 generations → 210 ka, against a published 50–60 ka. 4× too old. That failure was predicted: sweep 4 below reports on simulated data with known truth that a detector which fragments tracts rather than fusing them always dates the pulse too old, because a shorter measured tract inverts to more generations — there the error was 2.4× on a planted pulse. It is reported as a headline rather than buried, because a simulation that predicts the direction and the mechanism of an error later observed in real human DNA is doing more than one that merely scores well.

4 · The family, with no tree assumed. Two haplotypes drawn from the same population must go back 864 ka to meet in MSL and 641 ka in PEL, and the seven most internally diverse populations are all African. Two people from MSL are further apart than any European–East Asian pair in the panel (widest 847 ka).

5 · Patterson's D. Over 160 African × 640 non-African haplotype pairs, with a leave-one-window-out jackknife:

Archaic genome	ABBA	BABA	D	SE	Z
Altai Neanderthal	8,832,591	7,015,951	+0.1146	0.1047	+1.1
Vindija 33.19	9,068,240	7,008,880	+0.1281	0.1007	+1.3
Denisova 3	7,380,467	7,322,067	+0.0040	0.1243	+0.0

Both Neanderthals lean, the Denisovan is flat at 3% of that. Reference bias would push all three the same way. It is also not significant, and that is the more useful number: at this effect size the comparison needs ~7 Mb to reach Z = 3, emitted as mb_for_z3 so the limit is a number in the output rather than a hedge in the prose.

Engine

examples/dna/src/archaic.rs and real_panel.rs — no new dependencies.

Piece	Does
simulate_cohort	structured coalescent per window; introgression planted by reassigning a lineage to the donor deme, in geometric tracts
TraceEngine	embeds each window as an rvDNA k-mer profile vector, indexes it in a RuVector HNSW graph, computes exact Jukes–Cantor-corrected depth against retrieved candidates only
cluster_ghosts	two-means on log depth with a separation test
darwin	mutate one hyperparameter → sandbox on train → keep only past a margin → report on held-out
flywheel	promote confirmed ghost segments to references, run until two dry rounds
real_panel	1000 Genomes + archaic loaders, one bit per variant per haplotype, masked-XOR/popcount comparison

Seven binaries: trace-rv, depth-hist, discover, linkage-study, panel-study, real-dna, human-story.

Simulation results

• Clock — estimated vs true TMRCA, r = 0.9964, mean relative error 7.3%.
• Darwin — 20 proposals, 1 accepted; held-out F1 0.821 → 0.867. Held-out gain (+0.046) exceeds training gain (+0.034).
• Flywheel — converged after 3 turns; F1 did not move, and the loop reports that rather than manufacturing a gain.
• Lineages — mode 0 median 837 ka (planted 800), holds 76% of Ghost-A; mode 1 median 2.03 Ma (planted 1.8), holds 100% of Ghost-B. They do not separate per segment; what identifies the deep mode is Oceanian enrichment, 38.1% against 11.2% — a 3.4× signal that never touches the depth axis.

Five limit sweeps

1 · A ghost lineage vanishes below ~9 recovered segments (0.043% of the panel).
2 · Detection saturates at ~30 variable sites per window; below that the depth estimate turns to noise before the threshold does.
3 · HNSW reproduces 100% of the exhaustive scan's calls for 3.3× less work.
4 · Reading a window's neighbours beats reading it alone (F1 0.591 → 0.624; all 19 rescued windows genuinely introgressed; +0.089 held out). The control is the result: on an unlinked cohort the same rule moves F1 by exactly 0.0000.
5 · Size buys precision and costs recall (noise floor 140 → 32 ka). Composition beats count — 24 African-only haplotypes (F1 0.761) beat 36 balanced (0.744) — but the obvious corollary fails: that panel is strictly worse for every non-African query, and wins only because African queries are where the false positives are. Archaic reference value is steeply diminishing (three Neanderthal genomes → one costs 0.008 ghost precision; one → zero costs 0.216), and ghost recall is 1.000 in every arm: an under-referenced study invents lineages, it never misses them.

The flywheel bug, and why the fix is in the model not the tuning

The first implementation promoted any GHOST call to a reference and rescued anything within ghost_link_ka of one. False positives became references and validated more false positives: precision fell 0.84 → 0.22 → 0.19 over three rounds.

Two rules fix it, both from the population genetics: only primary calls may be promoted, and the rescue rule is relative — closer to a known ghost than to anyone alive, not merely close to a known ghost. flywheel_does_not_bootstrap_on_its_own_errors fails if either is removed.

Report

artifacts/index.html, titled rUv, self-contained at 457 KB.

Opens on a title sequence: letterbox bars retract, 34,000 points assemble out of a scattered sphere into a DNA double helix staggered from the middle outwards, the camera dollies from far to close, and the wordmark resolves out of blur — then scrolling flies the helix past the lens. Behind it, a nine-act scroll-driven sequence in raw WebGL 2, one GPU point per segment, 20,880 of them, morphing between analytic layouts computed in the vertex shader. The camera flies up one axis — coalescent time, 1.8 Ma to the present — then turns around and looks back down it, so the closing shot separates the two ghosts at their actual divergence times rather than by decoration. Section-level parallax throughout.

Two new animated figures for the real-genome section: a tract map with one row per living haplotype, continents banded and Africa first, revealed left to right along the chromosome — the empty African band is the control made visible — and a 26-population diversity chart that grows on scroll.

Why not three.js, which was asked for: the CSP blocks external scripts, so a library has to be inlined, and minified three.js is ~600 KB. An earlier 811 KB build of this page failed to load outright. Hand-written WebGL is ~4 KB and drives the same GPU pipeline.

Styled with the Cognitum design system — its tokens, 4px spacing base, 12px radius, four motion layers, Outfit + JetBrains Mono embedded as data URIs under the SIL OFL. Lineage colours stepped into the readable band and validated for colour-vision separation. Mobile-first: sticky bar, drawer menu with 44px targets, scrollable tables.

Verification

• cargo test --release — 194 passed, 0 failed
• cargo clippy --release --all-targets — clean; cargo fmt --check — clean
• Full scroll pass at 1440px and 390px: 0 horizontal overflow, no console errors, all nine acts fire, every chart and table populates

cargo build --release
./target/release/depth-hist     artifacts/data   # ~5 s
./target/release/trace-rv       artifacts/data   # ~10 min, Darwin dominates
./target/release/discover       artifacts/data   # ~2 min
./target/release/real-dna       artifacts/data   # ~1 s
./target/release/linkage-study  artifacts/data   # ~3 min
./target/release/panel-study    artifacts/data   # ~4 min
./target/release/human-story    artifacts/data   # ~1 s
node artifacts/build.mjs

data/real/ is checked in (~35 MB) so the real-genome stages reproduce without network access.

See ADR-016 and artifacts/README.md.

Sources

• Zhang, Biddanda, Johnson, O'Dushlaine & Moorjani, Recovering signatures of archaic hominin introgression using ancestral recombination graphs, Science, 30 July 2026
• Prüfer et al., A high-coverage Neandertal genome from Vindija Cave in Croatia, Science 358, 655–658 (2017) — the archaic genotype calls
• Green et al., A draft sequence of the Neandertal genome, Science 328, 710–722 (2010) — the ABBA/BABA D-statistic
• The 1000 Genomes Project Consortium, Nature 526, 68–74 (2015)
• Liang & Nielsen, The lengths of admixture tracts, Genetics 197, 953–967 (2014)
• ruvnet/metaharness — Darwin mode, flywheel

🤖 Generated with Claude Code

https://claude.ai/code/session_011Dhba1gV3hjg4tX9F3Q4C7