Stop indexing nibble-gram lengths that can never be hit

[ESultanik]

Stacked on #20.

Building the substitution alphabet cost roughly 1600 bytes of RAM per certificate byte, which put a hard ceiling on usable key sizes — a 2 MiB key pair needed over 4 GB.

Most of that bought nothing. A length-L hit needs the grams from every certificate to match at the same offset, so the keyspace is 16**(L*num_secrets) while the certificates supply only 2*min_cert_length offsets. Past a point the index holds one unique key per offset: pure memory, never a hit. Measured with two 256 KiB certificates:

gram length	distinct keys	keys reused	blocks used encrypting 4 KiB
1	256	256	3 (0.1%)
2	65,511	65,354	4,095 (99.9%)
4	524,260	25	0
8	524,281	0	0
16	524,273	0	0

Lengths 4, 8, and 16 were about 75% of the index and were never once used. The default level indexed all of 1, 2, 4, and 8.

Adaptive selection

select_nibble_gram_lengths derives the set from certificate size and secret count, and -1..-5 now cap it rather than choosing it:

secrets	key size	selected
2	32 KiB – 2 MiB	(1, 2)
3 or more	any	(1,)
1	64 KiB	(1, 2, 4) — the fixed list could never express this

What remains got cheaper too

• Index keys are packed into one bytes that interleaves the certificates' nibbles, so a key is a single slice of one interleaved buffer rather than a tuple of per-certificate slices. Measured ~2.2× faster to build than concatenating, at the same memory. key[j::num_secrets] recovers certificate j's gram; the gram length is len(key) // num_secrets, which is what the dictionary header needs.
• Nibble expansion uses two bytes.translate calls instead of a per-byte Python loop.
• Offsets are stored in array('I'). array('L') is native width — 8 bytes here, 4 on Windows — so it was both platform-dependent and twice the size required. The 'L' in index_type_map is deliberately left alone: struct's < prefix selects standard sizes, where L is always 4.

Result

2 x 128 KiB   0.57s /  294 MB  ->  0.07s /  28 MB     8x faster, 10x less RAM
2 x 512 KiB   2.85s / 1109 MB  ->  0.24s /  62 MB    12x faster, 18x less RAM
2 x   2 MiB  12.40s / 4334 MB  ->  0.95s / 145 MB    13x faster, 30x less RAM

The key count also stops growing with certificate size — 65,792 at both 512 KiB and 2 MiB, being the complete 1- and 2-gram keyspaces — so memory now scales only with the offset arrays.

Ciphertexts are unchanged in size: measured at 512 B, 4 KiB, and 16 KiB of plaintext, the adaptive index and the full fixed list differ by at most one byte, confirming the dropped lengths contributed nothing.

Also

• Fixes stop_when_sufficient, whose threshold was (16*L)**N — correct only for L=1 by coincidence, which is the only length -t passed.
• Throttles the status callback, which fired once per nibble offset.
• decode and encode raise MalformedCiphertextError / EncodingError instead of bare Exception; the shorter ciphertexts made decode's case reachable from a truncated file.

35 new tests covering expansion against the reference loop, pack/split round-trips, the packed layout matching what the index stores, the selection table, offset correctness, and the corrected threshold. 223 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01367hFob9sd4xpDmVT4uoFy