Community-label split-retry starves its own base case: max_tokens shrinks with batch size, single-community retries get only 88 tokens

[cshafer-edgepoint]

Summary

label_communities()'s adaptive split-retry (_label_batch_with_retry in graphify/llm.py) computes its output-token budget from the batch size:

max_tokens = _resolve_max_tokens(min(64 + 24 * len(batch_cids), 8192))

This scales the budget down as the batch shrinks. When a batch fails to parse and the recovery logic bisects it, each half gets a smaller max_tokens, and the recursion's base case — a single community — is handed only 64 + 24*1 = 88 tokens. That is small enough that the model's reply is frequently empty or truncated mid-string, which is itself a (or the) cause of the JSON parse failure the split was trying to recover from. The recovery path therefore makes the failure mode more likely at each level, not less, and the base case is the most starved of all.

Impact

On a large multi-corpus documentation graph (~3,300 communities, gemini backend via the OpenAI-compatible client), roughly half of all community-labeling batches failed with:

[graphify label] batch of 1 still unparseable at depth 3 (cids=[...]): Unterminated string starting at: line 1 column 2 (char 1)
[graphify label] batch of 1 still unparseable at depth 3 (cids=[...]): Expecting value: line 1 column 1 (char 0)

• Expecting value: line 1 column 1 ↔ an empty resp.choices[0].message.content (nothing came back within the token budget).
• Unterminated string ... ↔ a reply cut off mid-JSON-string.

Net result: ~1,755 of ~3,305 communities named on a run; the other ~1,550 silently fell back to Community N placeholders. No data loss (labeling is cosmetic), but the wiki/index is heavily under-labeled and the failures are noisy.

Diagnosis is confirmed, not theoretical

_resolve_max_tokens() already honors a GRAPHIFY_MAX_OUTPUT_TOKENS environment-variable override. Setting GRAPHIFY_MAX_OUTPUT_TOKENS=4096 and re-running the identical labeling pass over the exact same placeholder communities took it from ~1,755/3,305 to 3,305/3,305 named on the first pass, zero batch failures, no split-retry needed at all. The subgraph labeling pass went to 1,076/1,076 the same way. That single knob fully resolves it, which pins the cause to the token budget rather than the model or the prompt.

Suggested fix

The output budget for labeling should have a floor that comfortably fits the expected reply (a handful of short {"cid": "2-5 word name"} entries per community needs far more than 88 tokens of headroom once you account for the model occasionally being verbose), and it should not shrink toward the base case. A few options:

1. Raise the floor: max_tokens = _resolve_max_tokens(max(512, min(64 + 24 * len(batch_cids), 8192))) — cheap, keeps the per-batch scaling for large batches but stops starving the small ones.
2. Decouple the budget from batch size entirely for labeling (a fixed, generous default like 1024–2048), since the split-retry already bounds the input size; the output was never the thing that needed to scale down.
3. At minimum, document that GRAPHIFY_MAX_OUTPUT_TOKENS is the workaround for label-batch parse failures — it's currently undiscoverable unless you read _resolve_max_tokens.

Happy to test a candidate fix against the same ~3,300-community corpus if useful.

Note

This is distinct from #2068 (the build_from_json ghost-duplicate-merge basename-collision bug) — different function, different failure, same corpus. Filing separately.