Skip to content

Fix combining marks being dropped after an ASCII base - #140

Merged
christianparpart merged 1 commit into
masterfrom
fix/scan-text-drops-zero-width-continuation
Jul 26, 2026
Merged

Fix combining marks being dropped after an ASCII base#140
christianparpart merged 1 commit into
masterfrom
fix/scan-text-drops-zero-width-continuation

Conversation

@christianparpart

Copy link
Copy Markdown
Member

scan_text() hands the caller two positions: [result.start, result.end) is the text to print, and state.next is where scanning resumed to. The complex arm of the scanner handoff decided it had made no progress from sub.count alone and returned early, discarding the sub-scan's end pointer — but a zero column count is not a zero byte count.

A codepoint that joins the open cluster without widening it consumes its bytes and adds nothing: a combining mark, or a variation selector on a base that has no variation sequence. It reaches that arm on its own only when an ASCII base precedes it, because the ASCII fast scan hands over at exactly that boundary; with a non-ASCII base the whole cluster stays inside one scan_for_text_nonascii() call and never crosses it. So state.next moved past those bytes while result.end did not, and a caller that prints [start, end) and resumes at next never saw them. In Contour, printf 'éx' reached the grid as a bare ex, the accent gone; "ab" + U+0301 + "cd" lost the mark between the two ASCII runs.

Progress is now measured in bytes, which is what the question actually is. The trivial arm keeps using count, because there it means the same thing: no ASCII bytes at the front, hence nothing consumed.

scan.everything_consumed_is_reported pins the general property whose absence let this through: unless the scan stopped inside an incomplete UTF-8 sequence, the bytes it consumed and the bytes it reports are the same bytes. Comparing column counts — which is what the existing coverage does — cannot detect a gap between the two.

scan_text() hands the caller two positions: [result.start, result.end) is the text to
print, and state.next is where scanning resumed to. The complex arm of the scanner
handoff decided it had made no progress from sub.count alone and returned early,
discarding the sub-scan's end pointer -- but a zero COLUMN count is not a zero BYTE count.

A codepoint that joins the open cluster without widening it consumes its bytes and adds
nothing: a combining mark, or a variation selector on a base with no variation sequence.
Reached on its own only when an ASCII base precedes it, since the ASCII fast scan hands
over at exactly that boundary; with a non-ASCII base the whole cluster stays inside one
scan_for_text_nonascii() call. So state.next moved past those bytes while result.end did
not, and a caller printing [start, end) and resuming at next never saw them. "e" + U+0301
arrived as a bare "e"; "ab" + U+0301 + "cd" lost the mark between the two ASCII runs.

Measure progress in bytes, which is what the question is. The trivial arm may keep using
count, because there it means the same thing: no ASCII bytes at the front, hence nothing
consumed.

Both new tests fail without this: the targeted one on the shapes above, and
scan.everything_consumed_is_reported on the general property whose absence let it through
-- unless the scan stopped inside an incomplete UTF-8 sequence, the bytes consumed and the
bytes reported are the same bytes. Counting columns alone cannot detect a gap between them,
which is why the existing coverage did not.

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart
christianparpart force-pushed the fix/scan-text-drops-zero-width-continuation branch from 0709470 to fba8de2 Compare July 26, 2026 21:50
@christianparpart
christianparpart merged commit 56ae39a into master Jul 26, 2026
8 checks passed
@christianparpart
christianparpart deleted the fix/scan-text-drops-zero-width-continuation branch July 26, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant