Fix combining marks being dropped after an ASCII base - #140
Merged
christianparpart merged 1 commit intoJul 26, 2026
Conversation
scan_text() hands the caller two positions: [result.start, result.end) is the text to print, and state.next is where scanning resumed to. The complex arm of the scanner handoff decided it had made no progress from sub.count alone and returned early, discarding the sub-scan's end pointer -- but a zero COLUMN count is not a zero BYTE count. A codepoint that joins the open cluster without widening it consumes its bytes and adds nothing: a combining mark, or a variation selector on a base with no variation sequence. Reached on its own only when an ASCII base precedes it, since the ASCII fast scan hands over at exactly that boundary; with a non-ASCII base the whole cluster stays inside one scan_for_text_nonascii() call. So state.next moved past those bytes while result.end did not, and a caller printing [start, end) and resuming at next never saw them. "e" + U+0301 arrived as a bare "e"; "ab" + U+0301 + "cd" lost the mark between the two ASCII runs. Measure progress in bytes, which is what the question is. The trivial arm may keep using count, because there it means the same thing: no ASCII bytes at the front, hence nothing consumed. Both new tests fail without this: the targeted one on the shapes above, and scan.everything_consumed_is_reported on the general property whose absence let it through -- unless the scan stopped inside an incomplete UTF-8 sequence, the bytes consumed and the bytes reported are the same bytes. Counting columns alone cannot detect a gap between them, which is why the existing coverage did not. Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
force-pushed
the
fix/scan-text-drops-zero-width-continuation
branch
from
July 26, 2026 21:50
0709470 to
fba8de2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scan_text()hands the caller two positions:[result.start, result.end)is the text to print, andstate.nextis where scanning resumed to. The complex arm of the scanner handoff decided it had made no progress fromsub.countalone and returned early, discarding the sub-scan's end pointer — but a zero column count is not a zero byte count.A codepoint that joins the open cluster without widening it consumes its bytes and adds nothing: a combining mark, or a variation selector on a base that has no variation sequence. It reaches that arm on its own only when an ASCII base precedes it, because the ASCII fast scan hands over at exactly that boundary; with a non-ASCII base the whole cluster stays inside one
scan_for_text_nonascii()call and never crosses it. Sostate.nextmoved past those bytes whileresult.enddid not, and a caller that prints[start, end)and resumes atnextnever saw them. In Contour,printf 'éx'reached the grid as a bareex, the accent gone;"ab" + U+0301 + "cd"lost the mark between the two ASCII runs.Progress is now measured in bytes, which is what the question actually is. The trivial arm keeps using
count, because there it means the same thing: no ASCII bytes at the front, hence nothing consumed.scan.everything_consumed_is_reportedpins the general property whose absence let this through: unless the scan stopped inside an incomplete UTF-8 sequence, the bytes it consumed and the bytes it reports are the same bytes. Comparing column counts — which is what the existing coverage does — cannot detect a gap between the two.