fix: convert CJK built-in date formats in xlsx extraction (#736) - #738
Open
kevin-hans wants to merge 1 commit into
Open
fix: convert CJK built-in date formats in xlsx extraction (#736)#738kevin-hans wants to merge 1 commit into
kevin-hans wants to merge 1 commit into
Conversation
Calamine's `builtin_format_by_id` only recognizes the English built-in date IDs (14-22, 45, 47), so cells styled with the CJK built-in IDs 27-36 (e.g. `31` = `yyyy年m月d日`, `32` = `m月d日`) come back as `Data::Float` holding a raw Excel serial like `45761`. The downstream ingest LLM then treats those serials as anomalies. Side-parse the xlsx zip in `extract_spreadsheet` to find cells whose `cellXf` targets a CJK built-in date `numFmtId`, pre-convert the serial to an ISO date string, and swap it in when emitting markdown. Skipped for xls/ods and for xlsx workbooks with no CJK styles so the common path pays no per-cell cost. Fixes nashsu#736.
Author
|
@nashsu |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Ingesting an .xlsx cell formatted with one of Excel's CJK built-in date formats —
27–36in ECMA-376 §18.8.30, e.g.31=yyyy年m月d日,32=m月d日— surfaces the cell as its raw Excel serial number in the extracted markdown. A column that displayed2025年4月14日in Excel becomes45761, and the ingest LLM then flags the column as anomalous numeric data (#736).Root cause is upstream in calamine:
builtin_format_by_id(both 0.34 and 0.36) only knows the English built-in date IDs (14–22, 45, 47). Cells styled with 27–36 are taggedCellFormat::Other, soworksheet_rangereturns them asData::Float(serial)andextract_spreadsheetincommands/fs.rswrites the serial verbatim. Chinese / Japanese / Korean Excel emits these IDs whenever a user picks a locale-native date format from the built-in list, so it hits real files in practice.Custom
<numFmt>codes withnumFmtId ≥ 164(e.g.m"月"d"日") are unaffected — calamine'sdetect_custom_number_formatpicks up the leadingm/d/y/h/stokens correctly. Only the built-in-ID branch was blind.Fix
For
.xlsx/.xlsmonly, side-parse the workbook zip once before markdown emission:xl/styles.xmland collect everycellXfindex whosenumFmtIdis in27..=36and is not overridden by a<numFmt>entry (overrides are left to calamine's custom-format detector).(row, col)of every<c s="...">styled by a CJK datecellXf, and pre-convert its<v>serial to an ISO string (YYYY-MM-DD, orYYYY-MM-DD HH:MM:SSwhen a fractional time is present).extract_spreadsheet, aData::Floatcell with a matching coordinate uses the ISO override; every other cell goes through the existing formatter unchanged.Handles the 1904 date system and Excel's phantom
1900-02-29(serial 60 → no real calendar date, returnsNoneso the raw serial stays visible rather than being mis-dated)..xls/.odscontinue through calamine unchanged.Verified
numFmtId=31, serial44562) inextract_spreadsheet_converts_cjk_builtin_date_format_ids: red before the fix (| 44562 |in output), green after (2022-01-01).cargo test --lib→ 401 passed / 0 failed. All 33commands::fstests still pass, includinganydoc_extracts_generated_structured_document_fixtures, which exercises the styles-lessminimal_xlsxfixture and confirms the fast-path early-return does not misbehave.Notes for review
styles.xml/ sheet XML uses the same hand-rolled string-scanning style already used byextract_docx_markdownandextract_odf_textin this file, deliberately avoiding a new direct dependency onquick-xml(only transitively present through calamine today).builtin_format_by_idabout IDs 27–36. Once that lands and we bump calamine, the override map will simply be empty for every workbook and this code becomes dormant; no callers need to change.2022-01-01) rather than the workbook's original CJK glyphs (2022年1月1日). This matches what calamine emits for the dates it already recognizes and is what the ingest LLM interprets most reliably; happy to switch to locale-native rendering if preferred.Fixes #736.