Skip to content

fix: convert CJK built-in date formats in xlsx extraction (#736) - #738

Open
kevin-hans wants to merge 1 commit into
nashsu:mainfrom
kevin-hans:main
Open

fix: convert CJK built-in date formats in xlsx extraction (#736)#738
kevin-hans wants to merge 1 commit into
nashsu:mainfrom
kevin-hans:main

Conversation

@kevin-hans

@kevin-hans kevin-hans commented Sep 8, 2026

Copy link
Copy Markdown

Problem

Ingesting an .xlsx cell formatted with one of Excel's CJK built-in date formats — 2736 in ECMA-376 §18.8.30, e.g. 31 = yyyy年m月d日, 32 = m月d日 — surfaces the cell as its raw Excel serial number in the extracted markdown. A column that displayed 2025年4月14日 in Excel becomes 45761, and the ingest LLM then flags the column as anomalous numeric data (#736).

Root cause is upstream in calamine: builtin_format_by_id (both 0.34 and 0.36) only knows the English built-in date IDs (14–22, 45, 47). Cells styled with 27–36 are tagged CellFormat::Other, so worksheet_range returns them as Data::Float(serial) and extract_spreadsheet in commands/fs.rs writes the serial verbatim. Chinese / Japanese / Korean Excel emits these IDs whenever a user picks a locale-native date format from the built-in list, so it hits real files in practice.

Custom <numFmt> codes with numFmtId ≥ 164 (e.g. m"月"d"日") are unaffected — calamine's detect_custom_number_format picks up the leading m/d/y/h/s tokens correctly. Only the built-in-ID branch was blind.

Fix

For .xlsx / .xlsm only, side-parse the workbook zip once before markdown emission:

  1. Read xl/styles.xml and collect every cellXf index whose numFmtId is in 27..=36 and is not overridden by a <numFmt> entry (overrides are left to calamine's custom-format detector).
  2. If that set is empty — the vast majority of workbooks — bail out immediately, so the common path pays zero per-cell cost.
  3. Otherwise walk each sheet XML, collect the (row, col) of every <c s="..."> styled by a CJK date cellXf, and pre-convert its <v> serial to an ISO string (YYYY-MM-DD, or YYYY-MM-DD HH:MM:SS when a fractional time is present).
  4. In extract_spreadsheet, a Data::Float cell with a matching coordinate uses the ISO override; every other cell goes through the existing formatter unchanged.

Handles the 1904 date system and Excel's phantom 1900-02-29 (serial 60 → no real calendar date, returns None so the raw serial stays visible rather than being mis-dated). .xls / .ods continue through calamine unchanged.

Verified

  • Hand-built regression fixture (xlsx with numFmtId=31, serial 44562) in extract_spreadsheet_converts_cjk_builtin_date_format_ids: red before the fix (| 44562 | in output), green after (2022-01-01).
  • cargo test --lib401 passed / 0 failed. All 33 commands::fs tests still pass, including anydoc_extracts_generated_structured_document_fixtures, which exercises the styles-less minimal_xlsx fixture and confirms the fast-path early-return does not misbehave.

Notes for review

  • The XML parsing for styles.xml / sheet XML uses the same hand-rolled string-scanning style already used by extract_docx_markdown and extract_odf_text in this file, deliberately avoiding a new direct dependency on quick-xml (only transitively present through calamine today).
  • The right long-term fix is upstream in calamine — teaching builtin_format_by_id about IDs 27–36. Once that lands and we bump calamine, the override map will simply be empty for every workbook and this code becomes dormant; no callers need to change.
  • Output uses ISO dates (2022-01-01) rather than the workbook's original CJK glyphs (2022年1月1日). This matches what calamine emits for the dates it already recognizes and is what the ingest LLM interprets most reliably; happy to switch to locale-native rendering if preferred.

Fixes #736.

Calamine's `builtin_format_by_id` only recognizes the English built-in
date IDs (14-22, 45, 47), so cells styled with the CJK built-in IDs
27-36 (e.g. `31` = `yyyy年m月d日`, `32` = `m月d日`) come back as
`Data::Float` holding a raw Excel serial like `45761`. The downstream
ingest LLM then treats those serials as anomalies.

Side-parse the xlsx zip in `extract_spreadsheet` to find cells whose
`cellXf` targets a CJK built-in date `numFmtId`, pre-convert the serial
to an ISO date string, and swap it in when emitting markdown. Skipped
for xls/ods and for xlsx workbooks with no CJK styles so the common
path pays no per-cell cost.

Fixes nashsu#736.
@kevin-hans kevin-hans changed the title fix: convert CJK built-in date formats in xlsx extraction (#1) fix: convert CJK built-in date formats in xlsx extraction (#736) Sep 8, 2026
@kevin-hans

Copy link
Copy Markdown
Author

@nashsu
hello,would you plz review this pr?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Excel 自訂日期格式解析錯誤

1 participant