Both halves of the library — shaping and normalization (MNG / Hudum) — are verified against the same upstream corpora. CI runs the full suite on Python 3.9 – 3.13 on every push.
shape() and same_shape() are cross-validated against two upstream TSV suites:
| Suite | Cases | Pass | Notes |
|---|---|---|---|
mongfontbuilder/core-hud.tsv |
225 | 100% | curated regression set |
mongfontbuilder/eac-hud.tsv (GB/T 25914-2023) |
3513 | 100% | 5 cases excluded as UTN ↔ EAC xfail, matching mongfontbuilder's own pytest.mark.xfail set |
| Hand-written unit tests | — | 100% | shape / same_shape / joiner tokens (nirugu, ZWJ) |
normalize() / normalize_text() / normalize_written_units() are pure functions of shape with invariants checked in CI over every corpus encoding:
| Property | Result |
|---|---|
Round-trip — shape(normalize(x)) == shape(x) |
3757 / 3757 corpus encodings (100%) |
| Shape-canonicity — same shape ⟹ same Unicode output | 1993 / 1993 shape groups (100%) |
| Prefix-stability — word and word+suffix share their prefix encoding | 99.87% of real corpus pairs |
Scope note: normalization is implemented for MNG (Hudum) only — Todo / Sibe / Manchu load shaping rules but have no normalizer yet.
This project was generated with Claude Code (AI-assisted coding). The tests and key parts of the core code have been manually reviewed, and test coverage is extensive (corpus round-trip / shape-canonicity / prefix-stability plus the upstream cross-implementation suites). Treat this as a preview release — it should be fine for normal use; if you hit a problem, please open an issue or PR. Shaping logic is derived from UTN #57 v4 and mongfontbuilder.
Traditional Mongolian script in Unicode has a fundamental problem: the same visible word can be encoded in multiple different Unicode sequences. This happens because:
- Letters share glyphs — A and E look identical in medial and final positions; O/U and OE/UE share forms; QA and GA share forms depending on vowel harmony.
- Multiple encoding paths — The same tooth glyph (I) can be encoded as I, YA+FVS1, or even two separate I characters.
- Redundant FVS usage — Free Variation Selectors (FVS1–FVS4) can create equivalent sequences that render identically.
- Joining controls — nirugu (U+180A, the visible stem extender) and ZWJ (U+200D) force letters into their joined forms, and inside those joined contexts even more letters collapse to the same glyph (
nirugu+oandnirugu+urender identically). - Suffix particles after MVS/NNBSP — the same rendered suffix can be spelled with different letters (
MVS+aandMVS+eboth render the chachlag form;MVS+uandMVS+uerender the same connector).
This means:
- Search fails: Searching for "sain" (one encoding) won't find the same word in another encoding, even though they look identical.
- Deduplication breaks: The same word has multiple Unicode representations.
- Indexing is unreliable: Different encodings of the same word produce different keys.
This is a shaping-aware normalizer for Traditional Mongolian. It:
- Shapes the input using the full UTN #57 v4 shaping process (5-step conditional mapping)
- Compares glyph sequences to detect identical visual forms
- Normalizes to one canonical, FVS-pinned Unicode encoding — same shape ⟹ same Unicode, and it always round-trips
Example: All five of these encode the word "sain" (good) and look identical:
The normalizer implements a lightweight Mongolian shaping engine — equivalent to what HarfBuzz does with a font file, but using only the rule data from UTN #57 v4 and the mongfontbuilder project. No font files needed.
- Chachlag — Suffix forms for A/E after MVS (Mongolian Vowel Separator)
- Syllabic — Consonant/vowel context: onset, devsger, marked, masculine/feminine harmony, dotless
- Particle — MVS particle dictionary lookup for specific suffix words
- Devsger — I after a vowel (vowel_devsger) gets double-tooth form:
I → I+I - Post-bowed — Vowel forms change after bowed consonants (G, B, K, P, F)
normalize is a pure function of shape: any two encodings that shape identically produce the same Unicode output, and the output always round-trips — shape(normalize(x)) == shape(x). It is also prefix-stable. When these goals conflict the priority is round-trip > prefix-stable > shortest.
Per word:
- shape the input into its written-unit sequence. Structural characters — MVS, nirugu, ZWJ — appear verbatim as PascalCase
Mvs/Nirugu/Zwjtokens (nirugu renders a visible stem; all three are the evidence for a neighbour's init/medi/fina form). Split the shape at these tokens into chains; the tokens themselves are copied through unchanged. - encode each chain (right-to-left, so appending a suffix can't disturb what precedes it):
- partition + table lookup — the primary path. At each position take the single unit if the table has it (preferred — clean output), else the longest available multi-unit entry, and look up
(position, written-unit) → (letter, FVS)in an FVS-pinned table. Each value renders its unit regardless of neighbours, so the result is a deterministic, O(N), prefix-stable function of the shape. - velar-feminine refinement — a
G/Gxvelar's forward-coupled vowel (a/o/u) is swapped to its feminine partner (e/oe/ue) for clean output. - verify — reshape the candidate in full context; accept only if it equals the target chain shape.
- no search fallback — the table is total over the corpus (with FVS-first selection there are no gap chains left). If an out-of-corpus shape ever misses the table, the safety net returns the input unchanged (round-trip preserved, never a silent mis-encoding). A letter next to a joiner simply looks its unit up at the shifted (joined) position.
- partition + table lookup — the primary path. At each position take the single unit if the table has it (preferred — clean output), else the longest available multi-unit entry, and look up
- post-MVS suffix rule — a chain directly after MVS takes its standalone canonical (drop the MVS, normalize, re-attach), so the spelling never depends on MVS. One exception: chachlag
Aaafter MVS is written the bare lettera. (The isolate-I→i+FVS1spelling is pinned in the table itself — no post-processing pass exists.)
Prefix-stability means: if word A = word B + a suffix and their shapes share a prefix, the shared region encodes identically except the single boundary unit whose position changes (final in B → medial in A). The per-unit table delivers this for free — each unit's encoding depends only on its own position, never on its neighbours.
How the table is built (the selection method): offline, a context-independence battery fills each (position, written-unit) slot with the (letter, FVS) that renders exactly that unit in every probed neighbour context (the probes include a bowed consonant, so post-bowed effects can't hide). Candidate order is letter-major, FVS-first within the letter — an FVS exists precisely to pin a form against context, so the pinned variant of the right letter always beats its context-sensitive bare form. The result is exported and shipped as JSON; the battery lives in scripts/gen_normalize_table.py.
Note: output is FVS-pinned, not bare — each unit carries the selector that fixes its form independent of context. This is what makes "same shape ⟹ same Unicode" and prefix-stability hold. The per-unit table is exported as language-agnostic JSON (
mongol_norm/data/MNG.normalize.json); schema + consuming algorithm are in docs/data-format.md, so ports in other languages can implementnormalizewith just a JSON parser.
The exact canonical selection policy is frozen as mng-canonical/1. It is available as shaper.canonical_version and embedded in MNG.normalize.json. Applications that persist normalized search/index keys should store this version alongside them and rebuild those keys if a future release changes it.
mongol-norm is a single self-contained package on PyPI — the shaping/normalize data is bundled, no runtime dependencies:
pip install mongol-normOr from source:
git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .from mongol_norm import MongolianShaper
shaper = MongolianShaper(locale="MNG") # Hudum Traditional Mongolian
# Shape: get written-unit sequence
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']
# Compare: are two encodings visually identical?
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True
shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢ᠍ᠮᠠ")
# → False
# Normalize: one canonical, FVS-pinned Unicode (same shape ⟹ same Unicode)
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
# Normalize an already-shaped written-unit sequence
shaper.normalize_written_units(["B", "Aa"])
# → 'ᠪᠠ᠋'
# Every written unit is PascalCase, including shape() control output
shaper.normalize_written_units(["S", "A", "I", "I", "N", "Mvs", "Aa"])
# Encode authoritative HUD written-unit positions without inferring controls
shaper.normalize_positioned_written_units([
{"unit": "B", "position": "init"},
{"unit": "Aa", "position": "fina"},
])
# → 'ᠪᠠ᠋'
# HUD position is not Unicode topology: isolated FA borrows F:init
shaper.normalize_positioned_written_units([
{"unit": "F", "position": "init"},
])
# → 'ᠹ' (bare U+1839, no ZWJ); F:isol is unsupportednormalize_written_units() accepts an ordered Sequence[str] of shape units,
not nominal Unicode. Letter positions are inferred from unit order and structural
controls; explicit position records are not accepted by this API. All written-unit
names use PascalCase; structural controls are Mvs, Nirugu, and Zwj, exactly
as returned by shape(). Old lowercase or all-uppercase control aliases are not
accepted. The API never infers or inserts a structural control: ZWJ is present in
the output only when Zwj is present
in the requested sequence. An empty sequence returns an empty string. A malformed
outer input or a non-string item raises TypeError; unknown units and sequences
that cannot reshape exactly raise ValueError rather than being guessed or
partially encoded.
normalize_positioned_written_units() accepts an ordered sequence of exact
built-in {"unit": str, "position": str} dict records. Here position is the
position of the written unit in the authoritative HUD inventory, not a Unicode
letter's joining topology. It reuses normalize_written_units() rather than a
second encoding table. A complete multi-record chain starts with init and ends
with fina. An incomplete left or right edge gets an implicit Zwj; for example
B:medi, O:medi, G:fina is normalized as Zwj, B, O, G. A single init record
is the exception and is normalized without ZWJ, so F:init becomes bare U+1839.
A single medi gets ZWJ on both sides and a single fina gets ZWJ on the left.
F:isol is absent from the source inventory and fails closed. Mvs and Nirugu
use control; explicit Zwj input is rejected. A wrong outer/record/field type
raises TypeError; wrong keys, unit, position, chain positions, exact encoding,
or more than 1024 records raises ValueError. This API has no CLI subcommand yet.
normalize() operates on single words. For sentences, paragraphs, or mixed-script text, use normalize_text() — it normalizes each Mongolian word independently while preserving spaces, punctuation, and non-Mongolian text verbatim.
# Normalize a sentence (each Mongolian word normalized independently)
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢ᠍ᠮᠠ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ ᠨᠠᠢ᠍ᠮᠠ᠌'
# Mixed script: non-Mongolian text preserved as-is
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} inputs → {len(unique)} unique form(s): {unique}")
# 4 inputs → 1 unique form(s): {'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}After pip install mongol-norm, the mongol-norm command is on PATH (or run python -m mongol_norm.shaper ... without installing).
# Inline text
mongol-norm shape 'ᠰᠠᠢᠨ' # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ' # canonical form
mongol-norm normalize-written-units 'B+Aa' # → ᠪᠠ᠋
mongol-norm normalize-written-units 'BZwj' # compact PascalCase units
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ' # mixed script
# Pipe / stdin (use `-` as the text)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
echo 'B+Aa' | mongol-norm normalize-written-units -
cat doc.txt | mongol-norm normalize-text -
# File in / out
mongol-norm normalize-text -i in.txt -o out.txt
# Batch: one word per line in, one canonical per line out
mongol-norm normalize --batch -i words.txt -o canonical.txt
# Written-unit batch: one compact or '+'-joined sequence per line
mongol-norm normalize-written-units --batch -i units.txt -o canonical.txt
# Visual-identity check (exit 0 if same, 1 if different)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'normalize-written-units accepts compact PascalCase or explicit + boundaries.
Compact input must have one unique segmentation; ambiguous input fails closed and
must be rewritten with +. After parsing, the same exact-shape validation as
normalize_written_units() applies, so a syntactically valid unit stream can
still be rejected when it has no canonical MNG encoding.
normalize (single-word) skips non-Mongolian characters, so feeding it a multi-line file treats the whole thing as one word. Use --batch for one-word-per-line files, or normalize-text for free-form text.
cd mongol-norm
# Shaping + same_shape + normalize unit tests
python -m unittest tests.test_shaper -v
# Normalize properties: round-trip + shape-canonicity + prefix-stability
python -m unittest tests.test_round_trip
# Normalize-table export (compute == load)
python -m unittest tests.test_normalize_table
# mongfontbuilder core-hud (225) + GB/T 25914-2023 eac-hud (3513, 5 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud
# Or all together (auto-discovers every tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'The hand-written suite covers:
| Test class | What it checks |
|---|---|
TestShape |
shape() returns correct written-unit sequence (sain variants, the 5 shaping phases step-by-step, UTN-vs-EAC divergences) |
TestSameShape |
same_shape() correctly identifies visually identical vs. distinct encodings |
TestNormalize |
normalize() produces canonical output; idempotency; normalized result matches original visually |
TestNormalizeText |
normalize_text() handles multi-word, mixed-script, punctuation, empty input; idempotency; word independence |
TestNormalizeWrittenUnits |
public shape-unit input, structural controls, validation, and exact reshape |
TestNormalizePositionedWrittenUnits |
authoritative HUD pair inventory, borrowed-position variants, generation-time exact-position audits, no inserted/explicit ZWJ, and fail-closed records |
TestNormalizeWrittenUnitsCli |
inline/stdin/batch CLI input, canonical control spelling, and parser errors |
TestNNBSP |
NNBSP ↔ MVS equivalence (UTN model) |
Current totals: 214 tests (unit + property + 225 core-hud + 3513 eac-hud corpus runners), all green on Python 3.9 – 3.13.
- Search & Retrieval — Index Mongolian text with unique keys per visual word
- Deduplication — Detect identical words encoded differently
- Spell Checking — Normalize before dictionary lookup
- Corpus Linguistics — Consistent word frequency counts
- OCR Post-processing — Standardize OCR output that may use inconsistent encodings
- Input Method Engines — Validate and normalize user input
mongol-norm/ # the repo = the package (single, self-contained)
├── .github/workflows/test.yml # CI: Python 3.9-3.13 on every push
├── pyproject.toml
├── mongol_norm/
│ ├── shaper.py # tokenize / assign_positions / shape / normalize
│ ├── rules.py # the 5 shaping phases (iii1..iii5) mirroring iii.py
│ ├── _data.py # loaders for the bundled JSON
│ └── data/ # bundled shaping + normalize data
│ ├── MNG.json TOD.json SIB.json MCH.json
│ └── MNG.normalize.json # per-unit normalize table
├── scripts/ # dev-only generators (preprocess, gen_normalize_table)
├── docs/data-format.md # JSON schema, for other-language ports
└── tests/
├── test_shaper.py test_round_trip.py test_normalize_table.py
├── test_written_units_api.py
├── test_core_hud.py test_eac_hud.py
└── data/{core,eac}-hud.tsv # vendored from mongfontbuilder
mongol-norm has no runtime dependencies — the shaping/normalize JSON is bundled in mongol_norm/data/. Install with pip install mongol-norm.
- UTN #57 v4 — Unicode Technical Note: Encoding and Shaping of the Mongolian Script. The authoritative specification for Mongolian shaping rules.
- mongfontbuilder by Kushim Jiang — Source for the bundled flat variant tables in
mongol_norm/data/(preprocessed fromdata.variants/data.particles) and for thecore-hud.tsv/eac-hud.tsvregression suites we vendor intotests/data/. Both UTN #57 and mongfontbuilder are authored by the same person. - GB/T 25914—2023 — China national standard for Traditional Mongolian nominal characters; source of the EAC compliance test set.
- Claude Code — This project was developed with AI assistance. The shaping rules are derived from the above sources; Claude Code was used to implement and structure the engine.
| Locale | Script | Status |
|---|---|---|
| MNG | Hudum (Traditional Mongolian) | ✅ Full shaping + normalization |
| TOD | Todo | ⬜ Shaping rules generated, normalization WIP |
| SIB | Sibe | ⬜ Shaping rules generated, normalization WIP |
| MCH | Manchu | ⬜ Shaping rules generated, normalization WIP |
- Python 3.6+ (CI-tested on 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
- No runtime dependencies (shaping/normalize data is bundled)
MIT License — see LICENSE.
The shaping rules and bundled data are derived from mongfontbuilder (MIT) and UTN #57. Their required notices are retained in NOTICE.
库的两部分 —— 整形与规范化(MNG / Hudum)—— 都对照同一批上游语料验证。CI 在每次 push 上对 Python 3.9 – 3.13 跑完整套件。
shape() 和 same_shape() 对照两套上游 TSV 套件交叉验证:
| 套件 | 用例数 | 通过 | 说明 |
|---|---|---|---|
mongfontbuilder/core-hud.tsv |
225 | 100% | 精选回归集 |
mongfontbuilder/eac-hud.tsv (GB/T 25914-2023) |
3513 | 100% | 5 个 UTN ↔ EAC 分歧 case 跳过(跟 mongfontbuilder 自己的 pytest.mark.xfail 列表一致) |
| 手写单元测试 | — | 100% | shape / same_shape / joiner token(nirugu、ZWJ) |
normalize() / normalize_text() / normalize_written_units() 是 shape 的纯函数,以下不变量在 CI 中对每一条语料编码逐一验证:
| 性质 | 结果 |
|---|---|
往返 —— shape(normalize(x)) == shape(x) |
3757 / 3757 语料编码(100%) |
| 同形同码 —— shape 相同 ⟹ 输出 Unicode 相同 | 1993 / 1993 shape 组(100%) |
| 前缀稳定 —— 词与词+后缀共享前缀编码 | 99.87% 真实语料词对 |
范围说明:规范化目前只实现了 MNG(Hudum)—— Todo / 锡伯文 / 满文已加载 shaping 规则,尚无规范化。
本项目由 Claude Code(AI 辅助编码)生成;测试与部分核心代码经人工审核,测试覆盖比较充分(语料往返 / 同形同码 / 前缀稳定 + 上游跨实现套件)。当前为预览版,正常使用应无问题;遇到问题欢迎提 issue 和 PR。Shaping 逻辑源自 UTN #57 v4 和 mongfontbuilder。
传统蒙古文在 Unicode 中存在一个根本性问题：同一个可见词形可以用多种不同的 Unicode 序列编码。原因是：
- 字母共享字形 — A 和 E 在中间和尾部位置外形完全相同；O/U、OE/UE 共享形态；QA 和 GA 根据元音和谐共享形态。
- 多种编码路径 — 同一个齿形字形可以编码为 I、YA+FVS1，甚至两个独立的 I 字符。
- 冗余的 FVS 使用 — 自由变体选择符（FVS1–FVS4）可以创建渲染结果完全相同的等价序列。
- 连接控制符 — nirugu(U+180A,可见的连笔延长符)和 ZWJ(U+200D)会强制字母取连接形,而连接语境下更多字母塌缩成同一字形(
nirugu+o和nirugu+u渲染完全相同)。 - MVS/NNBSP 后的后缀词 — 同一个渲染出的后缀可以用不同字母拼写(
MVS+a和MVS+e都渲染 chachlag 形;MVS+u和MVS+ue同形)。
这意味着：
- 搜索失效：搜索同一个词的某种编码，找不到另一种编码，尽管它们外形完全一样。
- 去重失败：同一个词有多种 Unicode 表示。
- 索引不可靠：同一个词的不同编码产生不同的索引键。
这是一个形态感知的蒙古文规范化器。它：
- 使用完整的 UTN #57 v4 shaping 过程（5 步条件映射）对输入进行字形化
- 通过比较字形序列来检测视觉上相同的词形
- 规范化为唯一的、FVS 钉死的 canonical Unicode 编码 —— 同 shape ⟹ 同 Unicode,且必定往返还原
示例：以下五种编码都表示 "sain"（好的），外形完全相同：
本规范化器实现了一个轻量级蒙古文 shaping 引擎——功能相当于 HarfBuzz 配合字体文件所做的事情，但仅使用 UTN #57 v4 的规则数据和 mongfontbuilder 项目的变体数据。不需要字体文件。
| 步骤 | 名称 | 说明 |
|---|---|---|
| 1 | Chachlag | MVS（蒙古文元音分隔符）后的 a/e 后缀形态 |
| 2 | Syllabic | 辅音/元音上下文：onset/devsger/marked/阴阳和谐/dotless |
| 3 | Particle | MVS 小品词词典查找 |
| 4 | Devsger | 元音后的 i 获得双齿形态：I → I+I（vowel_devsger） |
| 5 | Post-bowed | 弓形辅音(G/B/K/P/F)后的元音形态变化 |
normalize 是 shape 的纯函数:任意两个 shape 相同的编码,normalize 输出相同,且始终往返成立 —— shape(normalize(x)) == shape(x),同时前缀稳定。三者冲突时优先级:往返 > 前缀稳定 > 最短。
逐词:
- shape 成书写单元序列。结构字符 —— MVS、nirugu、ZWJ —— 原样输出为PascalCase
Mvs/Nirugu/Zwjtoken(nirugu 是可见的连笔字形;三者都是邻居字母 init/medi/fina 形的依据)。按这些 token 切成 chain,token 本身原样拷贝。 - 逐 chain 编码(从右往左,这样加后缀不影响前面):
- 划分 + 查表(主路径):每个位置优先取单单元(输出干净),否则取最长多单元,查
(位置, 书写单元) → (字母, FVS)的 FVS 钉死表。每个值不依赖邻居就渲染出该单元 → 确定性、O(N)、前缀稳定。 - velar 阴性微调:
G/Gx前向耦合的元音(a/o/u)换成阴性(e/oe/ue),输出更干净。 - 校验:在完整上下文里重新 shape,只接受与目标 chain shape 一致的结果。
- 没有搜索兜底:FVS 优先的选择下,表对全部语料 chain 是完备的(缺口为零)。语料外的 shape 万一查不到表,安全网原样返回输入(保住往返,绝不静默错编)。紧邻 joiner 的字母只是按移动后的连接位置查表。
- 划分 + 查表(主路径):每个位置优先取单单元(输出干净),否则取最长多单元,查
- MVS 后缀规则:紧跟 MVS 的 chain 用其 standalone canonical(去掉 MVS、归一、再拼回),拼写不依赖 MVS。唯一例外:MVS 后的 chachlag
Aa写裸字母a。(孤立I→i+FVS1的拼写已钉进表本身 —— 不存在后处理。)
前缀稳定的含义:若词 A = 词 B + 后缀,且二者 shape 共享前缀,则共享部分编码完全一致,只有那个位置发生变化的边界单元不同(在 B 里是词尾、在 A 里变词中)。逐单元表天然保证这点 —— 每个单元的编码只取决于它自己的位置,与邻居无关。
表是怎么来的(选择方法):离线跑一个 context 无关性电池 —— 对每个 (位置, 书写单元),挑出在所有探测邻居上下文里都恰好渲染出该单元的 (字母, FVS)(探针含弓形辅音,post-bowed 效应藏不住)。候选顺序是字母优先、字母内 FVS 优先 —— FVS 的意义就是把字形从 context 里隔离出来,所以正确字母的钉死形永远优于其受感染的裸形。结果导出成 JSON 随包发布;电池在 scripts/gen_normalize_table.py。
注意:输出是 FVS 钉死而非 bare —— 每个单元都带着把字形固定住、不受上下文影响的选择符,这正是"同 shape ⟹ 同 Unicode"和前缀稳定成立的原因。逐单元表导出为语言无关的 JSON(
mongol_norm/data/MNG.normalize.json),schema 与消费算法见 docs/data-format.md;其他语言只需一个 JSON 解析器即可实现 normalize。
当前精确 canonical 选择策略冻结为 mng-canonical/1。可通过 shaper.canonical_version 读取，并写入 MNG.normalize.json。持久化规范化搜索键/索引键的应用应同时保存该版本；未来版本若发生变化，应重建这些键。
mongol-norm 是单一自包含包,已发布到 PyPI —— shaping/normalize 数据内置,零运行时依赖:
pip install mongol-norm或从源码安装:
git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .from mongol_norm import MongolianShaper
shaper = MongolianShaper(locale="MNG") # Hudum 传统蒙文
# 字形化：获取书写单元序列
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']
# 比较：两个编码视觉上是否相同？
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True
shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢ᠍ᠮᠠ")
# → False
# 规范化:唯一的 FVS 钉死 canonical(同 shape ⟹ 同 Unicode)
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
# 从已经shape好的书写单元序列直接生成canonical Unicode
shaper.normalize_written_units(["B", "Aa"])
# → 'ᠪᠠ᠋'
# 所有书写单元均为PascalCase，包括shape()输出的结构control
shaper.normalize_written_units(["S", "A", "I", "I", "N", "Mvs", "Aa"])
# 按权威HUD written-unit position编码，不推断或插入control
shaper.normalize_positioned_written_units([
{"unit": "B", "position": "init"},
{"unit": "Aa", "position": "fina"},
])
# → 'ᠪᠠ᠋'
# HUD position不是Unicode topology：FA的isolated variant借用F:init
shaper.normalize_positioned_written_units([
{"unit": "F", "position": "init"},
])
# → 'ᠹ'（裸U+1839，不含ZWJ）；F:isol不受支持normalize_written_units()接受由shape unit组成的有序Sequence[str]，而不是
nominal Unicode。字母位置由单元顺序与结构control推导；此API不接受显式
position record。所有written-unit名称统一使用PascalCase；结构control为
Mvs、Nirugu、Zwj，与shape()输出完全一致。旧小写或全大写control别名
不再接受。API绝不自行推断或插入结构control：只有显式包含Zwj时，输出才会
包含ZWJ。空序列返回空字符串。非法外层输入
或非字符串单元抛出TypeError；未知unit或无法精确重新shape的序列抛出
ValueError，不会猜测或返回部分编码结果。
normalize_positioned_written_units()接受由严格内建
{"unit": str, "position": str} dict record组成的有序序列。这里的position
表示权威HUD inventory中的written-unit position，不是Unicode字母在当前序列中的
joining topology。它直接复用normalize_written_units()，不再维护第二套编码表。
完整复合链必须以init开头、以fina结束；左端或右端不完整时自动补Zwj。
例如B:medi, O:medi, G:fina会按Zwj, B, O, G规范化。单个init是特例，
不补ZWJ，因此F:init输出裸U+1839；单个medi前后补ZWJ，单个fina只在
左侧补ZWJ。inventory中不存在的F:isol会fail closed。Mvs与Nirugu使用
control，显式Zwj输入被拒绝。外层/record/字段类型错误抛出TypeError；
keys、unit、position、复合链位置、exact encoding错误以及超过1024条record均
抛出ValueError。本API暂不增加CLI命令。
normalize() 作用于单个词。对于句子、段落或混合文字文本，使用 normalize_text() ——它独立规范化每个蒙古文词，同时原样保留空格、标点和非蒙古文文本。
# 规范化句子（每个蒙古文词独立规范化）
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢ᠍ᠮᠠ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ ᠨᠠᠢ᠍ᠮᠠ᠌'
# 混合文字：非蒙古文文本原样保留
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} 个输入 → {len(unique)} 个唯一形态：{unique}")
# 4 个输入 → 1 个唯一形态：{'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}pip install mongol-norm 之后,mongol-norm 命令即在 PATH 上(不安装也可用 python -m mongol_norm.shaper ...)。
# 直接传文本
mongol-norm shape 'ᠰᠠᠢᠨ' # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ' # 输出 canonical
mongol-norm normalize-written-units 'B+Aa' # → ᠪᠠ᠋
mongol-norm normalize-written-units 'BZwj' # 紧凑PascalCase单元串
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ' # 混合文字
# 管道 / 标准输入(文本位置写 `-`)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
echo 'B+Aa' | mongol-norm normalize-written-units -
cat doc.txt | mongol-norm normalize-text -
# 文件输入 / 输出
mongol-norm normalize-text -i in.txt -o out.txt
# 批量:一行一词输入,一行一个 canonical 输出
mongol-norm normalize --batch -i words.txt -o canonical.txt
# 书写单元批量:每行一个紧凑或`+`分隔序列
mongol-norm normalize-written-units --batch -i units.txt -o canonical.txt
# 视觉等价检查(相同退出码 0,不同退出码 1)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'normalize-written-units接受紧凑PascalCase或显式+边界。紧凑输入必须只有
一种合法切分；存在歧义时fail closed，调用方须改用+。解析后继续执行与
normalize_written_units()相同的exact-shape校验，因此语法合法的unit stream
若没有canonical MNG编码仍会被拒绝。
normalize(单词模式)会跳过非蒙古文字符,多行文件直接喂给它会被当成一整个词;一行一词的文件请用 --batch,自由文本请用 normalize-text。
cd mongol-norm
# 手写 shaper / same_shape / normalize 测试
python -m unittest tests.test_shaper -v
# normalize 性质:往返 + 同 shape 同输出 + 前缀稳定
python -m unittest tests.test_round_trip
# normalize 表导出(compute == load)
python -m unittest tests.test_normalize_table
# mongfontbuilder core-hud(225)+ GB/T 25914-2023 eac-hud(3513,5 个 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud
# 或一次跑全部(自动发现所有 tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'手写套件覆盖范围:
| 测试类 | 测试内容 |
|---|---|
TestShape |
shape() 输出正确的书写单元序列(sain 变体、5 步 shaping 分步测试、UTN-vs-EAC 分歧) |
TestSameShape |
same_shape() 正确识别外形相同 vs 不同的编码 |
TestNormalize |
normalize() 输出规范结果; 幂等性; 规范化后与原始词形视觉相同 |
TestNormalizeText |
normalize_text() 处理多词、混合文字、标点、空输入; 幂等性; 词独立性 |
TestNormalizeWrittenUnits |
公开shape-unit输入、结构control、校验与精确回形 |
TestNormalizePositionedWrittenUnits |
权威HUD pair inventory、borrowed-position variant、生成期精确position审计、禁止插入/输入ZWJ及fail-closed record |
TestNormalizeWrittenUnitsCli |
inline/stdin/batch CLI输入、control标准拼写与解析错误 |
TestNNBSP |
NNBSP ↔ MVS 等价性(UTN 模型) |
当前总数: 214 个测试(单元 + 性质 + 225 core-hud + 3513 eac-hud 语料跑批), 在 Python 3.9 – 3.13 上全绿。
- 搜索与检索 — 为每个可见词形建立唯一索引键
- 文本去重 — 检测编码不同但外形相同的词
- 拼写检查 — 规范化后再查词典
- 语料库语言学 — 一致的词频统计
- OCR 后处理 — 标准化可能使用不一致编码的 OCR 输出
- 输入法引擎 — 验证和规范化用户输入
mongol-norm/ # 仓库 = 包(单一自包含)
├── .github/workflows/test.yml # CI: 每次 push 跑 Python 3.9-3.13
├── pyproject.toml
├── mongol_norm/
│ ├── shaper.py # tokenize / assign_positions / shape / normalize
│ ├── rules.py # 5 步 shaping 阶段 (iii1..iii5) 镜像 iii.py
│ ├── _data.py # 内置 JSON 的加载器
│ └── data/ # 内置 shaping + normalize 数据
│ ├── MNG.json TOD.json SIB.json MCH.json
│ └── MNG.normalize.json # 逐单元 normalize 表
├── scripts/ # 仅开发用的生成脚本 (preprocess, gen_normalize_table)
├── docs/data-format.md # JSON schema, 供其他语言移植
└── tests/
├── test_shaper.py test_round_trip.py test_normalize_table.py
├── test_written_units_api.py
├── test_core_hud.py test_eac_hud.py
└── data/{core,eac}-hud.tsv # 来自 mongfontbuilder
mongol-norm 没有运行时依赖 —— shaping/normalize JSON 内置在 mongol_norm/data/。安装:pip install mongol-norm。
- UTN #57 v4 — Unicode 技术注释：蒙古文编码与字形化。蒙古文 shaping 规则的权威规范。
- mongfontbuilder(Kushim Jiang)—
mongol_norm/data/内置扁平变体表的来源(从data.variants/data.particles预处理而来), 同时也是我们 vendor 进tests/data/的core-hud.tsv/eac-hud.tsv回归套件的来源。UTN #57 和 mongfontbuilder 的作者是同一人。 - GB/T 25914—2023 — 中国国家标准：传统蒙古文名义字符、表现字符和控制字符使用规则; EAC 一致性测试集的来源。
- Claude Code — 本项目使用 AI 辅助开发。shaping 规则来源于上述数据；Claude Code 用于实现和组织引擎代码。
| Locale | 文字 | 状态 |
|---|---|---|
| MNG | Hudum（传统蒙文） | ✅ 完整 shaping + 规范化 |
| TOD | Todo（托忒文） | ⬜ shaping 规则已生成，规范化开发中 |
| SIB | Sibe（锡伯文） | ⬜ shaping 规则已生成，规范化开发中 |
| MCH | Manchu（满文） | ⬜ shaping 规则已生成，规范化开发中 |
- Python 3.6+(CI 实测矩阵: 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
- 无运行时依赖(shaping/normalize 数据已内置)
MIT License —— 见 LICENSE。
整形规则与内置数据派生自 mongfontbuilder(MIT)和 UTN #57,其许可证要求的署名保留在 NOTICE 中。