Project description • Who this project is for • Project dependencies • Instructions for use • Contributing guidelines • Additional documentation • How to get help • Terms of use
pydurma creates a clean e-text version of a Tibetan work from multiple flawed sources.
Benefits include:
- Automatic proofreading of Tibetan e-texts
- Creating high-quality e-texts from low-quality sources
- Doesn't require a spell checker (which doesn't exist yet for Tibetan language)
pydurma uses a weighted majority algorithm that compares versions of the work syllable-by-syllable and chooses the most common character from among the versions in each position of the text. Since mistakes—whether made during the woodblock carving, hand copying, digital text inputting, or OCRing process—are unlikely to be the same in the majority of the versions, they are unlikely to outrank the correct characters in any given position of the text. The result is a new clean version called a "vulgate edition."
vulgate (noun) /ˈvəl-ˌgāt/ or /ˈvʌlɡeɪt/: 2. a commonly accepted text or reading.
Medieval Latin vulgata, from Late Latin vulgata editio: edition in general circulation.
“Vulgate.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/vulgate. Accessed 23 Dec. 2022.
This less common sense of the term vulgate represents the objective of this project.
This project is intended for:
- Publishers who need clean copies of texts to publish books
- Developers who need clean data to train AI models
- Anyone who needs to proofread a Tibetan text and has access to multiple versions, such as in the BDRC library.
Before you start, ensure you've installed:
- python >= 3.7
- openpecha
- regex
- fast-diff-match-patch
To create a vulgate edition, you'll need:
- A reference pecha in the OPF format
- Several witness pechas in the OPF format (a witness is a version of a text)
Note To convert files into the OPF format, use OpenPecha Tools.
You can also convert scanned texts in the BDRC library to the OPF format with the OCR Pipeline.
To test pydurma, you can also use the OPF files in the text folder in this repo.
Assuming you've installed the software above and have OPF files:
- Clone this repo.
- Add witnesses in the OPF format into your cloned repo.
- Open
vulgatizer_op_ocr.pyin a code editor. - Update the paths to the actual witness folders in this code block:
def test_merger():
op_output = OpenPechaFS("ITEST.opf")
vulgatizer = VulgatizerOPTibOCR(op_output)
vulgatizer.add_op_witness(OpenPechaFS("./test/opfs/I001/I001.opf"))
vulgatizer.add_op_witness(OpenPechaFS("./test/opfs/I002/I002.opf"))
vulgatizer.add_op_witness(OpenPechaFS("./test/opfs/I003/I003.opf"))
vulgatizer.create_vulgate()
- Run
vulgatizer_op_ocr.py
The vulgate edition OPF will be saved in ./data/opfs/generic_editions.
- No code to detect transpositions.
pydurma creates vulgate editions in three steps:
- Preprocessing
- Alignment
- Vulgatization
Here is that process:
- text_alignment_tool
- CollateX
- MEDITE
- Juxta (sources, uses difflib)
- hypercollate
- helayo (paper)
- Versioning Machine
- nmergec ("en-merge-see")
- ocromore
- This was also done by Oliver Hellwig apparently, research to be one
- Sequence Alignment (Wikipedia)
- MSA: Multiple Sequence Alignment (Wikipedia)
- Smith–Waterman algorithm
- Needleman–Wunsch algorithm
- Spencer 2004: Spencer M., Howe and Christopher J., 2004. Collating Texts Using Progressive Multiple Alignment. Computers and the Humanities. 38/2004, 253–270.
If you'd like to help out, check out our contributing guidelines.
pydurma is licensed under the MIT license.