OCR Corpora Early 2025 Publication Thread

[ctschroeder]

OCR publication thread for early 2025

Do not close this issue until all checkboxes below are complete or have been rescheduled:

List of corpora:

In Processed OCR folder (need chapter divisions + sentence splitting+full automatic NLP processing like the bible corpora)

• Giron Legendes (11 docs)
  • chapter divisions should follow
  • Daughters Zenon 132
    • chapter divisions added/checked
    • pb xml:id numbers corrected
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
    • this document should be divided into 3: 1) BNF 132.1 ff. 19-20, 2) Leiden fragment, 3) BNF 132.1 f 21
  • Daughters Zenon 78
    • chapter divisions added/checked
    • pb xml:id numbers corrected
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Daughters Zenon Crawford
    • chapter divisions added/checked
    • pb xml:id numbers corrected
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 1
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 2
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 3
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 4
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Eve Serpent
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Marina BNF
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Marina Clarendon
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Sacrifice Abraham
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py

all documents above need to be moved to https://github.com/CopticScriptorium/auto-corpora when done

In GitDox

• timothy.alexandria corpus prayer.athanasius

  • translation span (work by E. Sturgis via @amir-zeldes in progress)
  • entities and identities
  • metadata

• apocalypse.paul (2)
  - [x] corpus name needed
  - [ ] other metadata updated
  - possibly error in data -- translation on p. 1043 begins with folio 24a but OCR coptic begins in the middle of folio 6a p. 533

• pscyril.alexandria

  • On Mary still in XML mode (auto tagging?)
  • entities and identities

• pscyril.jerusalem

  • on the cross
    • needs corpus name
    • metadata updated
    • chapter & verse need to be updated in spreadsheet based on open tags in XML
  • on Mary
    • needs corpus name
    • metadata updated
    • chapter & verse need to be updated in spreadsheet based on open tags in XML

• psepiphanius on Mary

  • translation span
  • entities and identities
  • metadata

• pschrysostom

  • translation span
  • entities and identities
  • metadata

• pscelestinus

• pstimothy.alex

• psote.psoi

• timothy.discourse

[ctschroeder]

I know we were going to put Giron in the "automatically processed straight from XML like the big bible corpora" bucket, but with the sentence splitting and chapter divisions required, I think it will be easier to do both in GitDox. Chapter divisions are easier to see in the spreadsheet mode, and you can use the entity mode to do the sentence-splitting. So I'm making the executive decision to put this stuff in GitDox. @amir-zeldes @EarlyCodices

[amir-zeldes]

Fine by me - 11 docs should still be manageable, I'd be more worried for really big stuff. Keep in mind that one potential advantage of "XML Bible mode" is that tokenization is done fresh each time as tokenizers improve, so if you're correcting any segmentation manually, GD can make more sense (or you could introduce pipes in the XML repo). If you want fresh re-tokenization each release, you should not put things in GD - that would be the other main reason not to choose it.

[ctschroeder]

I don't think we are checking segmentation manually, but this is a good point. We have lots to do so I'll ponder this. They already have pipes from Lydia's work. I found when I was working on other documents for the full-auto-from-xml mode, validation was a real pain for reasons I don't fully remember and probably repressed.

[ctschroeder]

Daughers of Zeno(n) is actually Life of Hilaria. Doc titles updated. @amir-zeldes can you confirm that we only need chapter not sentence divisions for the automatically processed docs? thanks!

[amir-zeldes]

Yes, that's right - if there are chapter tags I can automatically predict sentences within those (I just need to remember to switch that option on, but unless I'm very distracted I think I'll do it the right way 😅)

[ctschroeder]

Great. Thanks @amir-zeldes! I think I will leave the documents in with pipes but remove the pipes for the auto folder. I'll validate everything and fix the metadata here and then remove the pipes when copying over.

[amir-zeldes]

You can but you don't have to - I have a setting that auto-removes them if I know a document needs re-tokenizing (which is most of auto-corpora)

[ctschroeder]

@amir-zeldes what do you do about identities for these? (I assume entities=automatic)

[ctschroeder]

Note to self: Discovered that in Life of Hilaria ("Daughters of Zeno") Giron inserted text from aLeiden ms between folios 20 and 21. The text is continuous and in the proper place, but according to our guidelines this document should be divided into 3: 1) BNF 132.1 ff. 19-20, 2) Leiden fragment, 3) BNF 132.1 f 21.

[amir-zeldes]

@amir-zeldes what do you do about identities for these? (I assume entities=automatic)

I hadn't realized this, but as of now the auto-corpora online appear to have automatic entities and zero predicted identities, but also no metadata for entity or identity quality... We should probably change that. Adding the metadata is clear, but do we want auto identities in these corpora?

The pro would be having the annotations + uniformity, the con is that errors will also contaminate the people and places pages on CTS... To be fair, quality is somewhat decent, but some errors are inevitable in NLP.

[ctschroeder]

I think identity predictions are fine but the bible corpora might be a problem re errors, since it is generally messier than the others, and errors in bound groups and segmentation multiply the later errors.
I will add entities=automatic to the OCR'd corpora metadata. Is there anything else I should do?

[ctschroeder]

@amir-zeldes the "Emperor's Daughter" work is attributed to Theodosius of Alexandria. Not clear he wrote it. We already have a corpus for theodosius.alexandria which contains the encomium for the archangel Michael. Should I add Emperor's Daughter to the existing corpus or should I make a separate one? I had it in a life.empdaughter corpus before I realized there was already a theodosius one. Likewise I have to figure out the urn -- in the "lives" bunch/namespace or theodosius