Skip to content

Classify localization folders by name, not by the whole path - #30

Merged
biodranik merged 1 commit into
organicmapsfrom
ab/language-folder-classification
Aug 22, 2026
Merged

biodranik merged 1 commit into
organicmapsfrom
ab/language-folder-classification

Conversation

@biodranik

Copy link
Copy Markdown
Member

determine_language_given_path() falls back to scanning every segment of the path it is given for ^[a-z]{2,3}(-[A-Za-z]{2,4})?$, and generate_all_localization_files() handed it the joined absolute path of each candidate folder. Any short lowercase ancestor of the checkout matches:

/private/tmp/omim/iphone/Chart/Chart/ChartData   -> "tmp"   misfire
/home/ab/omim/iphone/Chart/Chart/Views           -> "ab"    misfire
/opt/build/omim/data/sound-strings/en_GB.json    -> "opt"   misfire
/Users/alex/Developer/omim/…                     -> None    ok

So every non-language subfolder of an output directory was treated as a language. With --include all that wrote full English files into source folders; with --include translated the output was empty and silently skipped, which only hid the same misclassification. Worst of all, generated output depended on where the repository happened to be cloned.

Both generate_all_localization_files() and find_translation_files() iterate the direct children of a single directory, so the language can only come from the child's own name — ancestors are meaningless there. Pass the name instead. _prepare_read_write() is left alone: it receives an arbitrary file path such as …/en.lproj/Localizable.strings, where the segment walk is the point.

That alone still left folders named raw and xml matching. Formatters with a fixed convention now return None rather than falling back to the generic regex: an Apple localization folder is always <lang>.lproj, an Android one always values[-lang].

The same flaw is in lib/twine/runner.rb; changed there too for parity.

Testing

Running organicmaps' tools/unix/generate_localizations.sh from a checkout under /private/tmp, against a clean tree:

before after
working tree after generation 1 file overwritten, 2 bogus files created clean
files generated 675 672
"would not contain any translations" skips 68 6 (all genuine)

The 672 legitimate files are byte-identical to what is checked in, and a second run is idempotent.

pytest: 134 passed, including two new regression tests. One drives the runner through a TemporaryDirectory, which reproduces the original bug naturally — macOS temp dirs live under /var/folders.

@biodranik

This comment was marked as resolved.

@biodranik
biodranik force-pushed the ab/language-folder-classification branch from 7c4be35 to 085e783 Compare August 21, 2026 18:53
@biodranik
biodranik changed the base branch from organicmaps to ab/ruff-fixes August 21, 2026 18:53
@biodranik

Copy link
Copy Markdown
Member Author

Rebased onto #31, which fixes the pre-existing Ruff findings described above, so this now targets ab/ruff-fixes instead of organicmaps. Merge #31 first and GitHub will retarget this back to organicmaps automatically.

The only conflict was the import block in test_integration.py, where #31's isort pass and this branch's new Runner import met. pytest passes 134 (132 + the two regression tests), ruff check is clean, and organicmaps' generate_localizations.sh still emits 672 files with 6 genuine skips and a clean tree.

@biodranik
biodranik requested review from strump and vng August 21, 2026 19:08
biodranik

This comment was marked as resolved.

@biodranik

This comment was marked as resolved.

@biodranik
biodranik force-pushed the ab/language-folder-classification branch from 085e783 to d9d0344 Compare August 21, 2026 19:46

@strump strump left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Base automatically changed from ab/ruff-fixes to organicmaps August 22, 2026 16:18
determine_language_given_path() falls back to scanning every path segment for
something matching ^[a-z]{2,3}(-[A-Za-z]{2,4})?$, and the two directory scans
handed it the joined absolute path of each candidate folder. Any short
lowercase ancestor of the repository matches: /tmp, /var, /opt, /home/ab. A
checkout under such a path made every non-language subfolder of an output
directory look like a language, so generate-all-localization-files wrote
into it.

With --include all (Apple, jQuery) that produced whole English files in
source folders; with --include translated (Android) the output was empty and
silently skipped, which only hid the same misclassification. It also made the
generated output depend on where the repository happens to live.

generate_all_localization_files and find_translation_files both iterate the
direct children of one directory, so the language can only come from the
child's own name. They now ask language_for_folder_name for it. The Apple and
Android formatters override it to accept only <lang>.lproj and values[-lang],
which is what folders named "raw" or "xml" were relying on not being checked.

determine_language_given_path itself is untouched, because it is also reached
from _prepare_read_write with a path the user named, where a missing language
silently falls back to the developer language: tightening it there would make
`consume-localization-file de.strings --format apple` import German as
English.

Mirrored in lib/twine so both implementations agree.

Signed-off-by: Alexander Borsuk <me@alex.bio>
@biodranik
biodranik force-pushed the ab/language-folder-classification branch from d9d0344 to 18b63ed Compare August 22, 2026 16:18
@biodranik
biodranik merged commit 0b5d80d into organicmaps Aug 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants