Match user-written text in every language the app ships - #827
Merged
Conversation
…rd list The GLP-1 side-effect chips in the mood form write the localised label into the flat `MoodEntry.tags` column, and three readers matched that label against a hand-written list of English and German words: the therapy timeline, the Coach snapshot, and the doctor report's GLP-1 block. The app ships six locales. Tap the nausea chip with the UI in French, Spanish, Italian or Polish and the day never reached the timeline, the Coach grounded its answers on an empty side-effect list, and a clinician read an empty table over a record that had three months of symptoms. Nothing on any of those surfaces said the table was a filter rather than the truth. The three lists had also drifted apart from each other. The chips are a closed set with stable keys, so matching the label was the defect and the key is the fix. `glp1-side-effect-tags.ts` holds the catalogue — seven keys, each with the i18n key its chip renders — and the mood form now draws its strip from it, so the write side and the read side cannot diverge again. `glp1-side-effect-tag-match.ts` builds the label-to-key index by reading `messages/<locale>.json` for every entry in `locales`, rather than transcribing the labels into a second list. Transcribing would have fixed today's four locales and reopened the same hole on the day a seventh is added; deriving means a new bundle is matchable as soon as it lands, with no edit in the matcher. Comparison folds case, accents and separators, so the English and German tags already in the database keep resolving and a straight apostrophe matches the typographic one the French bundle uses. All three readers now report the catalogue key instead of the stored string. The same recorded symptom yields one key whatever language it was tagged in, which also fixes the mixed-language output an account gets after switching UI locale; the doctor-report PDF resolves the key through the label map so the clinician reads it in the report's language. A tag outside the catalogue is still never counted — a free-text mood tag must not reach a doctor as an invented symptom, and the rest of somebody's day does not belong on a therapy timeline. It is no longer dropped without trace: the timeline route annotates the unresolved count on its wide event, so the classification rate stays answerable while the tag text stays in the database. The Coach's copy of the list also carried `vomiting`, `reflux` and `erbrechen`. Those are removed. No chip writes them, so they only ever matched a hand-typed tag in two of six languages; keeping them would have rebuilt the defect inside its own fix. Restoring either symptom means giving it a chip and a label in all six bundles. The response schema for `GET /api/insights/glp1-timeline` now enumerates the keys, and its description no longer documents the bilingual match as if it were the contract.
… or Polish
The free-text reference-range parser decided what a printed window meant from
an alternation of German and English words. The app ships six locales, and a
lab prints in the language of the country it stands in, so `jusqu'à 5,0`,
`hasta 5,0`, `fino a 5,0` and `do 5,0` all fell through to `low: null,
high: null`. The string was stored, the reading was then judged against the
biomarker's own band or against nothing at all, and no surface anywhere said
that the lab had printed a window the app could not read.
The vocabulary is TABULATED here, not derived from `messages/<locale>.json`
the way an app-owned word list would be. These words are prose a lab printed
on paper; the app owns no string they could be read out of, and the language
of a report has nothing to do with the language of the UI. So the table is
hand-written, keyed by the shipped `Locale` union so that adding a locale
fails to compile until someone has answered how that language prints a range,
and every locale's words are tried against every input at once. The cost is
written down at the table: a report in a language absent from it stays
text-only.
Three things had to change beside the word lists. Introducing words are now
stripped only on a second pass, because Italian "da", Polish "od" and French
"de" also state a floor on their own and stripping first read "da 3,5" as a
bare number. Input and table are diacritic-folded and their apostrophes
normalised, so a table row written "jusqu'a" meets a report's "jusqu'à" and
the curled apostrophe a word processor emits. And one-letter separators ("a",
"e", "y") are only accepted with whitespace on both sides, so "3.5e5" is not
read as a window. Decimal commas already worked and are pinned by the new
cases.
An unreadable window now has a name. `isUnreadableRange` separates "the report
stated nothing" from "the report stated something nobody could read" — the two
were indistinguishable at every call site, which is what let the French case
pass unnoticed. The manual form already showed the reader that the text was
kept but not read; the two ingest paths now count it into their wide event, so
an unsupported notation surfaces as a number rather than as silence.
The German and English lists are the previous alternation split word for word,
and the existing suite covering them passes unchanged.
…ipped locale
A biomarker minted from the labs catalog stores `t("labs.catalog.<slug>")` —
the name in the language of whoever minted it — and that stored name is what
reaches the FHIR exporter as free text. The exporter matched it against an
alias table written in English and German. So an account working in French,
Spanish, Italian or Polish exported "Cholestérol total", "Glucosa en ayunas",
"Glicemia a digiuno" and "Cholesterol całkowity" as text-only concepts with no
LOINC code, on the one document whose whole purpose is being read by another
system.
The names are DERIVED from `messages/<locale>.json` now, not transcribed into
this file. Both sides of the new slug map are app-owned identities with no
language in them, which is what makes it the one edit a new locale does not
need: `locales` grows, `messages/xx.json` lands, and the analyte names it
ships resolve the same day. Transcribing four more languages would have fixed
today's six and reopened the hole at the seventh.
The hand-written aliases keep precedence. They cover what the app owns no
string for — clinical abbreviations, the ways a person shortens a name — and
they were verified one at a time over several releases. A derived name landing
on the same key is a harmless duplicate; one landing on a different key would
be a translation typo re-pointing a verified mapping at another analyte, so
the build refuses it and the guard suite asserts the refusal held.
The lookup fold now drops accents before stripping punctuation. Without it a
name carrying diacritics lost them entirely rather than reducing to its base
letters — "Cholestérol total" folded to `cholestroltotal`, and "Hämoglobin" to
`hmoglobin`. That last one means the German umlaut spellings never resolved
either: the hand-written `hamoglobin` and `nuchternglucose` aliases only ever
caught the un-umlauted typing. Every pre-existing alias is ASCII, so the fold
is a widening and what resolved before resolves to the same key.
The lab form offers "negative" and "positive" as quick picks and writes the label it rendered, so the column holds the bundle's own string in the language the reading was recorded in. The SNOMED mapper matched that string against a list transcribed in English and German, so "négatif", "negativo", "ujemny", "positif" and "dodatni" all exported without a code — text-only on the one document meant to be machine-readable. The negative and positive arms read `labs.form.qualNegative` and `labs.form.qualPositive` out of every bundle now. The form writes those exact keys, so the write side and the read side cannot drift, and a new locale is coded the day its bundle lands. The detected / not-detected pair stays hand-written English and German, and that is the honest limit of this change: the app ships no string for those terms, they are prose a lab printed and a person copied in, and coding "non détecté" or "niewykrywalny" would mean inventing clinical vocabulary in four languages nobody here can check. This module's own rule is that an unverified code is worse than an honest `.text`. When they need coding they need a quick pick and a bundle key, which is the route the other two arms took. A test pins that they stay uncoded rather than leaving it to be discovered. Borderline stays uncoded as before; a test now pins that the derivation did not sweep its bundle key up by accident. Comparison folds accents as well as case, so a result typed without them still resolves.
…rman The gate that decides whether a transcribed document is worth an extraction call scored four classes of evidence and needed two. Three of them — the reference-range label, the analyte names, the report header — were written in English and German, so a French, Spanish, Italian or Polish report scored at most one: the units, which are the same everywhere. Auto-staging never fired for it. The document was lost from the flow silently, because the manual extract button stays available and nothing reads as broken. The text is folded once — lower case, accents dropped — and every pattern is matched against that, which on its own recovers a good deal: "Cholestérol", "Créatinine" and "Emoglobina" reach the existing stems as soon as the accents are out of the way. The reference-range labels and report headers of the four locales are added by hand, because they are prose a lab printed and the app owns no string for them. The analyte names ARE derived, from `labs.catalog.<slug>` across every shipped bundle: those are names the app itself displays, and a new locale should not need an edit here to be recognised. Names under four folded characters are dropped — "ALT" is German for "old", and a three-letter alternative in a gate this cheap would make every German letter a lab report. The acronyms worth having are in the hand-written half, where each was considered on its own. Which way this errs, stated at the code: a false negative loses a whole document from auto-staging, while a false positive costs one extraction call and stages facts a person then declines — nothing commits either way. So the vocabulary is widened, and the two-distinct-class rule holds the other side. The derived names extend the analyte class rather than forming a fifth one, so a document naming one analyte in two languages still scores one. Tests pin each addition on a case that fails the gate without it, and pin the prose that must keep failing it.
… class by identity The derived names extended the analyte class by list index. Reordering the signal list would have re-pointed them at the units or the report header with nothing to notice, so the class is a named constant and the check compares against it.
Three resolvers turned a user-authored word into an internal code by comparing it against a hand-typed English list. The app ships six languages, so five of them resolved nothing: `Fer` and `Żelazo` came back as `unknown_nutrient` while the settings card rendered exactly those words, and `poids` and `sommeil` reached no series at all. Stop transcribing. `nutrients.names.*` and `measurements.type*` already label these concepts in every bundle, so the match index is derived from `locales` instead — a seventh language works the day its bundle lands, with no edit at any call site. The nutrient tool's English labels now come out of `messages/en.json` too, so the wire wording cannot drift from the screen. The MCP contract is unchanged in shape and stays protocol-level English. What moves is which words are understood: `unknown_nutrient` / `unknown_metric` still mean "could not place this word", distinct from a placed name answering `no_data`. Nothing turns absence into data. Two findings the sweep did not predict: - `Puls` resolved to pulse-wave velocity, because the English display-name pass matches substrings and "pulse-wave velocity" contains it. An exact name now beats a loose English fragment. - French and Italian use one phrase for body fat and fat mass, and three bundles use one for fat-free mass and lean body mass. A word two metrics claim resolves to neither; answering kilograms to someone who asked for a percentage is worse than saying the word was not recognised. The known set is pinned so a new collision surfaces. The medication needles are the exception: they are drug names, with no bundle to derive from. They fold to a normalised INN stem rather than to a language, so `Metformin`, `Metformine`, `Metformina` and `Metforminum` are one entry, and `Semaglutyd` and `Atorwastatyna` cost nothing to add. A guard asserts no two molecules collapse onto one stem. Iron is the one class with no stem to fold to — it is named for the element — so its word list is hand-tabulated and says so. Every English resolution that worked before returns the same answer.
The three locale matchers were built in parallel and each grew its own fold. Two of them are the same function: the side-effect tag fold and foldForMatch agree character for character, so one now calls the other. The third stays. foldLabel decomposes with NFKD, maps sharp s to ss and joins with underscores; foldForMatch decomposes with NFD, leaves sharp s alone and joins with spaces. That is a disagreement about who typed the text, not an accident: one side of a lab comparison is a shipped label retyped or prose a lab printed, where tolerating a different spelling claims more than the fold can prove, while someone naming a metric to an assistant is composing rather than reproducing. Both docblocks now say so and name the other.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The application ships six languages. Thirteen places decide what happens to user-authored text by consulting a word list typed out in two of them, or in one. This closes the seven that cost a person the most.
What the defect looks like from the outside
You record a side effect by tapping a chip. The chip writes the translated label into a free-text column, so the same symptom is stored as
nauseaon an English account andNudnościon a Polish one. The GLP-1 timeline then matched that column against a hand-typed transcription of the English and German labels. Four languages contributed nothing, silently.The reported surface was the timeline. Two more readers had the same list: the Coach snapshot, and the doctor report's GLP-1 block, where a clinician read an empty side-effect table over a record that had entries.
Similar, elsewhere: a lab reference range printed
jusqu'à 5,0parsed to no bounds at all, so the reading was never flagged normal or abnormal.Cholestérol totalgot no LOINC code on the FHIR export. A French lab PDF failed the gate that decides whether a document is a lab report, so automatic staging never ran for it. Over MCP,FerandŻelazoresolved tounknown_nutrient, andpoidsandSommeilto nothing.Derive rather than translate
Translating a list four more times fixes today's four languages and reopens the hole the day a seventh is added. Where the app owns the words, the match index is now read out of
messages/<locale>.jsonfor every entry inlocales, so a new language is matchable the moment its bundle lands, with no edit to any matcher. Tests assert that property rather than the current list.This was possible more often than expected, because the vocabularies were already on disk: the side-effect chips are seven keys,
labs.catalog.<slug>carries 30 analytes in six languages,nutrients.names.*carries 26 codes in six, and the metric names come fromMEASUREMENT_TYPE_LABEL_KEYS.Where deriving does not reach, said plainly
jusqu'à,hasta,fino a,doare prose a lab printed, in a language unrelated to the reader's. There is no key. The table is hand-written and typed over the shipped locale union, so a seventh language fails to compile until someone answers how it prints a range.non détectéorniewykrywalnymeans inventing clinical vocabulary in four languages nobody here can check, against that module's own rule that an unverified code never beats no code. A test pins that they stay uncoded.Metformina,SemaglutydandLewotyroksynareach the right class off the existing English needles with no new entries. Iron is the exception and is labelled as one: preparations are named for the element, so there is no stem forEisenandŻelazoto meet on.Four defects found on the way, none of them reported
Accent folding was missing in the lab key normaliser, and it broke German.
normaliseLabKeydeletes every character outsidea-z0-9, soHämoglobinbecamehmoglobinand never met thehamoglobinalias. Same forNüchternglukoseandHarnsäure. The aliases only ever matched people who left the umlaut off. This is the primary locale, and it has been wrong for as long as the table has existed.Pulsresolved to pulse-wave velocity over MCP, because a substring pass matched English display names and"pulse-wave velocity"containspuls. Exact localised matching now runs ahead of the fuzzy pass.Four body-composition metrics collapse onto two words. French and Italian call body fat and fat mass the same thing; three languages use one phrase for fat-free mass and lean body mass. Judging that over the exposed slice alone saw no collision and would have answered kilograms to someone asking for a percentage, so ambiguity is judged over the whole label space and an ambiguous word is refused.
An unreadable reference range was indistinguishable from no range at every call site. It has a name now, and two ingestion paths count it.
Not silent about what does not match
An unmatched side-effect tag is counted on a wide event rather than surfaced: what was vanishing was catalogue tags in unsupported languages, and that is closed at the source; what remains is genuine free text, and putting it on a therapy timeline or in a clinician's PDF would be an invented symptom.
Two folds, on purpose
Three matchers were built in parallel and each grew a fold. Two are the same function and now share one. The third stays different: it decomposes with NFKD, maps sharp s to ss, and joins with underscores. That is a disagreement about who typed the text, not an accident — one side of a lab comparison is a shipped label retyped, where tolerating a different spelling claims more than the fold can prove, while someone naming a metric to an assistant is composing rather than reproducing. Both docblocks say so and name the other.
Break-proofs
Every fix reverted and re-run. Walking only
eninstead oflocalesreddens 43 tests across three files; cutting the side-effect index back to English and German reddens 16 across four suites; dropping the accent fold reddens the two German lab cases; unwiring the LOINC index reddens 22. Thirty-odd mutations in total, each isolated by its own case. English behaviour is pinned separately in every affected suite.Gates: typecheck, lint,
format:check,openapi:check, 21,822 unit tests, the touched integration suites, and the production build.