Julenmendieta/MILAB-6501_handleHugeInputs

[julenmendieta]

Greptile Summary

This PR changes embedding-mode UMAP to use a bounded CPU fit sample and batched projection, propagates embedding dimensionality through the workflow, and updates resource sizing and SDK dependencies.

• Embedding matrix: Long-format Parquet data containing one value for each clonotype and embedding dimension; loading changes from eager N×D materialization to batched assembly.
• Embedding dimension (D): The vector length from pl7.app/embedding/length; it is now passed as --dims, with first-row-group inference as a fallback.
• Fit sample: A seeded subset bounded by --max-sequences; CPU PCA and UMAP are now fitted on this sample before all clonotypes are transformed.
• PCA: Centered dimensionality reduction retaining 95% explained variance; CPU fitting becomes sample-bounded while GPU fitting remains fit-on-all.
• UMAP transform: Out-of-sample projection of each CPU batch through fitted PCA and UMAP models; GPU behavior remains full-device fit/transform.
• Degenerate vector: A near-zero post-PCA vector; these vectors now receive null coordinates and are excluded from UMAP fitting.
• Resource formula: Data-driven CPU RAM sizing for embedding mode is lowered from an eager-load multiplier to a 16–48 GiB range.
• PColumn: Platforma typed column abstraction; its local TypeScript alias now permits undefined column values following the SDK update.
• Dependency catalog: Workspace SDK, tooling, and Python runtime packages are upgraded and the lockfile is regenerated.

Confidence Score: 3/5

The PR should not merge until the CPU embedding output is streamed or otherwise bounded, because huge inputs can still exhaust the newly reduced RAM allocation.

Input feature matrices are streamed, but all output keys and coordinates remain resident and are copied repeatedly during concatenation, Python-list conversion, and DataFrame creation, so peak CPU memory still grows with clonotype count.

Files Needing Attention: software/umap/src/main.py and workflow/src/resource-formula.lib.tengo

Important Files Changed

Filename	Overview
software/umap/src/main.py	Introduces two-pass CPU streaming and streamed GPU loading, but the CPU output stage still accumulates and copies O(N) data, leaving a large-input OOM path.
workflow/src/main.tpl.tengo	Propagates the embedding-length annotation and exported matrix metadata into the calculation template.
workflow/src/umap-calculation.tpl.tengo	Adds conditional forwarding of embedding dimensions to the Python executable.
workflow/src/resource-formula.lib.tengo	Substantially lowers CPU embedding RAM based on streaming, although the remaining N-scaled output materialization undermines the stated bound.
workflow/src/resource-formula.test.tengo	Updates formula expectations consistently with the new CPU RAM floor, cap, and fallback.
model/src/index.ts	Broadens the local PColumn value type to accommodate undefined values after the SDK update.
pnpm-workspace.yaml	Upgrades Platforma SDK, build tooling, helper, and Python runtime catalog versions.
pnpm-lock.yaml	Regenerates resolved dependencies for the workspace catalog upgrades.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Long-format embedding Parquet] --> B[Stream clonotype batches]
  B --> C[Seeded bounded fit sample]
  C --> D[Fit centered PCA]
  D --> E[Fit CPU UMAP]
  B --> F[Transform each batch]
  D --> F
  E --> F
  F --> G[Accumulate all keys and coordinates]
  G --> H[Materialize TSV output]
  A --> I[GPU device matrix]
  I --> J[Fit-on-all cuML PCA and UMAP]
  J --> H

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
software/umap/src/main.py:1270-1277
**Output accumulation remains unbounded**

When a CPU embedding run contains enough clonotypes, this loop retains every key and coordinate batch, then duplicates them with `concatenate`/`vstack` and again as Python lists and a Polars frame, causing the process to exhaust the newly capped 48 GiB allocation despite the input matrix being streamed.

_{Reviews (1): Last reviewed commit: "Changeset" | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Context used - Terms is a types in codebase. Provide the list of ... (source)

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[greptile-apps]

Output accumulation remains unbounded

When a CPU embedding run contains enough clonotypes, this loop retains every key and coordinate batch, then duplicates them with concatenate/vstack and again as Python lists and a Polars frame, causing the process to exhaust the newly capped 48 GiB allocation despite the input matrix being streamed.

Prompt To Fix With AI

This is a comment left during a code review.
Path: software/umap/src/main.py
Line: 1270-1277

Comment:
**Output accumulation remains unbounded**

When a CPU embedding run contains enough clonotypes, this loop retains every key and coordinate batch, then duplicates them with `concatenate`/`vstack` and again as Python lists and a Polars frame, causing the process to exhaust the newly capped 48 GiB allocation despite the input matrix being streamed.

How can I resolve this? If you propose a fix, please make it concise.