plot_heatmap crashing on very wide matrices (fread segfault / duplicate row-names); switching to vroom fixes it

[jansen-ista]

Disclaimer, this issue was drafted using AI to help with formatting and best practices. Everything was checked by hand/me for errors and accurately describes the issue I observed.

Describe the problem

When running the plot_heatmap rule of
epigen/unsupervised_analysis (v3.0.3) on a very wide single-cell
expression matrix (~200 000 columns) the R script currently crashes
while reading the CSV files with data.table::fread().

Typical failure modes

• “duplicate row.names …” (see log below)
• Segmentation fault (*** caught segfault *** address … cause 'memory not mapped') on some runs

Replacing the four fread() calls in
scripts/plot_heatmap.R with vroom::vroom() and ensuring that
data.frame() is called with check.names = FALSE makes the rule run
successfully.

check.names = FALSE is important because base-R’s default sanitiser
turns hyphens (“-”) inside cell barcodes into dots (“.”); that breaks
later subsetting lines such as

data <- data[colnames(observations_distance), colnames(features_distance)]

causing an index out of bounds error.

Environment

item	version
OS	Debian 12 (bookworm)
R	4.1.3
data.table	1.14.8 (fails)
vroom	1.6.3 (works)
Snakemake	9.13.7
unsupervised_analysis tag	v3.0.3

Input file characteristics

• Header length ≈ 11 522 161 characters
• Typical data row length ≈ 3 937 555 characters
• ~2 034 rows × ~200 000 columns
• Uncompressed size ≈ 7.5 GB

Error log examples

At first I got the error message duplicate row.names as far as I could tell this seemed to be caused by fread not parsing the header correctly and somehow missing the row name column.

Error in data.frame(fread(file.path(data_path), header = TRUE), row.names = 1) :
  duplicate row.names: , ,-0.0, -0.1, ,-0., 1,-0., 8,-0, 85983, ,-0.11, -0.04
Execution halted
RuleException:
CalledProcessError in ...

After shortening my column names to prevent memory overflows or similar issues the error changed to:

 *** caught segfault ***
address 0x1457e82b47cf, cause 'invalid permissions'
Traceback:
 1: fread(file.path(data_path), header = TRUE)
 2: data.frame(fread(file.path(data_path), header = TRUE), row.names = 1)

I'm not completely sure wether these were really two different errors which is also why I include both in this issue.
In any way, switching fread to vroom fixed both of them.

Minimal code change that fixes the problem

- data <- data.frame(fread(file.path(data_path), header = TRUE), row.names = 1)
- metadata <- data.frame(fread(file.path(metadata_path), header = TRUE), row.names = 1)
- observations_distance <- fread(file.path(observations_distance_path), header = TRUE)
- features_distance     <- fread(file.path(features_distance_path),     header = TRUE)

+ library(vroom)
+ data <- data.frame(vroom(file.path(data_path), col_names = TRUE),
+                    row.names = 1, check.names = FALSE)
+ metadata <- data.frame(vroom(file.path(metadata_path), col_names = TRUE),
+                        row.names = 1, check.names = FALSE)
+ observations_distance <- vroom(file.path(observations_distance_path),
+                                col_names = TRUE)
+ features_distance     <- vroom(file.path(features_distance_path),
+                                col_names = TRUE)

Suggestion

• Replace data.table::fread() with vroom::vroom() in scripts/plot_heatmap.R.
• Call data.frame(..., check.names = FALSE) (or avoid the conversion and keep tibbles) to retain original barcodes/feature names.
• Add r-vroom to envs/ComplexHeatmap.yaml.

This change prevents crashes on very wide matrices and avoids silent
name mangling, while remaining fully backward-compatible for existing,
smaller datasets. Furthermore when running the code on my project (+200k cells) the import of the tables was still quite performant so this should also not represent a performance penalty.
Let me know if you have any further questions or concerns.
Best, Christian

[sreichl]

Hi @jansen-ista, thanks a lot for the elaborate issue description including potential fix. Much appreciated!

We have encountered this before. My solution back then was the addition of two new configuration parameter that subsamples the number of observations and/or selects the a fraction of features:

unsupervised_analysis/config/config.yaml

Lines 44 to 45 in a86e42b

	n_observations: 1 # random sampled proportion float (0-1] or absolute number as integer
	n_features: 0.5 # highly variable features proportion float (0-1] or absolute number as integer

Most of the time, a heatmap is used for visualisation of multidimensional data. Hence, a random subsampling and smart feature choice was enough for me. Have you tried this? Would that be good enough of a solution, or do you really want to always have all your data inside the heatmap?