Difficulties running cytokit on codex data

[jesusdpa1]

Hi,

I am trying to run cytokit on our acquired data, but I am not able to set up the ylam correctly nor the processing bash file. we ran 9 group of antibodies, 10 Z planes and 49 sections

Here is what I got:

YLAM

experiment.txt

SH

pipeline_execution.txt

Error

[jesusdpa1]

Update:

Manage to debug some errors I had in the code, but started getting the following error:
when using the following pipeline.sh

#!/usr/bin/env bash

bash -e /lab/repos/cytokit/pub/analysis/codex-spleen/pipeline_execution.sh

Ignore warnings relating to how the CODEX tif files were originally saved

export PYTHONWARNINGS='ignore::FutureWarning:h5py,ignore:ImageJ tags do not contain "axes" property:UserWarning:main'

DATA_DIR=/lab/data/images/src_CX_19-002_spleen_CC2-A/
BASE_CONF=/lab/data/config/experiment.yaml

Generate configurations for experiment variants

v00: Process data as-is with CellProfiler quantification (and DB export for CPA)

v01: Run with drift compensation and deconvolution (primarily for performance benchmarking)

cytokit config editor --base-config-path=$BASE_CONF --output-dir=/lab/data/images/output
add analysis '{"cellprofiler_quantification": {"export_db": True, "export_csv": True, "export_db_objects_separately": True}}'
set environment.path_formats "get_default_path_formats('cyc{cycle:03d}reg001/{region:d}{tile:05d}_Z{z:03d}_CH{channel:d}.tif')"
set processor.args.run_drift_comp True
set processor.args.run_deconvolution True
save_variant v00/config reset
exit

OUTPUT_DIR=/lab/data/images/output/$VARIANT
CONFIG_DIR=/lab/data/images/output/v00/config/experiment.yaml
echo "Processing experiment $EXPERIMENT (variant = $VARIANT, config = $CONFIG_DIR)"

Note here that the data dir for the processor command is equal to output dir

echo "Running analysis"
cytokit processor run_all --config-path=$CONFIG_DIR --data-dir=$DATA_DIR --output-dir=$OUTPUT_DIR

When changing the data-dir in the pipeline:

cytokit processor run_all --config-path=$CONFIG_DIR --data-dir=$OUTPUT_DIR --output-dir=$OUTPUT_DIR

Thanks in advance for your help,

[eric-czech]

Ah, you should change "run_tile_generator: false" in the config to "run_tile_generator: true" if you're running on your own data. I updated that config to have a comment in the future for anyone else that copies it. Turning that off is only necessary when you can't get raw images for an experiment (as was the case with the CODEX publication) and have to use assembled hyperstacks instead, which is almost never, so there should have definitely been a comment about it.

Also, I'd suggest:

• Adding an experiment name to the volume you're attaching to the container (i.e. change "/lab/data/images/output" --> "/lab/data/images/my_experiment_name/output" instead)
• Moving the raw image files (one per cycle, z plane, and channel) to a folder called "raw" and pointing the processor run_all command at that instead (the "output" folder should be at the same level as "raw")
• The emission_wavelengths don't look like they're set to meaningful values, so definitely disable deconvolution (add run_deconvolution: false to the processor.args property in the config)

[jesusdpa1]

Hi Eric,

Thank you very much! I'll do the modifications to the path. After changing "run_tile_generator: true" it started working but gave me the following error:

from my experience I know that RTX have a problem handling memory allocation. The way I have been able to solve this problem is by limiting the usage of the memory to 0.8 of the total memory with

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.8
session = tf.Session(config=config)

is it possible to do this with cytokit? Do you think that I am presenting another problem?

pipeline_execution.txt

experiment.txt

[jesusdpa1]

Solved it. Had a bad math calculation

previous = tile_height: 1007 -> now = tile_height: 1008

But got the problem I was expecting on the GPU memory

[eric-czech]

Hi @jesusdpa1 ,

A couple thoughts:

• One thing I also forgot to mention is that you probably want run_crop: True in the config (processor.args property) if the images overlap one another. That too was a quirk of analyzing non-raw data for the publication and will save precious GPU memory.
• You could modify the code in the container related to TF gpu configuration (this should help you find it), as everything under /lab/repos/cytokit in the container is editable and any changes to that code would take effect immediately. Having an unbounded GPU memory fraction and allow_growth = True is usually a good strategy for not getting unnecessary memory errors though (which is the default), so turning it even lower probably won't help. Someone else submitted an issue like this where they saw those CUDA memory errors yet the program didn't actually crash. I have yet to see that but in this case, did it actually crash or not? You should have gotten a big traceback if it did.
• It looks like deconvolution is still running -- shut that off to save some GPU memory since the wavelengths aren't configured

[jesusdpa1]

Hi Eric,

Thank you very much! everything help, the program is now running smoothly,