modkit (V0.6.4 )pileup hangs mid-run (process alive, 0% CPU) on large metagenomic BAM

[ran525]

Summary

modkit pileup consistently stops making progress partway through a large
metagenomic modBAM. The process stays alive but sleeps at 0% CPU indefinitely.
It stops at a different point on each run, so it is not a specific contig.

Environment

• modkit 0.6.4 (conda/bioconda, micromamba env)
• Linux, SLURM cluster, NFS-mounted storage
• Data: PacBio Revio HiFi, MM/ML tags written by PacBio jasmine v26.1.3
  (--keep-kinetics), aligned with pbmm2 v26.2.99 --preset HIFI --sort
• Reference: metagenome assembly (metaMDBG), 5,211 contigs, 462 Mb,
  contig names contain = characters (e.g. QIBS1__C572_L=1861861=)
• BAM: 91 GB, 1,607,903 primary alignments, 100% mapped
• Mod codes present: A+a. (6mA), C+m? (5mC), C+h? (5hmC)

Command

modkit pileup QIBS1.meth.aligned.bam QIBS1.methyl.bed
--ref scaffolds.fasta.filt
--threads 32
--log-filepath modkit_pileup.log

log infor

[ArtRand]

Hello @ran525,

Is there a small(er) modBAM that can reproduce the error?

[ran525]

Hi @ArtRand,

Thank you for your reply.

I remember that I also tried splitting the data into smaller chunks, but the results were still not very good.

I then switched to the older version, modkit v0.5.0, and it worked well for my PacBio HiFi data.

Could you briefly explain the main changes between modkit v0.5.0 and v0.6.4? I'm wondering if there were any major algorithmic or default parameter changes that could explain the difference in performance.

Best wishes,
Ranran

[ArtRand]

Hello @ran525,

There are no major algorithmic differences other than the changes here, however the data structures in v0.6.z are more optimized. Could you paste the output of $ modkit modbam check-tags $bam --head 100 here? I may be able to advise more.

[ran525]

Hi, @ArtRand , thank you for your reply.
Here is output of $ modkit modbam check-tags $bam --head 100.

modkit modbam check-tags "$BAM" --head 100

checking tags on first 100 reads
input modBAM contains 63 (63.00%) failed records
num PASS records: 37 (37.00%)
num records: 100
errors:
+-----------------------------------------+-------+-------+
| error | count | pct |
+-----------------------------------------+-------+-------+
| conflict-explicit-prob-greater-than-one | 62 | 98.41 |
| MM-tag-missing | 1 | 1.59 |
| total | 63 | 100 |
+-----------------------------------------+-------+-------+

valid record tag headers:
+------------+-------+
| tag_header | count |
+------------+-------+
| A+a. | 37 |
| C+m? | 36 |
| T-a. | 36 |
| C+h? | 21 |
| G-h? | 21 |
+------------+-------+

invalid record tag headers:
+------------+-------+
| tag_header | count |
+------------+-------+
| A+a. | 62 |
| C+h? | 62 |
| C+m? | 62 |
| T-a. | 62 |
| G-h? | 40 |
+------------+-------+

modified bases:
+--------+--------------+----------+------+
| strand | primary_base | mod_code | mode |
+--------+--------------+----------+------+
| + | C | h | ? |
| + | C | m | ? |
| + | A | a | . |
| - | G | h | ? |
| - | T | a | . |
+--------+--------------+----------+------+

Error! input modBAM contains 63 (63.00%) failed records.

Thank you very much. Looking forward to your reply.

Best wishes,
Ranran