Skip to content

Pipeline is slower for files which are combined (input files with same prefix) #36

@peterdekker

Description

@peterdekker

When running the pipeline for input files with the same prefix, the files are combined to one output file. Now I am testing on many-core hardware, it becomes apparent that combining files makes the pipeline much slower, especially in the OCR step. Probably this happens because parallelization across multiple CPU cores cannot be applied.

This is not a problem in itself, but I think it is good to notify users that files with the same prefix will be combined, and will have longer processing time. Or give them a choice to enable/disable combination. Now, a small difference in input, gives a large difference in processing time.

EDIT: I saw there is a "Reassamble PDF" option in the webinterface, but same-prefix files were also combined if I disabled this option.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions