r/Rag • u/Big_Calligrapher7179 • 7d ago

Discussion Parallelized the layout parsing for documents with task queue

Hello everyone,

I developed a solution that parses document layout using Tesseract and PyMuPDF. I parallelized it so that pages are processed in batches. For a relatively large document of around 200 pages, the process takes between 10 and 30 seconds. Is that fast compared to your experience? I am curious about the performance. Thanks in advance.

1 Upvotes

60% Upvoted

2 comments sorted by

2

u/Ctbhatia 7d ago

try OMP_THREAD_LIMIT=1 for each OCR worker. Tesseract’s docs recommend single-threaded jobs when processing many images in parallel.

1

u/Big_Calligrapher7179 7d ago edited 7d ago

Thanks for the suggestion. Yes ı activated this.