Comparative Analysis of PDF Parsing Models Reveals Trade-offs in Accuracy and Speed

A new comparative study of PDF parsing capabilities has highlighted significant performance disparities between Vision Language Models (VLMs) and traditional OCR-based systems. The benchmark, shared via a technical repository by researcher alaamroue, evaluated eight different parsers across 14 distinct capabilities, including complex table rendering, LaTeX formatting, and handwriting recognition.

Performance Benchmarks and Model Accuracy

According to the findings reported by the researcher, the model “Chandra” demonstrated the highest level of fidelity across the tested categories. The report indicates that Chandra achieved a perfect score in 14 of 14 categories, successfully rendering merged-cell HTML tables and maintaining correct LaTeX for both display and inline text. Notably, the model was the only one in the test set to preserve italics on a specific 1904 document and correctly identified a physical stain on a page by skipping it rather than attempting to interpret it as text.

However, the study noted significant limitations in other models. While LightOnOCR-1B showed efficiency with a processing speed of 7.9 seconds per page on an L4 GPU, it exhibited issues with mid-sentence text drops and hallucinations regarding handwriting. In contrast, Chandra’s high accuracy came at a higher computational cost, requiring approximately 91 seconds per page on an L4 GPU.

Other models showed varying degrees of failure in specialized tasks:

  • Handwriting Recognition: The study described the handwriting column as a “massacre,” noting that XBerg, LiteParse, and PDLA returned noise or no data when faced with cursive text.
  • Error Patterns: PaddleOCR-VL was noted for “inventing” text where none existed, while LightOnOCR-1B produced confident but incorrect text when attempting to read illegible stains.
  • Structural Errors: Granite-Docling was reported to leak raw “DocTags” into its output, and XBerg was noted for its speed but limited capability in complex scenarios.

Technical Observations and Community Feedback

The benchmark results have prompted technical discussion regarding the scalability of these models. While high-accuracy models like Chandra provide superior fidelity for sensitive documents, industry observers noted that the speed and compute requirements may pose challenges for large-scale ingestion tasks, such as processing 100,000 document chunks.

Independent feedback from the technical community highlighted specific edge cases not captured in the primary summary. One reviewer noted that while Chandra performed well, it occasionally struggled with document structure, such as failing to insert newlines for headers or missing footers and headers in specific newsletter layouts. Another user pointed out that Chandra occasionally added unsolicited descriptive interpretations to charts, which may exceed the requirements of pure data parsing.

The researcher has made the raw outputs, test files, and scripts available in a public GitHub repository (alaamroue/pdf-parser-bench) for further verification. The study suggests that for high-volume workflows, a hybrid approach—using faster models like XBerg for simple tasks and high-fidelity models like Chandra for complex documents—may be the most efficient strategy.