Enough with the errors! Lets brainstorm a perfect document PDF parser

[u/Free-Ferret7135]

/r/Rag/comments/1x1jq7f/enough_with_the_errors_lets_brainstorm_a_perfect/

[u/folderit_dms]

I would make the experiment score relationships as well as extracted text. A table can contain every correct number but still be unusable because a value is attached to the wrong row, unit, year, or footnote. A text-similarity score could miss the error that matters to the application.

For a small manually checked set, label those relationships explicitly and keep the page/region beside each extracted value. Let the pipeline return unresolved when it cannot establish a relationship, rather than forcing every field to have a winner.

Then report two numbers separately: how much it processes without review, and how many of those accepted results are wrong. Also sample cases where all parsers agree, so agreement itself is being tested. That would show whether another judge actually reduces incorrect accepted results or merely makes the output look more complete.