AWESOME or Overhyped? OXR Document Parsing, Tested With My Own Ugly PDFs

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: AWESOME or Overhyped? OXR Document Parsing, Tested With My Own Ugly PDFs · Channel: Prompt Engineer 48 · 10:21 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI infrastructure and hardware, AI models, Open source and software ecosystems, RAG, search and information retrieval

Summary

The creator tests OXR, a document-parsing model whose authors claim it ranks first on three benchmarks, using the provided examples and several documents of their own. After the free API fails and local hardware proves insufficient, they rent a GPU and work through setup issues. OXR handles a research paper, formulas, handwriting, and complex tables well in several tests, but produces serious errors on a demo page and a homemade receipt. The creator concludes that OXR is promising, but the benchmark ranking remains unverified, the sample size is small, and the license has important restrictions.

Key points

OXR is described as a 1.3-billion-parameter model, although its Hugging Face listing shows about 1.53 billion parameters.
The authors claim top rankings on three document-parsing benchmarks; the creator could not verify those claims.
The free API returned repeated 429 errors, and the creator’s laptop did not meet the stated 24 GB GPU-memory requirement.
Running OXR on a rented RTX 4090 required troubleshooting virtual environments, model downloads, and service ports.
A cropped Chinese math exam produced LaTeX formulas and 42 blocks in about 1.5 seconds through the API.
A demo-page run produced a wall of repeated characters in the header area; the creator did not establish the cause.
A 15-page arXiv paper was processed in 11.8 seconds, with its title, author block, abstract, equations, tables, captions, and figures identified.
A blurry, rotated receipt was parsed with incorrect table alignment, missing address lines, and no total value.
The license includes conditions for commercial use and hosted services; the creator notes that the demo footer only says Apache 2.0.
The creator’s conclusion is that OXR is worth watching, not proven or flawless; their tests were too limited to establish statistical accuracy.

Chapters

Start	Chapter	Summary
00:00	The claims and the test plan	The creator introduces OXR’s benchmark claims, explains its parsing features, and notes a discrepancy in the listed parameter count.
01:46	API failure and GPU setup	The free API repeatedly returns rate-limit errors, so the creator rents a GPU and troubleshoots the installation.
04:36	Testing examples and a research paper	OXR handles several example documents and a 15-page paper well in some respects, though a demo run shows a repeated-character failure.
07:36	Receipt failure and license restrictions	A homemade receipt reveals table and text errors, and the creator reviews the license’s commercial and hosted-service terms.
09:13	Verdict and experiment cost	The creator calls OXR promising but unverified, summarizes its failures and strengths, and reports an estimated 24-cent GPU cost.

Quotes

So either I just stumbled onto a gold mine or the read me is overselling. — 00:00

I didn’t test it off, so I can’t tell you that’s the cause, but it is a real failure and it’s on camera. — 05:38

So, it understood this was a table and got the structure wrong. — 08:08

It’s promising, not magic. — 09:45

Concepts and entities

OXR, document parsing, layout model, recursive self-improvement loop, RunPod, vLLM, Apache License 2.0

Full transcript (original language)

00:00 Okay, I need to show you something and honestly, I don’t know yet if it’s real. This is OX R. The authors say a 1.3 billion parameter model ranks number one on three document parsing benchmarks. Three and the repo has about 180 stars. That is tiny. Things this good usually have thousands. So either I just stumbled onto a gold mine or the read me is overselling. I’m going to find out on camera with their own examples and a couple of ugly documents of my own. >> [music]

00:32 >> Hey, I’m Prompt Engineer. I genuinely don’t know how this one ends. So stay with me. Let’s find out together. Quick version of the claim. You give it a PDF or an image and you get markdown or JSON back with coordinates for every block. 13 element types, tables, formulas, code blocks, even chemical structures and stamps. And here’s the design choice I like. Layout analysis is a separate model you can swap out while OX R does the reading and the reading order. So if your documents are weird, say invoices, you can plug in your own layout model. They also say it was post

01:05 trained with a recursive self-improvement loop. Roughly, find the pages the model gets wrong, train on those, repeat without human labels. That’s their description. The technical report says coming soon, so I can’t check how that part works. What I can check is the output. And one small thing on the model card, the read me says 1.3 billion parameters, but hugging face lists about 1.53 billion. Small thing, but I noticed it.

01:46 First plan, use their free API from my own machine. The read me says no sign up, no key. So I installed the Somar command line tool, downloaded three of the read me examples and ran the first one. Nothing. Wrong flag. The output folder has to exist already. Fine, made the folders, ran it again. 429 too many requests. It retried for almost 4 minutes, then gave up. Second file, same. Third file, same. I can’t tell why. Maybe a lot of people are trying it right now. Either way, that plan is dead and free API is not a plan. Okay, so can I just run it here? It’s my laptop. RTX 4060, 8 GB of VRAM on

02:18 Windows. The readme says Linux and 24 GB of GPU memory just to get started. So, no, I’m not going to pretend I squished it in real model, real GPU. I rent one. The math is simple. 24 GB is the floor, so I need a 24 gig card, and the driver has to support CUDA 13. I pulled live prices from the RunPod API. RTX 3090, 24 gigs, 22 cents an hour on community cloud. RTX 4090, same memory, 34 cents. Both marked low stock. I went for the 4090 at 34 cents an hour and hit create.

03:00 Capacity error. I changed the CUDA floor, pinned the data center, tried again. Capacity error. Switched to the cheaper 3090. Same thing. Three tries, three failures. And honestly, that’s the part nobody shows you. So, I paid more, secure cloud, 89 cents an hour for the 4090. And I always check the create response because the quoted price and the built price can differ. Cost 0.89, same as the quote. Pod running. First thing on the pod, NVIDIA says me RTX 4090, 24 GB driver 595, CUDA 13.2,

03:30 Ubuntu 2404, Python 312. No Docker here, so I followed the readme’s run from source path instead. This is where it got real. First install script, pip refuses. Externally managed environment, that’s Ubuntu protecting itself. So, I switched to UV virtual environments. Then I realized that meant no HF command. So, the model download silently never ran. I caught it, ran it

04:02 from the right environment, and 3 GB came down in about 6 seconds. Then the VLLM start script tried to call pip inside an environment that has no pip. And then the layout service said port 8081 was already in use. That was runpod’s own engine X. I moved it to 8091, edited the config, and finally all three services were up.

04:36 One number I like, the model weights load in about 3 seconds and take 2.6 GB on the GPU. With all three services running, Nvidia-smi showed about 23 gigs used, and most of that is VLLM reserving cache, not the model. First test, the readme examples. I grabbed the A3 exam paper and sent it three pictures. That’s it, three blocks, all pictures, no text. My stomach dropped, then I opened the image. The readme example is in the document. It’s a poster. Original on the left, layout in the middle, parsed preview on the right. I fed it the whole poster, and it

05:08 correctly saw three pictures. My mistake, not the model’s. So, I cropped out the original panel and ran it again. Through the API, that came back as 42 blocks, 34 texts, six titles, two pictures in 1 and 1/2 seconds. A Chinese math exam, and the formulas come back as

05:38 LaTeX, like x = 1 + the square root of 2002 over 2. But then I ran the same image on the demo page, and it took 43 seconds. Look at the right side. The header area comes back and then a wall of the same repeated character. The demo had header and footer recognition switched on. I didn’t test it off, so I can’t tell you that’s the cause, but it is a real failure and it’s on camera.

06:13 The complex table came through with the merged header cells kept as HTML row spans and column spans, which is where a lot of parsers lose the plot. The first call took 13 seconds because everything was warming up. The English handwriting came back in about a second with the paragraph text readable. I’m only comparing against the poster’s own preview, so I’m not claiming accuracy numbers. I’m saying it didn’t fall over.

06:46 Now, something real. Attention is all you need. The actual arXiv PDF, 15 pages, two columns, equations, tables, 11.8 seconds for the whole thing. It found eight formulas, four tables, nine captions, and five figures. Title, author block, abstract in the right order, and the abstract text I checked matches the paper.

07:36 The demo interface makes this easy to see. You upload and you get preview, source, and layout with the boxes and the reading order. This is the receipt page, which brings me to the part where it doesn’t work. I made this receipt myself, a blurry, rotated phone photo with items, prices, tax, and a total. The correct total is 34.32. Now, look at the table, the prices are shifted one row from the items. Oat milk has no price, and the first price sits on a blank row. The address lines are gone, and the total has no number. The subtotal and tax numbers are in there,

08:08 but on the wrong labels. So, it understood this was a table and got the structure wrong. Synthetic receipt, one image, so please don’t read this as a benchmark, but it’s the first honest failure, and I’d rather show it.

08:42 Before you build a business on it, read the license. I pulled it off the pod. It’s Apache 2.0 plus extra terms. Commercial use is free only while you and your affiliates stay under $2 million in total funding and under $2 million in annual revenue. Go over either, and you need a separate license. If you run it as an online service, you must credit So Mark OXR, and you can’t offer a competing hosted document parsing API at any size. Internal use is fine. Also, the demo page footer just says Apache 2.0. The license file is the one that

09:13 counts. So, Goldmine, here’s my honest answer. The speed is real in most runs, from under a second to about 13 seconds on a single 24 gig Plus that 143-second loop, merged tables, formulas, Chinese text, and a real paper looked right to me. But it failed twice. It looped on the A3

09:45 header in the demo, and it scrambled my receipt. The number one on three benchmarks is the author’s claim. The technical report is still coming, and five documents from me prove nothing statistical. It’s early, 180 stars, a custom license, and a free API that was falling over. I can’t confirm the ranking on what I could test. It’s promising, not magic. Total bill for the experiment. The pod ran about 16 minutes at 89 cents an hour, roughly 24 cents by my math. I terminated it, and the pod list is empty.

10:21 If you’re looking for cheap and reliable GPUs, I use Runpod, and the link is in the description. So, did I find a gold mine? I think I found something worth watching. Drop the ugliest document you own in the comments, and I’ll run it through OXR and show you what comes back.