Lift by Datalab: 90.2% Accuracy, 3x Faster... and the Number They Left Out

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Lift by Datalab: 90.2% Accuracy, 3x Faster… and the Number They Left Out · Channel: Prompt Engineer 48 · 09:52 · Watch on YouTube

Topics: AI evaluation and benchmarks, Data, databases and embeddings, AI governance and regulation, Multimodal models

Summary

The video reviews Lyft, Datalab’s open-weight 9-billion-parameter model for extracting structured JSON directly from PDFs and images. It explains the schema requirements, installation options, and document-processing workflow, while noting limits such as unsupported schema features and undocumented minimum GPU memory. In Datalab’s benchmark, Lyft achieves 90.2% field accuracy and is about three times faster than Gemini Flash, but only 20.9% of documents have every field correct. The presenter recommends testing it on your own documents and adding validation or human review when complete accuracy matters, and cautions viewers to check the weights’ license.

Key points

Lyft processes page images directly and returns JSON constrained to a user-provided schema, without a separate OCR stage.
The model is designed to return null for fields that are absent from the document, a feature the presenter considers especially important.
Schemas should be kept simple; features such as enums, anyOf, oneOf, refs, and additionalProperties may prevent constraint decoding from compiling.
Users can run Lyft through a VLM server or Hugging Face mode, use a Python interface, or create and test schemas with Schema Studio.
The minimum GPU memory requirement is not documented; the server is described as supporting GPUs from the T4 upward.
In Datalab’s benchmark, Lyft scored 90.2% field accuracy, compared with 91.3% for Gemini Flash and 95.9% for Datalab’s hosted API.
Lyft’s median latency was 9.5 seconds per document, versus 28.1 seconds for Gemini Flash; Datalab says the latency figures should be treated as relative.
Only 20.9% of documents had every field correct with Lyft, compared with 40% for Gemini Flash and 44.4% for Datalab’s hosted API.
The benchmark was produced by Datalab, and the presenter advises testing the model on users’ own documents.
The code is Apache 2.0, while the weights use a modified OpenRAIL-M license with usage restrictions; commercial use beyond the stated terms requires a commercial license.

Chapters

Start	Chapter	Summary
00:00	What Lyft does and how it works	The presenter introduces Lyft as a PDF-to-JSON extraction model and explains its schema-constrained, single-pass approach and null handling.
02:33	Schemas, installation, and tools	The video covers schema design limits, command-line and Python usage, Schema Studio, and the lack of a documented minimum GPU memory requirement.
05:10	Benchmark results and the missing metric	The benchmark shows Lyft’s field accuracy and speed, then reveals its lower full-document accuracy and the implications for validation.
07:45	License, hosted API, and who should use Lyft	The presenter explains the weights’ license restrictions, contrasts the hosted offering with the open model, and gives recommendations for trying Lyft.

Quotes

One model, one step, no OCR stage, no repair code. — 01:32

abstension by default — 02:03

So roughly three times faster and that claim is true. — 06:11

But if you’re building a pipeline where every field has to be right, this number matters more than the headline. — 07:15

If you need every field right on a long document with no human in the loop, I would start with the hosted API or add a verification step on top of lift. — 09:21

Concepts and entities

Lift, Datalab, Gemini Flash 3.5, Qwen3.5 9B, Schema Studio, Marker, Surya, Chandra, JSON Schema, Schema constraint decoding, OpenRAIL, Apache License 2.0

Full transcript (original language)

00:00 All right, in this video we will look at Lyft, a new open model that turns a PDF into clean JSON. Now, mind you, some of the numbers going around about it are not the full story. So, here is a post on X that says, “Everyone is sleeping on this new JSON extraction model. It says open weights. It matches Gemini Flash 3.5 and it is three times faster.” Some of that is true and some of it needs a closer look. So what I’ve done is I’ve gone through the GitHub repo, the data lab blog post and the full benchmark table and I’ll show you what is really

00:30 in there. Here is the plan. First what Lyft is, then how you use it. Then the benchmark where I’ll show you the good number and then not so good number and at the end who should actually use it. Let’s start with the problem. You have a stack of PDFs, invoices, contracts, research papers and you want the data inside them as clean fields like invoice number, total and the line items. The usual way to do this is a chain. You run OCR to get text. You hand that text to an LLM and then you write extra code to repair the JSON when the model breaks

01:01 it. And every link in that chain can fail. OCR can mangle a table. The LLM can invent a value and the JSON can come back with a missing bracket. Three places to debug. And there is a fourth problem that people forget. When a field is not in the document, a lot of models will still fill something in. a madeup tax ID or a madeup due date and nobody notices. So what is lift? It is a 9 billion parameter vision model from datal lab. You give it a PDF or an image along with a JSON schema and you get back a JSON object that matches that

01:32 schema. One model, one step, no OCR stage, no repair code. The model looks at the page images directly and reads each field straight from the page. Datal lab says the output is always valid JSON of the right shape because the model decodes against your schema. This is called schema constraint decoding. It also handles multi-page documents in one pass. So if a value spans two pages, the model sees the whole document at once. That is really useful and it is trained to leave a field null when the answer is not there. The block calls this

02:03 abstension by default and honestly this is the feature that matters most to me. Data lab is the team behind the OCR tools marker. Surya and Chandra and they say Lyft is their first model built purely for extraction. Here is the quick list from the blog 9 billion parameters single pass multi-page. The code is Apache 2.0 and the weights are on hugging face. So you can download them. Now let’s look at the schema because this is the part you will write yourself. A schema is just standard JSON schema. Here we have invoice number as a

02:33 string, total as a number and line items as an array. And you can see here each field has a description. Write one for any field whose name is not obvious. This is the only prompt engineering you really do with lift. Then there’s required. Mark a field required only when it must appear. Fields that really are not in the document come back null. So don’t force everything to be required. Now one important thing. Keep the schema simple. Strings, numbers, integers, booleans, arrays, and nested objects are all fine. But avoid enum any of one of ref and additional properties.

03:04 Why? Because the constraint decoder skips schemas. It cannot compile. and then the guarantee of valid output gets weaker. So this is a real limit and it is one extra thing you need to know. Now how do you run it? Let’s go ahead and look at the install. It is a single line pip install Lyft PDF that gives you the light install which expects a VLM server. Then you start the server with Lyft VLM. Behind the scenes this launches a docker container and it scales the batch size to the memory of your GPU. It supports H100, A100, L4DS,

03:36 A10, L4, 409,309 and even the T4. So this is not only for data center cards. And then you extract lift extract. Then the input PDF and output folder and d- schema pointing to your schema file. Really simple. You can also pass the schema in line as a JSON string or use the name of a save schema and limit the pages with page range for example 0 to 5 and page 8. Point it at a folder instead of a file and it processes the whole directory. For each

04:07 file you get the JSON and a second metadata file with the page count token count and any error. If you don’t want a server there is a hugging face mode pip install lift PDF with the hf extra add- method hf and the weights load on your own machine. It needs storage torch and they recommend flash attention. And for Python, you import extract from Lyft, pass the document and the schema and read result.extraction. If extraction fails, that value is none and the metadata keeps the raw output for debugging. There is one more tool,

04:38 schema studio. It is a small streamlit app. You run Lyft app and you can build, save and test your schemas against your own documents. It is a really good place to start. Now the question everyone asks, how much GPU memory do you need? And I have to be honest, I could not find a minimum anywhere in the readme or the blog. What we know is that it is a 9 billion parameter model and the server tunes itself for cards from a T4 up. So I won’t promise you a number here. I will measure it in the next video. Now the benchmark because this is where the

05:10 story gets interesting. Data lab tested on 225 documents, 6 to 64 pages each with about 11,000 scored fields. And they planted hard cases on purpose. Values across pages, exhaustive lists, fields that must be left null, near miss distractors, and values you have to add up from several places. Scoring is exact match against ground truth with a tolerance for numbers. And every model got the same page images and had to extract each document in a single pass. So that part is fair. Here’s the table

05:40 at the top. Datalab’s own hosted API 95.9% field accuracy. Then Gemini Flash 3.5 at 91.3 and then Lyft at 90.2. Below Lyft we have Azure Content Understanding at 83.4, New Extract 3 at 81.5 and QN 3.59B at 76.3. So among the models you can host yourself. Lyft is clearly on top. It is 13.9 points ahead of the base QN model of the same size and almost 9 points ahead of new extract. Now compare

06:11 Lyft with Gemini Flash. It is 90.2 2 against 91.3 that is a gap of 1.1 so nearly matches is fair matches is a little generous and speed the median is 9.5 seconds per document for lift Gemini flash takes 28.1 and the datal lab API 30.8 eight. So roughly three times faster and that claim is true. One note from data lab itself, those latencies are with eight concurrent requests. The local models on a single GPU with V LLM and the others over an API. They say

06:43 treat it as relative, not absolute. But now look at the other column. Full document accuracy. This is the share of documents where every single field is correct. Gemini Flash gets 40%, the data lab API gets 44.4 4 and Lyft gets 20.9%. So that is about half of Gemini Flash. And look at this Quinn 3.59B the base model gets 24% which is higher than Lyft and Azure gets 22.2. So on this metric Lyft is not ahead. Now let me be fair. Full document is a really harsh test.

07:15 These documents run up to 64 pages with a lot of fields and one wrong field fails the whole document. But if you’re building a pipeline where every field has to be right, this number matters more than the headline. In practice, you will want a validation step or a human check on the fields that count. So what does the post get right? Open weights, yes. Three times faster, yes. Nearly matches Gemini Flash on field accuracy, yes. What it leaves out is that on whole documents, it is about half. And one more thing, this benchmark is data labs

07:45 own built by the people who made Lyft. That does not make it wrong, but I would test it on your documents before you trust it. Also, all the models were served with their default or recommended settings. So, a tune setup could shift these numbers a little. Now, the license because open weights does not always mean free for everything. The code is Apache 2.0, which is the easy one. The weights use a modified open rail m license. It is free for research or personal use and for startups under $5 million in funding or revenue. And the catch is that you cannot use it

08:17 competitively with Datalab’s own API. If you want to remove those limits, you need a commercial license from their pricing page. For a lot of you watching personal projects and small startups, that is fine. But if you are bigger company, read the license before you build on it. There’s also the hosted side. Datalab says their managed platform runs an improved extraction stack with higher accuracy than the open weights plus per field verification, citations, and confidence scores. That is the 95.9% row in the table and you get $20 of free

08:49 credits every month or you can try the public playground without setting anything up. So this is a normal open core story. The open model gets you started and the hosted API is where they want you to go for the best accuracy. Nothing wrong with that, but know it going in. So here’s my take. What I like it is one model for a job that used to be three steps. The output is valid Jon. It leaves fields null when the data is missing and it is fast. What I would watch full document accuracy is low. The schema has limits. The GPU requirement

09:21 is not documented. And the benchmark comes from the team that built the model. Who is this for? If you process lots of documents like invoices, forms, and reports and you want to host it yourself, then Lyft is worth a test. Especially when speed and cost matter. If you need every field right on a long document with no human in the loop, I would start with the hosted API or add a verification step on top of lift. That is my honest read. And if you just want to try it, the simplest way is the playground or schema studio with a few of your own documents. Look at the nulls

09:52 first because that is where models usually fail. In the next video, I will rent a GPU, run Lyft on real invoices and a research paper and tell you how much memory it actually uses and how many fields it really gets right. So, in this video, we have seen what Lyft is, how to use it, and what the benchmark really says. Links are in the description. Let me know how it goes, and I’ll see you in the next one.