Transformers.js v4.3: Structured Output in the browser

Author: Hugging Face · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Transformers.js v4.3: Structured Output in the browser · Channel: Hugging Face · 10:46 · Watch on YouTube

Topics: Prompt and context engineering, AI developer tools, AI content generation, Local and open-source models

Summary

Transformers.js version 4.3 introduces structured output, alongside an overhauled documentation-generation system and community contributions. Structured output uses constraint decoding to restrict which tokens a model can generate, making it possible to enforce formats such as JSON schemas and regular expressions. The new Transformers structured output package plugs into Transformers.js through its existing logits processor, without requiring changes to the core library. The video also describes the package’s JavaScript-only implementation and performance work, and invites viewers to try it and share feedback.

Key points

Transformers.js 4.3 adds structured output, documentation improvements, bug fixes, and community-provided features.
Unconstrained generation may produce inconsistent formats, making its output less predictable for applications to parse.
Constraint decoding filters out tokens that do not conform to the requested structure.
The new Transformers structured output package is an add-on that uses the existing logits processor in Transformers.js.
A JSON processor can enforce a supplied JSON schema, including required properties.
The package also supports regular-expression constraints.
A recipe generator demonstrates how schema-constrained JSON can populate an application layout.
The initial WebAssembly prototype was slow and required downloading a 1 MB library; a JavaScript-only implementation eliminated that dependency.
Performance testing across different models and tokenizers showed minimal overhead from constraint decoding.

Chapters

Start	Chapter	Summary
00:00	Transformers.js 4.3 release	The release introduces structured output as its main feature, alongside documentation improvements and other updates.
01:34	Why structured output matters	The video explains how unconstrained model responses can vary and introduces constraint decoding as a way to control generated tokens.
03:12	Implementing constrained generation	A new structured output package uses Transformers.js’s logits processor to apply JSON schema constraints without changing the core library.
06:22	Examples and implementation details	The video demonstrates schema-constrained recipe generation and explains the approach to validating generated tokens against constraints.
09:05	JavaScript implementation and performance	The JavaScript-only implementation removes the WebAssembly dependency, achieves minimal overhead in tests, and supports JSON schemas and regex.

Quotes

the big one is structured output — 00:00

the technical term of structured output — 03:12

it actually did generate just JSON — 05:19

created a complete JavaScript only implementation of LLM guidance — 09:05

we barely have any overhead — 10:12

Concepts and entities

Transformers.js, Transformers structured output, structured output, constraint decoding, logits processor, LLM guidance, JSON Schema, regex, Hugging Face, Gemma 4, Granite, LFM 2.5

Full transcript (original language)

00:00 We finally released Transformers.js version 4.3 and it’s packed with really cool features. Yes, it has been a while since 4.2 and the main reason for that is that we spent a lot of time with the new Hugging Face Canvas library that released last week. But let’s have a look at this one. As you can see from the release notes, the big one is structured output and I will spend a lot of time talking about this in the rest of the video. We also took over or made a complete overhaul of our documentation process or

00:31 the process that generates the Transformers.js documentation. We do that or it has always been the case that the documentation was generated by the comments and the type system that we have in in our code. Problem is that it was based on a couple of libraries that have been a bit outdated. They didn’t fully understand the type system that we have. So now we have a completely new system mostly written by us by hand that does that for us. We also have a couple of new bug fixes,

01:02 couple of new features and most of them have been provided by people like you, by the community and we are super happy to see that so much community engagement. Um Um that that we have so much community engagement in this new release. >> [gasps] >> But let’s have a look at structured output. Let’s take this example here. We do have a system prompt very instruct the agent to extract personal details from a given input prompt and then we have the user prompt with information about that user.

01:34 If we then click generate, it warms the pipeline and then it generates a markdown snippet that has a JSON block and there we have some a JSON with name and age. That looks okay. I mean, we can parse that. We can strip out the backticks and the JSON and then we have the object. We can work with that. Problem is that we never or we can’t be sure that it will always generate the same um block of JSON. Maybe it will change the the structure of the JSON. Maybe it won’t use JSON at all because

02:09 it wants to use XML or markdown. We we just don’t know. That’s where structured output plays a big role. So, if we have a look at this little graphic here, when an LLM generates text, it does that token by token. So, it basically one run of the model um returns a list of tokens with the scores how likely it is that this one should be the next token. Here we have uh a text “When was Stanley

02:41 Boyle born?” And what you can see on the left-hand side is that we have a couple of tokens that could be the next token. We have born, birth, date, birthday, and given. Another thing that you can see is that some of them are green, some of them are red. So, some are allowed, some are not allowed. Born, for example, is not allowed. Birth is not allowed. Birthday is not allowed. So, it has to pick date. Next token is of. Of is allowed, so it can pick pick this one. So, the idea of constraint decoding, and that’s

03:12 basically the technical term of structured output, is that instead of picking one of the available tokens, it will strip out some tokens that are not allowed by some constraints. If we look at the implementation, here we can see that we have the pipeline from Transformers AS. And now we have a new package called Transformers structured output. Let’s have a look at the um at our GitHub repository. In version four, we did some re-rearrangement in the GitHub

03:46 repository, too. And there we do have this new packages folder, and inside packages, for the longest time, we only had the Transformers package. Now, we also have the Transformers structured output package. And the idea with this restructuring is that we were able or we are able to actually ship new packages, um, that we can use as plugins or as as add-ons to Transformers.js. And this structured output package is basically the first one that we that we shipped.

04:17 So, we can import the structured output processor from this uh Transformers structured output package. If we look at the code, we can see that here we have our generation process that adds new tokens to the to the streamer. And right now, we do have we have no constraints. Now, we also have this new JSON processor. And the new JSON processor is coming from the structured output processor. We have to pass the the tokenizer, and then we have the JSON schema, and the exact schema that we

04:48 expect. And the cool thing is, to ship this library, we haven’t or we didn’t have to change anything in Transformers.js, because Transformers.js already did support or did support already the logits processor. And that is basically a way to hook into the last step of generating a new token, the decoding step. And in the decoding step, we have all the the the the tokens that are or that could be um possible next tokens. And here, we can hook in, and we

05:19 can basically strip out all the tokens that we don’t want to support. So, we can pass this new uh JSON processor to the logits processor. There we go. And if I now run the same example again, there we go. You can see that now it didn’t generate markdown, it actually did generate just JSON, because it starts with the curly brackets. And instead of name it picked full name. Why?

05:50 Because we told it to use full name as a property, and that property is actually required. So, we basically forced the model to follow an exact exact same schema. We can do the same thing with regex too. So, instead of using type JSON schema, we can also use the type regex and then pass the regex. And what will happen next is we then click generate, it will generate a regex, or it will generate the person

06:22 equals Bob Johnson, and age will be 25. So, we basically have the same system prompt, the same user prompt, but because we force a very specific a very specific structure, um it generates the structure that we that we want. So, we have the constraint decoding that allows us to um to to be very specific about the structure that we expect. If we take a look at a diff- different example, here I built a little um a little recipe generator. And what I can do now is I can

06:55 generate a new recipe, and it will then generate that by following the exact JSON structure that I want it to follow. So, we have a title, description, we have uh the ingredients, and if you look at the response, that’s the JSON it generates. And I can then parse the JSON in my application, and it will then fill basically the layout that I have. And that’s only possible because I know the exact structure that it will

07:27 generate, and I can rely on the structure that um that it will generate. Let’s have a look at some of the um technical details behind the feature. So, being able to pass a a structure that it needs to follow has an issue or has been an issue that has been open since a couple of months. So, I really wanted to see how we could solve that. And the main problem is that yes, we have the constraint provided by the application and then we need to have

07:58 some kind of package that basically validates new strings against those constraints. So, we have already generated strings, already generated output and then on each step we basically need to validate all the tokens against those constraints. There is a library called LLM guidance that has already been used for this since quite some time and a lot of projects already are based on that. So, we have the prompt API that uses that.

08:30 We have VLLM, llama.cpp. So, it’s pretty well known in the in the community. So, the first step as a prototype was to basically create a WASM wrapper around the Rust LLM guidance implementation and to then also create a little adapter that hooks into the generation process in inside Transformers.js. I did that and it did work. It was quite slow and we also had to download this 1 MB library as a web assembly file which wasn’t wasn’t optimal. So, the next step is that we basically

09:05 created a complete JavaScript only implementation of LLM guidance, of course with the help of a lot of coding agents. And that went really, really well and it allows us allowed us to completely eliminate the the 1 MB JavaScript web assembly library. But we didn’t stop there. We also invested a lot of time into the into the into the performance side because as you can imagine, whenever a new token is added, we need to basically regenerate the whole mask of allowed

09:38 next strings and we need to then compare that mask to the tokens that the model generated. It did take a couple of of iterations, but as you can see now I created a little test case where we have um the Gemma four model, we have the Granite model and we have the LFM 2.5 model. So we have different models, different tokenizers and different test cases. And if we look at the um the tokens per second and the difference, we can see that we barely have any overhead. So yes, sometimes it

10:12 adds it’s a little bit slower if we have constraint decoding enabled, but it’s very very minimal. And we’re super proud to now have a package that is super fast and should work with any uh with any tokenizer. Now if we head over to the repository, you you can see that we have some instructions here. So all the um the the schemas that are supported, we have a JSON schema, we also have regex, um we do have it documented here. Please feel free to try it out. We would love to get some get your feedback on

10:46 that new product, that new package. And yeah, thank you so much for watching. Please check out the new version, Transformers Trace version 4.3. It is out now. You should try it and um have fun with it. Thanks. Bye.