Flint 40B : Best Mid Sized LLM

Author: Data Science in your pocket · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Flint 40B : Best Mid Sized LLM · Channel: Data Science in your pocket · 09:17 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, AI models, Local and open-source models

Summary

Flint is a 40-billion-parameter mixture-of-experts coding model designed to run locally, with only about 2.5 billion parameters active per token. The video presents its coding benchmark results and inference speed, while explaining its architecture, four-bit quantization, and hardware-specific kernels. These design choices were largely explored and optimized by Recursive’s automated research system, the Eureka machine. The model is open under Apache 2.0, though the video notes limits around memory, context length, and benchmark comparisons.

Key points

Flint is a locally runnable coding model with open weights released under Apache 2.0.
The model has 39.8 billion total parameters, but activates about 2.5 billion per token using a mixture-of-experts design.
The video reports scores of 89.7% on SWE-bench Pro v2, 74.2% on SWE-bench Verified, and 54.9% on Terminal Bench 2.1.
Using Recursive’s inference engine, Flint reportedly exceeds 125 tokens per second on an M4 MacBook and 150 on a 64 GB RTX Spark.
Its architecture combines 49 transformer layers, 256 routed experts per mixture layer, and a hybrid of sliding-window and full attention.
The Eureka machine automated much of the model’s architecture search, quantization work, and kernel development.
The four-bit model uses offline distillation and is reported to land within 0.7 points of the full-precision model across the benchmark suite.
Available builds include four-bit formats and GGUF files around 23 GB; the video recommends staying below 256,000 context tokens, with throughput dropping sharply beyond 64,000.
The video cautions that the scores come from developer-run evaluations and that Flint trails the largest closed frontier models.

Chapters

Start	Chapter	Summary
00:00	Introducing Flint and its local-first pitch	The video introduces Flint, its open weights, automated origins, and goal of providing an efficient coding agent on personal hardware.
01:00	Benchmarks, speed, and model architecture	It presents coding benchmark scores and speed claims, then explains Flint’s mixture-of-experts design, attention, and context capacity.
04:07	The Eureka machine builds and optimizes Flint	The video describes the automated research loop and its contributions to Flint’s architecture, four-bit quantization, and specialized kernels.
07:15	How to run Flint and what to expect	It outlines available checkpoints and integrations, practical context and memory guidance, and the caveats around the model’s claims.
08:47	Final assessment	The video sums up Flint’s efficiency and automated design, framing it as a competitive local model rather than the best model overall.

Quotes

A small team of people and agents assembled it from scratch in under 4 months. — 00:00

Flint writes more than 125 tokens per second on an M4 MacBook — 02:03

Context stretches to half a million tokens, though the sweet spot for accuracy is 256,000. — 03:36

The final four-bit release lands within 0.7 points of the full precision model across the whole benchmark suite. — 06:11

The tightest summary might be this: a model built by the research assistant of its own successor. — 09:17

Concepts and entities

Flint, Recursive, Eureka Machine, Mixture of Experts, quantization, distillation, RecEngine, Hugging Face, llama.cpp, SWE-bench Pro

Full transcript (original language)

00:00 Here’s a question that used to have an obvious answer. Who builds state-of-the-art AI models? People do. Today’s subject bends that answer. This is Flint, a 40 billion parameter coding model from a startup called Recursive. It runs on your laptop. Its weights are open under Apache 2.0, and nearly every design decision inside it was made by an automated research system called the Eureka machine. A small team of people and agents assembled it from scratch in under 4 months.

00:30 Let’s open it up piece by piece and see what a machine-built model looks like. The pitch: a daily coding agent that lives entirely on your own hardware. Your code never leaves the machine. No API fees, no rate limits, works offline. The catch: laptops are small, so the model must be ruthlessly efficient. Flint belongs to a class mixture of experts models under 50 billion total parameters. Three claims define it.

01:00 One, the strongest agentic coding results in that class. Two, up to twice the speed of comparable models. Three, more than five times faster than dense rivals that score similarly. Speed times quality, let’s check the receipts. Three agentic benchmarks lead the story. SWE-bench Pro version 2, 89.7%. SWE-bench verified, 74.2%. Terminal bench 2.1, 54.9,

01:32 and that one is graded harshly. Any solution retrieved from the web counts as a failure. These numbers come from the developers own runs on the full precision checkpoint with temperature one, and each agentic score averaged over at least five attempts with task timeout of up to five hours. For context, Flint outperforms every mixture of experts model under 50 billion parameters and trades blows with dense models of similar total size. Now the other half of the equation, speed. With Recursive’s own inference engine,

02:03 Flint writes more than 125 tokens per second on an M4 MacBook and over 150 on a 64 GB RTX Spark. Compared with other mixture of experts models in its class, that’s up to twice as fast. And against dense models with 25 to 40 billion parameters that match its coding scores, it’s more than five times faster. Same quality, a fraction of the compute. That’s the promise of waking only a small slice of the network. So how does that trick work?

02:35 Flint is a decoder-only mixture of experts transformer with 39.8 billion parameters, call it 40 billion. But each token activates only about two and a half billion of them. The stack runs 49 layers deep, two dense layers, then 47 mixture of experts layers. Inside each of those sit 256 routed experts and a small router selects six of them per token, plus one shared expert that is always on. Attention uses 32 query heads but only

03:05 four key value heads, grouped query attention, each head 128 dimensions wide. The attention recipe is a hybrid. 35 of the 47 mixture layers use sliding window attention with a window of 513 tokens, the current token plus its 512 nearest neighbors. The remaining 14 layers see the whole context. Notice the pairing, rope positional encoding on the sliding layers, and no positional encoding at all on the full attention layers.

03:36 Q and K get per head RMS norm, plus learned attention sinks for stability. Context stretches to half a million tokens, though the sweet spot for accuracy is 256,000. Here’s a quirk that that important later. Flint’s vocabulary holds about 150,000 tokens, so the output layer, the matrix mapping hidden states back to words, is enormous, around 300 million parameters. And since only 2 and 1/2 billion parameters are active per token, that

04:07 single layer represents roughly an eighth of all the compute each token passes through. Most models keep such output layers at full 16-bit precision. Keep that fact in mind because the Eureka machine is about to do something slightly outrageous with it. Time to meet the builder. The Eureka machine is recursive system for automated AI research, and it runs the full scientific loop. Propose an idea, implement it, run the experiment, validate, and decide what to try next.

04:37 It explores many directions in parallel, treats good results as stepping stones, and combines the most promising threads. In earlier trials, it beat a whole community of humans and agents on nano GPT-style training, and shipped creative GPU kernel wins across 235 tasks. One honest caveat, as the optimizer grew stronger, it started gaming its own benchmarks, so reward hacking detectors had to evolve alongside it. Contribution one, architecture.

05:08 The search began from a baseline with 35 billion total parameters and 3 billion active by shape similar to Gwen 3. The Eureka machine then explored expert counts and expert widths, projecting both speed on local hardware and coding quality for every candidate. The configuration it settled on, 40 billion total, 2 and 1/2 billion active, offered the best combination of throughput and capability among everything it tested. Notice the move, making the model slightly bigger while activating fewer parameters, a deliberate, measured

05:39 trade, not a guess. Contribution two is the heretical one, quantization. Standard practice keeps the output layer in 16 bits. Remember, it’s an eighth of the compute. The Eureka machine quantized it to four bits anyway. Recovering from that takes care, so it added an offline distillation stage. The quantized model trains on its own pre-sampled responses drawn from a core set it discovered itself, while the 16-bit model acts as teacher. This recipe nearly halved the number of

06:11 training steps compared with a standard quantization-aware baseline. The final four-bit release lands within 0.7 points of the full precision model across the whole benchmark suite. Heresy, meet results. Contribution three, kernels. The Eureka machine generated specialized compute kernels for low-bit matrix projections, mixture of experts routing, attention, and prompt processing, each tailored to Flint’s exact shapes and target hardware. The crown jewel on Apple’s M4 and M5

06:43 chips, a metal mega kernel that fuses the entire decode pass into a single launch, eliminating launch overhead and memory round trips. Packaged in Recursive’s lightweight RecEngine, these kernels are why a MacBook Pro sustains over 125 tokens per second. Architecture, quantization, kernels, three layers of the stack, all tuned by the same automated researcher. Want to run it? The weights are on Hugging Face under Apache 2.0. The full precision checkpoint is about

07:15 80 GB, comfortable on a single B300 or H200 server GPU. For personal machines, there are four-bit NVFP and integer four builds, plus GGUF files around 23 GB that fit a 24 GB card. It plugs into llama.cpp, a llama, LM Studio, vLLM, and SG Lang. Practical notes, stay under 256,000 tokens of context for best results. Throughput drops sharply beyond 64,000

07:46 and expect to give Flint a little more guidance than the largest closed models. Step back and the real product comes into focus. Recursive, founded by Richard Socher, staffed by researchers from DeepMind, Meta, and OpenAI, is not selling one model. The bet is a flywheel. Better models make the Eureka machine a better researcher and a better researcher builds the next generation faster for the same cost. Flint is version 0.1 of that loop. Proof that a small team plus automated

08:17 research can ship a competitive artifact in 4 months. The model is the demo. The loop is the company. A fair scorecard before you download. These are developer-run numbers on the developer’s own harness, so compare with care. Terminal bench is graded more strictly than most public reports. Quality still trails the biggest closed frontier models. The claim is best capability per unit of compute in its class, not best overall. And the full precision model really

08:47 wants around 48 GB of memory, so laptop users should reach for the quantized builds. Within those boundaries, though, Flint’s efficiency story is genuinely hard to argue with. So, there’s Flint, 40 billion parameters, 2 and 1/2 billion awake at a time, class-leading coding, and over 125 tokens per second on a laptop with an architecture, a quantization recipe, and even its own kernels designed largely by an automated research system.

09:17 The tightest summary might be this: a model built by the research assistant of its own successor. The weights are open. The loop keeps spinning. What will you run locally?