Cloudflare Clef vs Jev: I Tested Q4_K_M Locally in Ollama

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Cloudflare Clef vs Jev: I Tested Q4_K_M Locally in Ollama · Channel: Prompt Engineer 48 · 13:02 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, Local and open-source models, Multimodal models

Summary

The video tests Cloudflare’s small Clef Flash Q4_K_M package locally in Ollama on an RTX 4060 laptop, using a support-ticket example and a still image. Schema-constrained requests produced valid JSON, but repeated outage requests disagreed about the right team, illustrating why format compliance does not guarantee decision quality. The image example returned a plausible gesture classification, while the local run demonstrated generated answers rather than the native probabilities produced by Clef’s separate joint schema head. Published comparisons with Jev are discussed separately from the laptop results, which the video presents as a limited experiment rather than an accuracy benchmark.

Key points

The test used a quantized Clef Flash package in Ollama on a laptop with an Nvidia RTX 4060 GPU and 8 GB of video memory.
The installed package was reported as about 6.8 GB, while the quantization card listed the backbone file at about 5.84 GB.
Asking for JSON in the prompt alone produced invalid JSON; supplying a schema through the runtime produced parseable output.
Two repeated outage requests disagreed on the support team: one selected technical and the other billing, despite both marking the issue urgent and critical.
An invoice request was classified as billing, not urgent, and minor.
A cropped still image was classified as a peace sign, with no drink being held; this was only a single-image test.
The inspected local quantized file and requests did not load Clef’s separate joint schema head, so the outputs were generated strings, not native option probabilities.
The video distinguishes its local inference timings from Cloudflare’s published evaluation results and cautions against comparing them as if they came from the same setup.
The practical recommendation is to test the actual application workflow, use explicit schemas, validate fields, and review ambiguous or consequential cases.

Chapters

Start	Chapter	Summary
00:00	Decision models and the test question	The video introduces decision models, explains constrained choices and uncertainty, and sets up the local Clef test and Jev comparison.
03:05	Local setup and support-ticket demo	The video describes the laptop and quantized Ollama setup, then tests support requests with both prompt-only JSON and schema-constrained output.
07:16	Image test and the decision-head limitation	A cropped still image yields a plausible gesture result, followed by an explanation that the local path did not use Clef’s joint schema head.
09:53	Published comparison with Jev	The video presents Cloudflare’s published evaluation scores and separates them from the local laptop measurements.
11:28	Practical recommendations and verdict	The conclusion recommends application-specific evaluation and summarizes the local package’s promise and its inference-path limitation.

Quotes

The schema is simply the menu of decisions. — 01:02

Passing the first question does not settle the second. — 02:03

A repeated outage request still with zero temperature and the same allowed fields returned billing instead of technical. — 06:13

These are generated strings that we parsed as JSON. — 09:21

That is the difference between a working local experiment and claiming we reproduced Cloudflare’s complete decision model. — 12:31

Concepts and entities

decision model, schema, quantization, Cloudflare, Clef, Clef Flash, Jev, Ollama, joint schema head, Bartowski, RTX 4060

Full transcript (original language)

00:00 Cloudflare has released KFFT a family of decision models. I wanted to answer one practical question. Can we take the small version put it into a llama on an ordinary laptop and actually use it. So that is what I tested. Just two demos today. A support request and an image. And there is an important surprise in the results. Loading the weights is only part of the story. We will keep this focused. First, what makes a decision model useful? Then the local setup. the two demos and a comparison with the

00:31 original Jeff model from Typesafe. The goal is to understand what you can run today and what the results really mean. There is no giant collection of examples here. We will spend time looking carefully at the few examples we actually ran. Imagine your application receives a support ticket. Someone has to decide which team handles it and whether it needs attention. Now, a normal language response gives you a paragraph. Your application needs a value, something like technical, billing or sales. A decision model is built around that situation. You give it the

01:02 information and define the available answers before the model makes its decision. The schema is simply the menu of decisions. One question might be a yes or no. Another selects a department. A third chooses a severity level from an ordered scale. That makes the output easier to put into ordinary software. The surrounding application knows what each field means. It can route the ticket, raise a notification, or leave the ticket in a queue without trying to interpret a long explanation. There is another useful ingredient,

01:33 probabilities across the allowed options. Think about a message that could belong to billing or technical support. Instead of forcing every situation into a confident answer, an application can use uncertainty to decide when a person should take over. The threshold is an application choice. You still need examples and evaluation to find a threshold that makes sense for your actual customers. A constrained answer space helps with integration, but it does not make the decision automatically correct. Billing is a valid department even when the message

02:03 should have gone to technical support. That distinction becomes very visible in our local test. Keep two questions separate while watching. Did the output follow the format and did the selected answer make sense? Passing the first question does not settle the second. The original Jev model is the comparison here. Types safe introduced it as a system one model with unstructured state going in and typed probabilistic decisions coming out. That is a useful baseline for understanding this category. The comparison in this video

02:35 uses the original announcement and published evaluation results. I did not run a live Jev request on this laptop. Though there is no homemade jev timing to show, the smaller CloudFlare release is Clef. Its model card describes a 9 billion parameter backbone with visual inputs plus a separate joint schema head that scores the decisions. The weights are available under Apache 2.0. That combination is why local experimentation is interesting. We can inspect the release and test a quantized version

03:05 rather than only reading about an API behind a hosted service. Here is the machine used for this video. An Nvidia RTX 4060 laptop GPU with 8 GB of video memory. Oama version 0.35.0. The installed model is the Bartowski Cleft Flash Quantization Q4KM. It was already present when I started the test. So, these are inference checks. I’m not presenting the download as something that happened during this recording. The quantization card lists the backbone

03:36 file at about 5.84 GB. Lama shows about 6.8 GB for the installed package which also includes the visual projector. Those are different size measurements. During the test, Lama reported a mixture of CPU and GPU execution with about 87% on the GPU. This is a laptop run with a modest context, not the published hosted setup. The basic local route is straightforward. Select the hugging face repository and the Q4KM tag through OAM.

04:07 Our requests then go to the local OAMA endpoint. For these tests, I used a48 token context and a short output limit. Those settings keep this experiment small. They do not test a long context window and they do not tell us how a large production workload would behave. The good news is that the model loaded and responded. We have actual output from the local machine. That is already useful if you want to know whether the quantise package can run on this hardware. But a successful load alone does not tell us whether Ola is

04:39 executing the same decision mechanism used by Cloudflare. To find that out, we have to look at both the output and the release files. Our first demo is a short support request. Checkout has been failing for every customer for the last hour. We ask for three fields. Urgent, team, and severity. In this simple menu, technical is the intended team because we are describing a widespread service failure. Critical is a reasonable impact level and urgent should be true. The input is deliberately short so we can inspect the answer without hiding behind

05:11 a huge prompt. The first request asked for JSON in the prompt. Here is the actual output. The model said urgent was true and mentioned technical and critical but the string was not valid JSON. Some keys and values were missing the required structure. That is a real failure for software that expects a JSON object. The cold request took about 18 seconds inside Ola. Most of it loading the model. Next, I supplied an actual JSON schema through AMA. Instead of

05:41 relying only on the words return JSON, the schema defines urgent as a boolean and gives team and severity a fixed set of allowed strings. The resulting output parsed correctly. This is an important improvement for an OAMA application. It shows the benefit of constraining the format at the runtime level. It still leaves us with the question of decision quality. The first schema constraint outage request returned technical urgent true and critical. A llama reported about 2.27 seconds for that warm

06:13 request. That is a plausible answer to the example. If this were the only output we showed, it would look like a very clean success. But there was another result from the same demo family and it changes how confidently we should describe the experiment. A repeated outage request still with zero temperature and the same allowed fields returned billing instead of technical. It kept urgent true and critical. The wall clock time was about 1.96 seconds. I’m keeping both results in the video. This is a small local experiment so we cannot diagnose the cause from one

06:44 repeat. What we can say is that this tested path did not give stable team selection. To check the other side of the same routing demo, I used an ordinary invoice request. Please email a copy of my invoice. There is no outage and no payment dispute. I can wait until next week. That returned billing, urgent false, and minor. The wall clock time was about 1.55 seconds. This answer fits the input well. So, we have one clear success and one case where repeated outputs disagreed. The support demo

07:16 tells us something quite specific. The quantise package can respond locally and Olama can constrain its generated answer to valid JSON. It also shows why one good screenshot is not enough to judge reliability. An application that automatically roots every ticket should be evaluated on a meaningful set of examples. Our two inputs are an illustration of behavior, not an accuracy benchmark for KF or Jev. For the second demo, I used the webcam image visible in the supplied screenshot. I cropped the embedded image so the model

07:47 sees the hand gesture rather than the entire desktop. This is a still image test. I did not record a live webcam stream. The question is simple. Which hand gesture is visible and is the person holding a drink? The allowed gestures are fixed before the request. The image requests returned peace sign and holding drink false. Looking at the input, those answers make sense. Two extended fingers are visible and there is no drink in the hand. The wall clock time was about 1.35 seconds for the

08:18 request. Again, this is a generated JSON answer through Olma. It is useful evidence that this package accepts images, but it is only one picture. The linked webcam repository demonstrates a different execution path. It uses the original release and its reference inference code to produce type decisions with probabilities from webcam frames. Its documented quick start targets an Apple Silicon Mac with at least 32 GB of memory. Our Windows Olama experiment uses a quantized package. The repository is a useful reference, but I did not run

08:50 its Mac setup on this laptop. Now we reach the important detail. The official release lists separate files for a joint schema head. The reference loader explicitly loads that head alongside the backbone. It uses the head to score the allowed options. I checked our local quantise file and its listed tensors. The inspected file contained the usual backbone tensors and I found no separate joint schema head being loaded by the requests we sent. The request logs make the distinction even clearer. Olama

09:21 reports generated output tokens for these answers. The constrained image response used 21 output tokens. The support responses used around 30. These are generated strings that we parsed as JSON. The official decision interface score schema options through its head. We should not call our generated strings the native probability output even though the package has cleft in its name. So the honest claim is that we ran the cliff flash quantise backbone package through Olama and tested its generated answers. We verified local

09:53 loading one visual input in constraint output formatting. We did not reproduce the official decision head in Alama. There are no native option probabilities in these logs and I will not invent confidence bars to make the demo look like the original decision interface. For the comparison with Jev, we need to switch evidence sources. The following numbers come from Cloudflare’s published evaluation, not from our laptop. Cleft Flash scored 93.11 on API Bank versus Jev at 88.19, but Jev led on when to

10:25 call 80.97 versus 65.58. The published picture has strengths on both sides. The workflow results tell a similar story. Customer service was 77 for Cleft Flash and 76 for Jeff. Invoice processing favored Jeff 61.8 8 against 57.1. Agent trace observability also favor Jev 71.6 against 69.8. These are reported evaluation scores. They do not certify the performance of this quantization and

10:56 they should not become a blanket claim that one model wins every task. Cloudflare also reports a much lower median latency for its hosted cliff flash evaluation than for Jev in that same evaluation. That is an interesting published result, but our local numbers answer a different question. Different hardware, different inference path and generated output instead of the decision head. Putting the laptop time beside a hosted Jeff time and declaring a winner would erase those differences. Keep the measurements labeled by where and how they were produced. What would make me

11:28 choose one of these approaches? I would start with the application. Does it need images? Does it need local weights? Does it need type probabilities through a supported interface? Then I would test the actual workflow on examples from that application. Published scores help decide what to investigate. They are much less useful as a substitute for testing the deployment path you will actually use. For the amma path we tested. I would keep the explicit schema and validate the selected fields. I would also review the cases where the

11:59 answer is ambiguous or consequential. The outage repeat is a good reason to do that. If the goal is the full decision model experience, the next implementation step is a runtime that actually loads the joint schema head. Asking a generated answer to include a confidence number would not fix that gap. My verdict is encouraging but specific. The Q4KM package runs locally on this 8 GB laptop. It accepted the image and produced the expected gesture result. Schema constrained support output was valid with mixed routing

12:31 behavior across repeated outage requests. The key limitation is the inference path. This Olama test did not execute the official decision head. That is the difference between a working local experiment and claiming we reproduced Cloudflare’s complete decision model. That is the useful way to look at CLF today. The open release makes experimentation possible and the visual input is a meaningful addition to this category. The original Jev remains an important comparison with competitive published results on several tasks. We

13:02 kept this video to two demo families and save the actual logs. If you try the same local route, check the output and the runtime because the model name alone does not tell you which mechanism is running.