LlamaCPP on Native Windows | This is HUGE

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: LlamaCPP on Native Windows | This is HUGE · Channel: Prompt Engineer 48 · 09:13 · Watch on YouTube

Topics: AI developer tools, Model inference and optimization, Local and open-source models

Summary

The video demonstrates a native Windows llama.cpp app that runs local models from the system tray, without requiring Linux, WSL, or a manual terminal setup. The presenter installs it, downloads Gemma models, uses the chat interface and overlay, and connects to the local OpenAI-compatible server through curl and Python. Tests cover vision, tool calling, structured output, and code generation, followed by speed measurements and practical notes about GPU memory and model size. The presenter recommends it as an accessible starting point for local AI on Windows, while noting that the project is new and results vary by hardware.

Key points

The Windows app is installed through an installer and runs from the system tray; the presenter says it requires no manual Python or CUDA toolkit setup.
Models can be selected and downloaded from Hugging Face through the app, and they share the Hugging Face cache with other tools.
The app includes a chat interface and a keyboard shortcut for opening a chat overlay.
A local server runs on port 9931 and supports the OpenAI API, allowing connections from curl, Python, and compatible apps.
The app’s command line and tray interface share the same model cache.
The presenter demonstrates vision, tool calling, JSON-schema structured output, and code generation using the local server.
Gemma 4E 4B generates at roughly 56–59 tokens per second in the presenter’s tests; Gemma 3 4B reaches roughly 61–69 tokens per second.
Prompt processing was much faster than generation in the tests: around 2,800–3,100 tokens per second versus roughly 58–69 tokens per second.
The presenter reports that two resident models used nearly 8 GB of GPU memory, limiting what fits fully on an 8 GB card.
The presenter cautions that the app is an early release and that performance depends on the system being tested.

Chapters

Start	Chapter	Summary
00:00	Introduction and project overview	The presenter introduces local Gemma 4 performance on a Windows laptop and explains how the native app differs from earlier Windows setup options.
01:33	Installing the app and downloading a model	The presenter installs the tray app, reviews its model list, and downloads Gemma 4 from Hugging Face.
02:37	Chat, tray controls, and developer access	The video demonstrates the chat interface and overlay, then shows how to access the local OpenAI-compatible server and shared model cache.
04:21	API use and local model capabilities	The presenter uses curl and Python and tests vision, tool calling, structured output, and code generation through the local server.
07:02	Speed results and hardware considerations	The presenter compares model speeds and GPU memory use, then discusses which model sizes are practical on an 8 GB card.

Quotes

No Linux, no WSL, no Mac. — 00:00

There’s no Python and no CUDA toolkit to set up by hand. — 01:33

The app runs a llama.cpp server on port 9931 and it speaks the OpenAI API. — 04:21

So, reading is roughly 50 times faster than writing. — 07:33

If you have a Windows 11 PC and want local AI without a terminal, this is the easiest way I’ve seen to start. — 08:41

Concepts and entities

llama.cpp, gmml-orgs/llama.cpp, Hugging Face, Gemma 4, Gemma 3, OpenAI API, WinUI 3, RunPod

Full transcript (original language)

00:00 This is Gemma 4 answering on my Windows laptop at about 58 tokens a second and the whole thing is controlled from one icon in the system tray. No Linux, no WSL, no Mac. Now, mind you that this is the AGB RTX 406 laptop that I have here. Everything is running on my local system. In this video, I’ll install it from scratch, run it three different ways, and show you the real speed numbers, including the one that surprised me. Let’s go ahead and test this out. So, if you’ve tried running models

00:31 locally, you know the story. The nice tools showed up on Mac first, and on Windows, you ended up in a terminal building llama.cpp or setting up WSL. Now, we have a native Windows app, and it’s on the Hugging Face blog right here. You can see that it’s a WinUI 3 app that sits in the notification area. You pick a model from the Hugging Face hub, and it downloads in one click. The page says, “No terminal, no build step, and no admin rights.” And hold on to this because later I’ll show you the same server answering plain curl and

01:02 Python calls. Let’s look at the actual repo. It’s gmml-orgs/llama.cpp, and you can see it’s MIT license with about 142 stars and 18 forks as I’m recording this. The latest release is V0.12 from the 10th of October. So, this is a very new project, and I’ll say that plainly it’s early, so expect rough edges. On the llama.app site, there are docs for installation, quick start, the command line, the server, the web UI, and the API server. We’ll use pretty

01:33 much all of those in this video. Okay, so let’s install it. I go to llama.app, click download, and Windows opens an installer. Here’s the dialogue, and it says llama is ready, publisher Hugging Face. So, I’ll click launch. Now, this app has no window of its own. It lives in the system tray next to the clock, and if you don’t see it, click the little arrow to show hidden icons. That’s honestly the whole install. There’s no Python and no CUDA toolkit to set up by hand. And if llama.cpp isn’t on the machine, the app installs it for

02:04 you. The docs list other ways to install, too. A one-line install script for other platforms, pre-built binaries, Docker, and building from source. But on Windows, the desktop app is the recommended route. Now, click the llama icon and you get this flyout. On the top, there’s a recommended models list, and you can see Gemma 3, Gemma 4, and a few others. Each one with the size on disk and whether it has vision. That matters because you can check if it fits your GPU before you download anything. I pick Gemma 4 and click download, and you can see the live progress right here.

02:37 The models land in the normal Hugging Face cache under your user folder .cache huggingface hub. So, llama.cpp and other Hugging Face tools share the same copy. You download once. All right, the model is loaded, so let’s open the web UI. This is the llama.cpp chat interface running on this machine. First, I’ll ask which model are you, and you can see that before the answer, there’s a reasoning block. Gemma 4 thinks first, and the web UI shows you that thinking, which is useful for checking why it answered the way it did. Now, let’s try something more real. I type, “Can you interview me on

03:12 knowledge of LLMs?” and it asks me a first question about the core difference between traditional AI models and the architecture behind modern LLMs. I answer it, and it grades my answer, and then moves to the training phase, then RLHF, and then prompt engineering. So, this is a full study session running on my laptop with no API key and no login. Back in the tray, the models panel shows what’s in my cache, Gemma 3 4B, Gemma 4 E 4B, and a small 2B coding agent model. You can load and unload each one from

03:47 there. There’s also a shortcut. The block says to press Alt and Space from anywhere in Windows to open a chat overlay, so you can ask your local model something without switching windows. And the latest commit in the repo adds llama.cpp deep links, so a link can open a chat with a prompt or load a model. Now, here’s the part for developers. The app runs a llama.cpp server on port 9931 and it speaks the OpenAI API. So, in a terminal, I run curl localhost:9931/models.

04:21 You can see the three model IDs, each with status loaded or unloaded, and the context length is 131,000 tokens. For Gemma 4, it also lists text, image, audio, and video as inputs. Now, in Python, I use the normal OpenAI client. I just changed the base URL and the key can be anything. My first try returned empty text because I set max tokens to 200 and Gemma 4 spent all of them on reasoning. When I raised it to 1,500, it used 450 tokens and about

04:51 1,500 characters of that was reasoning before the three-line haiku. So, if you call it from code, give it room. And this isn’t just my idea. The llama.cpp app homepage says it directly. Point any app that works with OpenAI at localhost:9931/v1 and it lists coding agents, editors, and chat apps. There’s also a command line. If I type llama help, it lists serve, CLI, update, download, version, and licenses. And llama CLI cash list shows the same three models that tray app downloaded, so the

05:26 app and the command line really share one cash. The docs also show llama CLI with -hf and a model name to chat in the terminal and llama serve to start the same server without the app. Now, let me try things you can’t do in a plain chat window. These are four tests I ran against the same local server one after another. First vision, I send it a screenshot from my own recording and ask what’s on the screen. It answers in two sentences, a subscription confirmation message and a sidebar listing the

05:58 models. That took about 11 seconds with 434 tokens just for the image. Second, tool calling. I define one function, get disk free, that takes a drive letter and ask how much free space is on my D drive. The model doesn’t guess, it returns a tool call with the function name and the argument D in about 5 and 1/2 seconds. Your code runs the function and sends the result back. And that’s how a local agent works. Third, structured output. I give it a JSON schema with four fields and a sentence about this release. It returns

06:31 name llama windows version v0.12.0 date 10 October 2026 license MIT valid JSON in under 6 seconds. Fourth, code. I ask for Python script that prints the first 15 Fibonacci numbers. I take the code block it gives, save it to a file, and run it. And you can see it prints 011235 all the way to 377, about 4 seconds. So, that’s chat, vision, tools, structured output, and code from one

07:02 tray app, and nothing left the machine. Okay, so the number I promised, I used the server’s own timing fields, same machine, same prompts. Gemma 4E4B generates about 58 tokens a second. I measured 59, 56, and 58 across three prompts. Gemma 3 4B, the quantized one is a bit faster, between 61 and 69 tokens a second. The surprise was the prompt side. When I send a prompt of about 4,000 tokens, Gemma 4 processed it at around 2,800

07:33 tokens a second, and Gemma 3 at around 3,100. So, reading is roughly 50 times faster than writing. The docs have a page on exactly this, prefill versus decode. Prefill is compute bound. It handles the whole prompt in parallel. Decode is memory bandwidth bound, one token at a time. That’s why long document goes in fast, but the answer comes out at reading speed. Also, mind you, the first call to a model that wasn’t loaded took about 34 seconds because the server had to load it. After that, it’s quick. One more comparison, same question to

08:06 both models. Explain what a GGF file is in two sentences. Gemma 3 answered in 62 tokens. Gemma 4 spent 292 because it reasons first and still ran at about 62 tokens a second. So, the speed is similar, but the reasoning model uses more tokens per answer. So, who is this for and what should, you know, my GPU is 8GB and Nvidia has me short about 7GB used and almost 8 at the peak with two models resident. So, a much bigger model won’t fit. The app is on release 0.12, so it’s new.

08:41 The reasoning in Gemma 4 uses tokens, so set your limits higher and I tested one machine, one GPU, so your numbers will differ. If you have a Windows 11 PC and want local AI without a terminal, this is the easiest way I’ve seen to start. And about size, the llama.cpp page shows example models on a PC, a 27B at 19GB, a 20B at 12.1, and Gemma 4E 4B at 4.59. On an 8GB card, only the last one fits fully on a GPU. So, that’s the class of

09:13 model I’d start with. If you’re looking for cheap and reliable GPUs, I use RunPod and the link is in the description. So, in this video, we have seen how to install llama.cpp pick and download a model, chat with it, and call it from curl and Python on our own machine. Go ahead and try it on your own system and let me know which model you ran. Next, I’ll test how far that 8GB card can really go. So, check out the next video.