The 50ms #AI gateway running locally with #Ollama and #Nimble: an Open #Jev
Read the Portuguese translation →
Author: Matt Williams · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: The 50ms #AI gateway running locally with #Ollama and #Nimble: an Open #Jev · Channel: Matt Williams · 00:31 · Watch on YouTube
Topics: Model inference and optimization, Local and open-source models, AI safety
Summary
Section titled “Summary”The speakers discuss Ollama support for Nimble and Tev 1, describing a local AI gateway that can screen emails for prompt injection before passing them to an LLM. They say Tev 1 is a smaller model that is about 30% faster than Jev while giving correct answers just as often. The conversation emphasizes response times of roughly 50–70 milliseconds and compares that speed to the time it takes to move and click a mouse.
Key points
Section titled “Key points”- Ollama 0.35 supports Nimble and Tev 1, with the speakers noting there may be other supported options.
- A model can check an email for prompt injection attacks before handing it to an LLM for further processing.
- The speakers say the screening speed is comparable to Jev.
- Tev 1 is described as a smaller model that is 30% faster than Jev.
- The speakers say Tev 1 gives the right answer just as often as Jev.
- Reported response times are around 60–70 milliseconds, with Tev often taking about 50 milliseconds.
- One speaker compares these response times to the time it takes to move and click a mouse.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Ollama support and email screening | The speakers discuss Ollama support for Nimble and Tev 1, including using a model to check emails for prompt injection. |
| 00:00 | Tev 1 speed and accuracy | Tev 1 is described as a smaller model that is 30% faster than Jev while answering correctly just as often. |
| 00:31 | Milliseconds-scale response times | The speakers cite response times of about 50–70 milliseconds and compare them to moving and clicking a mouse. |
Quotes
Section titled “Quotes”Ollama now supports nimble and Tev 1 and maybe others. — 00:00
it’s a smaller model 30% faster than Jev. — 00:00
Tev is often 50 50 milliseconds. — 00:31
it’s as fast as you can think and drag your mouse basically — 00:31
Concepts and entities
Section titled “Concepts and entities”Ollama, Nimble, Jev, Jev 1, LLM, prompt injection
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 With 0.35 of Ollama, Ollama now supports nimble and Tev 1 and maybe others. >> No way. >> Yeah, and the speed is exactly the same as Jev. You know, now you’ve got this this model that says, “Are there any prompt injection attacks in this email?” No, okay, great. Hand it off to the LLM to process further. And the speed is exactly the same as Jev. In fact, with Tev 1, it’s a smaller model 30% faster than Jev. And and it seems to spit out
00:31 the right answer just as often as Jev does. >> Yeah, I mean just pretty fast I think is the main thing. >> 70 60 70 milliseconds and Tev is often 50 50 milliseconds. >> For most people, if I show something on the screen and you have to move your mouse and click on it, it’ll take you 60 milliseconds. So it’s as fast as you can think and drag your mouse basically, which is it’s non-trivial. >> Phenomenal.