Your GPU Neighbor Can Hear Your Model — Branislav Brzak, Stealthium

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Your GPU Neighbor Can Hear Your Model — Branislav Brzak, Stealthium · Channel: AI Engineer · 13:06 · Watch on YouTube

Topics: AI infrastructure and hardware, Local and open-source models, Privacy and data protection, AI safety

Summary

The talk describes timing side channels that can reveal information about LLMs running on GPUs shared with other tenants, without accessing the victim’s memory or using a traditional exploit. On the same H100 GPU using MIG, the researchers report identifying model architecture and depth, token speed, and—in some conditions—the kind of workload. A separate attack across GPUs connected by NVSwitch can also reveal model fingerprints, though it loses the layer-count signal and is less accurate. The speaker concludes with mitigation advice focused on tenant placement, understanding hardware trade-offs, and monitoring GPU activity for suspicious workloads.

Key points

The attacker times activity on shared GPU hardware and reads only its own memory; the attack does not require root access or a traditional exploit.
On MIG, compute units, memory capacity, bandwidth quotas, and L2 cache are partitioned, but GPU-wide memory barriers can still affect timing across instances.
The talk reports identifying model architecture and depth with 97% accuracy and token speed with 99.3% accuracy on real H100 systems.
Layer-count fingerprints help distinguish models whose token rates are very similar; the described fitting method does not require a training set.
With speculative decoding enabled, timing can also help classify workload types such as prose, code, JSON, or retrieval.
An NVSwitch-based attack can work from another GPU in the same server, but the layer-count signal is lost; the reported model-identification accuracy is 79.2%.
The speaker says the probes can be detected by monitoring kernel launches and identifying repeated access patterns with little data returned to the CPU.
Recommended mitigations include avoiding placement of untrusted tenants beside sensitive models, understanding the trade-offs of MIG and sharding, and actively monitoring GPU workloads.

Chapters

Start	Chapter	Summary
00:01	GPU timing side-channel overview	The speaker introduces the threat of learning about a neighboring tenant’s LLM by observing timing on shared GPU hardware.
02:07	MIG isolation and the timing primitive	The talk explains which GPU resources MIG isolates and how GPU-wide memory barriers can still produce a timing leak.
04:12	Recovering token speed and model depth	Timing traces reveal token rates and layer-count fingerprints that can help identify models, including ones without prior training examples.
07:24	Workload classification and NVSwitch attacks	The speaker describes classifying text workloads under speculative decoding and presents a less precise model-fingerprinting attack across GPUs.
11:03	Detection and mitigation	The talk presents suspicious kernel behavior as a detection signal and recommends careful tenant placement and active GPU monitoring.

Quotes

Timing is the one thing that isn’t isolated. — 02:37

The attacker only ever reads its own memory. It only times its own reads. — 03:09

This is not a trained machine learning classifier. — 05:46

79% against a 20% baseline is still real fingerprints not noise. — 11:03

First placement is something operators actually control. — 12:03

Concepts and entities

Branislav Brzak, Stealthium, Multi-Instance GPU (MIG), NVLink, H100, Qwen, Llama 3, speculative decoding, Delty

Full transcript (original language)

00:01 Hello everyone. This talk is about the danger of your LLMZ information leaking through shared GPU hardware. If you’re running a model in the cloud on a GPU you share with other tenants, someone next to you can learn things about it. They can tell what model you’re running, how fast it’s responding, and even what kind of work is doing right now. All without touching your data, just by watching timing. That’s what this sock covers. What a code tenants on the same hardware can learn about your LLM from timing alone and what you can do about it. To start, I’m Brian Sabbrzac, a cyber

00:32 security researcher employed at Stealthium. At Stealthium, we specialize in securing and providing observability into the environments AI systems run on. My area of focus is low-level GPU related security research. That’s something this talk covers. When you share a GPU between tenants, you would expect full isolation, but the isolation provided may not prove to be sufficient. In the case of this attack, the attacker doesn’t even need any special permissions like root access. It doesn’t need any traditional exploit either. The only thing it does is time

01:05 the parts of the hardware that still shared among tenants. This is a demonstration video of what the victims and the attackers dashboards look like. On the left, we have quentry model doing. On the right, the attacker uses a side channel that quentry leaks into as it’s running. From that it determines exactly what model it is and what kind of text is producing right now. These are just the headline claims but not the whole story. We can tell which model it is architecture and depth

01:36 within 97% accuracy. We can tell how fast is the coding within 99.3% accuracy. We can tell when and how much the busy periods token counts and the answer length. What we can tell what kind of work it’s doing, codes, J pros, JSON or retrieval. What we cannot tell though are the actual prompts, answers or contents, just timing. And all of these on real H100s serving real LLMs. If you’re using Nvidia hardware, GPU

02:07 isolation is achieved using multi instant GPU or MIG for short. Each isolated MAG slice has its own physical compute units and dedicated VRM that provides hardware isolation on those parts. In our example, the LLM we’re serving runs on a slice with three compute units and 40 GB of VM. The attack runs on just one compute units and 10 GB of VM. What’s isolated are compute units, memory capacity, bandwidth quotas, and

02:37 false errors. The L2 cache is partitioned per instance too. There’s no cross instance eviction. So it’s not shared in that sense. The actual leaks are memory barriers or members which are GPU wide. One instances barrier still touches every L2 partition on the card. Timing is the one thing that isn’t isolated. So when one tenant is busy, the other tenants owns get slightly slower. The key point is the a GPUwide memory barrier takes effect in every L2

03:09 partition on the card. The next slide builds the whole attack on that one fact. The vector is that the attacker only ever reads its own memory. It only times its own reads. It never touches the victim’s memory. There are three steps. All from its own small buffer, time the load and repeat every five microsconds. This primitive and the MIG member leak itself isn’t our discovery. Credit goes to the researchers behind behind bars at using security. A big thanks for their

03:41 work. Now try looking at the timing differences with the naked eye comparing your tenant idle versus serving an LLM. You wouldn’t even be able to discern them. The median latency is only three microsconds higher. To the attacker though, those three microsconds are all it needs. That’s enough to gather all the information we already talked about. Quentry with 8 billion parameters is running on the victim slice throughout this one. All captured data was taken

04:12 from a live model. At the top is GPU activity over time, one burst per token. Below is the frequency spectrum of the attackers on latency trace. There is a fundamental at 81 Hz which is just tokens per second plus harmonics since each token is a mix of distinct frequencies not one clean one. This is a real trace one token in the time domain. The token shows a total of 39 bumps. The bumps with green peaks are the layer invocations while orange ones

04:44 are additional between layer processing. In a later slide, we will demonstrate how we determine the actual layer counts even with those extra bumps are present. Here we look at the decode speed itself. Just by timing, we found the token ray completely bind. So speed alone is something an attacker can recover accurately. But speed by itself isn’t a clean fingerprints. Quentry 8 billion and llama 38 billion they quote only 1.6% apart in their token rate. So from

05:15 speed alone, you’d think they’re the same model. It narrows things down, but it doesn’t nail it. We need something sharper. Layer counts. That’s what the next few slides get into. We use the same fold and average stick from before. This time we run it once per model instead of once per token. What comes out is a bit count that lands almost exactly on a whole number. The mean distance from an integer across 22 traces is just 0.028. 0 to8. That’s the

05:46 tell. If this were just a fixed hardware artifact divided by different token rates, we’d expect factions, not whole numbers. Each count works out to the model’s layer count plus one to true, one to three additional picss. Every model keeps its own signal fingerprints, a mix of tempo and harmonic structure. To be clear, this is not a trained machine learning classifier. There’s no training set, no labels, no configuration. It’s under 30 lines of code. The method checks each

06:17 candidate’s depth from 16 to 48. It measures how strongly that depth’s expected signature shows up in the real trace. Then it picks whichever of its best for quen 8 billion 36 layers wins clearly by four point times over the next candidate. Because this is a best fit test, not a lookup table. It also works to models it never seen before. That includes proprietary on unreleased ones. since there’s nothing it needs to have seen in the first place.

06:52 And this works surprisingly well. The layer fit classifier also outputs the confidence level. Across our testing, only two least confident answers ended up being wrong ones. This same method also picks up fine tunes and distillations that that share the same base model. That’s because the architecture and layer count carry over unchanged. Take three related models as an example. Llama 38 billion from meta has 32 layers. Llama 3.18

07:24 billion also from meta uses that same 32 layer backbone with minor upgrades and R1 is still llama 8 billion from deepseek is an improvements on top of that. The per layer timing comes out to only 1.6 600 seconds median latency shifts of each other. The rest of the signal looks the same. Regardless of this, our classifier is still able to discern between them. Now we ask a deeper question on the same channel. Not which model, but what is

07:55 doing right now? We look at four workload types. Pros, code, JSON, and retrieval. We give a work that’s on every half second based on the last 5 seconds of timing. out of 40 half second words with 31 rights and four full responses were correctly identified by majority votes. This is the same classifier from the video earlier misses included. One condition is worth stating this was measured with speculative decoding turned on. That’s an optin in VLM, not

08:26 the default. Here’s the mechanism. How predictable the text is changes how many drafted tokens get accepted at each step. that changes step time and that’s the signal we’re picking up. We tested 11 configurations dense and mixture of experts BF16 and FB8. The same figure page showed up every time just at very different strengths. 70.2% spread on GPT or 20 billion versus 0.9% on GLM 49 billion. 10 out of 11 round pros fastest

08:58 when the chance alone would only give you one in four. So say the attacker already knows the model from the earlier slides and specular quality happens to be on. It can also tell what kind of text is coming out. Now we take the shade chip out of the picture entirely. MLG needs the attacker on the same physical die. This attack doesn’t. The setup here is an eight times H100 server connected over an MV switch fabric. The victim runs tens of parallel across GPU

09:30 0ero and one. The attacker just needs any other GPU in that same server, GPU 2 in this case. Per access over NV switch is a real intended feature. It’s how multi-GPU jobs normally work, not a bug being abused. MLG tensor parallelism also can’t coexist on the same device. So this is really a different population of deployments, not just the same attack repeated. The probe runs on GPU 2. It pointer

10:00 traases memory that actually is on GPU0 over envelop along the way. An idle hop takes about 890 nconds. Once the victim starts working that climbs up to 300 microsconds. These are three half seconds me captures and you can actually count the tokens by eye. What we lose compared to MLG is the layer bit is below the noise at this distance. So over envelink the

10:32 fingerprints we we’re left with is mostly tempo not layer counts. Here we test the five models on identical serving setup. Only the model itself gives away which one it is. This method gives 79.2% accuracy. Now it fails exactly where you’d expect the tempo to fail. Llama 3.5 8 billion and R1 is still are both 8 billion models that the code three hertz apart so they blur together where MIG’s

11:03 layer counts would have separate them easily. The takeaway is that the envel worse than MG’s since it loses the layer bit. But worse signal to noise ratio isn’t the same as safety. 79% against a 20% baseline is still real fingerprints not noise. First, some good news. Our own product at work called the attack straight out of the box. Delty monitors every single kernel launch across every GPU, analyzes

11:33 suspicious workloads, and runs them through a full alert pipeline, which is something not every vendor can do. Here is what’s made this one stand out. A single kernel relaunched about 3,000 times, hitting the exact same 40 kilobytes of memory over and over instead of moving to new data. only one kilobte of data was sent back to the CPU the entire time because the timing itself was the payload not the output. In real terms that is under 600

12:03 milliseconds start to finish with fewer than 50 tokens of a reply. We quot it before the response was ever half typed. So three takeaways. First placement is something operators actually control. So do not put an untrusted tenants. MIG slides right next to sensitive model. Second, know the trade-off. MIG rules out peer-to-peer communication, meaning there is no tens of parallelism on MIG. Sharding over envels

12:35 what leaks to timing alone rather than the layer counts. Third, watch the C GPU itself. Make sure the host infrastructure is actively verifying that no malicious workloads are running on the hardware. That’s exactly what Seltium does by catching these probes straight from the Chrome behavior. Sharing a GPU means timing alone can give away the model, the speed and the workload. That’s true even across envelink without needing any exploit at

13:06 all. Deltium sees this happen in real time. Most organizations still don’t have that kind of visibility. Thanks again to the researchers of Behind Bars. This whole talk builds upon their work. Thank you all for watching.