Preemptible compute with Pavneet Ahluwalia
Read the Portuguese translation →
Author: Together AI · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Preemptible compute with Pavneet Ahluwalia · Channel: Together AI · 03:45 · Watch on YouTube
Topics: Model inference and optimization, AI infrastructure and hardware, AI models
Summary
Section titled “Summary”Together AI’s preemptible compute is interruptible with up to five minutes’ notice and is priced 50% below on-demand compute. It uses the same underlying GPUs and infrastructure, and is intended for workloads that can tolerate interruptions, such as batch inference, development and testing, and short research runs. Users can add preemptible GPUs to new or existing clusters; available capacity is fulfilled asynchronously and billed by the minute. The video also shows how to identify preemptible nodes and use labels to keep critical workloads off them, then demonstrates scaling an existing cluster.
Key points
Section titled “Key points”- Preemptible compute can be taken away with up to five minutes’ notice.
- It is discounted by 50% compared with on-demand pricing.
- It uses the same Nvidia GPUs, CPU architecture, and InfiniBand fabric interconnect as other compute types.
- It suits short-lived, non-critical workloads that can handle interruptions and restarts.
- Preemptible GPUs can be added when creating a cluster or to an existing cluster.
- Capacity is fulfilled asynchronously when spare capacity is available, provided it is included in the cluster shape.
- Preemptible compute is billed sub-hourly, so users pay only for the time used.
- Node names and labels help users identify preemptible nodes and schedule workloads away from them when needed.
- The video says preemptible compute is available with Kubernetes, with Slurm support planned.
- Preemptible nodes can provide an option for accessing spare capacity when on-demand capacity is limited.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:02 | What preemptible compute is | The video introduces preemptible compute, its interruption notice, discount, and underlying hardware. |
| 00:33 | Suitable workloads and cluster setup | The presenter describes workloads suited to preemptible compute and begins showing how to add it to a cluster. |
| 01:37 | Availability, billing, and node identification | The video explains asynchronous fulfillment and sub-hourly billing, then shows preemptible nodes and their labels. |
| 02:44 | Scaling an existing cluster | The presenter shows how to add preemptible nodes to an existing cluster and discusses using spare capacity. |
| 03:15 | Adding a preemptible node | The presenter reviews and purchases an additional preemptible node, which will appear in the cluster after a few minutes. |
Quotes
Section titled “Quotes”Preemptible compute is by definition interruptible — 00:02
we’ve discounted it by 50% on top of on demand pricing. — 00:02
whenever we have spare capacity available — 01:37
you’ll only pay for those few minutes. — 02:10
That’s all you need to scale. — 03:45
Concepts and entities
Section titled “Concepts and entities”Together AI, Pavneet Ahluwalia, Preemptible compute, On-demand compute, Reserved compute, Kubernetes, Slurm, InfiniBand
Full transcript (original language)
Section titled “Full transcript (original language)”00:02 Hi. Let’s look into our latest compute type, preemptible compute by Together AI. This joins our two existing compute types, standard or on demand, and reserved compute. Preemptible compute is by definition interruptible, which means that it can be taken away with up to a maximum of 5 minutes notice. As a result, we’ve discounted it by 50% on top of on demand pricing. Under the hood, it’s the same underlying compute, same Nvidia GPUs, CPU architecture, and InfiniBand fabric interconnect. [clears throat]
00:33 What is preemptible compute good for? Essentially, it’s a best fit for burst short-lived workloads that can handle interruption and restarts gracefully. Think about non-critical dev test workloads, batch inference, or fine-tuning runs that runs for a few minutes up to a couple of hours. Non-critical research work where you may need a few nodes just for a short period of time to run maybe some hyperparameter optimizations or experiments. So, let’s get into it. The You can add preemptible compute to both new clusters when creating them or
01:07 to an existing cluster. It looks pretty much the same. You go to your create cluster page, you pick the GPU type you want to use. So, you here you’re adding your on demand compute, and let’s say you want to augment that with a few preemptible nodes, you can just enable it, pick how many GPUs you want to add, and that’s it. The good thing about preemptible GPUs is they’re asynchronously fulfilled. That means whenever we have spare capacity available, as long as you defined it in your cluster shape,
01:37 we will fulfill your cluster with those preemptible GPUs. So, you don’t have to worry about constantly checking for availability and scaling up your clusters. Let’s give this a name. Let’s say demo preemptible. Add a shared volume. Let’s call it test one. Today, preemptible is only available with Kubernetes but we’ll be adding Slurm support shortly. And that’s it. We’re done. You can just preview your cluster and confirm and create. Preemptible compute is billed sub-hourly. That means
02:10 if only if you only get preemptible nodes for a few minutes, you’ll only pay for those few minutes. So, we have an existing cluster here running with the same name. Let’s look at how the experience looks like. You can see that this has two healthy nodes. One is a standard node and one is a preemptible node. In the event timeline, you can actually see how the nodes transition and when the nodes were fulfilled. Now, you don’t necessarily want to put critical workloads on preemptible, right? So, we’ve named the nodes in such
02:44 a way so that they’re easily identifiable. And you even have node labels so that you can do scheduling uh and uh scheduling prioritization and affinity based on those labels to make sure that critical inference pods, for example, don’t end up on preemptible compute. Let’s look at how you would add preemptible compute to an existing cluster. All you have to do is go to scale. You can scale both your on-demand GPUs. You can see right now 16 additional GPUs are available.
03:15 Or you can add preemptible nodes. Now, you can see here there are way more preemptible nodes available than on-demand nodes available. So, when you’re looking for burst capacity and you’d absolutely need compute it as soon as possible and you can’t wait, tapping into preemptible is a very good option to get access to our spare capacity as well, which is not always uh available through on-demand. So, here I’ll just add one more node, review, and purchase.
03:45 That’s it. That’s all you need to scale. Now, in a few minutes, you’ll see that a new node will appear here as preemptible compute.