Polars 2.0 Says 5× Faster... So What's Changed?

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Polars 2.0 Says 5× Faster… So What’s Changed? · Channel: Better Stack · 08:16 · Watch on YouTube

Topics: Data, databases and embeddings, Software engineering with AI, AI developer tools

Summary

Polars 2.0 makes its streaming engine the default for queries, which can improve performance but means operations such as joins and group-bys no longer guarantee row order unless requested. The release also removes or changes several APIs, with errors generally pointing users toward replacements, and tightens some type and concatenation behavior. The advertised 5× speedup is Polars’ own expectation, not a benchmark result presented in the announcement, and streaming does not yet provide true out-of-core execution. The video recommends trying 2.0 for new projects but waiting if code depends on row order or uses group-by dynamic while release-candidate issues are still being addressed.

Key points

Polars has had both an in-memory engine and a streaming engine; in 2.0, queries move to the streaming engine by default.
The streaming engine processes data in smaller pieces through a query pipeline, rather than requiring the whole dataset to be loaded at once.
Joins, group-bys, unpivots, and other operations no longer guarantee row order unless users explicitly request it.
Adding maintain_order = left to a join restores the original left-side order in the demonstrated example.
Polars 2.0 removes or renames APIs, including melt to unpivot, and errors often identify the replacement to use.
Adding signed and unsigned 64-bit integers now produces an int128 result instead of a float, according to the video.
The claimed 5× speedup is Polars’ expectation; the video notes that the announcement includes no benchmark table.
Streaming is currently chunked and pipelined, but does not yet allow datasets larger than RAM through true out-of-core execution.
The video advises waiting to upgrade if code relies on row order or uses group_by_dynamic while release-candidate issues remain.
The Rust crate is still version 0.55, so the video says the 2.0 release does not yet apply to Rust users.

Chapters

Start	Chapter	Summary
00:00	What changes in Polars 2.0	The video introduces the default engine change and explains why losing implicit row-order guarantees is connected to it.
01:02	Streaming and row order	It compares the in-memory and streaming engines, demonstrates changed join ordering, and shows how to request the original order.
03:06	API cleanup and stricter behavior	The video reviews removed and renamed APIs, changed type behavior, and errors that guide users toward replacements.
04:40	Performance claims and current limitations	It places Polars among other data tools and explains the 5× claim, the lack of benchmarks, and the absence of true out-of-core execution.
06:12	Concerns, release-candidate bugs, and upgrade advice	The video covers risks around row order and release-candidate bugs, then offers guidance on whether to upgrade.

Quotes

By default, every query you’ve already written moves onto a different engine. — 00:00

It’s just acting different. — 02:34

The 5x number is Polus’s own expectation. — 05:11

It does not yet mean your data set can magically be bigger than RAM. — 05:41

It’s exciting because Polers is using a major version bump to really just clean up the foundation underneath everything that we’re already using. — 08:16

Concepts and entities

Polars, pandas, streaming engine, in-memory engine, DuckDB, Dask, Spark, Python, Rust, Better Stack

Full transcript (original language)

00:00 The biggest competitor to Pandis is about to drop version 2.0. The Polar update is almost here, and the biggest change is probably not the one you’d expect. By default, every query you’ve already written moves onto a different engine. You don’t change a line. Polar also stops promising that your rows come back in the order you gave them. That sounds like regression, though it isn’t. It’s the price of making the first change possible. As Polers gains more traction with 2.0, know it’s giving pandas a run for its money. Let’s take a

00:30 look at the biggest update yet. All right. Now, from the start of time, pandas has always been the go-to for most data work. It’s simple. It’s used by the masses. Not only works great for data analytics, but it’s also key in machine learning as well. Then a while back came Polus. It’s always been faster than pandas. We’ll talk about that now. With this major 2.0 0 update. It may just be getting closer to more adoption. Since you got the quick backstory,

01:02 Polers has quietly had two engines for a while. The one most people have been using is the in-memory engine. You can think of that as like a warehouse. It loads your entire data set onto the floor. The warehouse then runs each step of the query across the whole floor. And that is extremely fast as long as everything fits in RAM. The streaming engine works differently. Instead of one giant floor, it’s more or less like an assembly line. Now, Polers chops the data into small pieces the maintainers call more cells. Those pieces are sized

01:34 to fit in your CPU cache. Then they’re pushed through the query plan. Now, different parts of the query can keep moving instead of waiting on the whole data set. That’s where the speed comes from, and eventually it’s also the path toward working with data larger than your memory. But there’s a catch. If you have eight workers on an assembly line, they don’t necessarily finish in the same order they started and neither does polers. So in 2.0, joins, group buys, unpivots, the docs literally say, etc.

02:04 No longer guarantee row order unless you explicitly ask for it. If you enjoy coding tools to speed up your workflow, be sure to subscribe. We have videos coming out all the time. Right, so here’s what this actually looks like. I’m going to UV pip install pre-polar. And remember that pre flag, we’re going to circle back to that in a second. Okay, I’ve got a lazy frame with the keys 2110. I left join it to another frame. Then collect. And look at the result. Same code as

02:34 polar one. Order is no longer guaranteed. Same code. It’s just acting different. Now I add maintain order equals left to the join. I can execute it again. I’ll run it here again. And the original order is back. So this isn’t polar randomly shuffling around our data. It’s polar saying if order matters, you tell it. You have to actually tell it. And there’s another nice detail here. Run the explain function on a query. It’s not hiding the behavior. You just have to know to look for it. That’s the big behavioral

03:06 change. But 2.0 is strange because this isn’t really a feature release. It’s actually just a cleanup. And there’s a decent amount being cleaned up here. Originally pandas polars. I have read CSV which is now just scan CSV collect underneath that lazy frame profile is gone. Melt now becomes unpivot. Join nils becomes nils equal. Casting an integer directly to categorical is removed. You can use cat 2 now to help

03:37 with this. Casting a string directly to a date has been removed. That’s string to date. Add a signed and unsigned 64-bit integer together and Polers now gives you int 128 instead of turning it into a float and silently losing more precision. And there’s even more coming. You can add concat that now refuses mismatched heights instead of trying to guess what you meant. This is actually a smart choice on Polar’s part. Every removed thing throws an attribute

04:08 removed air that tells you what replaced it. call melt and the error literally tells you to use unpivot with index and on all these errors more or less become the guide on how to actually use it. Run the code, break something, fix exactly that break based on the error and just keep going. And that gets to the reason Polers has become a daily driver for so many in the first place. Pandis pushes a lot of problems to runtime. Polers tries to catch them early through all this. So you can call collect schema and you can

04:40 find out the types are wrong before polers reads a single row. It’s just anticipating what’s about to come and it’s becoming more and more efficient. So where does polus actually fit? Well, duck db is SQL first. Pol is an expression API built for Python. There’s also dask and spark. Those are distributed. Pol is focused on one machine. The open source library itself is open source. The polar cloud is not. And that key thing changes things once we get to performance. Now, this announcement says the new streaming

05:11 engine is easily five times faster. Now, I’ve played around with Polars over the years, and yeah, it it is faster than Pandas, but I’ve never gotten any crazy speeds like five times faster. Try it out on some larger data sets. You will definitely see a speed difference. It’ll vary each run and with each data set, but you’re going to feel the difference and actually see it in runtime. And this is probably the most important distinction in the whole release. The part that would let the engine spill to disk, true out of core execution, hasn’t landed yet. So today, streaming means

05:41 chunked and pipeline. It does not yet mean your data set can magically be bigger than RAM. The 5x number is Polus’s own expectation. There’s no benchmark table in the post anywhere. And honestly, most people seem to be fine with that. A lot of users seemed happy with a useful yet boring release. There’s no giant feature dump. You don’t really have to learn anything new. those errors as you’re coding, they’re going to tell you what to replace it with. Just breaking cleanup and sever being used the way it’s supposed to be used.

06:12 But two complaints keep coming up repeatedly. The first is row order. If maintain order defaults to false, you can create some nasty bugs in the code because nothing crashes. Your numbers can still be completed correct. They’re just attached to rows in a different order. That’s much harder to notice than an exception. The second complaint was the word streaming here. People argued that it’s confusing for an engine that still isn’t truly out of core. Then there are actual release candidate bugs. Since the RC landed, there’s been a p

06:44 high issue where group by dynamic throws a daytime out of range air on the streaming engine. call the limit method while limit isn’t early exiting after join and string to date time can return nil where it used to just raise errors which is why I told you to remember the install command pre pre 2.0 this is still a release candidate and right now it behaves like one which is fine right it’s not the full version yet one more thing the rust crate is still version

07:14 0.55 so if you’re using polers from rust there’s no polars 2.0 know for you yet, but I’m going to assume that most of us are probably just going to be using Python here. Now, should you upgrade? Well, for a new project, yeah, I’d do it right. We are adapting to new technologies, new updates. If your pipeline already uses sync paret or sync CSV, you’ve effectively been using streaming this whole time anyways. But there are a few use cases where I just wait, right? If your code depends on that row order, just don’t upgrade and hope. Go grap for group buys that don’t

07:46 have a sort after them. If you’re using group eye dynamic, I’d just wait for all this to settle down. And I mean, this is cool to see where this most likely is going. This probably will be 2.0, but it’s not the official release yet. You could get warmed up with it, but it’s not quite there. The thing I keep coming back to is how unusual this release feels. Polar 2.0 ships zero new features, and it still changes what your existing code does. The new engine is faster on my machine, just not five times faster. So, 2.0 0 isn’t exciting

08:16 because it adds a bunch of new stuff. It’s exciting because Polers is using a major version bump to really just clean up the foundation underneath everything that we’re already using. I’m Josh from Better Stack. If you enjoy coding tips and tricks like this, be sure to subscribe. We’ll see you in another