Run Live Energy Trading, Risk, and Analytics in Databricks

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Run Live Energy Trading, Risk, and Analytics in Databricks · Channel: Databricks · 09:13 · Watch on YouTube

Topics: Data, databases and embeddings, AI models

Summary

The video presents a Databricks approach to energy trading that combines live operational systems with analytics, addressing the risks and maintenance costs of separate hot and cold data stacks. Spark real-time mode processes streaming events, while Lakehouse provides managed Postgres for transactional workloads, with data governed through Unity Catalog. A demo shows a live trading desk, real-time risk alerts and forecasts, plus historical analytics and natural-language questions through Genie. The proposed architecture brings live trading, analytics, and machine learning together on one platform and a shared copy of the data.

Key points

Energy markets have become more volatile, and 15-minute settlements increase the frequency of records, orders, and risk recalculations.
Stale data can lead to breached risk limits, mis-hedges, and missed intraday moves.
Separate hot-path and cold-path systems can create conflicting versions of data, freshness gaps, fragmented governance, and costly maintenance and synchronization.
Spark real-time mode supports continuous streaming with sub-100 ms event processing; the speaker cites a customer example processing over 260,000 events per second at sub-100 ms P99.
Lakehouse is described as managed Postgres on Databricks for millisecond reads, transactions, and upserts.
The live desk demo displays prices, P&L, positions, and risk, with alerts as risk approaches desk limits.
Machine-learning models trained on the same data can provide real-time price forecasts and trade signals.
Embedded dashboards and Genie let users explore historical data and ask questions in natural language.
The architecture combines operational and analytical workloads on one copy of data, with governance, lineage, permissions, and audit through Unity Catalog.

Chapters

Start	Chapter	Summary
00:00	Energy markets and the two-stack challenge	The speaker describes market volatility and faster settlement intervals, then explains the data freshness, governance, and maintenance problems of separate hot- and cold-path systems.
02:11	Databricks real-time trading demo	Spark real-time mode and Lakehouse support the operational workload, demonstrated through live prices, positions, risk alerts, and machine-learning forecasts.
04:56	Historical analytics and Genie	The demo switches to historical dashboards and uses Genie to compare buy and sell volumes for top counterparties.
07:04	Architecture and key takeaways	The speaker walks through the market-feed architecture and summarizes how one platform and governed data copy support trading, analytics, and machine learning.

Quotes

the SLAs at the desk are measured in milliseconds. — 00:35

this is your fully managed Postgres on Databricks, which is native to the platform. — 02:43

as this approaches the limit, um this here will fire our risk alerts. — 03:49

you only have one version of the truth. — 08:42

Concepts and entities

Databricks, Spark real-time mode, Spark Structured Streaming, Lakebase, Unity Catalog, Genie, OLTP, OLAP, Victoria

Full transcript (original language)

00:00 [music] >> Hi everyone. I’m Victoria. I’m a solutions architect at Databricks, and today I want to talk to you about energy trading on Databricks. So, let’s have a look at the energy markets. [clears throat] Um, there’s been a bunch of changes over the last years. So, first off, volatility has highly increased. In 2025, there were over 500 hours of negative prices in both the Netherlands and Belgium. Additionally, in 2025, the EU day-ahead market moved from 1 hour to 15-minute

00:35 settlements, which means you now have four times records, orders, and risk recalculations. And then also, the SLAs at the desk are measured in milliseconds. Now, if you have stale data, that means you breach your risk limits, it leads to mis-hedges, and missed intraday moves. So, what companies have done is that they have built two independent stacks. One stack for your live trading desks. So, this is your hot path data, your operational data, or OLTP. And then you have your cold path for any

01:07 risk backtesting, machine learning, and regulatory reporting. So, your OLAP system. Now, um, maintaining two systems here comes with a bunch of challenges. So, first off is you have two versions of the truth. Your data between your hot path and cold path might not match. Um, you have a freshness gap for the risk control. This typically run in a batch job, maybe overnight. Uh, and it doesn’t reflect the current state of the data.

01:38 You have fragmented governance and lineage between the two systems because you need to set it up twice. And then you need to maintain two systems, which means you have double the maintenance effort, and you have expensive sync pipelines between the two. Finally, you have your machine learning, which lives somewhere in the middle between these two systems. It needs the data from the cold path, but it runs live on the hot path. And yeah, it becomes quite difficult to maintain these two systems and quite expensive. So, what we propose as solution is that

02:11 now you can do this actually directly on Databricks. So, historically you would have run your analytical or your OLAP system on Databricks. And with two recently released technologies, we can now also support your uh transactional databases, so your OLTP operational stack. Um what makes this possible are these two things. So, you have your Spark real-time mode and Lakehouse. So, Spark real-time mode allows you to continuously continuously stream data uh via Spark structured streaming

02:43 real-time mode. And this allows us to get sub-100 ms event processing. Recently, for example, we did this with a customer where we processed over 260,000 events per second at sub-100 ms P99, and this is scalable. Lakehouse then is your sync, and this is your fully managed Postgres on Databricks, which is native to the platform. Um so, this is your OLTP system where you really have your millisecond reads, transactions, and upserts. And all of this data is governed by Unity Catalog.

03:15 So, this is how we bring in the operational part into Databricks where you have your analytical part. Now, let me show you what this actually looks like in a Databricks app that we created. Now, just open this app here. And this is now your live desk. You can see your prices, your P&L, your positions, your risk, all updating several times a second as the market moves. You can scroll down here a bit. We have our instrument prices, so for example, our Brent crude, our German and French

03:49 power spots, our euro to US dollar conversion, EU carbon, TTF gas, what have you. We can go down here. We can see all of our positions updating live. And then down here we see our risk. Um so we have risk again in the same system. We can see this move live against the desk’s limits. And as this approaches the limit, um this here will fire our risk alerts. So we get live updates about any risk that we are facing and um we can act on it immediately.

04:23 Now additionally, you need machine learning to like uh generate your trade signals, anticipate risk, and so on, act proactively. And these models uh will be trained, of course, on the same data that runs here. And then you deploy them on Databricks and you can have them directly here. So when you click on this, for example, or this might be integrated directly into your um bull intro and workflow, um you can see here the price forecast happening as well in real time. And this can generate, for example, your trade signal.

04:56 Yes. So this is our trade desk. And now we also want to perform some analytics. Now, of course, you can go in Databricks, you can analyze all of your data in detail, you can use SQL, you can use Python, you can use Spark. Not everyone might be able to use these and not everyone might know how to write these languages. So let’s go up here and go into our analytics and genie tab. And what we can see here is an embedded dashboard that shows the more um cold path data or historical data.

05:28 Um this can be updated as uh frequently as you like, of course, but here now we have more of a historic view that we might want to analyze and backtest and so on. So you can see here, for example, our total trades, we can see our volume traded. Um we might want to compare the a um traders, like how well they perform. And now I might want to perform some additional analysis on this. And I can go here directly and ask a question to Genie. Um so it will give me some suggested questions. I’m just going to

06:00 ask it, “Please for our top five counterparties by number of trades, compare our total buy volume versus sell volume with each.” Now what this does is Genie lives directly in your Databricks workspace, and it understands the data that you have because it understands how it flows through your system, into which dashboards, and so on. And you can define semantics on top. So Genie now goes ahead and analyzes that data, and then as it already has done now, it will give us a response. So

06:32 we can see here, okay, it’s super balanced, which makes sense because obviously the original data I’m simulating here, um and we have fairly even um distribution between the buy and sell volume. We could see the slight differences. So ANBV is the leader with 410,000 trades, and it’s nearly balanced with a slight sell bias, and we can see the exact numbers right here. Amazing. So we’ve seen how both the Trade Desk and any analyst would interact with this

07:04 data. Let’s go back and see what the architecture behind this is. So at the very beginning, you have your source, and your source is your market feed. So it could be your eyes, your Apex, or any Kafka stream, for example. And this feeds into Spark structured streaming real-time mode stream, which is basically just a very simple implementation of real-time mode. Um and this uh has a sink, uh the Lakehouse. So Lakehouse again, the managed Postgres, and this is then your operational database. Now Lakehouse is of course part of your

07:38 Lakehouse, so you have again one copy of the data in all in one place, and this is where you can then build on top your bronze, silver, gold layer and so on. Um to then go ahead to analytics. At the same time lake base is feeding the back end of the app. So if you want these real time visualizations, this is where they come from. Now on top of the lakehouse, you will build your analytics. So what we saw before in the app, we have the embedded dashboard. You have your AI BI genie and then you can build out your semantic

08:09 layer with metric fuse and ontology. And then on the same data, you will train your machine learning models and deploy them right in place for both the visualizations in your app and your trade signal generation. And then again this feeds back into the app where it all comes together. So what’s the takeaway here is with data bricks, you can combine your analytical and your operational systems. You can see risk as it moves live, your P&L, your value at risk, your exposure, all of this. Nothing needs to be

08:42 like a nightly batch job. It can all happen here in one place. You have one copy of the data, so only one platform to maintain, less effort. You don’t have to have any expensive sync pipelines and you only have one version of the truth. And best of all, you only have to govern this once through Unity Catalog which will apply to everything, your app, your BI, your machine learning. And throughout all of this, your lineage, permission and audit are maintained end to end in one system that

09:13 you have to maintain. Thank you very much. >> [music]