Configure Azure Private Link for Databricks

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Configure Azure Private Link for Databricks · Channel: Databricks · 21:22 · Watch on YouTube

Topics: Cybersecurity and software security, Data, databases and embeddings, AI infrastructure and hardware, Privacy and data protection

Summary

The video explains how Azure Private Link can keep traffic between users, Azure Databricks workspaces, compute, and Azure resources off public endpoints. It focuses on classic (back-end) Private Link for classic compute-to-control-plane connections and inbound (front-end) Private Link for user access, then demonstrates creating endpoints, configuring DNS, and testing connectivity. The demo confirms that the workspace can be made inaccessible from a public network while remaining reachable from a transit virtual network. The video closes with guidance on regional browser authentication endpoints, DNS, connectivity to other resources, and serverless workspaces.

Key points

Azure Private Link provides private connectivity and reduces reliance on public endpoints, helping address security and compliance needs.
Classic Private Link secures connections from classic compute to the Databricks control plane using a Databricks UI API private endpoint.
Inbound Private Link secures user access and uses a Databricks UI API endpoint plus a browser authentication endpoint for single sign-on.
The demo uses a VNet-injected Premium workspace and a transit virtual network; the required setup includes Azure permissions for VNets, private endpoints, and DNS records.
Private DNS integration creates records that allow workspace and browser authentication hostnames to resolve to private endpoint IP addresses.
The demo validates classic Private Link by resolving the workspace URL to a private IP from a cluster.
After public network access is disabled, the workspace is reachable from the jump box in the transit network but not from the public internet.
Browser authentication private endpoints are regional, so one is required for each region where Azure Databricks workspaces are deployed.
Classic and inbound Private Link use the same control-plane subresource, and a hub-and-spoke setup can share an endpoint and centralize DNS.
Classic compute still needs connectivity to storage and a Databricks-managed event hub; the video recommends service endpoints for storage and event hub access.

Chapters

Start	Chapter	Summary
00:00	Private Link overview and architecture	The presenter introduces Azure Private Link, explains its security benefits, and outlines the supported inbound, outbound, and classic connection types.
02:03	Inbound and classic endpoint requirements	The video describes the endpoint subresources for classic and inbound access, including browser authentication, and explains how users reach endpoints through a transit network.
04:07	Prerequisites and architecture for the demo	The presenter covers permissions, workspace requirements, the transit network design, and the recommendation to host browser authentication in a dedicated workspace.
05:09	Create and verify classic Private Link	A Databricks UI API private endpoint is created in the workspace VNet, then a cluster lookup confirms that the workspace URL resolves to a private IP.
09:21	Set up inbound Private Link and browser authentication	The demo creates a front-end endpoint in the transit VNet, deploys a workspace for browser authentication, and creates the browser authentication endpoint.
14:34	Check DNS and test private access	The presenter reviews the private DNS records and compares public and private name resolution and workspace access from a local machine and the jump box.
16:39	Disable public access and validate the result	After terminating compute clusters, public access is disabled and tests show that the workspace is accessible from the private network but not from the public machine.
18:44	Operational considerations and wrap-up	The video discusses regional endpoints, hub-and-spoke DNS, custom DNS, connectivity to storage and event hub, exfiltration protection, and serverless workspace applicability.

Quotes

With private link, you basically remove the dependency on these public endpoints and enable end-to-end private networking — 01:01

its purpose is to secure connections from classic compute to the control plane. — 02:03

Usually, this is achieved by having them created in some sort of transit virtual network — 03:05

the browser authentication private endpoint is regional. — 18:44

For serverless workspaces, classic private link is not applicable. — 20:49

Concepts and entities

Azure Private Link, Azure Databricks, Databricks UI API, browser authentication, private endpoint, private DNS zone, VNet injection, Databricks CLI, Filipe

Full transcript (original language)

00:00 Hey everyone, my name is Filipe. I am a sales solutions engineer here at Databricks and today I want to walk you through how to set up your Azure Databricks workspaces with private link support. First of all, why do we configure private links for Azure Databricks workspaces? Azure private link creates a private secure connection between users, Azure Databricks workloads, and your Azure services. Ensuring your network traffic is fully private and removing the dependency on

00:30 public endpoints. Let’s take a look at this high-level architecture diagram. As you can see, we have the Azure Databricks control plane and serverless compute. We also have the classic compute plane hosted in the virtual network that you manage, as well as your own resources such as storage accounts, databases, or even on-premises resources. And we also have users and applications that must access the Azure Databricks workspaces, Unity Catalog, compute resources, and so on. By default, all this communication

01:01 between resources happens through public endpoints. With private link, you basically remove the dependency on these public endpoints and enable end-to-end private networking, ensuring that your traffic remains isolated from the public internet. This helps organizations address security and compliance requirements while minimizing the risk of data exfiltration and reducing the attack surface for sensitive workloads. There are three types of private links supported by Azure Databricks. First, we have inbound or front-end

01:33 private link, which allows you to secure the connection from users to workspaces. We also have outbound or serverless private link, which secures the connections between serverless compute and your Azure resources. And we have classic or back-end private link, um which enables a secure connection from classic compute resources to the Azure Databricks control plane. In this video, we’re going to focus on inbound and classic private link, and we will discuss outbound private link in a separate video.

02:03 Starting with classic private link, its purpose is to secure connections from classic compute to the control plane. To set it up, you must configure a private endpoint in your account linked to your Databricks workspace with a sub resource type of Databricks UI API. After this is configured, your compute resources will connect to the control plane using this private endpoint IP address, and this traffic will be routed through Azure private link with no exposure to the public internet. Inbound private link, on the other hand,

02:34 secures connections from users to Databricks workspaces. For this setup, we actually have two different endpoints. One is a Databricks UI API endpoint, the same sub resource we configured for classic private link, and we must also create another private endpoint with a sub resource type of browser authentication to connect private link to Azure Intune ID for single sign-on and authentication requests. We also have the newer general access private endpoint, which is used for securing traffic to the account-level

03:05 resources such as the account console and Genie One. Now, these endpoints must be accessible to end users. Usually, this is achieved by having them created in some sort of transit virtual network, and then you configure access for your users to the cloud with either a VPN connection or ExpressRoute. After the connectivity is in place and the users can resolve DNS queries through these private endpoint IP addresses, they will be able to access the workspace privately. Before we get into the setup, I’d like to highlight the private endpoint sub

03:37 resource types that Azure Databricks provides. The main ones are the workspace endpoint with the Databricks UI API sub resource, which handles REST API calls for inbound and classic private link, and the web authentication endpoint with a sub resource type of browser authentication used for single sign-on requests. We also have the general access endpoint for securing traffic to account level resources and the service direct private endpoint for performance intensive services like zero bus and lake base.

04:07 These two endpoints will also be covered in separate videos. All right, we’ll now go through a quick demo of the private link setup and before we do so a few prerequisites. If you want to do the setup yourself, you must have Azure permissions to manage virtual networks, create private endpoints and manage DNS records. And we’ll start the demo from a VNet injected workspace, which must be on the premium plan. For the inbound private link, I will use a transit virtual network with a jump

04:38 box virtual machine, but in real production scenarios, you should have a solution that provides connectivity for on-premises resources such as a VPN gateway or express route. And finally, it is a best practice to host the browser authentication private endpoint in a dedicated workspace to avoid a single point of failure. So, this is our proposed architecture. My workspace is VNet injected and this workspace VNet you see on the right side where the Databricks UI API endpoints for classic or back end private link

05:09 will be created. On the left side, we have the transit virtual network where my jump box virtual machine is deployed and we will create the Databricks UI API endpoint for inbound or front end private link access along with the browser authentication workspace and the browser authentication private endpoint. So, I’m starting from my workspace resource group and first I’ll navigate to my VNet injected workspace. And here, I can see if I click see more,

05:40 I can see the what’s the virtual network where this workspace is VNet injected. Um now what I’m going to do is navigate to settings on the left and click networking. Uh, and then private endpoint connections. I will click to add a private endpoint. And then I will fill in some details over here. First thing I need is to provide a name. In this case, I’m just going to copy it from here. Um, and then I will click next and go to

06:11 the resource section. Um, for the target subresource, I will click on the drop-down and choose Databricks UI API. And I will click next for virtual network. Here, I will select my workspace virtual network. So, you can see I have my workspace VNet over here. And I’ll make sure that this is in the correct subnet. So, as you can see, I have two subnets that are uh used for VNet injection, and I configured an additional subnet for the private

06:42 endpoints. Uh, once this is selected, I’ll keep the other options as defaults and click next for DNS. I will choose to integrate with a private DNS zone. And I will create it in the same resource group where my workspace is hosted. As you can see, this is my workspace RG. And then I will click next. Uh, I’ll just skip the tag section over here, go to review and create. And then I will click here to create the private endpoint. Um,

07:13 All right, so this is initializing the deployment, and it should take a few minutes before this is completed. So, let’s wait for it, and then we’ll be right back. All right, so as you can see, my resource is now created. So, I was able to deploy this private endpoint, and I can even click go to resource and see the private endpoint created over here. Um, so next thing I’ll do is just test the deployment and make sure my workspace is really configured with back-end privately. So, I’m navigating back to my workspace resource group. I’ll click on my VNet

07:45 injected workspace and I will launch it. Here you can see my workspace is deployed. Um the first thing I will do is navigate to compute. And now I’ll try to create a compute resource. So, the first thing you need to do to ensure that uh private and the classic private endpoint is working nicely is to have a compute resource spin up. So, I’m just going to name this test cluster or something. Um I’ll untoggle a few things. Make sure this is not too large. And then I’ll hit

08:15 create. So, creating a compute resource takes around 5 to 6 minutes. So, I’ll just wait for it to be completed and then we’ll be right back. All right. So, my cluster is finally created and now I’m just going to go to a notebook that I created over here. And I’m going to run a simple NS lookup command. Okay? So, I’m just trying to validate the DNS resolution for the workspace URL. So, I can just copy it from the browser over here.

08:48 Um then I’m running a shell command. So, this is why I have this magic command for a sage over here. And then I’m going to run an NS lookup. Make sure that this is attached to my test cluster that I just created. All right. And then I’m going to run this command. All right. So, as you can see um the result of this uh NS lookup command is a private IP address. All right? Which confirms that my connection from the classic cluster to the Databricks control plane is being handled through

09:21 private link and going through a private IP address. In fact, if I want to double-check, this should be exactly the same IP address as the private endpoint that I just created um in the video as well. So, next we’ll go for inbound private link. So, the first thing is I’m going to navigate to this resource group where I have my transit virtual network, as you can see over here, as well as my jump box virtual machine. Okay? So, the first thing I want to do is configure the Databricks UI API

09:53 endpoint for front-end private access. Okay? So, first thing I’m going to do is navigate back to my VNet injected workspace. I’m going to go again to network private endpoint connections, and I will create a private endpoint. So, here I have the resource group. Um well, in this case, I can actually choose to deploy this endpoint in the transit resource group, so that’s what I’m going to do.

10:26 Okay? Now, I need to provide a name for this private endpoint. In this case, I’m just going to copy it here. Then, I’ll click next. I will choose the same sub resource. In this case, it’s Databricks UI API, then I’ll go to the virtual network section. Here, instead of selecting the workspace VNet, I’m going to select my transit virtual network. Um make sure this is connected to my private endpoint subnet that I configured in this transit virtual

10:57 network as well. And I will click next for the DNS section. I’ll keep the integrate with private DNS zone option enabled as well. And here, I’m just going to make sure that the resource group is in the is the transit resource group and not the workspace resource group that I configured previously. So, if this is all correct, I’ll just click next, then I’ll click review and create, and I’ll create this resource as well.

11:27 All right, so now my resource is just created. I can click here and go to the resource. You can see my uh inbound private endpoint is configured. So, that’s the first step. Next, uh we’re going to go ahead and create the resources that are required for browser authentication. So, here in my transit resource group, I will go ahead and create a new resource. And the first thing I need to do is create an Azure Databricks workspace to

11:58 host the web authentication private So, I’m just going to go here, click create Azure Databricks. Um I’ll deploy this in the transit resource group, and I’ll give this workspace a name. Let me just copy it here. There we go. This is my the name of my workspace. Uh I’ll make sure this is Make sure this is in the same region as your other workspaces or your other resources as well. Um for the workspace type, I will choose

12:29 serverless for now um because I don’t want to manage any sort of infrastructure resources for this workspace, so I’ll just keep it as a serverless workspace. Then, in the networking section, I will actually disable public network access and not configure access uh private access for this workspace intentionally. So, it’s not actually accessible for any users. Um once this is done, I can actually skip the remaining sections. And I’ll just go ahead and create this resource. So, my serverless workspace deployment

13:01 succeeded. I can now click go to resource. And now we’re going to configure the browser authentication private endpoint. So, within this workspace that I just configured, I will navigate to networking. Um then I’ll go to private endpoint connections and create a new private endpoint. Okay. First thing I need to do is provide a name. Um this is the name of my private endpoint. And I’ll click next

13:31 and go to resource. For the target sub resource, I can choose browser authentication and then click next for virtual network. And then I’ll make sure to select my transit virtual network. So, I’m just going to choose my transit VNet over here and I’ll choose my private endpoint subnet as well. Um now I can click next for DNS. And then I’ll make sure to keep this integrated with my private DNS zone. Um and for the resource group, make sure

14:03 that you select the same resource group where you deployed the front-end private link. So, in this case, I’m choosing the transit resource group, which is the same RG that I deployed the other endpoint for front-end access. Um once this is done, I can just click next, skip the tag section, and I’ll just go ahead and create this resource. All right. This is once again going to take a few minutes, so let’s wait for it and we’ll be right back. This deployment is completed. I can go

14:34 to the resource. Here’s my private endpoint. And I’ll just go back to the transit resource group. And what I want you to take a look at is the private DNS zone that was created. As you can see, if I go in here, navigate to DNS management and record sets, I can see the A records that were registered in here. So, I have one A record for the workspace. As you can see, this is the workspace URL. So, workspace.privatelink.azure-databricks.net. And then I have two records that were

15:05 automatically created for the browser authentication private endpoint. Now that this is created and this private DNS zone is linked to my transit virtual network where my jump box virtual machine is deployed, we can now start testing this. So, to test this, I’m actually going to do this approach over here where in the left side, I have my terminal, which is in my local machine over here.

15:36 On the right side, I have a PowerShell window from that jumpbox virtual machine that’s deployed in my transit virtual network. All right. The first thing I’m going to do is run an NS lookup command in both places. And what we’re going to see here is that my local machine resolves to a public IP address, as you can see, while my virtual machine resolves to a private IP address, which is the private IP address of the

16:08 front-end private endpoint that I configured. Okay? And now, if I try to connect to the workspace from both locations, and here I’m using a Databricks CLI command with a profile that I preconfigured, um I can just run this, and I’ll run this uh in my virtual machine as well. And as you can see, I can connect to the workspace both from the public internet as well as using my private endpoint. So, what I’m going to do next, now that I can see that this connection is

16:39 working for um for my virtual machine, is I can navigate to my workspace. So, in this case, I’m going to go back to my uh VNet injected workspace over here, and I’m going to disable public access. So, I can go to the networking section, and here instead of allow in the allow public network access option, I’ll just disable this, and I’ll choose no Azure Databricks rules for the NSG rules, and hit save. So, this update is going to take a few

17:11 minutes, um and once this is finished, we’ll be right back. Of course, this is going to fail because I had some classic clusters spin up. So, first I need to uh turn off those uh compute resources. Now that I’ve terminated all my uh compute clusters, I can actually disable these options and click save. Now, this is going to take a few minutes. Uh it should be completed, so we’re going to have to wait for this as well.

17:42 So, now that this is completed, we can do some additional testing to make sure that only uh private access can uh go through the workspace. So, first thing, if I try to launch this workspace, we’ll see that I cannot really access it, right? Yeah, as you can see, it just gives me this message, “I’m not authorized to access through the workspace.” And of course, because my local machine is coming from a public IP address. So, if I were to try to run that same command that I did to validate connectivity, it should no longer work

18:14 from my local laptop. As you can see, I’m having the same error message. Now, if I try to run the same command from my uh Windows virtual machine over here, yep, there you go. You can see how this works for my uh my jump box virtual machine, but it doesn’t work for my local uh for my local desktop. The reason is because once again, my virtual machine is connecting to the workspace through a

18:44 private IP address. So, this proves that I can only access this workspace if I’m connected from um this private network over here. Okay, before we wrap up, let’s take a look at some additional considerations. Regarding private endpoint arrangements, the browser authentication private endpoint is regional. You must configure one browser authentication private endpoint for each region where you deploy Azure Databricks workspaces. Also, both classic and inbound private

19:16 links use the same private endpoint sub resource type for control plane connectivity. In hub and spoke scenarios, you can use the same endpoint for classic and inbound private link. So, in the hub and spoke architecture, you can set up something like this where your workspace virtual network is peered to this transit or hub virtual network, which consolidates all private endpoints. With this uh with this option, you can also centralize DNS resolution by having a single private DNS zone and linking it

19:46 to both virtual networks. Regarding DNS configuration, for inbound private link and for virtual networks configured with custom DNS, you must ensure hostnames can resolve to the correct private endpoint IP addresses. Usually, this involves configuring your DNS servers to forward queries for all Databricks domain to Azure’s internal DNS. Make sure that you forward the domains you see here listed to ensure proper DNS resolution to your workspace and for web

20:17 application. Now, for connectivity to additional resources, classic compute clusters also require connectivity to storage accounts, mostly blob and ADLS, and an event hub managed by Databricks, which cannot be placed behind a private endpoint. For these resources, there are a few options for connectivity. The recommended approach is to use service endpoints for storage account and event hub access, and optionally, service endpoint policies for exfiltration protection.

20:49 For data exfiltration protection to public domains, you can use a firewall or virtual appliance and allow list the required domains in your firewall. And lastly, you can use a NAT gateway for outbound public access, but this is usually not a best practice since there’s no traffic inspection or as exfiltration protection. And finally, for serverless workspaces, classic private link is not applicable. But you can still configure inbound and outbound private link as required.

21:22 Hope you found this video helpful and thank you for your time. >> [music]