Unlock Graph Insights from BigQuery Without Moving Your Data

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Unlock Graph Insights from BigQuery Without Moving Your Data · Channel: Neo4j · 22:39 · Watch on YouTube

Topics: Data, databases and embeddings, AI infrastructure and hardware, AI models

Summary

Ben Lackey explains how Neo4j Aura’s virtual graph can connect to data in BigQuery, enabling graph exploration without moving the underlying data. He describes how graph knowledge layers can support AI by grounding responses and making them easier to explain. In a live demonstration, he selects a public FDA food-safety dataset, configures a virtual graph, and uses an AI-generated data model to explore its relationships. The presentation concludes with ways to learn more and deploy Neo4j through the Google Cloud Marketplace.

Key points

Neo4j and Google have integrations across products including Gemini, Google Distributed Cloud, and BigQuery.
Knowledge graphs can help ground AI results, improve accuracy, reduce token usage, and make responses more explainable.
Traditional ETL can require moving data from a warehouse and maintaining duplicate copies, creating organizational and data-sovereignty challenges.
Neo4j Aura virtual graph leaves data in BigQuery and provides a graph abstraction layer for querying it.
Virtual graphs can combine data from multiple sources, including BigQuery, Neo4j, Databricks, and Snowflake.
The demo uses a public FDA food-safety dataset from BigQuery as its example.
A BigQuery data source requires project and billing details, plus a service account with appropriate permissions.
Neo4j offers AI-assisted, manual, and schema-based options for creating a graph data model.
The resulting virtual graph can be explored with Neo4j tools such as the query workbench and Bloom.

Chapters

Start	Chapter	Summary
00:09	Introduction and Google partnership	Nathan introduces the session, and Ben outlines the topics: Neo4j and Google, AI and knowledge graphs, and a BigQuery demonstration.
02:16	AI, knowledge graphs, and BigQuery	Ben explains how knowledge graphs can ground AI applications and describes using BigQuery data to create a graph without moving it.
05:25	Virtual graphs and data architecture	The presentation covers the challenges of ETL and duplicate data, then introduces virtual graphs and composite access to multiple data sources.
08:06	Finding a BigQuery dataset	Ben opens Google Cloud Console, browses public BigQuery datasets, and selects an FDA food-safety dataset for the demonstration.
11:13	Deploying and configuring a virtual graph	He navigates through Google Cloud Marketplace to Neo4j Aura and shows how to configure a BigQuery data source and create a virtual graph.
15:51	Generating a graph data model	Ben demonstrates the available modeling options and uses AI to generate an editable graph model from the BigQuery data.
18:29	Exploring the graph and next steps	He explores connected FDA dataset records in Neo4j, then closes with resources for learning more and deploying through the Marketplace.

Quotes

leaving the data in the underlying data warehouse — 06:29

storing them in a graph database can help us look at the connections between the data — 10:09

I really like the generate with AI one. — 16:24

it’s possible to take that underlying data in BigQuery, virtualize it and now explore it in Neo Forj. — 21:06

Concepts and entities

Neo4j, BigQuery, Google Cloud, Neo4j Aura, Virtual Graph, Gemini, Google Cloud Marketplace, Bloom, Food and Drug Administration, Ben Lackey, Nathan Barney

Full transcript (original language)

00:09 [music] Hi everyone, welcome to this week’s edition of Neo4j connected intelligence. Once again, I’m Nathan Barney, partner marketing director here at Neo4j. In this series, we explore how graph technology integrates with the broader AI ecosystem to solve complex data challenges. Today we’re focusing on how to unlock graph insights from Big Query without moving your data. Joining me

00:41 today is Ben Lackey, senior director of partner architecture here at Neo4j. Hi Ben, thanks for joining us. >> Hey there. Thanks Nathan. Uh yeah, so I’ll talk through a few slides about how Neo Forj works with Google. I’m going to talk through how we work with AI and knowledge graphs and allow you to build an enterprise knowledge layer. And then I’m going to go through a demonstration of how BigQuery works in the Google Cloud Console, how you can use Neo Forj

01:13 through the Google Cloud Marketplace, and then finally when using Neo Forj through the marketplace, how you can deploy Aura and one of its components called virtual graph uh that connects up to BigQuery. So uh let’s get started, [clears throat] right? So first off uh Neo Forj and Google have been working together for a long time uh since we established a strategic partnership and then coming

01:44 out of that built all manner of deep integrations between different products and these include things like Gemini, Google distributed cloud and of course uh BigQuery which we’ll go into more now. running on Google. Uh Neo Forj has over 500 uh customers. These are a few examples of um publicly referencable logos which use Neo Forj on Google Cloud

02:16 though of course there are many many more. Thinking about how Neo Forj works with AI specifically uh we often observe a two-part data flow. So on the left side of this diagram you see a number of different data sources and these can be structured or unstructured data sources and we see our customers using tools like Gemini AI tools to process that data and write it

02:48 into a knowledge graph uh stored in the Neo Forj graph database. Once that knowledge graph is available, uh we see users building applications on top of Gemini and then Gemini making calls down into the knowledge graph uh to better ground its results, improving accuracy uh and reducing the number of tokens that various requests require. This approach also increases explanability of

03:20 responses as you can introspect the knowledge graph to understand uh what the AI did and why. Of course, in the context of using Neo forj with BigQuery specifically, our data source wouldn’t be unstructured, then it would be a very structured data source in a BigQuery table and uh then running Neo Forj on top of that table without having to move

03:51 the data. In this case, we use Gemini to extract a graph data structure from that underlying uh rows and column structure in the BigQuery database. So this is kind of the flow we observe today and that many of our customers are using. uh at the same time we see the industry going to a enterprise knowledge layer kind of architecture and while I

04:24 wouldn’t say we have any customer today who has deployed every little icon you see on this uh diagram together uh we do have customers who have deployed portions of this architecture uh and that’s what we’ll talk through today. So in our case today, we’re considering a use case where our data is residing in BigQuery and then we’re building a knowledge layer on top of that that other applications and agents

04:54 and tools could make use of. It’s really exciting to see how this is all developing and this picture is sort of how we see it today. I really wonder uh if we come back in six months if it will have continued on on this path or if once again AI will throw us some exciting curveball and we’ll be doing even more unexpected things. It’s a great time to be in this space.

05:25 Right. So we’ve talked about these agent infrastructures and sort of the value of building them and how you build them. Now if we specifically consider uh a use case where your data is in BigQuery and you want to build one of these enterprise knowledge graphs, we realize there are some common objections and barriers that come up and these really revolve around your data

05:55 existing in one system and then wanting to use a enterprise knowledge layer system like Neo forj. Now, traditionally that required ETL of the data, moving it from that data warehouse into Neo forj and then maintaining two copies of the data. Uh that has all sorts of barriers uh data sovereignty issues um organizational challenges and so on. So we have uh some

06:29 new technologies where we’re seeking to address this and this is where virtual graph comes in. So virtual graph is a component of Neo forj’s Aura SAS platform and what it does is avoid all those challenges we were just looking at by leaving the data in the underlying data warehouse in this case BigQuery and then allowing us to put a graph abstraction layer on top of

07:01 that and query that using all the normal tools and agents and so forth that Neo Forj provides. We’re also working on um some more advanced functionality let us say uh so this is composite query functionality where you can have multiple underlying stores for a single composite uh virtual graph in Neo forj and as an example you

07:33 might have data residing in bigquery you might have data residing in a neo native Neo forj database And then you could also have other virtualized data stores. Uh we support uh data bricks and snowflake as well. And the benefit of this is across all these different data sources you can virtualize a single knowledge graph that agents can interact with uh to reason about the underlying

08:06 systems. Right? So, at this point, uh, we’ve made it through our slides. Uh, I’m going to pop over to the Google Cloud Console and we’ll look at some of this live and then I’ll be back to talk about next steps. So, let me exit out of our slides. And over here, I already have a tab open to the Google Cloud Console. This is one of

08:36 those. There are many like it but this one is mine. Uh in this case you can see I have the project open to Neo forj business development and that’s sort of the day-to-day project that I build demos uh with Google with. Uh at the same time if I go over here you can see there’s a project for BigQuery public data and it’s kind of cool. Google maintains uh public data sets. Uh some of these are public domain, others are

09:08 ones that Google has acquired and then shares with us. Uh but these are a great starting point for just playing around uh both with BigQuery and uh then as a result with Neo Forj’s virtual graph. So I can select that project and now in here in the search bar I’m going to type BigQuery. It pops up product data warehouse analytics. Sure, it definitely is that.

09:39 And I’m going to click on that and wait for it to load. Also, dismiss this dialogue. And uh you can see here that there is a list of different data sets available in BigQuery. And there are a whole bunch of these. Uh, I’ve been poking around and sort of checking out ones that have interesting graph data structure because some data sets are represented well as

10:09 rows and columns and therefore are great in relational databases. Other data sets have lots and lots of connections between the different data points. And while they can be stored in a traditional database, storing them in a graph database can help us look at the connections between the data and uncover um new information that wouldn’t otherwise be apparent. And one of those I I sort of thought was interesting was a data set from the FDA,

10:41 the Food and Drug Administration in the US. And these are the folks who make sure that food and drugs are safe. uh and it’s a database of uh safety event violations. So this would be things like uh bagged lettuce is sometimes a problem and there’s contamination in that and then that gets reported and a case gets tracked in here. So I I thought that was kind of neat and uh this is a data set

11:13 that we’ll now look at running in Neo Forj’s virtual graph. So I am going to exit out of BigQuery and I’m going to search again for Neo Forj Aura and Neo Forj Aura is uh Neo Forj’s software as a service of our product including virtual graph which I’m talking about. Uh looks like there are a couple results for it and the second result is the

11:44 marketplace product. So, I’m going to click on that. And that directs us over here to the marketplace listing page for this. Uh, there’s a subscribe button. Uh, of course, I am in the BigQuery public data uh, project, which I, as a non-Gler, do not have right access to. So, I’m going to switch here back to the project I use for playing around with these things. And you can see

12:14 unsurprisingly I have already subscribed to this. Uh so in this case I’m going to hit manage on provider and this is going to punch me out. It’s the technical term of the Google cloud console and take me into the Neoraj Aura console and that goes through a few redirects. In this case, I guess I have recently enough authenticated that it carried um my Google credentials all the way

12:44 through and dropped me in the console without needing to reauthenticate. So, that was sort of nice. Uh in here in Neo Forj Aura, you can see I’m in an org called marketplace and then I’m in a project called Ben Google Cloud Marketplace GCMP. And here I have um instances for Aura. In this case, I already have two running. Uh I believe this is an evaluation we’re doing with Google uh

13:15 for a new integration. And this is one of my gen Gemini enterprise agents. But uh over here, if I click on virtual graph, uh you can see I actually already have a virtual graph running. And it says the data source here is unsurprisingly FDA food. Um, of course, if I didn’t have this running, the way to go through and create a virtual graph would be unsurprisingly to create a

13:45 virtual graph. Click that button. Uh, if I do that, uh, you see the different details. I can select the region I’m going to deploy in. I can select the size of the graph. And then I select which data source the virtual graph is going to use underline. And it’s possible to add new data sources. Simply click this button. And there are currently three different options supported for data sources. Snowflake,

14:16 data bricks, and of course what we’re excited about today, BigQuery. If I click on that, a configuration menu comes up. Uh in this I give the name of the data source. Uh I’m just going to show you what this would look like. Uh in our case it’ be FDA food. I give the project uh that it came from. So uh and it even has a helpful prompt here in the description. Uh BigQuery uh public data

14:47 is the name of that project. The way BigQuery works, it decouples um compute and underlying storage. Uh so I provide the billing project ID. In this case, this would be my project, the Neo Forj business dev. And then finally, I provide the name of uh the data source. And you also have to create a service account and grant it appropriate

15:18 permissions. That account is going to be in your billing project uh where it’s going to run all this stuff. Of course, in my case, I already made the data source. So, I’m going to exit back out of here. Uh, and we just select the existing data source. To create the virtual graph, I simply click next. And you can see I’ve already been through here and I’ve created some data models, but I’m going to show you how you would make a brand new one. Just

15:51 clicking uh create new data model. And uh then I hit next and it thinks for a moment and in a second it should pull up a number of different options for our graph data model. And we provide different modeling tools. uh there are tools to uh use AI to create a graph data model as well as uh tools to define

16:24 it manually or from a schema. So you can see here uh it has introspected the data source and it’s pulled in uh the different rows in that underlying uh BigQuery table and then we’re presented with these different options. Uh I really like the generate with AI one. Uh this is of course using AI under the hood to look at the BigQuery table and

16:54 then create a graph data model for us. So once again it it thinks for a moment uh introspecting all that data and building a data model for us. Here we are. It has generated a data model and we can see uh if you remember this data set is about food safety. So there’s a recall node in here. Uh there’s the name of a firm. Uh there’s

17:25 an enforcement event uh which I I believe is probably the FDA uh creating a case. And then um there’s an adverse event report and a product it’s related to. And one of the neat things about this data model is it’s an iterative approach. So we’ve created this, but we can go through and edit it if it doesn’t exactly meet our needs. Uh sort of interacting with the AI and uh providing

17:57 feedback on what it generated. Of course, I’m just going to leave this as is and hit create virtual graph. So, at this point, it uses that underlying data source we were just looking at uh that references BigQuery and it stands up a virtual graph instance on top of that. And this thing will behave just like any other Neo Forj instance. It takes a few minutes to deploy. So

18:29 rather than waiting for it, I I’ve done the Julia child approach and we already have one running we can look at. So let’s look at it. Uh as with other Neo Forjs, there are these very standard options to interact with the database. We have our query workbench and then we have Bloom, our business intelligence tool for graphs. I’m going to click on query and

19:01 it should connect here. Okay, it took a second to pick up the connection and uh we can see it’s thinking and loading the database. Looks like it had loaded 500,000 some nodes in with 300,000 relationships between them. And I can click on a node type and that will automatically generate a cipher query. In this case, I’ve searched for enforcement events and returned 25 of

19:31 them. I can zoom in and the names of these guys aren’t overly exciting. Appears to just be a number, but I can click on those and expand any neighbors they might have. Any connected notes. So it looks like in this case, Trafa Pharmaceuticals is part of enforcement event uh 99314. And because we expanded this node, we know there are no other enforce no other

20:04 firms related to that enforcement event. Uh at the same time, um we get this node of firm type. And that wasn’t the most exciting graph. I’m just curious if we expand some of the neighbors if we’ll find anything more exciting. Uh here we see a few more uh disconnected uh disperate subgraphs. So kind of fun to pop those guys open

20:36 and explore. Uh similarly we can run a query for an adverse event and we see that node type um and we can expand those guys and see what they are connected to. Oh and that hey we got a little super node there. Uh we have a whole bunch of things connected up to that guy. Um,

21:06 I could write more queries to explore this. I could also go back to Bloom and use that to explore this. But at this point, we’ve kind of seen that it’s possible to take that underlying data in BigQuery, uh, virtualize it and now explore it in Neo Forj. And we could of course do this with any of the BigQuery public data sets and you could of course do that with the data in your own BigQuery instances using the Google Cloud Marketplace to deploy Neo Forj

21:37 Aura and then setting up a virtual graph on top of that. So, at this point, I’m going to end our demo and we’re going to go back to the slides and then I’m going to hand over to Nathan to take it away. >> Great. Thanks, Ben. I think you’ve given everyone a pretty good sense for how to or how Neo4j virtual graph can help you get more value from the data uh you already have in Big Query. Um, so thank

22:09 you for that. Of course. So for our for our viewers uh to explore the documentation or see more use cases or get some more information, visit us at neo4j.com. Uh you can access a host of additional materials by accessing these QR code links as well. And you too can do what Ben just did. Just visit the Google Cloud Marketplace to make it happen. For those of you with specific project requirements or technical hurdles, feel free to reach out to the team. We’d love

22:39 to hear from you. You can email us at ecosystem@neil4j.com. We’d love to help. Thanks again for joining this week’s connected intelligence. And thanks again, Ben, for your presentation. We’re here every other week exploring how graph powers the next generation of AI. Thanks again and goodbye all. >> [music]