Road to NODES 2026 - Build Your First Knowledge Layer

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Road to NODES 2026 - Build Your First Knowledge Layer · Channel: Neo4j · 2:01:25 · Watch on YouTube

Topics: AI agents, Data, databases and embeddings, RAG, search and information retrieval

Summary

The workshop demonstrates how to build a foundational knowledge layer with Neo4j that connects structured warehouse data and unstructured documents for AI agents. It introduces three useful graph shapes: connections for a semantic layer over warehouse schemas, document outlines for navigation, and themes discovered with graph data science. Participants use Neo4j tools and Claude Code to build and query these structures, then combine them to answer repair and management questions with evidence. The presenters also discuss scaling, zero-copy tradeoffs, domain concepts, and updating document graphs.

Key points

A knowledge layer sits between agents or users and source data, connecting semantic knowledge, schemas, business rules, and other context.
The workshop uses simulated auto-repair data, with structured work orders and vehicle tables alongside PDF manuals, bulletins, and recall notices.
A Neo4j semantic layer represents warehouse schemas, tables, columns, and joins so agents can write more reliable text-to-SQL queries without copying warehouse data.
Document outlines model libraries, documents, sections, ordering, and links, enabling agents to navigate documents rather than relying only on vector search.
The document structure is loaded deterministically, providing a simple starting point before adding entity extraction or domain concept nodes.
Full-text search can be constrained to a section of the document hierarchy, while outline traversal lets agents follow document links.
The Leiden algorithm groups linked documents into themes, which can help surface recurring patterns and connect them to warehouse data.
The Neo4j CLI and its skills help agents write Cypher, use Graph Data Science, inspect schemas, and manage Neo4j resources.
Zero-copy semantic layers improve an agent’s understanding of data, while copying data into Neo4j can be preferable when graph query performance is essential.
Document graphs can be refreshed programmatically; version history can be modeled with revision IDs or managed in an external version-control system.

Chapters

Start	Chapter	Summary
00:00	Workshop access and credentials	The presenter introduces the Graph Academy course, shared API credentials, and responsible use of the temporary keys.
02:40	Use case and knowledge-layer foundations	The workshop frames an auto-repair scenario and explains why graphs can connect structured warehouse data with unstructured documents.
13:24	Setting up the development environment	Participants are guided through CodeSpaces, credentials, a Neo4j sandbox, and starting Claude Code.
26:29	Context shapes and the semantic layer	The presenter explains the idea of shaping context for an agent and distinguishes semantic layers from ontologies and virtual graphs.
35:00	Building connections to warehouse data	The workshop loads warehouse schema metadata into Neo4j and visualizes tables, columns, and join relationships.
41:36	Querying the warehouse through the semantic layer	An agent uses Neo4j schema context to construct and run a SQL query, followed by discussion of scale and zero-copy tradeoffs.
53:39	Document outlines and navigation	The presenter introduces a document graph that represents folders, PDFs, sections, ordering, and links between sections.
1:02:19	Using the Neo4j CLI and outline skills	The workshop uses Neo4j skills and a specification-driven workflow to generate Cypher for document outlines and explore linked sections.
1:09:43	Searching and navigating document content	The presenter builds full-text search over document titles and sections, demonstrates scoped searches, and checks document counts.
1:15:05	Domain concepts and graph visualization	The group discusses why the starter workflow omits domain-concept extraction and how to inspect and visualize the knowledge layer.
1:21:28	Finding themes with graph data science	The workshop aggregates document links and applies Leiden community detection to identify and describe recurring themes.
1:33:52	Connecting themes to repair activity	An agent uses document themes and warehouse queries to identify where repair issues are concentrated.
1:36:02	Adapting the workflow and using the Neo4j CLI	The presenter discusses applying the approach to other domains, then reviews CLI capabilities and agent skills.
1:43:03	Combining graph shapes to answer questions	The agent traverses document and warehouse data to answer repair, documentation-gap, and recurring-pattern questions.
1:55:28	Updating documents and closing remarks	The presenters discuss refreshing document graphs and handling revisions before closing the workshop with access and startup-program information.

Quotes

The problem is, how do you combine that data — 05:18

what shape of data do I need to give it? — 30:09

This is all going to be zero copy. — 35:00

we want to give our agent a table of contents where it can navigate. — 54:42

these two things live in different sides of our data. — 1:43:33

Concepts and entities

Neo4j, Graph Academy, Databricks, Claude Code, Neocarta, Neo4j CLI, Graph Data Science, Camada de conhecimento, Camada semântica, Modelo de grafo de propriedades, Busca zero-copy, Leiden, Cypher, Neo4j Aura

Full transcript (original language)

00:00 Here we go. We can build knowledge layers, but sharing our screen is still difficult for us sometimes. So, there’s there’s basically two links that are going to be important for you all, and hopefully they’ll get posted in the chat here really quick. And if they don’t, um Alex or Ryan, just help me out and uh put them in there. So, there’s the actual workshop course, which if you open it, it’s a Graph Academy workshop. You’ll go and sign in if you’re not already a part of Graph Academy, enroll in the

00:32 workshop, and there’s directions there. You can even do it without us talking. You can do it yourself. Um the one thing you will need as well is there’s a credentials file. Um and if you go to that link, there’s going to be an Anthropic key and a uh a Databricks key, which we’ll be using for our warehouse. Now, I want to just say we are giving an Anthropic key for everyone to use. I We’re all adults. We can all use keys responsibly. It I’ll be watching it. So, if if someone just just, you know, use your common

01:05 sense. Don’t take it and and go off and run with it and do other things outside of the workshop. We’ll turn it off after the workshop is over. If it turns out that someone is abusing the key, unfortunately, I’ll have to shut it off, and then, uh you know, it’ll be deactivated for everyone. Um if you have your own key that you want to use, if you have if you want to use Cloud Code with your own Anthropic key or just your own subscription, you’re perfectly fine to do that as well. It won’t make a difference for the workshop. It’s just we want to make sure everyone has access to Anthropic and Claude if you don’t

01:37 already because we’re going to be depending on Claude Code. Um so, I’m going to go through some slides very, very quickly before we start just to level set on the idea of what we’re doing here. So, um we talk about knowledge layers a lot uh and and Neo4j. And I really wanted to create something that uh just sort of from a concrete level, how do you actually build something like this? A simple version, not like the big complicated six to eight layer version that you might end up with in a big

02:10 enterprise, but like what’s your like 101 knowledge layer look like? Um so we got a great team here to put together a workshop for you. Uh it includes Ryan who’s on the call, uh who’s a senior partner architect, um who’s helped particularly with the connectors to Databricks and some of the back-end stuff. Um Ben Squire, who’s our developer, one of our developer advocates, who’s helped test the course and walk through it and sort of how do you make it uh sort of a better learning experience.

02:40 Um and then myself, and I’m an AI researcher here at Neo4j. Um so what we’re going to do is we’re going to take a business use case around Think of it like an auto repair chain. Um so we have these in in the United States where you there is these big groups that run like oil changes, and they’ll do like other diagnostic checks and stuff, and you can bring your car into them. And imagine that they have a ton of documents like bulletins about

03:11 cars and recalls and stuff, plus they might also have transactions that they keep inside of an operational store and then eventually put inside of a data warehouse. So like all past repairs, um you know, and like different transactions they’ve had with customers uh and so forth. And there’s multiple different types of personas that might be interested in different problems around that business. So for example, a technician might have a problem around finding a particular fix. You might have more senior

03:41 individuals inside of the organization wondering about like how do I um actually get AI implemented um so everyone can use it, and how can I ask more analytical questions for reporting? Like, you know, what are we missing? What are our blind spots? All this sort of stuff. Um and then you have AI engineers, which might be a lot of us on the call that actually has to have to build a co-pilot for all of this stuff, right? Um and in this case, the data that will be working with, this is all simulated, um but it comes in two parts,

04:13 a structured part and an unstructured part. So, as I was saying before, we have a lake house, um and inside of the structured part is going to be tables around your work orders. So, your vehicles, different codes for those vehicles, procedures, the work orders, and then different parts that are being used. And they have primary and foreign key relationships uh that are implied, I think, inside of uh Databricks and some of these other data warehouses uh like BigQuery, for example. Um and then you’re going to

04:46 have your documents, which in this case we’re just modeling with PDFs. These are going to be manuals, uh service bulletins, and recall notices. So, basically like safety or issues uh that a manufacturer has with a certain vehicle um that’s important for someone who’s going to be, you know, building or or fixing um fixing vehicles. Um and so, I we’ve made it to a point where retrieval from those data sources is not a problem, right? You can retrieve documents and warehouse data fairly easily. Uh there’s text to SQL

05:18 tools often by these vendors that have data warehouses like Snowflake and Databricks, and there’s vector search to retrieve from documents. So, like getting the data is not the problem. The The problem is, how do you combine that data um and how do you basically navigate through it in a reasonable way uh where the AI model has all the context it needs. And there’s certain very specific instances where this becomes a really big problem. For example, if you want to really navigate your documents like say, “Hey, I have a

05:51 work order and then that connects to a series of manuals and safety bulletins.” Just doing like vector search on these documents, you’re searching, but that search may have holes in it or it may like over index on just like one piece of documentation and give you a lot of duplicate results. So, you have this like precision recall problem where you’re not like recalling all the information you need and sometimes if you have a really low like K return, you suffer with your precision. Um so, you know, you you need to go a little bit

06:25 beyond just semantic search and maybe navigate the links inside your documents, the structure of it. Um and similarly, if you have to look at sort of ask questions about negatives, like what doesn’t exist in my data or where is something missing, that’s very hard with vector search. Um and of course, trying to find connections between different things. So, like how do I sort of connect up my unstructured data and find the connections to my structured text to SQL data and make sure that, you know, even within side of

06:57 my uh tables that I’m connecting between the tables efficiently according to sort of the business language or this is sort of the business semantics uh that people who actually work at the business use. Um and so, we’re going to look at three specific shapes uh to help us with this. Um so, those are going to be creating tables of contents or trees um out of our documents. Uh we’re going to be looking at connections, which is like how do you form paths, particularly between your structured uh lakehouse data or or data warehouse data, and

07:32 themes, which is going to be how do you extract communities and surface patterns that uh no one really named beforehand. So, like so, tell me about my like basically if I don’t know what I’m looking for, but I want to sort of uncover, you know, contradictions or errors where I’m missing or or common patterns inside of my data, reoccurring themes, how do I how do I find that? Um and I’m sure a lot of you if you’re here, you might already know a lot about graphs, but I’m just going to explain Neo4j and

08:06 how we think about graphs very very quickly, so everyone’s um sort of up to the same level because we’re going to be using graphs to express these shapes and get over these problems that I was talking about before. So, when we refer to a graph here at Neo4j, we’re often referring to a property graph data model. There’s other types of graph data models. At the end of the day, they all model effectively the same thing. Um this one is just very very simple and easy to understand. So, there’s three

08:37 elements inside of a property graph data model that are important to know about. There’s nodes, which represent people, places, and things, right? So, entities. There’s relationships, which are the verbs or associations between those people, places, and things. So, you can see here, you can have a person own a car, or you can have a person drive a car, or you can have relationships between people like a person is the brother or a sister of another person.

09:09 And then the other really key part is are the properties, which are could be strings, they could be dates, they could be vectors, for example for vector search, uh they could be lists of things, they could be dates, they could be uh numbers, all these different things that can go on both attributes of nodes and relationships. That helps with search and filtering and all sorts of stuff that we’d use. Um and the idea, right, is that it’s almost like pre-joined tables because instead of having primary and foreign keys, I can

09:40 go between nodes multi-hops very easily. Everything’s just uh already you know already pre-joined together so you can just zip through everything. Um and when we talk about the knowledge layer which is the which is our title for this um it might be helpful just to explain what this is, right? So like a knowledge layer you can think of it as existing between your agents or your end users and your source data. And this

10:11 knowledge layer where Neo4j fits inside of the stack can hold lots of useful information, graphs of various types. Knowledge graphs which tell you about your semantic knowledge, how things are related together uh ontologies that tell you about your semantics and your schemas and how they relate to the real world and define meaning in your data memory can go in here, business rules, procedural knowledge so like how do you actually do tasks? And all these things

10:44 help your AI models become more accurate. They give you a form of governance and explainability um and you can distill and connect a lot of your agent knowledge and human experience inside of this knowledge layer and make it portable so you can plug any agent into it and you can share this knowledge between agents. Now there’s lots of components to a knowledge layer and what we’re focusing on today is really more going to be this knowledge graph side and this ontology or semantic layer side with a little bit of business rules

11:16 mixed in. And in other talks we you know there’s lots of information that we provide on memory, how do you make cool these cool memory graphs or virtual graphs and context graphs and procedural knowledge and I cover those in other talks as do a lot of my colleagues and you can easily extend what we’re doing here, but here I want to give you sort of that basic foundation for a 101 knowledge layer, which is why I’m focusing mostly on these two and a half areas. Uh and this is what it’s going to look like uh when we jump into the workshop. So, basically what we’re going to build

11:50 today is you’re going to have your agent here uh and your lakehouse. In this case, we’re going to use Databricks. You’ll have your structured tables and documents I was going over. Um we are going to make a semantic layer graph with something called Neo4j. That’s going to be like an ontology in a way. Um and someone’s asking if we can have access to the slides. I can share them after. If you go to the workshop, um

12:20 the workshop has these images in there, too. So, it doesn’t have every image that I showed you, but it like this diagram is inside of the workshop. So, you’ll you’ll have access to these. Um but, yes, I can um I can look at just sharing them here uh after after the talk. Um the semantic layer, again, is going to be a reflection of what’s in the warehouse, and it will tell the agent how to write really good text to SQL. And then, our documents we’re actually going to parse and ingest into the graph, and I’m going to do this in a way

12:52 that’s item potent. Um so, I’ll get into that when when we get to that piece, but often times people struggle with entity resolution because or not entity resolution, but entity extraction. Um some of those workflows can be a little bit tricky if you approach them for the first time. I’m going to do something here that focuses more on the structure of documents. So, it’s very very simple deterministic code that you can then enrich later with entity extraction if you want. So, again, with that sort of 101, here’s just the starting knowledge layer type of theme. Um so, with that in mind, we’re going to

13:24 jump into the course. Um and let me switch my tab over. So, uh can someone tell me, Martin or Alex, uh did we post in the chat the links to these two things here? Let me check. >> Yes. >> Okay. So, so so everyone has this. Okay. >> Yeah, these are posted and also pinned to the top of the um of the workshop chat. So, if you don’t see them right

13:55 now, then please scroll look at the pinned messages. There is the workshop content and there is also the the credentials content that you shared in this Google Docs file with the API keys. >> Okay. Awesome. Perfect. So, here’s here’s the start of the workshop. Now, I already did most of this overview. Um so, you know, you you’ve got that already. The next thing I want you to do is go to the your environment piece. Um and you’re going to open so, I’ll

14:27 walk through these steps. Uh you’re going to open in GitHub CodeSpaces an environment to do this if you want to follow along. Now, if you want, you can run this locally, too. Like the all the code for this is open source, so you can go to the repo that has all the code, and you can start running this locally. You can hook this up to Orin, whatever you want to do. Um if not, you can go ahead and, you know, create a new one in in CodeSpace. I have already got a CodeSpace here that I’m working on, so I’m not going to

14:59 press any buttons here, but that’s what you would do. Um once you’re in here, uh basically what you need to do is it’s going to take a while to spin up, so this will this will take like 3 to 5 minutes to spin up. Um there’s an environments file, and you’re going to need to fill these things in, and that’s where the credentials file that we have from before is going to come in. So, basically what you’re going to do if I scroll down here is there’s an Anthropic AI key and there’s these data warehouse keys.

15:32 Here’s the workshop credentials. It feels very naughty of me to be showing workshop credentials on screen sharing them. But, we’re going to tear all this down afterwards. So, and you know, people please use this responsibly. So, I am and then basically I’m going to I’m going to go into my environment and I’m going to paste all that in. And then, oh, actually I pasted over the wrong thing. So, there’s the Anthropic key and then the warehouse stuff

16:02 you’re going to want to put down by where we have all the warehouse stuff down here. So, I’m going to put that in down here. And then basically you have that in and then you’re going to need to start a sandbox for Neo4j. Now and it will give you directions here on how to do that. You have to go to open sandbox. And then, once you’re here, I already have a graph data science sandbox selected cuz I was, you know, running

16:36 this beforehand in preparation. But, for you, you would go to a new project and then I don’t have it for me because I already have one, but it should be under your own data. Not the ordinary blank sandbox, but blank sandbox with graph data science and it should show you in a picture here what that looks like when you’re setting up. So, it’s blank sandbox graph data science. There is a way to hook this up with Aura, too. So, while you guys are going through and doing this, I’ll say that

17:08 I at some point we’re going to update this course to use Aura. The only thing blocking that is a technicality around how to do some of the GDS stuff when you make projections cuz it uses a different mechanism there for I don’t want to get into it for this for this workshop. It’s a it’s a it’s a technical detail. Um and we actually say Ryan had put in down here kind of what you would need to fix in the workshop to make it work with Aura, which is our cloud offering. Um but for today we’ll use sandbox.

17:43 After today, I’d actually recommend looking at if you go back to that credentials and you look at these directions, I’d recommend uh going in here and and following that and using Aura instead of a GDS sandbox. Um the experience is just going to be uh likely a lot better. Okay. Um so anyway, once once you get this created, you’ll open up your sandbox. For you, it should be blank. Um basically uh

18:16 if I if I open my sandbox, um it it will come up with credentials and uh you basically just have to go through and plug in your connection details. Um the username for this is just going to be Neo4j and then your password. And then when I connect to mine, it’ll show me that I have um sure we’ll save. It’ll show me that I have stuff inside of my graph cuz I was running before. For you, it’ll be nothing cuz it’s just

18:47 a fresh uh blank sandbox. Um once once that is up, you basically take your credentials. Um you’ll when you go through and do this, you’ll download credentials. Um it’ll give you a file to download and you take those and you put them into your environments file. So for me, um they came in I have mine here, so I just go copy. And then I would go to my

19:23 my code space, and I just stick them in right here inside of my environments file. Uh Once all of that’s done, and many of you might still be waiting for all this to spin up. Uh you’re going to start Claude, and it will bring you through a series of steps. Uh so if I go here and and another thing too is this bash script, this was uh the bash terminal rather was initiated often,

19:57 it’s a race condition, before the environment uh file was created before you put your credentials in. So you can either source from your bash here. Uh there’s some directions for doing that here, but what I do that’s a little bit easier is I just create a new bash terminal. And then I can say Claude. And if you want it to be a little bit easier, I say enable auto mode. So it asks me less questions. Um and then you’ll go through and

20:29 select, you know, whatever you you want here. When it asks you for your API key, say yes. Um enter it to continue, use recommended use recommendation settings. Trust the folder. It’ll ask you about an MCP server, which we have here. This is going to be to talk to Data Bricks, and you are going to say yes. I just say use all here. Um and then you should get your uh Claude terminal. Now, if any of those

21:01 directions, if you selected like the wrong option uh inside of here, it will tell you how to fix that. Um, so if you need to restart, you can remove from settings. If you pick the wrong MCP thing, you can you can reset project choices here. And then when you’re done, you can you can ask this question. You should get zero for this question. Um, and for me, it uh, it won’t be zero because I uh,

21:35 What are we doing here? For me, it won’t be zero because I already started loading stuff um, inside of the inside from the workshop. Uh, the the other thing here, too, is we’ve tested this workshop mostly with Opus, but on older versions of Opus. I might be a little bit of a daredevil here. I will actually Well, I don’t need to open a new a new script for this. Well, but I’m I am going to try this with Sonnet. I I was testing with Haiku

22:09 earlier this morning, and it was working. Um, but I’m going to try it with Sonnet. And uh, if it doesn’t work, which I think it will, then I will go ahead and switch back to Opus. You know what? I’ll stop here and maybe leave some room for questions. We’re about 25 minutes in. Um, let’s see if we have any questions in the chat that we need to answer. Um, I actually think everything here is answered.

22:40 >> I think Yeah, Martin uh, had graciously answered most of the questions. I think the one that came up a couple of times was how long will the API keys be available for people if they want to watch you today and do the workshop at their own time later? >> Um, we can’t keep them around for that long. So, I know for the Anthropic key, I’ve got to shut that off pretty much right when the course ends. I’ll I’ll let it linger for maybe a little bit after that. Um hopefully you can get um you know a Cloud subscription. It

23:13 shouldn’t be too expensive for you to just run through the whole course. You’re just answering having Cloud answer a few different prompts. Um and it’s not doing any entity extraction or anything. Ryan, for the Databricks one, uh what do you think? Can we keep that alive for a day or do we want to shut that off immediately? >> We’re going to have to shut that off immediately as well. >> Yeah. Yeah. But uh there is if you if you look at the course, there’s there’s an option to use BigQuery as well as your backing data warehouse. Uh and that

23:44 one I leave up all the time cuz it’s it’s super cheap. So and it shouldn’t change the way the workshop functions at all. Uh so you can go ahead and and use the use the BigQuery option. I might need to add the BigQuery key back to the workshop credentials, but I am more than happy to do that. Uh in fact though, I think it’s in the uh it’s in the environment already. Um yeah, it is. So uh never mind. I don’t need to do that. It’s already it’s already automatically done for you.

24:18 >> Yeah, Nile, you you’ll get an email with this recording from today um and with the materials as well. So you can um you can continue this uh anytime and you can obviously decide how you want to run it. Watch Zach and Ryan do it and then do it afterwards with the caveats that obviously the keys [snorts] aren’t going to be disabled or you just continue doing it today and >> [gasps] >> the workshop in itself is obviously forever yours and you can do it at any time uh as you like it. >> Yeah, and then um and there’s another question here. You can definitely run

24:51 this locally. Uh it’s fine. We Yeah. We we provide we provide this so that it’s you know everyone has the same playing field and they can all do it but I I if it were me I’d be running this locally too. Super Okie Dokie. So thank you. Thanks and then how are we I don’t know if there’s a way to get feedback from chat. Is is everyone like uh I don’t know is there like a way to thumbs up if everyone is like if everyone is good on the environment set

25:24 up or we need more time? Yeah let us know in chat if you if you’re there yeah I see thumbs up flying in a couple of people lies just say more time please a little bit more time okay. Let’s let’s give it give it a bit more time. Yeah we’ll give it we’ll give it a couple more minutes there’s nothing wrong during the waiting period if you want to move through just press continue. And it will bring you to to the next

25:57 piece so I’ll give it a couple minutes for everyone to catch up but you’re welcome to go through and check things out and start running things yourself as well if you’re if you’re already there. Sounds good. Cool I’ll just let me see if there’s anything in chat that we can maybe talk about while people are are getting ready yeah previous workshops I shut that. Yeah no I think um people are either ready or getting getting ready.

26:29 Okay. Hi sorry there was there there was a question from Max which is any comments in relation to running eval’s over this knowledge graph. Using one model versus other options if you had any had any thoughts on that. So running eval’s comparing different models was that the question? >> Yeah, exactly. Yeah, and yeah. >> Yeah, so for this workshop, I don’t have

26:59 like any eval harness set up, um, but but you can easily do so, uh, and you know, it’s funny because when I first made this workshop, uh, earlier this year, I remember that I was biasing myself towards Opus and more advanced models, but as the models have gotten better, at least within inside of the Anthropic universe, I’ve been able It seems like I’m able to use smaller models there. Uh, so it would be it would be an interesting test because

27:30 you know, where when we provide a knowledge layer, in theory, what we’re doing is we’re almost like distilling human knowledge for a model, and we’re giving it a graph to follow rather than having to do a lot of reasoning and routing on its own. So, in theory, the bigger lift should be for the smaller models. And it but that doesn’t always happen if the use case is very complicated, and it sometimes like and again, I I think I’m seeing it more the case now that smaller models are

28:01 doing better. I’m mostly using Anthropic, so I’ll I’ll preface with that. Um, whereas in the past, it’s been complicated enough that you would you would kind of want a bigger model for some of the more like for some of the examples that required six or seven hops. Um, but I don’t I don’t think that’s the case anymore. Uh, so I would be interested in adding like a harness to this a benchmarking harness and actually really seeing like how well uh, the different models compare. And then obviously like comparing it to, you know, just using

28:32 BigQuery or Databricks by itself is is always very interesting. Um, and what I find is with I mean the whole reason we set this workshop up, right, is because it’s it’s actually very hard to do this sort of stuff without some sort of semantic layer on ontology relating between the unstructured and the structured data. >> No. No. >> All righty. >> Super. I think um we should we should probably go ahead. I think you know, if you had could maybe catch up with this this one sentence on what you mean with

29:05 shapes and dimensions. >> So, we talked about um context comes in shapes is you know, the so the verb which is used with this workshop. And what I mean by that is as AI engineers I and talk about this as if AI engineering is this discipline that’s existed for decades. You know, I think as we’re I think let me rephrase that. As as all of us are struggling to kind of keep up and learn what’s happening with AI, we get very focused on the

29:37 tools. And this happens with retrieval a lot. Like, oh, I need to do text to SQL or oh, I need to do rag or oh, I need to write a cipher query, a graph query to pull something from the data. Uh but often times if you take a step back and you think about more like, okay, as you know, not to personify an an LLM, but like from the perspective of the LLM, what does it need what context does it need to see to be able to appropriately answer a user’s question or perform some

30:09 sort of autonomous action. And when you phrase it that way, it’s more like what shape of data do I need to give it? Like, what does the view need to look like? If you if you think about like a materialized view, like like what does that need to look like? And then work backwards from there as far as what tools and queries you’re going to need to put together. So, instead of thinking about it as like I need to make a cipher query to access the data, think about it more like I need to give my AI model a table of contents that it can navigate. Now, how

30:43 do I build that? Is it a Cypher query? Is it a SQL query? Is it something else? Like, how how do I do that, right? So, that’s that’s the idea behind that. >> Perfect. That makes sense. Thank you very much. Let’s Let’s continue. >> All [snorts] righty, let’s keep going. So, we’re going to move on to our first shape, which is this connections shape. Uh and so, the idea behind this is how do we be explain to the model or to our agent how to navigate our warehouse data. And for this, we’re going to be focused

31:15 specifically on the structured data side. Uh and there’s some metrics that we talk about here in terms of like how well text to SQL works and all this sort of stuff. And the field’s constantly changing, models are evolving. Um you know, the different vendors put out stuff for for what is effectively text to SQL all the time. But, what we’re trying to get the AI model to understand is what does the shape of this data look like? Um what connects to what and why does it connect to what? And to do this, we’re going to build a semantic layer

31:48 or an ontology in a sense. And I have a little table here that kind of tries to explain so the different levels of thinking when it comes to ontology versus a semantic layer versus a virtual graph. Uh and it’s a lot of words, but basically, an ontology is about interpretation and sort of reasoning through the data. A semantic layer is more about, well, how do I actually query the query the data to just match the business terminology? So, it’s a little bit more like uh functional in a sense. And then we have these concepts of virtual graph, which we’re not going

32:20 over today. And that actually um you know, that that would put a graph physically on top like a virtual graph on top of a warehouse where you can actually talk to your warehouse with SQL. So, it brings it to the it brings Cypher to the query layer, basically. And then it pushes down into SQL. So it’s like you can query your your Databricks warehouse as if it’s a graph. We’re not going to go over that today. Um we’re going to go over a semantic layer today. And we’re going to use this tool called Neocarta. Um Neocarta is a labs project. Uh and

32:55 it’s something that we’re continuously thinking about how to integrate into the rest of our product offerings. A labs project is something that we work on often. These are going to be open source. And uh think of them as a reference architecture uh tools that you can use and that you can incorporate into your solutions, but they’re not really part of our official product offering um and supported by engineering. And basically what Neocarta does is it sets a you can connect it to your warehouse.

33:27 Uh in this case we’re using Databricks, but there’s BigQuery. I think it works with Postgres and you know a whole bunch of others. Um and you can form this uh well, this is not the semantic layer. Where’s the semantic layer graph? Or maybe this was it here. Um this is more I should showing you how how the querying works, but what I want to show you is something like this, right? So it’s uh your database and how it’s schema, all the tables and the references and

33:59 everything that uh basically like a mapping of your logical or physical data schema that lives inside of a graph. Um and these reference keys, which are hard to see, but I’ll show you after it loads, that tell you how things join together. And these can be inferred by a query log, for example. So a lot of times in data warehouses, you don’t have explicitly defined primary and foreign keys. They might come from comments on the tables or they might come from a query log, for example, uh or some other form of data.

34:29 And Neocarta helps you map that over. And then there there’s other things that this can do specifically around adding I wonder if it’s in here. Um Yeah, like adding a business glossary which we’re not going to do here cuz we’re we’re just doing like a one-on-one example. Um like business terminology and stuff that you can overlay on top of the schema. Uh so that that can be very very useful basically to get your AI model to understand like how all of this works.

35:00 And again here we’re not actually going to import any data from Databricks into Neo4j. This is all going to be zero copy. What we’re going to put in Neo4j is this mapping. And then your agent is going to use that to then write better text SQL once it understands the shape of the data, right? Um and we will continue on. So we’re going to call this Python script here.

35:33 So I’m going to just copy it. And I’m going to go back into my uh project here. I’ll open up a new terminal. And I will type in the command. And I’m being a little bit risky here cuz I already have the uh all the lakehouse data in my workshop. So I’m going to be writing it twice. That might screw me up later. Um but what this will do

36:06 is it it looks great. It was item potent. So so it only loaded. It didn’t it didn’t uh it didn’t duplicate the data in there. So that’s good to know. Um it’s going to write tables and references into there. And if you if you looked at the way that script works, let’s see where is it? It’s in the load. Um build connections. Basically what it’s going to do, the important part here is where it calls and get from Neo4j. Um and then where’s the actual Here’s where it it instantiates the the

36:41 Databricks schema connector. Um opens up a driver. Runs this. Um but I think um this is just a count. So, here is where we do it. So, in this library basically make that a little bit bigger. You’re you’re just saying, “Hey, ingest uh basically uh from Databricks and then the schema here um is uh basically reading in um the Databricks schema. So, these these are

37:14 the environment variables that we filled in from before. Uh so, that’s what that’s what ran. And then uh what you can do to see what it creates inside of Neo4j, I can go back to my workshop. And one thing here, too, by the way, it says six tables and five references inside of the catalog. I um screwed things up um for Ryan. And I almost blamed it on Ryan, actually, which was very unfair cuz I forgot that I screwed things up. I was I was playing around with AI search inside of

37:45 Databricks. And so, there’s two tables related to documents that I loaded in, which don’t belong in there. Um but they don’t affect the course. So, you’ll see eight tables and five references. But anytime you enter a question for the workshop, pretend that there’s only six in there. And we will have to go through and remove them at some point. Um This Here’s Here’s the query that’s sort of going to show you what this looks like. So, I’m going to I’m going to press copy here for this for this bit of Cypher. And I think a lot of you by now know uh

38:17 what Cypher is. Cypher, though, just so just for those that are unfamiliar, is our graph query language. It looks like SQL, but it does this like pattern matching. What I’m saying here is I’m saying, “Hey, get me a path where you’re going to start at a day at a node labeled Databricks, and then go out on uh basically any type of relationship for like as many hops as it takes, and get me all the paths.” I’m basically here saying, “Return everything that’s connected to this database node.”

38:53 I wouldn’t do this with a large database, by the way, because you can imagine if you have hundreds of thousands of nodes in here, it’ll it’ll try to pull all them back. Um I don’t have like a return P like limit two, right? In which case it’ll return back just a little tiny piece of it. Um I’m saying, “Return everything.” And it will bring me back the entire connected component. Uh and what we’re going to look at here is we have our database, Databricks. We have our schema, which is our AutoFix

39:23 schema, that’s the use case that we’re working with. And then you can see here we have our tables. So, here’s like our parts table, our work orders, our procedures, um our vehicles, all this cool stuff. And uh what these do is these tables then connect to our columns. Uh so, these are basically the names of our columns inside of our tables inside of uh inside of Databricks. Uh and then we can have sample values in

39:53 here as well. And then I have highlighted here these reference joins. Maybe I’ll make them a pretty color. What about blue? Make them big, okay? And these are our joins uh from primary to foreign key. And there’s multiple ways that this can be inferred uh by Neo4j. Obviously, if you have an explicitly named primary foreign key like in uh in a relational database, it can achieve that. Um it can

40:24 also do if there’s comments uh a metadata like different data warehouses have different ways of doing this. I think Snowflake’s is comments. I forget what it’s called inside of Databricks, but you were you would basically say like oh like this is a you know foreign key to you know so-and-so table. Um and then it can also do it via a query log. And so that’s really in practice, right? Cuz a lot of these data warehouses as I was saying before don’t have uh they they have like these implied uh join keys. Uh which can be a hassle if you have hundreds of tables. So this example is very very simple, um

40:58 but I think it’s nice for just illustrative purposes to kind of understand how this works. Um and keep going to the wrong thing. Um there’s different things that uh different operations. It can get full metadata schema. It can list schemas. It can context by the table type. Uh you have it. There’s an MCP server uh that Neo4j that’s how it’s consumed by the agent. There might also be a cloud, I think in there, but we’re you we’re using the MCP server for this. Uh and um the we give the agent the ability to run this uh this script here

41:36 this uh run SQL script. Uh so I can go ahead and just copy this and make sure my agent works. Um let me actually just start a new cloud here. And then I want to make sure the model is I’m using Sonnet. Cool. All right. And I will go ahead and ask which vehicles received a certain part, use the MCP to get the warehouse schema, and write the SQL and run it with with the thing here.

42:07 And it should go away and do that for me. And what it will do is uh basically use that MCP server to read the schema from Neo4j. And then it will use that to run the SQL. And while it’s working on that Oh, brought back.

42:40 So, it went ahead and answered our question. Um went on 521 vehicles. And then let me just say what MCP points did you use for this? And it should tell me what it used. Get context by column and get full metadata schema. So, it basically went to you know that uh our Neo4j Neo4j, it

43:15 got that and then I can say like what uh query did you run in SQL? And then once it got that it ran this other query in SQL. And you can see it looked like it ran two. Um and it it had a couple joins in there. So, anyway, uh that’s the that’s the basic pattern here. Obviously, this is very simple, right? Because what we’re doing here is you know, we we have our um

43:47 our schema is only like six tables. Uh so, it’s not that hard. Um you know, I I wanted something that’s visually easy to see. Uh but you can imagine people will use this our customers will use this when they have hundreds of tables. Um and it helps substantially. And then you can also imagine, which is another area where this is used, if you have data inside a Databricks, then you also have some data maybe inside a BigQuery or something else. It looks like was someone saying something was that you, Alex?

44:17 >> Yeah, I was No, because, you know, I would didn’t want to interrupt, but yeah. Um there there is a couple of repeat questions on where to find the Neo4j URI URI and and password from the sandbox, and I think it’s in the it’s in the Graph Academy course, but it’s also in the sandbox view. Maybe you could quickly show that one more time so people know where >> So yes, I will. So So um if you’re here and this thing is running Now, first of all, when you did this, it should have downloaded a credentials file, and it will be in that downloaded credentials file as well. That’s easier.

44:50 Uh but if not, you go to See how there’s this thing here? You can go to connection details, and then don’t copy my password, please, and screw up my database, but this is this is where it lives. Um and you can you can copy to clipboard and then put them in that way. >> Super. >> And this might be a good place to stop for questions. Let me just see here. Uh Yeah, there’s there’s some details in here if you wanted to see around um kind of how things are namespaced inside of Databricks uh in terms of the schema and

45:27 everything. Uh and then you can also um I think this this will just show me some info about uh Where’s my I go to Hm. Yeah, it’ll just it’ll just show me the dialect that it’s using Databricks SQL and Databricks and then the and then the table. Um of course, we have the option to use BigQuery here, which if you’re viewing this after, that’s probably what you’re

46:00 you’re to want to be doing. You’re going to be wanting to do the the big query version. Um which will use it a different dialect in that case. Um for here say six tables and five joints to continue in the course. Don’t say eight cuz there’s two that the docs chunks and the docs something there’s like two docs tables in there. They don’t belong in there. So it’s six and five. Um and that that will allow you to to move on to uh the next part. So this is a good place to stop Alex if there’s other

46:33 questions that we have. So let’s see. >> Yep. Yep. Yeah, we have a couple. Um one is uh a longer one, but I think it’s interesting because you know what it it it connects with what you said earlier. This is a small data set. This is small demo size, but Oscar asks, “What what if you have millions and millions of registries in in in an in an enterprise grade? We have to see millions of nodes. Which is the key number of nodes between being optimal for agent querying and not being optimal? Do we have to have layers like memory cache or something like that?” >> [snorts]

47:04 >> So right now with inside a Neo4j, the the nodes and the the relationships are really referring to columns and tables. Uh so, you know, the from the deployments that I’ve seen realistically in enterprises, you’re talking hundreds, maybe thousands of tables in really big cases. Is it possible to get to millions of nodes in that representation? I mean, maybe. Um the way Neo4j works is it it can do text searches to search in like

47:37 to search parts of the graph. So it doesn’t have to pull back the full metadata schema, which would obviously blow the context window, right? Or or it could. Um so there’s ways So, you know, Neo4j and the tools that we’re developing along those lines um will scale well for larger sizes. In terms of millions of nodes, that would be very large for a semantic layer, although it might be possible. Maybe I’m wrong about that. Uh for other forms of retrieval and memory, for example, you can obviously

48:09 get to millions of nodes. Like if you’re storing session history inside of a graph and you’re doing entity extraction from that, you can easily get up to, you know, millions to tens of millions to billions of nodes. Uh and in those cases, the way we do retrieval, uh you can do vector search inside of Neo4j to pick up uh particular nodes and then just use that Cypher traversal language that I was talking about before uh to go from there. Um you’ll see a little bit of this inside of our next thing about trees and navigation. Like you can start

48:41 on particular documents and then do like three hops out, right? And that scales perfectly for larger data. That’s That’s what we specialize in at Neo4j is like good graph reads as your data set get gets larger, right? Um so that’s all possible. And then obviously with things like caching, um yeah, like I I mean um you know, I I would I would imagine that some caching strategy for graph queries would be helpful. Uh there is like we within side of the Neo4j implementation, we we have our own

49:15 page cache that there’s if you do like the self-managed deployment in Aura, we we manage that ourselves, but in the self-managed deployment, you can like play around with the amount of memory on that and stuff. So hopefully that answers your question. Uh I don’t see it written down um here and I’m just missing it in the comments and the in the questions piece. Probably just don’t see it, but hopefully that helps. >> Hopefully, yeah. But I’m sure. Um cool. Uh Well, I see a bunch of other I think Martin answered most of these already, so I think we are

49:49 probably good to continue. >> Okay, I see a more. Maybe I’ll I’ll take some time uh to answer. >> The The rest of the course, by the way, Alex, it’s um I we can we can as we go through this, we can go faster and faster, but like getting the foundation I think is is really important. So, I’ll take a little bit more time here. >> Um are there disadvantages on zero copy approach versus copying the data into Neo4j? Uh and that’s a really good question. Um so, right here right we’re not Yeah, we’re not copying the data into Neo4j

50:22 and were you doing that uh semantic layer? Uh and there are disadvantages to that. Um one of the big disadvantages is going to be on your multi-hop uh accuracy and importantly your latency. So, we I see this sometimes with big like supply chain or routing use cases. Like let’s say you want to find the shortest path between two points. That’s an arbitrary length query. If you’re like, how do I get like a package between, you know, across a country, across multiple warehouses. Like it

50:55 could be two warehouses, it could be like five stops, it could be no stops. Like you don’t know. And those types of queries just perform better inside of a graph database. And so, it makes sense to move your data into the graph database. Uh I think the the biggest question there is like is the blocker for you that the query performance and that the the actual mechanisms of the query depend on a graph for performance? Is that the blocker? Then moving your data

51:28 into Neo4j would be smart. If on the other hand, the issue is really more well, like in theory I can do this in SQL and it’s fine. It is just that my agent doesn’t understand how to do it with the current tools that we have inside of my data warehouse or or my SQL or whatever I’m using. That’s a different type of question because for a lot of these like the examples that I was showing you here, right? It was you know, we were doing um like uh what? Like two hop joins or something. It wasn’t it wasn’t that

52:01 atrocious. Yeah, like two two joins. So, it’s doable in sequel, but it might be a situation where your agent just doesn’t understand it cuz like the business terminology and everything and like how things link together might be really complicated cuz there’s maybe too many or too much business terminology and that’s when the zero copy approach is really good. So, to answer the original question, the disadvantage is really going to be you don’t have the graph query performance. It’s really the short answer, right? You just have a better semantic understanding and potentially a

52:32 better way to structure your query together. All righty. Um Cool. >> There’s a question on from Constantine on Neo4j. Does it have HIL or does it generate a semantic model with one shot? >> Does it have HIL? >> know what Yeah, I don’t know what HIL means to be honest with you. >> Human in the loop. >> Okay, I just didn’t know if you were in the loop. [laughter] >> Yes, that could be it.

53:07 >> Um do you want to expand on that a little bit Ryan or >> Yeah, it’s it’s basically just a a one shot right now. So, Neo4j basically reads reads the metadata, builds a semantic layer, and you can do a manual view, but there’s not like an iterative process for that loading of the metadata right now. We are definitely expanding on that to add, you know, a you know, a human in the loop process. That’s kind of the future. We’re not right now. >> Right. Awesome. Okay, great.

53:39 >> Thank you. >> Um let’s go ahead and move on to our navigation piece here really quick. So, now we’re moving on to a different shape and we’re going to be talking about a table of contents and navigation. So, I call this a couple different things. Sometimes I call it outline, sometimes you’ll hear me call it trees, other times you’ll hear me call it a table of contents, and I apologize for that. The The words have been jumbled as I’ve moved across different projects.

54:09 It’s all the same thing, though. The idea here is that sometimes when in we’re going to be moving out of the structured data world and we’re going to start to talk about our the document side of the database. These are going to be the PDF documents. And our natural inclination right now is to do vector search and chunking on PDF documents. And that’s great and we can keep doing that. But, a lot of questions might involve more navigating those documents. And the idea here is that we want to give our

54:42 agent a table of contents where it can navigate. So, just like a human, if you open a book and you look at a table of contents, how do I jump from, you know, one section to another section and how do I know what contains what? We want to give our agent that exact same view so that instead of just doing like a control F, it actually goes through and understands the structure of the library that it’s looking at in terms of the library of PDFs and going to the right thing.

55:14 Uh, and this is This is the view or the shape that I want to give my my agent. Basically, here’s the technical library, there’s bulletins, and there’s going to be recalls and manuals and stuff also. And then you’re going to have documents. And those documents are going to contain different sections. And those might link to other sections. So, this this So, this basically means, “Hey, this section on this condition of this Falcon 2.0T cylinder misfire thing links to a

55:46 misfire diagnosis that’s inside of our manuals on the Falcon engine.” And they have these URIs here. They’re kind of long that help identify down to the document, but they’re hierarchical, right? So, the So, like like I have the technical library, my bulletins, and then a document would be like that whole URI technical library bulletin, and then PDF, and then a section you can see here would be like

56:16 these, you know, you have a hash symbol and then the the section. Um so, I can I have this big containment view. Uh and that’s helpful because it it allows my agent to do more of an exhaustive search where it understands everything. This is what the graph data model would look like together, right? So, you have your containment relationships and sort of these has relationships. So, you start out at the folder level, then you go down to the document, and then you we do have next section relationships as well. So, there’s a

56:48 sense of ordering, which is going to be important for a table of contents, right? Because when we have multiple entries, we don’t want them to be random. We want them to almost be like page. So, you would like if there was, you know, a diagnostics section after a condition or in this case a cause section after a condition, it’s it’s ordered uh in the way the document has it. Um and then the links to relationships, which are super important. Um and what we’re going to do with this uh is basically we’re going to load it directly into

57:20 Neo4j using this script. And the great thing is, let me run it while I talk to you about it. And actually, I’m not going to physically run it. Um I’m pretty sure it’s idempotent. So, if you run it and idempotent means if I run it twice, uh, I’m not going to duplicate it in my data. Um, I’m going to skip running it though, just in case, um, I’m wrong and I made an error making the script and it actually isn’t idempotent. But, this process has the capability to be completely idempotent. Meaning, once

57:51 you load it. So, go ahead and run run this yourself. Um, and, uh, basically when you run it, it’ll create all of those document nodes from from the PDF docs. We actually have a copy of them here, I think. Um, where would they be? They’d be under corpus. So, we loaded them into, uh, cloud storage, I believe. Um, if not, then they’re just pulling from here. But, it’s, you know, whether or not you load from cloud storage you load from locals pretty trivial. Uh, these are actually the, not the PDF,

58:25 these are not the PDF’d ones. Um, these are the markdown ones, but we also should have in here the Would it be under Would it be under docs? Give me 1 second here and if I can’t find it, I’ll just look at the markdown ones. Um, sources, PDFs. Here they are. This is what, uh, Oh, it’s going to show me it in a in a gobbled up view. That’s no fun. Um,

58:58 Do I have it pulled? I don’t. Um, well, they’re PDF documents. If you look at the markdowns, you know, the it’s just a PDF of the same markdown here. So, that’s that’s what they look like. And so, what that script does is it goes through and it says, “Okay, like, I have the different sections.” And it just has a deterministic parser that knows where the sections are. These links are, um, uh, you know, they’re not they’re not structured like normal URL links. Uh you

59:32 can you can easily make the code so you structure it like normal URL links, but they they refer to other documents here and it will deterministically parse those into other nodes. Uh and what that what that looks like uh once it’s done loading, I can go down here and I’m going to copy another piece of my Cypher and go back to my window here. And I’m going to say, “Hey, like go to the uh library uh root node, basically, which is at the library’s like the top of it’s a root node of this whole thing,

1:00:05 and go down uh four zero to four hops down, and I’m going to limit it to 50 cuz there’s actually a lot of nodes here and I don’t want to pull everything back.” Uh and then I can even select If I go down here and I go to hierarchical layout, it will lay it out hierarchically. So, you can see I have my folder. There’s actually three folders here, right? Because we have the um the bulletins, which it which it went down, but then there’s also manuals and we calls. And then you can see the documents and then all of the sections and the contents of that section underneath that. Um

1:00:40 you know, and then the the has relationships and highlight them in orange. So, those are the has relationships. And then there’s also these links to relationships that we have, which you can highlight here in pink, and those go to different documents. And then like I can, you know, go to these documents and expand out and create this cool like, you know, interconnected graph thing where I can start to understand how everything uh connects, uh which is all really nice. And so, one thing this is kind of related to to Some of you are probably familiar with that whole Copart C like LLM Wiki thing where you have like documents in

1:01:14 markdown and you have your your Wiki where you have the LLM like extract your research into entities and uh concept pages. This same process can be used to deterministically load that type of wiki into into a graph. Uh and I like this to start because we don’t do there’s no entity extraction here, which is great. Like I might at a certain point want to say, “Hey, like I actually want to extract parts and problems and concepts from these documents inside of my graph

1:01:46 further.” And instead of just having you know these like folder document sections, subsections, like you know, nodes, I’d have, you know, part node and and person node and all this stuff. And that’s cool. Um but the great thing about this is it’s deterministic. You can get started running it immediately. It’s it’s all copacetic. It works great. And then you’re you’re sort of off to the races with your with your knowledge graph, which is why I like this example. Um But anyway, what what we do once we have that, let me erase this page here. Um

1:02:19 is there’s um a spec that we have around the outline format, by the way. Um that just that just describes kind of how how everything works. But I’ll get into that a little bit here in the next section. Why why I mentioned that and why it’s important. Um one of the things that we do in this workshop or we’ll do for the next 40 minutes, however long I have you here, is we’re going to start to use uh the Neo4j CLI. Uh the Neo4j CLI is

1:02:50 this thing here. So, I know I’m I’m going to be sort of showing a lot to you, but anyone that’s done text to Cypher 2 years ago with Neo4j and was like frustrated by how shitty Oh, sorry. How bad it was, uh which is normal. Um things are getting a lot better now uh with better models, but also because we’re creating tooling that gives you the ability to uh have a a CLI where the agent can actually run Cypher commands

1:03:20 very easily, even spin up infrastructure for you. But really importantly, it can also install these Neo4j skills, which we keep updated. And uh these skills tell your agent how to do like work with all of our drivers, work with graph data science, work with um MCP, and write Cypher, the query language. Um which means that if you’re using this, your your agent can write really good Cypher. Uh so instead of and and the idea here

1:03:51 is like I want to build that shape that I was talking about earlier. So I want to build the um where is a picture of it? Well, I want to build that outline shape, that table of content shape. And how do I get my agent to do that without me knowing how to write what is actually not going to be trivial Cypher? So if we if we I’ll copy this cuz this is going to be our um our command that we’re going to give our agent to do to basically build that for us in real time.

1:04:25 Um and I have all this open. Close some of this off. So if we go into our skill, we have a whole skill here that’s going to help the agent basically do everything for us uh a little bit down the road in the workshop. But there’s our actual outline uh script, and you see here we have this wire where and this is going to happen to you a lot where you’re going to kind of know like

1:04:57 um like what you want to write, like you’re going to have basically like a spec for how to build the query, but you’re not going to know how to do like specific parts of the query. So what we have in here is a spec that’s like telling the agent like this is kind of how I think I want to put it together. And what we can then go and do is I can ask my agent that question.

1:05:29 Uh I’m saying, “Hey, use the Neo4j Cypher skill.” I have another spec in docs outline format and this built spec here. So, the spec in doc outlines format is here. This explains the format that I want it to produce. Yes. Maybe I didn’t start this with the auto thingy. Um so, this this this document explains how I want to put everything together for the agent. See? It’s nice and pretty. Um And so, this this pattern I just wanted

1:06:00 to have it in the workshop so you can see uh Oh, did it finish already? No, it should still be working. See. What just happened here? Okay. And in case this doesn’t work, too, so it it already it already filled in the um the query. So, if I go to outlines um

1:06:32 You see here it it already uh apparently put the query in. Um although I’m I just want to make sure of one thing here before I move on too far. Nope. Oh, don’t do this. Yeah, it did modify it. Yep. So, you can see here

1:07:05 it uh went ahead and it it put in basically the rest of the query that I needed. Um so anyway, this pattern’s useful because it’ll it’ll help you um in your adventures with writing things when you don’t know how to use Cypher. Uh there is inside of here, too, a solutions folder, which I told it not to look at. So if it’s getting it wrong, you can just copy from the solutions um if you if you get a little bit lost there if you don’t want to wait for your model to uh to get through it all. And then but then basically once once that’s done, get rid

1:07:37 of this good thing. I can um I’m going to open a new window. I can go ahead now and start running my my outline script. So I’m going to go ahead and copy that. Go back in here. And this is going to return the whole outline, so it’s going to take a while. It’s going to be huge. Um see, so like return the whole thing and not really very useful. Uh but what you can do with this is you

1:08:11 can say, “Hey, go down to depth one or depth two or whatever.” Uh and then when you do that, it’ll uh the agent can use that and then I can do like, “Hey, go down to deep.” And then it will instead of just doing the high-level ones, it’ll give you under each one like all the all the pages. And then the other cool thing that I can do is I can be like I can take one of these. So say that one. And then I can go, “Oh, do do it for

1:08:41 then I think I just put it at the end. Yeah. So and then it’ll give me just for just for that one document, what are the See, just for this Ridgeline whatever thing, uh what are the different sections and the things that it links to? And then I’m like, “Oh, it links to this. Let me go check out that page.” Right? So then I can be like, “Oh, I can go here.” And then I can check out that. So, you can see how this might be useful for an agent over just vector search cuz you can sort of like traverse through these documents in an intuitive way. And um

1:09:13 it’s a little bit hard to see, but if I make it smaller, you’ll be able to see. And then I can I can have the agent kind of loop through and get all these different connections. Uh so, that’s that’s really nifty in the sense that um the agent that’s actually consuming this doesn’t need to understand Cypher. It just needs to understand what a table of contents looks like, which, you know, a lot of agents already know how to do cuz it’s it’s a pretty easy human task. All right, let me let me see what’s next in this guy here. Um

1:09:43 there’s also a search query. So, I don’t have vector search in here, um but I do have full-text search. And so, the same pattern that we did before with uh what I was showing you for outline, we’re going to do with full-text search. And that’s going to allow it to search the titles and the contents of each PDF document and section. So, I can go back to the cloud session that I had. And I can say, hey, use the Neo4j Cypher skill.

1:10:16 And then I have a spec contract inside of search. In this case, I don’t have a whole like document around it like I did for outline cuz it’s a little bit simpler. I can have it go and do that. The um the spec documents See, it made its little change there. Um inside of skills, I can go and look at search. And then uh I did have this built from spec. And yes, you can make the change.

1:10:48 Uh and then once it makes that change, this will hopefully pop up with the change here really quick. Um Yep. All right, so it went and did that and then I don’t know why it’s not updating here, but if I were to open it again, maybe. Um But it basically what it it Why are you not Hold on here. Maybe I just don’t know how to use uh

1:11:22 It might just be because it’s a sibling reference. It did It did update in the search document. Same pattern. I gave it a spec. It went ahead and uh updated it. It did update it in the cipher here. I’m just an idiot and I don’t know how to read. Uh so, you can see it updated here uh where it it basically is using um a full-text search uh here and then it’s also doing this other cool thing which I’ll show you in a second. Uh basically you can do a full-text search

1:11:52 and then um I can I can ask for like top K documents. So, if I go back uh to my window here and someone yell at me in the comments or something if this is too hard to see. I promise I’ll try to read it. Uh so, I can go and it will give me a my top documents that are searched with that URI. These are actually sections. Um and you know, I can expand top K to

1:12:24 be bigger like 10 and then it will give me 10. Uh that time the time before it was four. And then the other thing that I can do here is I can say, “Well, do it do a search, but do the search under somewhere inside of this library.” So, the whole library is hierarchical cuz it’s all these containment relationships, right? Uh so, one thing that might be useful is I just want to restrict my search to being underneath uh recalls.

1:12:55 So, I say under recalls and then it will only give me documents that are under the the recall uh under the recall subsection of the library. And you can do that with any URI. Uh and again, I don’t use vector search mainly not because I don’t like vector search, but just because vector each vector is different and each embedding provider like they’re they’re not compatible. Um so, to make this easy to use, I just didn’t do vector search. You can easily put it in here and it would it would work fine. Just from an interoperability perspective, so I don’t

1:13:28 have to deal with, you know, some people want to use open source models, some people want to use Gemini, some people want to use OpenAI, some people want to use Anthropic and then you end up like needing to like make up, you know, an option for for each different key. So, this is easy cuz, you know, it’s just it’s just text search. Um And then for check your understanding, it’s the first or I guess it’s 183 and 985 documents. So, to do this question, which is it’s basically if you were doing this by yourself, it’s a learning question to help you move on to the next

1:14:01 step. Um you go back to your Let me move this over here so I stop getting confused with that. You go back to here and uh you say, “Hey, count the number of documents and sections I have.” Just to double-check, 183 985 um and that is the same number uh that we have here. So, just this quick sanity check. Uh and I think for this one um these next parts I think are fine to skip actually because

1:14:34 a lot of this is um just sort of walking through kind of how the cipher works internally. Um and then there’s some optional library practice. So, these two sections here, navigating the shape and optional library practice, um are fine to skip for now. Maybe what I’ll do here is I’ll stop, and I’ll see if we’ll take a little bit of a break, cuz I feel like I’ve been talking about a lot of different things. We’ll see if we have any other questions, um and kind of do a pulse check of where everyone’s at

1:15:05 in the room. >> Super. Sounds uh good. Let’s um Let’s hear in chat. Let us know if you’re struggling with something, if if you are uh up to date with where we are, or if if you need a little uh repetition of some some of the exercises. How long is the break? Uh well, a a a few minutes maybe. Uh We’ll try to get through a couple of um of questions. >> The questions, yeah.

1:15:37 >> Yeah, go go for it. Um and uh yeah, let us know uh if you are in uh in and not not not kept up with what uh what Zach went through. There is one question um that I see we could we could probably talk about uh that came in a little earlier. Um and that was do we only need to document structure nodes, uh and no domain concept nodes?

1:16:10 So. >> So, here, yeah, so I I was talking about this a little bit initially. There’s um To start, I’m not doing domain concept nodes. That was an explicit decision by me for two reasons. One of them is it’s easy to do what I’m doing right now. So, for people getting started with the knowledge layer, this step is is simple, and it gets you off and running very quickly.

1:16:43 The second piece is that for the type of example we’re looking at, an an AI agent to a great degree already has the ability to infer entities and concepts that exist inside of your documentation. And so if you have a way for the agent to navigate that documentation and search it effectively, then in theory it can do a fair amount of the work without explicit entities. Of course, where that starts to break down is, hey, I’m now you know, I now have a huge amount of

1:17:16 data and one of the interesting things in this graph is we actually have two disconnected graphs. So we have like a semantic layer graph and a document graph. If I was to go and actually look at what this looked like, what’s a good query for this? Let’s just match P equals I’m going to I’m going to bring back all the data. How about that? So, once I can get my Cypher right, and then return everything

1:17:46 and fix my spelling. This won’t This won’t crash. Return my whole graph and do it in a force layout. So, see See what this graph looks like right now? We have um and it it’s still limited the amount of documents that it returned. I think. Here’s my document graph. And then here is my on the bottom here, here is my semantic layer ontology

1:18:17 graph. There’s no relationships between the two. I’m letting my agent infer the relationships between them. Where extracting domain concepts would help a lot is if I wanted to link between these two graphs, which at a certain point is probably smart to do. As I get more and more data and I get like, you know, you think about it like if if I was getting really technical inside of like different you know, like car specs and engineering knowledge, I might not want to rely on my agent’s general understanding and

1:18:49 pre-training to do that. I’m going to I’m going to want relationships here. So, those domain concepts will be helpful as you get more data and as the domain knowledge gets more specialized to connect and resolve these together. At the scale that I’m working with, this I can I can get away with and I can have my agent kind of infer the connections between. >> Cool. >> So, yeah, long answer, but hopefully it’s helpful. Let’s um >> Should we Yeah. You you say.

1:19:19 >> Well, we can do one or two more questions and then I think we should move on. >> Okay. Uh I think most of these go a little bit down different paths, so let me see if there’s something that is more closer to what we um I have here. There’s one question on is there a way to document and view the data model or the knowledge layer? I think what you just showed probably. >> Um Yeah, I mean, so we have the view and browser here. Um you can call the schema.visualize

1:19:52 here. Um why is this telling me that I’m wrong? Visualization, that’s why. Spelled it wrong. And you can you can look at what your data model looks like. So, you can see again we have the Here is the um the Neo4j data model. Here is my document data model in the database. Um also with inside of the Neo4j CLI, I’ll just jump ahead and I’ll I’ll show it to you really quick. Um

1:20:24 you can do stuff like this where I can I can actually call for the schema. >> Mhm. >> So, go back to my thing here. Say, “Hey, can you bring me back the schema?” Um and it’ll tell me, “Here’s my constraints.” So, like unique IDs. I’m zooming out just so it it renders well inside of the inside of my terminal. But, you can write this out to JSON, too. Um and then I have my indexes.

1:20:56 I have my relationship paths. All this sort of stuff. Um and my nodes. So, there’s different ways to to pull these schemas and visualize them, if that’s the question. >> Okay. Cool. >> [gasps] >> Uh let’s save the others for later so that we at least have time to com- plete what we want to show in this workshop. >> All right. Awesome. So, uh I’m going to move on to the next shape now. And this is going to be the theme shape, which um might be one of my favorite shapes. Not that I have a favorite, but I kind of like it because

1:21:28 it uses graph data science. And I used to be I was the our first graph data science technical marketing person, so I just I like it when I get to when I get to touch graph data science. I’m going to pull back um another view of the graph here, which I’m going to say, “Get me all of the links to relationships.” This is when a document links to another document. And the interesting thing is when you look at this graph once it comes up cuz I’m pulling back a thousand things, it might take it a little while here. Um

1:21:58 you’ll start to see different clusters will form in the graph. So, like you’ve got like this thing over here, right? Which seems to be centered around this ABS stuff. Um and sensors, right? And then you’ll have like other stuff over here. Uh control module. What’s this guy? Um communication control module again on this node. So, you start to understand that as you as you aggregate more and

1:22:30 more data, the links between your documents are going to start hinting at deeper underlying patterns. I think that’s pretty simple. I think we understand that kind of from a graph perspective. So, what we what we want to do is find a way to understand what these clusters mean and like importantly for, you know, you can imagine in this this is more goes towards like imagine you’re like a VP of auto fix or something or you’re making a report for senior leadership and you’re like, what are like the big things that we’re documenting? How do those things relate

1:23:03 to what’s actually going on in the field and what’s being fixed on the line? And like, is there a place where we’re missing documentation? How do we sort of do or even if it’s like a customer segmentation question, you can imagine there’s all uses for this type of clustering. We we want to find a way to do that. And what we want to be able to do is find an easy way to give our um this is going to be hard to view because of how this window is is rendering this, but basically, I want to create these themes where I can name themes inside of

1:23:38 the graph. So, here what I’m doing is I’m I’m going to say, well, if I have all these documents, how can I group them into these groups? And the names instead of doing something and again, we can do this this thing where we like use an AI model to create this domain-specific name that it infers from every group. What we’re going to do here is instead just say, what are the top shared target documents and most linked documents inside of each of these

1:24:08 clusters that we identify. And that in a sense will kind of give us an inferred name cuz like we’ll get, you know, these like um communication stuff of in this one, we’ll get another one about brake stuff. And then we’ll get another one that’s like all about like pressure sensors. Like those are the documents being referred to. And so even without like having a very fancy process that like does this sentiment naming thing, um, just using the documents themselves, we can kind of infer a lot, um, by

1:24:43 creating this shape again for our agent to look at. And uh, we’re going to do this with, um, with a with a graph algorithm called Leiden. And we have a way inside of that graph algorithm to say like how tightly interlinked these different, uh, communities or themes are going to be. Um, and it’s a it’s con- it’s conduct- conductance. I always say conductance, but I think it’s, um, where is it? I’m I’ll pronounce it here somewhere. Um,

1:25:14 yeah, conductance. Um, which is basically, um, it’s it’s a measure of like how It’s like how many are how many links are in between the clusters versus inside a cluster or a theme. And we can use that to measure kind of, you know, how well the the algorithm did or how naturally the graph clusters, um, just to start with. So we have a themes format markdown document. So in the same agentic coding spirit, uh, inside of here, I’ll make this bigger again.

1:25:51 We have our themes format document. And where we describe the shape that we want, we describe the types of things that we want on our headers, the number of documents, the number of groups and themes, shared cross-references, um, how we want to define cohesion, uh, Like by tightly moderate loosely interlinked, top shared targets, most linked stuffs. Um

1:26:23 the algorithm itself will produce theme IDs for every node. That’s how the community detection detection algorithm works. And I’ll show you what the query actually looks like once our AI model effectively helps us write it. So there it is and then the scripts file uh in the same sense we have our sort of spec right? Build from spec. Um we’ll see commands in here like GDS graph project and some of some of this

1:26:54 stuff. Um The Leiden calls uh we make I think what are we doing from spec? Um I think the the the hint here is like how do we uh create the the graph projection basically? Um and we’re actually going to provide a little bit more guidance around the here’s how we call the Leiden algorithm. Uh and the conduct the conductance metric con- conductance gosh the conductance metric.

1:27:24 I said it right the first time. Uh and then basically what we’re doing here is we’re going to be doing this Leiden algorithm that will say hey how can I cluster the nodes based on these links? Uh and then we’re going to write that to the graph. That’s what a lot of this is and then some of these queries are is just logic on how do we pull that back out of the graph again to create the view? And I recall why we have the spec here now that I was just talking about it.

1:27:55 What we actually want to do here is cuz links are between sections. So when I had this graph up not this one when I had this up before uh if I look at the types of nodes here there’s sections. We actually want to aggregate this, I think, to the document level. In that way, it’s it just creates a cleaner graph. And then we want it to be a bipartite graph or an undirected graph. Or no, do we want it to be undirected?

1:28:29 Now I’m talking too much. Links to target collapse both ends to their owning document. So we collapse it up to the document level. So that we’re not cuz otherwise we would have documents with multiple sections that link to something and it it’s it becomes a lot cleaner if we basically aggregate the graph up to documents and look at just links between documents. And then we do Yes, undirected type. So we do do the we do do the projection undirected, which

1:29:00 just basically means that when two when one document links to another it’s it’s sort of like a bidirectional link. It it helps the lighting algorithm run more effectively the way that it’s supposed to instead of having a directed relationship. We’re basically saying if document A links to document B, then document B is like just as related to document A as document A is related to document B. It’s not like document A is related to document B and document B isn’t related to A.

1:29:31 So I will leave it there. I probably said too much about that cuz I don’t want to make this a course in like graph projections and undirected graphs versus directed graphs. It’s probably going a little bit too deep. But if there’s a question on it, we I can elaborate on it a little bit more. So if I continue on here, what I’m going to do is do the wire here. So

1:30:02 go back into my agent and I’m going to it to create this again. Now, if you notice here what I’m doing is I’m saying, “Hey, use the Cypher skill, but also use the graph data science or GDS skill.” That’s in charge of running that Leiden algorithm and doing those projections. And you see it just did the projection here. Um and it was able to do that document aggregation. Um it just matches sections to the documents. And then did this like URI

1:30:36 split thing. Yes. I’ll let it run here for a second, and I’ll go through and copy basically what it now will look like to run this themes query. Um so, it looks like it did that well. And if that works, then we can go to our bash script here or our bash terminal, rather.

1:31:09 And we can run themes. And then it will bring it back. Um and we’ll get our view here. So, now we see these are the basically themes, 13 themes throughout our documents. We have our so, this BCM and communication stuff. Brake stuff is T2. Theme three or T3, pressure sensor, pressure sensor stuff. Here’s the documents that are involved. And you just keep going down, ignition coils, um catalytic converter,

1:31:43 um oxygen sensors. Um so, if you know a little bit about cars, uh this stuff should be making sense why it’s grouped together. Um in inside of the same uh inside of the same groups. So, we get to understand the themes inside of our documents. And then what we can do with that um oh, we can also turn the So, I’m not going to run this I mean, I can. There’s a gamma parameter that controls granularity, and you could turn up the

1:32:16 granularity. It’s not going to make a huge difference inside of this uh data set. But, basically if I turn gamma up to two, I think one is the default. It’ll give me more communities if I do that. And that might be useful like, you know, sometimes, you know, you want to tune these algorithms. Yeah, it So, it gave me 14 themes instead of 13. So, it’s not a huge difference here. Um but there’s there’s ways to basically tune these algorithms.

1:32:48 Um this question uh I actually think I need to go back uh to our graphic academy people and fix this. Every document should have a unique theme ID, um I believe. If that’s not the case, then I might have to go back and fix it. But, the way that you answer this question to move on, you just say most documents carry a theme ID grouped into two or more themes. Um I think the rule is actually supposed to be that each theme has a unique each document has a unique theme ID.

1:33:21 Um but it may also not have a theme ID. So, there there’s um And oh, no, wait. Now I understand the question. Okay, I’m I’m getting my one to many and many to one uh things confused. Forget everything I’m saying. Theme IDs are going to be community IDs. So, there’s 13 community IDs. Each document will have a single theme ID. Um and or they might have a null theme ID if they weren’t able to be put into a group. That is the correct answer to this question. A is what you would say to

1:33:52 move on um to the next piece here. And uh what happens next? There’s some cool questions you can try from here. So, like, where issues concentrated, for example, um, I can go into my, uh, agent view here. And, I can ask it this question about like, “Hey, where are issues concentrated?” Sometimes with some of these smaller models, I have to give it a reminder to use the auto fix scale. Um, or else it’ll go in and it will like try to it’ll think you’re talking about

1:34:27 like GitHub issues or something. Um, and it it will know to use the themes here to to work that out. And it will look at like it’ll basically do this process of like, “Oh, let me, you know, go and look at the different themes and then, you know, get everything, you know, concentrated and then look at, you know, how they connect, everything else.” Um, inside of the warehouse. So, then it will fire up like the warehouse piece. Just look at the work orders, which it’s doing now. Uh, so that’s that’s kind of cool to

1:34:58 see. We’ll ask more questions along these lines as we get, uh, deeper into the course. Um, and so it it’s able to find like, uh, you know, the original question is where issues concentrated. Concentrated in ignition coil and misfire repairs. Um, that’s for the come backs and the volume doesn’t concentrate anywhere in particular cuz the 15 troubleshooting codes I pull each have about 300 workers work orders. Um, so it ran this by looking at the themes and then going back to the warehouse and doing a SQL

1:35:29 query to pull back like the different codes and stuff that fit into those themes. So, kind of cool, uh, the way it’s the way it’s able to connect those two pieces of data together. Um, and then from there, I’ll stop here really quickly cuz I think again, um, for these for these next ones, uh, it’s going to be going into Cypher a lot and I don’t want to really do that given the amount of time we have in this course, sort of optional practice. So, I’ll stop here very quickly. We’ll take a couple minutes for questions and then we’ll move on to our next two

1:36:02 lessons. The one lesson is going to be really short, like 5 minutes, and then we’ll spend the majority of our time on that last lesson. >> Sounds good. Thank you, Zach. Um there is one question uh that came up with a theme rather that came up a couple of times is um what would be the challenges for adopting the workflow that we we implemented here or shown here today in a different context such as, you know, telco or supply chain or in any other other use case? What would that be?

1:36:35 You know, how would that work if you have different >> Well, yeah, I’d say that everything that we’re doing here, if you actually look at our schema, it’s um it’s a it’s not a use case specific schema, right? Like look at how these nodes are labeled. We have documents and libraries and sections. And then we have uh the like schema stuff. So, this could if you just looked at this schema, this could be for telco or supply chain or fin serve or anything,

1:37:08 right? Um so, it this can be a starter for just about any use case. Now, that being said, you know, your data kind of needs to be in a certain spot for this to make sense right out of the box. Like you need documents that are somewhat well-structured for, you know, this hierarchy of documents and links to to be able to fire correctly. And likewise, you kind of need like a data warehouse or something or some sort of relational database for this to work. Um

1:37:40 you know, putting your data into that format may or may not be challenging depending on where you’re coming from. But the great thing about this is that most projects will start out with some form of structured and/or unstructured data. Uh and if you can start with something close to here and augment it like for Telco or supply chain and by me necessary to like have other elements that said like in the supply chain you might have to do some structured data import, same with like Telco like you might want a digital twin of your power

1:38:11 grid in there at some point. Uh but this is sort of like your think of this is almost like your your foundation starter type of ingredients that you can then build on and augment and maybe you take half of the schema and use it or 70% of it and then you you try to you know fit in some of your other pieces. So, hopefully that makes sense. >> Yeah. Cool. And maybe one more. >> yeah. >> Yeah. As another [clears throat] um is whether the semantic layer is meant for managing documents or for managing data

1:38:44 in a data mesh. For example, on top of data data product endpoints and then you know how how should the agents basically consume the data through through these API endpoints or through a direct graph uh connection? >> So, that’s a good question. So, if I understand it correctly, the the question is not like well, I don’t really want to access the data through SQL, I want to access it through some other microservice or like endpoints. Um >> Yeah. >> Because the data is exposed differently through an API. So, Ryan, I don’t know, do we have anything in Neo4j yet that

1:39:18 does like a mapping to a data mesh or API endpoints or like I know we have virtual graph for some like Microsoft products, but >> Yeah, I don’t I don’t know not that I know of. We we have an MCP server which agents can use and that’s the typical pattern I’ve seen but yeah, but not like a data mesh which would actually be a great addition. >> Yeah. Yeah, so I don’t think we have that yet, but it would be an interesting idea and if you, you know, yeah, I would love to see something like that and I and you know, it’s we think about these things

1:39:50 quite frequently. Um but that yeah, that would be a great addition. I don’t think we have it here yet though. >> Sounds good. All right. >> Thank you. >> Cool. All right, we only have 20 minutes left. So let’s see if I keep us honest about being able to get through everything. The next section I’m not going to spend a lot of time on because we already talked a little bit about it and it’s about our Neo4j client that we’re using. So this client I already went over the shell here. It’s very easy to install. Uh it won’t

1:40:25 just work with um with client either. You can use it for code X um and for a ton of different types of agent harnesses. Uh you start with the bash script and it will auto sense the the harness that you use. I think at some point here it’ll say uh all the different uh clients it works with. Maybe in quick start. Well, it’s in somewhere in this documentation it will say um all the different agent harnesses it works with. Um but it’s quite a diff

1:40:58 it’s quite a few. It’s pretty much all of them. Uh and that allows you to run Cypher directly inside of the shell. So for example, uh when I was back on my thing here, I can write my Cypher queries in the shell and get results back. So quick Cypher query and then it will give me results back. I

1:41:28 can format that to JSON or or however um as well. And then uh it will also allow me to get schema information, like we showed before. And uh what’s great about this is that it the agent can now access this CLI. And in addition to accessing this CLI and understanding how to use the skill, uh it can also do things like uh you know, spin up infrastructure for you. So, if you’re you can manage Aura, for example, from the CLI. You can add

1:42:00 databases, uh get information about them, spin them down and up. Then the other thing that’s really important was what I was talking about earlier, which is getting these skills. So, that spec-driven development that I was showing you before was through that Neo4j CLI and that skill bundle that it installed. If I go there again, right? It’s all of these skills. Uh so, if you look at um for example, the GDS skill, you get these, you know, skill MD files that tell the agent how to use it,

1:42:32 what to check for. It’ll have references to all the latest and greatest. So, instead of your agent using like 3-year-old Cypher from some, you know, I don’t know, from some, you know, Stack Overflow, you know, thread that it scraped from like 2021 or something, it’ll use all of our new Cypher um and and everything you need to to run really effectively. So, I highly recommend if you’re doing any form of agentic coding, uh to check out the Neo4j CLI because

1:43:03 we’re using it here and it helps you uh use Neo4j very, very quickly. That is all I want to say given the amount of time that we have on the Neo4j CLI. And so, the last thing that we’ll do is start putting this together into some more complicated questions. So, um if you sort of move through the course to this last lesson, uh if we some of these questions that we’re asking, and we saw it a little bit in

1:43:33 what I had put earlier is like if we ask about you know, like, you know, what fixes um a certain issue on a car, it’s going to look at documentations. And then it’s going to need to look at candidate parts. And these two things live in different sides of our data. One of them lives in our documents and then the other one lives on the warehouse in the structure data side. Um and then it needs to return that at, you know, the answer to that question with with evidence. Um and so what we’re attempting to do

1:44:05 here is basically keep this process honest. And we’ve developed a whole skill inside of the workshop here. Uh if I go to the skill to do that. So, you can see uh there’s certain things that I that I’m saying don’t go to cuz just for the sake of the course, like don’t look at our solutions, right, to answer questions. Um describes the graph,

1:44:35 describes the outline, the search tool, and the themes tool. Um you know, how to basically use uh read different aspects of the database. Um this sort of shapes first mentality that we’re using. Some examples or exemplars of things that it needs to uh that it needs to think about like at certain types of questions and how to answer them. Uh and then basically some common sense stuff like grounding your answers, um using evidence, all this all this

1:45:08 kind of stuff. And it has those outlined in the search and the themes uh from before. And so we can go ahead and if we go to continue, let’s continue. We can start asking these questions and having it traverse both sides of it. So, the first one is for a VIN and let me just plug it in and have it cuz it sometimes it takes it a minute here to to answer. So, while it’s spinning, we’ll talk about the question that’s being asked. So, here I say, “Hey, for this VIN number with a certain code, what was the

1:45:42 fix on similar vehicle?” So, to get this information, it’s referring to something in the database. And then, it’s going to have to look at fixes for similar vehicles. And then, you’re going to have to explain the steps and the shapes that you use and I’m telling it not to use the Neo4j client, mainly because um sometimes it will try to sidestep our procedures like our search and themes and outline procedure and just use the Neo4j client by itself. It already returned a response.

1:46:12 And what I like to do is just look at sort of the steps that it used. So, in this case, it searched for the issue inside of our documents. It used the warehouse shape to look up the vehicle stuff. It filtered to a specific code and then it did a second search under this bulletin to confirm the suppression and repair procedure um for there’s there’s one part that basically like superseded another part. They think it was like faulty or or retired it or something. It’s like the IC 2042B comes after IC 2042A.

1:46:49 Um Sonnet is going really fast today, which is nice. Normally, I’m used to this taking a little bit longer when I use uh Opus. So, anyway, it went through and it basically searched both sides of the graph with that, our knowledge layer. Um we can ask more complicated questions, too. Like, for example, here, this sort of a state that’s proving a negative. And in this case, what we’re asking is more of like a higher-level question that someone making a report, not necessarily someone working on the um

1:47:21 on the floor of, you know, like a a mechanic might ask. But this is are there mismatches between our documented procedures and the actual problems in the field, what documentation is missing, and what documentation isn’t being used. So that’s that’s an interesting question that if you’re like, you know, thinking about this from a management perspective, um do we have all the documentation we need? Are we running our, you know, business effectively basically?

1:47:51 And this one might take a little bit longer to run. Um but while it does that, what I’ll do is I’ll go over and cuz we only have 10 minutes left and I don’t want to like take forever with the with the workshop, is here’s here’s another type of question that might be interesting, um which is similar to what we were looking at before, which is what are the common patterns across our bulletins and recalls, and how many of our cars are affected by these patterns? And bulletins and recalls, right, are like

1:48:22 issues with cars, with vehicles. So this is a this is a question of saying like, well, what are the issues and how many of the repair orders that we get in are actually having these issues? And break it out into common themes, so it’s not like every single issue by itself. So what I’ll do here is maybe I’ll just go into this next fast sec path bash uh terminal here, and I’ll enable auto mode

1:48:53 with an extra dash there, and I will go ahead and ask this question. Um we’re we’re also on Let me just check that we’re on Sonic. And we’ll go ahead and get that question spinning. Some of these questions, because they’re more complicated, they’re just going to take longer. And that’s fine. I mean, I I we expected this. I expected this to begin with. So, if it takes a few minutes for these to return, um that’s totally fine.

1:49:25 Um but, what’s going on here is it’s doing more exhaustive searching, and it’s going between the two sides of our graph to be able to to be able to find these responses. Um I’ll take Well, that it might take a minute here to run. So, let me see if there’s something else we can talk about in the meantime. Um cuz we’re right about at the end of the at the end of the workshop. The right fix for this, it actually came back on our first one. Um

1:50:00 we were looking at it in the response. There’s this revised coil ignition coil that it’s supposed to use to to fix this issue. That’s the answer to our first question, by the way. So, it’s B here for this one. Um proving a negative, um I believe these are we’re waiting on these to come back here. Uh Actually, one of them came back. So, this was the question about are there mismatches between our documents and procedures and actual problems in the field and what documentation is missing? So, here what it did was

1:50:34 uh it went to the warehouse, um and it ran some SQL basically to enumerate all the codes. Um and then in here, it chose to do a search with search.py. Um and it has some limitations here, which are interesting. Um because I really wanted it to actually use the outline.py. If you look here, it couldn’t enumerate codes that appear in documents, but not in the warehouse. The search can’t list an outline.py show document titles, not codes inside. So, here’s an interesting example of

1:51:08 where like I really wanted it to use outline.py to connect documents together to what was in our work orders. And it didn’t here because the document codes weren’t part of the title of the documents. So, to those people asking like, well, when should I extract nodes from my uh documents? This would be a really great place to do it, right? Because then you would have the ability to navigate the connections between uh in this case your specific parts instead of just using search, which is really what I intended. Um but you can see here

1:51:42 how some of the formatting of the data is really important. And it’s still able to do this by looking at the data warehouse and then performing the search. Um outline would have been better cuz instead of doing an exhaustive search, it would have been able to have basically jump between all the documents and links. Um and it would have made this return much faster. Um but you can see basically the answer that it gave here. Um there are mismatches in both directions. Uh the clearest one is the two high-volume troubleshoot codes that

1:52:13 the library doesn’t mention at all, which is 335 and 340. And in fact, if we go back uh to our question here, it enumerated field codes agents of the document library, which pair of fault codes is failing in the field but has zero documentation anywhere in the library. And that would be this one here. Um and yeah, so that would that would that would complete the lesson if if we um but then but then for the last question, um cuz we have a few more minutes left. And then I’ll go to your questions after this.

1:52:44 Uh we have what are the common patterns across our bulletins and recalls and how many of our cars does this affect? Uh it finished returning that. And we sort of answered this question before, I think, kind of when we were on that theme section. Um but here basically uh we see it returned 13 system-level repair family. So, those are the themes directly. Um the bulletins and recalls repeat one pattern. A single name component fails. The documents show which part failed and

1:53:16 many say revise part replace it. The fleet is at least 821 of them or 48% have a repair using one of the four superseded parts below. Um and so it it lists those parts here. Uh so, some of this goes back to how we synthesize because I synthesized it to be a lot of this sort of superseding replacement part here. But you can see it’s like identifying like, “Hey, we have, you know, a ton of, you know, these ignition coil and timing belt issues.” Um and those show up sort

1:53:51 of under that um that one theme that contains them. Uh so, for comparison, the revised parts have few come backs. The other families follow the same shape with a lower comeback rate and vehicle counts. So, this is saying like the family, these are like the the families and then all the um parts in vehicle fixes underneath them. Uh so, the theme shape cover the whole library across all questions um about the different recalls and and

1:54:21 bulletins. So, basically what it’s doing here in summary is it’s identifying these themes and then the fixes that fall under these themes. I wish it visualized it a little bit better here cuz it’s just sort of saying the it’s not saying the theme these fall under. Um and it could be my my use of Sonnet instead of Opus. Uh but it can identify like the ignition coils, um you know, are are a big area for example, a big theme inside of this repair shop or or series of shops. So, anyway, that’s that’s the workshop.

1:54:56 Um there’s another section inside of here that covers basically like the fact that you can port this pattern to not just to data bricks but really any data warehouse. I’m not going to go through that here cuz a lot of it’s more conceptual. Uh so, you know, just just to sort of bring the point home, this here is is what we went over, this pattern. And this pattern works with any warehouse and in theory any sort of set of documents in cloud storage. You would

1:55:28 have to make some changes to the scripts, but it’s schema agnostic, so it’s meant for you to apply anywhere. So, with that, I’m happy to answer questions uh until we reach the end of time and I can go a little bit over Alex if you if you’d like. >> Super. Uh amazing. Uh thank you very much Zach and Ryan for for the workshop. Thank you very much to Martin uh for answering questions in the in the back. Thank you everybody for for watching. Um I think we covered uh all of the questions. I don’t think Oh, well, there

1:56:02 is one. All right. Let’s Let’s do a couple of it maybe before and if people have to drop off, thank you for for watching today. Thank you for your question. Thank you for participating. Root notes continues next Thursday, so if you are not registered for the next workshop, that’s going to be covering virtual graphs. I have shared the link in chat. Um check it out, uh register, and join us next Thursday. Hopefully, you registered for notes if you like the event and if you like what you see here, then please share it also on your social media. Um would be nice to get a little bit more folks to to join our workshops and and notes itself. And uh with that, um

1:56:35 yeah, I think we have a little bit of time uh to cover um a couple of questions. Other than that, uh feel free to to drop off, obviously. There’s one um that asks, “How difficult or easy is it to update document graph if there are changes in the documents?” So, the great thing about the way we’re loading data here is that it’s all uh programmatic. So, you can re So, if there’s a change in a document, you can if the document’s sort of corpus is small enough, you can remove it and

1:57:07 re-upload the whole thing. There’s another project that I have called knowledge index or key. If you search for Kaparthi Wiki with Neo4j and my name Zach, you’ll find a blog on it. And that project actually has a way to do a change in part of the document tree. So, like imagine you had, you know, you were loading like tens of thousands of documents or more, you wouldn’t want to like blow the whole database away and reload the whole thing. And so, there’s a way to like focus on part of that tree and like just

1:57:37 reload that. So, I’ve open-sourced code for that. Um and so, yeah, that’s, you know, with with this simple thing where we’re going off of document structure, that’s pretty easy to do. There’s some complications around it, but the but it’s it’s definitely possible to update it and it’s not too hard. >> Amazing. I just checked I think I found the link uh to your blog. I just shared it in chat. Yeah, exactly. It’s this one. Uh KG also found it. Perfect. Um

1:58:08 And then, another question, how do I handle document with versions? I think that’s the same question, right? Close to the same. >> Yeah, I mean, it’s it’s similar, right? So, like uh a lot of times it and it depends like for for some workflows, people or store their documents somewhere where there’s version control, the source documents, whether that be in cloud storage, you can have version control, or if it’s in Git, you can version control it. And then, you use the graph as a means of searching, like a downstream sort of not the data the file system, but where you actually

1:58:41 expose it for search. Uh so, that’s one way to handle it if you don’t need to read revisions. If you do need to read revisions, you can have you just make your schema so that you have like a revision ID or you have a document that points to its revisions. And then whenever you do a query, you can like use latest revision as part of the properties. It’s a little bit more complicated. Um but you would basically make your own custom data model if read of revisions was important. If reading of the revisions isn’t important, you

1:59:14 just want the latest, use some other version control system with cloud storage or the file system or whatever would be my first instinct. >> Cool. All right. I think that’s that’s it uh at the end of the workshop. Thank you very much again uh Zach Ryan for for running us through, for helping uh with questions, and obviously for developing the workshop in the first place. I think it was very very interesting, very very uh good feedback in chat. As I said in the beginning, the workshop in itself on Graph Academy will be available for you forever. Um as we said also in the

1:59:47 beginning, the API keys for Entropic and the uh uh the Entropic um access will unfortunately not be able to keep uh up, but you can like we said, you can use uh BigQuery or you can use your own um API keys to to run it. It should be relatively low um low on the cost side, so it should be possible for you to to replicate that on uh your own um and um the video is this was recorded, so the video um will be in your inbox probably um tomorrow at this time. Uh it’s going to be processed and will it

2:00:20 will be ready for you, so you can obviously watch this at any time. And yeah, thank you for watching, thank you for attending, thank you for questions, thank you for participating, and paying attention. Um and I hope to see you next uh week on uh Road to Nodes with virtual graphs or somewhere else uh at any event or conference. Until then, have a good one. See you soon, and bye-bye. >> All right. And one more thing maybe before we go, we have a startup program. Um um so I’ve I’ve been told Alex we should mention this and

2:00:54 I’ll do what I’ll do is just go to I think what is it? neo4j.com/startup. >> /startup, yeah. >> Um so if you are a startup and you want some free Aura credits, uh go ahead and apply. Um and you know, we offer help and we offer free Aura usage. Um so it’s a pretty simple application process. Uh and yeah, we just wanted to we just wanted to throw that one out there before before you left.

2:01:25 All righty. Okay, thank you. I am done now. All right. Thank you, everyone.