Welcome and thanks for joining us today for building the AI native enterprise together. I'm Anthony Ampelius, Alliance leader here at Atlan, focusing solely on the Databricks partnership, where our goal is to help customers deliver trusted, accurate and ROI driven business outcomes on their AI initiatives. I'll be your host today and with me, a great colleague and very close friend, Gene Arnold, partner solutions engineer who manages the Databricks partnership with me here at Atlan. And I'm really excited to have Gene on and get a chance to really dive deep into this.
So here's the problem I wanna start with today. And as we look through the problem, we'll also look at how AI leaders in this space are really overcoming it. As we know, every enterprise is experimenting with AI. You got genie spaces, agents, co-pilots, really a whole stack. But there's a gap between experimental AI that shows well and AI production that actually delivers the business outcome enterprises are striving for. We really call that the AI context gap. And that's not just us saying it.
Ali Godsey, the CEO of Databricks, put it really well recently. He said, there's a reliability gap. Agents are not able to do full-blown tasks end-to-end and get it right. But it's not because they're not smart. And we need to train huge models that are gigantic with massive GPUs. They're missing the context. And that's exactly the gap today. On one side, you have powerful AI tools that can reason over your data. On the other, you have the business context. What the metrics mean, which data is trusted, who's allowed to see what.
That's still locked in heads across business domains, disparate systems, wikis, and tribal knowledge. Without that context, agents hallucinate, governance breaks down, and trust roads. Closing the gap is exactly what brings Databricks and Atlan together and takes you from experimentation to AI in production. Databricks Genie is a phenomenal tool. It's really the intelligence layer. It queries, reasons, acts, and answers. Atlan is the context layer, the semantics, quality, governance, and overall business knowledge that makes every Genie answer trustworthy, precise, and explainable. With that, I'm going to pass the ball over to Gene.
And we're going to dive a little bit deeper into Databricks Genie, explain how the leaders in AI are using Databricks for enterprise intelligence, how they gather insight, build, and take action. From there, we're going to dive deep into bridging that gap, bringing Genie and Atlan's context layer together. And as I pass this over to Gene, again, you're all going to see a couple of polls popping up. So make sure we answer so we can get the right feedback. Gene, over to you. Perfect. Let me make sure I get this right.
And boom. All right. Great. Again, Gene Arnold. I'm one of the global partner sales engineers here at Atlan. I do specialize in our Databricks relationship. And I can't wait to tell you about it. I have to tell you, I've been so excited about this webinar because AI, Atlan, Databricks, it's a passionate space that I love to be in. So let's go have some fun. Please ask questions. We've got some super smart SEs monitoring the chat. So put your questions in there. We've got you covered. So the first thing we want to talk about is simply this.
The Databricks system in general already handles context in its own way, understanding the fact that you have Unity Catalog for governance. It's able to take a look at the context inside of Unity, and it can make some amazing decisions based on what it already knows. We're going to learn more about how we can extend that in just a little bit. And Genie gives business teams that governed natural language analytics tools that people want. If you haven't created the Genie space yet, I urge you to go in and give it a try.
I was blown away by how intelligent this tool is. I just dropped a few tables into the space, Genie space. I gave it some content. It figured it all out. And it was able to answer questions within that context window quite accurately. And you're seeing lots of people adopting this. And lots of Genie spaces are being created because it is that easy to get the ball rolling. Add some data in, and you can start asking the Genie space some questions. But the context is limited to what that Genie space can actually see.
There we go. Poll was up. Okay, great. Now, when we move this into the enterprise runtime, it's not just about Genie spaces anymore. Now we want to create agents. Okay, what's an agent? Well, an agent could be multiple Genie spaces working together. To handle a specific task. Okay, an agent is supposed to be able to do something for you while you are possibly doing something else. Or answer a larger question than you might be able to answer on your own. That's where agent bricks come into play. And they're amazing. You can put in all your Genie spaces.
Right? You can add in MCP servers. You can wire them all up. And you can have them do things for your business. They are incredibly intelligent. I want you to kind of double click on that because they are. They're incredibly intelligent. Now, Databricks delivers that governed intelligence. All the workflows, right? The Genie, the Genie code, agent bricks. It puts that all together and does a phenomenal job.
What we're going to show you in a few moments is how we can bring in the context to make this a very, very, very powerful story. So, why are we all here today? Let's just jump right into it. AI is a safe bet. It's a real bet. In fact, we've got solid numbers to back that. Lots of money is getting invested into the AI space. A lot. Okay? And it's constantly increasing. And the deployment of AI agents is also increasing over and over and over. So, what are we seeing here?
We are seeing everybody jumping into the AI boat, if you will. Let's all jump in. It's going to be a great ride. Let's jump into the AI boat. But let's go a little bit down here and take a look because the boat's got a few leaks. All right? Here's the deal. Companies aren't seeing the ROI they're expecting. Okay? They're reporting the fact that the value they were expecting to get out of these agents, maybe not there. And even worse, all the time that's being invested in deploying these agents, it's not yielding what they were expecting.
Meaning, we created the agent in dev, but it never really made it to production. So, hold on. We've got everybody wanting to jump in the boat. But wait a minute. The boat's sinking. But they still want to jump in the boat. And the reason why is because of this. It's intelligent. Databricks is proving that. It's built on intelligence. People understand that AI is incredibly intelligent. What they're learning, though, is that intelligence alone doesn't cut it. Databricks is crushing it when it comes to the intelligence. I love working with Databricks. I really recommend you make a genie space if you haven't.
If you want to step it up another notch, create an agent brick. All the tools you need are there on the intelligence level with Databricks. But at Atlan, we figured something out.
There's a little bit more to this. The performance factor, the performance side, is more of a sum of or a formula of intelligence and context together. There's more to it. You can't just have intelligence alone because with only intelligence, it doesn't have enough context to know what to do when it's asked a question. Now, what does that mean? Well, let's take a look at this in a couple other slides. Customers right now fall within these quadrants. If we take a look at the left-hand side, the high, okay, is high context.
Companies with high context and obviously the opposite would be low context. If we look at the bottom, the left would represent low intelligence and the right would be high intelligence. Let's move forward. I only want to focus on these two customers right here, these two businesses right here. Because we have customers, we have businesses that are using incredibly intelligent tools. Spot on, amazing tools. But they're hallucinating. They're ungrounded. Why? Because the model, the intelligence, it's going to give you an answer, my friends. If you put in a question, it's going to do its job.
That's what an LLM is designed to do. Its whole existence is to give you back feedback. You ask it a question, it's going to give you an answer. That's what it feels it needs to do. It's going to do it one way or the other. What we're going to help you with is making sure those answers are trusted and accurate.
Okay? Because the intelligence is there, but the context that the intelligence needs, right, is missing. Okay, let's really nail this home. I absolutely love this slide. It is probably my favorite slide. I show this every single time. Let me explain why. Every one of you has been in a chatbot before, right? Everybody's got a chatbot. You can ask questions. Everybody's got a copilot or everyone's got a chatbot someplace. Here we are looking at an example. Simple example. How much simpler could this be? What are the top 10 new shows? I mean, talk about a gentle, easy question.
But behind the scenes, this is incredibly complicated. Let's break this down. There's the user context. Well, what does that mean? Well, who's asking the question? Because if it's the marketing department or the editorial department, their idea of what's important to them is different, right? So now you have to think about the user context. Without the model knowing who the user is, it doesn't even know how to even assemble the answer. Okay. Let's go further. What does new mean? Okay. That's a term. Okay. What does new mean? If you are just a human being and I asked you that question, you still have to try it.
What does new mean? Is it the last 90 days, 30 days? I don't know. How is a model supposed to know what new means without the proper context? And top 10, I don't know what top means. Is it views? Is it watch time? I don't know. What are we gauging these on? The fact of the matter is, Databricks has the data context, okay? The joins, how everything works together. It's already there. That context is there. It's baked into the intelligence. Together, right? This is why I love this slide, because it's easy to see.
Together, we can answer this question now. Atlan owns this. We have all the context we need to answer this question. And Databricks has all the intelligence it needs to answer the question. But without each other, the question's not going to be answered properly.
All right. Let's get to the fun part. Our customers are hitting three walls right now, and we're going to help you climb over all of them. The first one is called context bootstrapping. What's context bootstrapping? You have a lot of context. You have a ton of context. All right. But there is a thing called a context window. An LLM can only retain so much information. It's a window. It's limited. There's even something called context rot. You can type it into a chatbot and learn more about it. But the fact is, you have to create a repository of context so that your agent knows what to focus on, because you can't stuff it all in.
So somebody has to help create this context repository. Bootstrap this whole thing together. Got you covered. Now, after you have that context created, right? You've bootstrapped it. You know what context you want to work with. You have to test it. You can't just send this out live and hope for the best. Remember the figure I showed you earlier, why so many are failing? Because it looked real good when we asked it three questions. I guess it's going to be great for 300. It doesn't work that way. It just doesn't. So what Atlan does is it creates questions and questions that it knows what the answer is.
You'll see this. It creates the question. It knows what the answer is. It takes the context. It just bootstrapped. Sends it to the agent and says, here's the context. Here's the question. You give me an answer. I will compare that to what gold looks like, what true looks like. And we will test. And it's a fly reel effect. We ask a question. We test. We look at the results. We tweak the context. And we keep on going until we get to the point where we feel it is ready to be deployed.
Now, you own your context. You should own your context. Context is what makes you special. Anyone can use any model out there. Models are a commodity right now. I can grab any model you can grab. But it's your context that makes you special. And that context should be portable. And where you use it should be up to you. We'll show you how easy it is to push it out to Databricks. We are going to get into the actual demo now. Ask your questions down below. I can see that they're starting to happen right now.
That is awesome. I love the fact that you're asking questions. But I want to show you some product. So I'm going to click off this button now. All right. And I'm going to come over to this button right now. Okay. And I am going to jump into this right here.
Excellent. Welcome to Atlan. Now, what we do first to make this whole three-wall story happen is we will offer you the ability to connect to almost anything. How can you possibly build up context unless you have some kind of foundation, a data layer, a knowledge layer, right? You have to have these foundational layers to build on top of. I can't bootstrap on Vapor. Luckily, Atlan has a very deep and rich connector framework. We can bring everything in. In fact, I can come over here and type in Databricks, and we're going to bring in all the Databricks assets.
And we can bring in all of the SQL logs. I can build that lineage. I can see what's popular. But the important part is it's not just about one system. It's not. It's just not. It's about everything. And that's what makes Atlan so special. Atlan brings it all in. We know all about your context. Now, what happens when we bring all this in? Because we bring in a lot. When we bring this all in, we're able to create something called an asset. Now, we have lots of different asset types, but we're going to focus on one type right now.
This is a table. Now, this is really important because I'm going to use Atlan to help create context around this table. Give me a description, right? Tell me about data quality issues. Who owns it? Tags, terms. Tell me about all the columns. Give me descriptions about the columns. I want you to tell me everything you can about this asset table. Why? Think about what we talked about earlier. How in the world is an LLM going to know where to get the answer from without the proper context to know that that's where you got to go, right?
That's the table you have to drill into to get the information you're looking for. But it doesn't really end there. Check this out. What about context further out? What about the fact that Atlan's able to plug into everything, right?
Everything. Because here is the deal, folks. It's not just about this table. It's not. It's about everything that touches this table. Everything upstream. Every single thing downstream. Everything on this screen is context. Everything on this screen describes this table. It's not about just this table. So this is where we get to the part that I was mentioning earlier. Databricks has this level of context. Well, the table lives in Databricks. They certainly know what it is. But it doesn't know all of the rest of surrounding information. It doesn't know about all the terms that are attached to it, right?
It can't. It's okay. It's incredibly intelligent. That's what Atlan's for. We're going to give it all the context it needs. Okay? That's why the Better Together story works so well. Let's keep going. Because how are you supposed to possibly curate all this, right?
We've got all this intelligence that's happening over here. We know about how the table is being used. Who's asking questions about the table. What the query looks like. We know everything about this. But that's a lot of information to curate and create. So Atlan. Whoops. Atlan. Atlan has agents too. Now, what our agents help you with is that this is the beginning of the first wall, right? This is the, okay, great, Gene. Love it. But let's focus what we're going to create context on and curate to a specific scope, right?
Only take a look at specific Databricks assets, and we're going to go and rip through them. Make sure they have descriptions, readmes. Let's focus on this context window because I need to create an agent that's going to focus on this content. So I need it to be curated properly. So what Atlan allows you to do is just that. We can come in here and create these different agents, these new collections, and group it in a way that it's going to get your data ready, your metadata, everything inside of Atlan. Get your context prepped, right? The bootstrapping. Get it all put together so that when we move to the testing, we have everything we need to test on.
First wall. Okay. That sounds good. Next part of that first wall. Let's actually build this context repository. This is where Atlan shines. Because we know everything about everything in your data state, I can come in here and ask a question and say, Hey, describe the context repo that you want to build, right? Create me a context repository for sales, metrics, and revenue. And now I can type whatever I want here. And what it will do is it'll rip through everything Atlan knows about your business and put together a context repository that's properly formatted for an agent to address the questions that that type of agent is going to be asked.
Now, this is something that would normally take months to do because where do you find it all? How are you supposed to know where all this is? But Atlan already does. In fact, let me give you an example of one. I happen to like this one. Here is the Mick Context Burger demo analyst. Now, look, we're building out all the skills, right? In fact, we're learning about everything. We're building out all of the different models, right? Views. We're building that all out. We're creating the proper context that an agent would need to answer a question within its domain. Okay? But is it going to work? I mean, you shouldn't have to do this all day long.
And so what we do is exactly what I told you about earlier. We'll create a series of test questions. All right. And we're going to run those questions. Remember what I said earlier. We know what it should look like. This is what was generated by the agent when we offered it the context. And we were running tests. All right. And we know what we want it to look like. Oh, that's a quick click on the mouse. Let's do a test. Let's get the flywheel going. Let's make sure the context is in shape, if you will, is correct to answer the questions that it's going to be asked.
So this is now that second wall that we helped me get over. This is the wall called testing because we already did the bootstrapping. Now let's make sure what we have is actually going to work. Okay? That's what you're seeing right here. We run our tests. Cool. Now that we have it all tested and we're comfortable with what we have, let's deploy it. Let's make this portable. Now, obviously, we're going to want to push this out to Databricks, but we do realize that our customers want this to be portable. You have the option to push your context anywhere you want, just like you have the option of owning your context now. You should. It's yours. It belongs to you. Now you can push it out, deploy it to wherever you need to.
So what does that look like? Well, if I pushed it out to Databricks, it might look like something like this. An actual genie space. We will make the genie space for you. We will add in all the information about the genie space. We're actually building genie spaces directly from Atlan. What data should be in the genie space? We put that all together for you. Remember, that's part of the bootstrapping, wiring this all up. Here are all those metric views that we created. Remember the testing? Well, here are all the instructions that you would normally have to keep on tweaking and running and tweaking and running.
Nope. It's all right here. So we will build these out for you. Or if you want to, you can do this right from genie code and connect our MCP server into Databricks and actually build this same thing out directly inside of Databricks using Atlan's MCP server. Because everything I just showed you is available through our MCP server. Just plug it in and go. And then it even gets better. After that, you can take all those genie spaces that you created and catalog them back in Atlan. Because this is really important.
I don't know if you're seeing this or not, but I definitely am. Genie spaces, these different agents, are starting to become the new dashboard, right? I'm not going to create a dashboard. I'm just going to create a genie space and you can ask it a question. But just like with dashboards, without them being cataloged, you end up with a gazillion of them. Everyone creates the new dashboard for the West Coast sales analysis dashboard. Well, why are there 15 of them? Because you didn't know 14 other people made them before you.
Because it wasn't cataloged. You didn't know. So all of the genie spaces actually get cataloged inside of Atlan. So that means you have people that understand how to make the genie spaces. They do their job. Atlan helps them through the process. They come right back into Atlan because folks that need to use the genie spaces need to know where to go. Everything's cataloged. When I mean everything, I'm even talking about the data that's wired into the genie space is cataloged. So you even know what's going on. And just to make sure you like the genie space, you can 100% come in here and try it.
You can talk to the genie space directly from Atlan, just like you could open up a dashboard inside of Atlan to make sure it's the one that you like. And give it a run. Okay, this is cool. This is the one I need to use. Now I'll go into Databricks and continue from there.
Perfect. I want to end this demo on a pretty cool example. What I'm going to show you now, this is technically two chatbots running side by side. The left chatbot, okay, the left chatbot is just an intelligent chatbot. It's really smart, and it's going to try to answer a question with whatever context it currently has. Remember, an LLM is going to find a way. You're going to see that. The other side, right, the right chatbot, same intelligence, same data set, same everything, but this side, okay, this side has context. Watch what happens.
It's a fictitious scenario over here where someone has a problem, and they're going into the burger joint chatbot and asking you the question. But it's really important to take a look at what's happening here because on the left, you'll notice that everything is in the data level because that's all the intelligent LLM has to work with. Okay, so it is doing everything it can to figure out how to answer this question with, in this case, the limited context it has because it's going to answer that question for you. And it gives you a rather intelligent answer considering what it had to work with.
Okay, what about the right-hand side? Take a look at this. You're going to see the icons change because it's dipping into Atlan to figure out how to properly answer the question and figure out where to go within the data to get the answer it needs. Okay, Gene, just show me what this is even doing for me. Well, look, your son, he didn't know the son's name was actually Ethan. Why? Because it didn't know where to go. It didn't know where to go to get the answer for that. So basically, it took a look at the context and made an assumption and gave you an answer.
How about the refund? Well, the refund is completely wrong. It's wrong because the left-hand side didn't know how to calculate a refund correctly. So it just, it found something. The right-hand side had the context to properly figure it out. So the left-hand side is showing you why companies are building these out and not bringing them into production. Because, well, they're basically giving money away over here and they don't even know who they're talking to. The right-hand side with proper context. This is going into production now. This is something that can be used.
Okay. What I'm going to do is take us right back over here to the slide deck. Click on slideshow. And it should bring, there we go. Anthony, I did it right. I clicked on the right button, buddy. Take me home. I love it. Good stuff. And honestly, going through the demonstration, I love seeing that last piece where you can see what it looks like on both sides. With Atlan and without Atlan, because you can get the good understanding of the answer could be good, but what happens if it's not great?
And with that context that you showed there, Gene, it's great. You're able to drive forward. You move fast. But you don't miss anything that could hold somebody back, right, from an allergy-related issue or a future fine or, you know, scenario in the future. So really appreciate that, Gene. Everybody on right now, feel free to put the questions in the Q&A. We've had a lot that you've already submitted, and Pablo and Ryan have done a great job answering it. The first one I see, Gene, from Krish is, where does the context reside, in customer space or in Atlan?
Ah, great question. So I would definitely say both. So the context is stored inside of Atlan, okay, and you can access it through the MCP server or Metadata Lakehouse. And then use it and do whatever you want with it. Or remember the deploy option. I am taking that context and sending it out, deploying it to an end result. So that's why I'm saying both. But the best part is it's always accessible to you. Our Metadata Lakehouse gives you access to every single nook and cranny of Atlan. So essentially, it's always yours, but it does have to live somewhere.
Absolutely. Great question, Krish. And you actually came out with another one as well. How do you ensure, oh, looks like it might have just been answered here, but here we go. Okay. How do you ensure that there is no duplication of context as the data is scattered? Oh, good one. So you've probably heard of the human in the loop phrase, okay? Well, we call it human on the loop. Because if I have to go and curate 100,000 items, I'll show you what human in the loop looks like. It looks like this.
Click, click, click. I'm just going to click the mouse and say yes to everything. So what we do is we do something called human in the loop. We're going to bring a human in when there might be a situation that has to be solved. So think of your duplicate entry. Well, through our process, if something doesn't look right or there's a situation, we bring the human in to address it. The other great part is since we have the entire metadata lake house, that allows us to help scan and see what's going on.
There's many checks and balances we can do to figure out if there's a duplicate or not. It's certainly something we can address. I love that. All right. Here's the next one for you. What happens when the context layer has gaps? Like if a table doesn't have business definitions or lineage mapped out yet, does Genie still respond? How does it answer quality? How does the answer quality differ? Right. Great question. Now we're getting into the engineering and development of an agent. Okay. This is where you have to start putting in hard guardrails.
This is where you have to understand how to build the instruction and the system prompt to define those guardrails. Okay. So that's where you have to start setting rules where you have to explicitly tell the agent, if you cannot come up with this answer, this is the answer you should give. And let me explain to you why I said it that way. If you cannot get the answer, this is the answer you should give. Why I said that is because directly because what I've said earlier, the agent wants to give an answer.
It's why it lives. It wants to give you an answer. So you have to give it that way out. So you have to say, if you're not sure, then this is what you say. It's like, fine. That's great. I didn't know, but I still get to do what I like. So this is where you start to get into the whole testing and understanding how agents actually work behind the scenes. So a lot of that work is done inside of the system prompt. And then we could always get deeper if we had an actual specific use case.
This is where we get into the stuff that I like to do and start to play. But that is what I'm going to give you right now for an answer. And maybe one day we'll meet and I'll give you a longer answer. I love it. So how about this one, Gene? If the data is in, let's say, Databricks and Snowflake, both, are you able to create a single context and duplicate? Yes, that's our superpower. Remember when I showed you that lineage, okay? The data could originate in different places. This is what's important about Atlan, because if we're not the middleman, then you may be defining context differently where the data resides.
That's not good. Okay, let's just go with a simple one. How you define PII. That should be a business standard. It doesn't matter where a social security number is. Your definition of how to work and address PII is the company standard. So that's a major power play that we have, because when you're answering a question, the context should be coming from everything that the company knows. So that is definitely what we can do. In fact, when you ask that question in the context enterprise studio, it's ripping through everything to answer that question for you.
That's perfect. So I know we're very close on time here. Akshay had a little bit of trouble joining before. Akshay, before we wrap up, is there anything you want to kind of touch on from what you're seeing from the Databricks perspective in terms of how strong Atlan's context layer is for the best, highest quality Genie results? Everybody, sorry about not being able to join. I look at Databricks and Atlan as perfect complements. The way Databricks is trying to solve the problem is essentially bridging the gap between the technical teams, the business teams, by democratizing data and AI across all personas.
Philosophically, the gap that exists or if there is something that would unify all of that would be is the context. And we have thousands of customers who have data on Databricks already. Databricks Genie does a pretty good job of understanding the data, the semantics, stitching together everything. Add on to the business context that companies like Atlan provides and all the awesome integrations that we have built with these partnerships. We are truly one click away from those insights. More exciting stuff will be announced at the Data and AI Summit as well, where we'll be available to host more questions, Gene and I.
But I would say this is just the starting. We have awesome things coming. Yeah, I completely agree. Coming into Atlan a little over a month ago, I was able to see firsthand how strong this partnership is built. And it's all focused on how we make this integration stronger, better, that allows every customer to be able to move at the speed of the business. They want to get to the business outcomes faster. They want to drive their AI initiatives forward. And to see the way that this will really enable our customers to deliver the AI success they're looking for, it's truly remarkable.
And even to actually what you were saying about the context. When we kicked this off, I talked about all these, quote, you know, the big part about the reliability gap is it's missing the context. It's not the, you know, the huge models or the massive GPUs. It's literally the context that's going to enable this success. And I feel we've done a great job together building this out to make our customers successful. Absolutely. Well, I love it. I know I think we got to majority of all the questions. Feel free to keep sending them in.
Gene and I and Akshay, we're here to support you all. But at the end of the day, you know, as we wrap up, I just want to thank everybody for joining today. If you're attending a data and AI summit at the end of next month, please let us know. We have some amazing sessions, some incredible events that we're hosting, and we'd love to meet with you there. So in the chat, I posted Atlan's link for DAIS, and we'd love to take this forward and help you all be more successful and become more of an AI native organization. So Akshay, Gene, thank you so much for joining today and hosting this with me. And let's take this forward, team. Appreciate it.
