See how Atlan's context layer makes AI ready to ship

Prukalpa Sankar

0:00 All right, let's go. When we started Atlan, we dreamed of a room exactly like this one. Practitioners, leaders, builders all in one place. So very, very, very excited to welcome you all to Atlan Activate, our flagship product launch event today. Wow, today feels surreal. Less than a year ago, we were at the last Atlan Activate, and we opened with a provocation that in the dawn of the internet era, Bill Gates had said, "Content is king." And as we dawn on this new era of AI, context is going to be king. That was our bet. And since then, the industry has coalesced around our vision.

0:46 A few weeks ago, at the Gartner Data and Analytics Summit keynote in Orlando, they went on stage and they said, "Context is king." Gartner, main stage, "Context is king." Why is context having its moment right now? What's going on? In the last year, context has had its moment. We are in this in this moment where context has gone viral on X. Context is becoming a real job. Context engineers are becoming a real title. What's really going on? Why is context in the spotlight? We believe the answer lies in a paradox. On one hand, the models are getting exponentially smarter. Every single day, the models are getting smarter.

1:35 About two years ago, it couldn't pass the bar exam. Today, in novel reasoning, when it's asked to solve a problem that it's never seen before, 87%. But on the other hand, if you ask a CEO, they report no financial benefit. 56% of CEOs say zero financial benefit from AI. So what's going on? The models are getting more and more intelligent, but just not useful. Every time I'm faced with a hard question about AI, I typically like to go to understand humans, and the answer is hiding in plain sight. Cognitive intelligence does not determine real-world effectiveness. In fact, it's a very small predictor of real-world effectiveness.

2:22 Less than 10% of job performance variance can be explained by IQ. It makes sense, right? Think about your best performing employee. Are they someone who also scored the highest on the SATs, or are they the person that works really hard and learns really fast and takes feedback really well? I think we all know the answer. Today, in the real world, companies care about performance, the outcomes you actually deliver in the real world. Performance is a function of intelligence, cognitive horsepower, how fast you learn, but also context, knowledge, skills, and tools that are built doing the work.

3:07 This equation explains the difference between AI that is actually intelligent and AI that is actually useful. In the last decade, we have compounded on one of those axes. Intelligence, we've compounded over thousand x. Every six months, we are compounding two x in reasoning. It's also commoditizing. But on the context axis, context has barely moved. Situated knowledge of your business. Yeah, we've moved some data to the cloud in the last decade, but largely, it's still logged into dashboards and Slack threads and, you know, the heads of humans. We believe this is the next frontier, contextual intelligence.

3:54 There is no intelligence without it being contextual. I'm gonna say that again. There is no intelligence without it being contextual. It's purely an academic construct. We believe that in the next decade, the companies that find a way to truly take their institutional knowledge and context and operationalize it are the ones that are going to win with AI. Typically, when I say this, I... You know, people tell me, "All right, I'm bought in." But how do we help AI build context about our businesses? Again, I go back to human constructs, the same way we train and onboard humans. I want to introduce you to Maya.

4:35 Maya works in support at, let's say, Mech Context Burgers in Chicago. Her shift is 11 to 7. She does about 40 calls a day. Most of them are easy. Sometimes the call is hard. Today, there was a hard call. She had a customer that called in saying that her son had a dairy allergy, and it's registered in the family plan. And today, the order came, one of the burgers had cheese in it. The kid's meals are also missing from the burger. The mother is really angry. Her kids are hungry. She wants a full refund. She wants to talk about the membership. She also threatens social media.

5:16 Maya has 90 seconds, 90 seconds to make a decision and handle this call. What does she need to do to handle this well? Well, first, Maya needs context and knowledge about what is true. Who is calling? If it's, you know, a valued loyal customer, or is it a serial complainer? What happened? Why did the order get messed up? What plan is the customer on? She needs to know knowledge about what is true. But knowledge is not enough. Maya also needs to have skills. There were three issues that the customer talked about, ranging from safety to a missing order. She needs to be able to triage this. To be able to read the customer.

6:04 That is the customer really threatening social media? Are they really distressed? And she needs to be able to understand the remedy and size the remedy. And last, she needs to have tools. How do I use the CRM? How do I use the billing and refund engine? How do I actually process this refund for this customer? Maya, she's a veteran. She she handles the situation with ease. She talks about the child by name. She, she apologizes. She refunds the order. She adds an extra credit. The customer gives rave feedback. But how did Maya get here? Maya joined MechContext just over a year ago. And when she joined, she started reading up the wiki.

6:48 She started understanding the matrices, the escalation trees. She went to classroom training. She was taught about the refund decision tree. She understood how to use the tools. But then Maya started learning on the job. In week four, Maya shadowed Jamie, and she saw Jamie giving a refund to a customer when it wasn't really MechContext's fault. And Jamie looked at her and said, "We believe that when it's a 50/50 situation, customer always wins." Maya also started making mistakes. Her first mistake, she issued a refund to a customer who had eighteen prior complaints in the last one month. She realized to always look at the history of the customer before processing a refund request.

7:37 Maya got manager feedback. Maya was coached on the job. Maya handled edge cases as like food poisoning. That is how Maya got great at her job. So the question ahead of us, as we start deploying AI, how can we give a customer support AI agent the same contextual intelligence that Maya has? And so to walk us through that, I am really excited to welcome my amazing co-founder, Varun, to take us through building contextual intelligence in real life.

Varun Banka

8:11 Thanks, Prukalpa, for kicking us off, and great to meet everybody joining us from around the world. If you're a math nerd, and you're still struggling to process that formula Prukalpa talked about, let me make it easier for others.

Varun Banka

8:24 What we are talking about is to ensure Maya just not just know the right— knows the right thing but does the right thing in the right way with the right tools and with the right guardrails. And today, we are gonna talk about how the world around us is shaping up. This is where all companies are. Prukalpa talked about that, hey, we move data to the cloud. Yes. Our business runs on a system of record. The we move system of record via pipelines to a system of analytics. We make hundreds of decisions on our documents, emails, chat thread, and all of them stays.

8:57 Today, I'm not gonna talk about this whole diagram, but we are gonna ground this to help understand how we are gonna go about activate demos today. We're gonna go through three parts of the demo today. Number one, we're gonna go through context layer, understand how do we get started. How do we get even started with hundreds of assets staying in data layer? Second, we are gonna go to the agent layer and talk about, while we've given knowledge, how do we create those agents which can help trust? And third, at Atlan, we always believe that context needs to be portable and interoperable. We'll talk about how we've provided the interface to make it happen.

9:36 Now, hat tip to everybody who's on this, watching this screen. If you're an AI leader right now, just take a screenshot of this. Share it on your Slack, Teams with your AI builders. If you're an AI builder joining us, right, discuss pros and cons, hardest. Reach out to us, hardest... Ask us your hardest questions. Reach out to us with your open-ended questions. If you are someone like me who's curious, use AI as a thinking partner and learn about how the world is shaping up. And if you're somebody building in the industry in the same ecosystem, use this as a reference. So what are we gonna do today?

10:12 We're gonna see how do we get started, solve the cold start problem with context layer. Second, how do we use the context layer to ship trusted agents like Maya? And last but not the least, we'll talk about portability, so you get future-proofed irrespective of how the agent layer keeps evolving. So maybe as we head to the demo, just clear your context windows, and let's go.

Varun Banka

10:37 So here's the first demo we are gonna going after. How do we bootstrap our context layer? I'm gonna switch to the Atlan world, and we'll go from there. Let's just first start and move into our connectors to bring in all the existing context that is in the data layer. What you have, Atlan has hundreds of connectors, both from a breadth and depth point of view, to bring rich metadata from your existing, existing data layer. And for this example, I'm just gonna open up a Databricks connector. It's self-serve, or you can just put in your credentials, and we are good to go.

11:10 And not only this, I'm sure some of you are smiling because we've— this has gone through years and years of punches with enterprise security, authentication, networking. We've gone through the hard, hard grid work for doing that. Once you do that, what automatically happens is we create a data map for you. What you're seeing right now is I just moved on from setting up a connector to an actual table that is on your data layer called Subscription Health. What you're seeing right now, maybe let me, let me make it easier for everybody in the room to just imagine what this looks like. This is Subscription Health table.

11:48 Like, for Maya to answer those questions, it needs to find these tables. Where, where's the health of the customer? Where is it coming from? Where the invoice is getting stored. It, it can look upstream. It can go further upstream if the data is getting processed from somewhere. Maybe this data might be coming from CRM. Maybe there's an SAP instance which is there. From a downstream point of view, it could see that, hey, there is actually a BI dashboard that's talking to it. I've— So there might be some metrics which might be there. And we don't just stop here. We take one level deeper. So maybe let me look at a column— PLN disc text.

12:23 Now, not only it helps us understand how these tables are connected, but when your agent has to traverse through this map of your data to understand what even columns to pick up, it can pick up, find the right columns, understand what business logic was written, why was this written in a certain way. It could read through SQL queries. Now, while it can f-maybe find the right information in the map, but how does it trust this? Last year, we launched native data quality capabilities on top of Atlan.

12:51 What this means now is for this plan description text, we have all the categorical variables, the guardrails, like what's the minimum, what's the maximum length so that when it decides to run those SQL queries, it has all the rich information to make the right queries so that it does not need to make guesswork. But this is about the map. But what about the language? Let's go to all the columns that this table has.

Varun Banka

13:16 What you're seeing right now is some names, beautiful column names that you're seeing on your screen, and I'm sure some of you are laughing because this is what real world looks like. These are how the columns are described. Documentation does not exist. And it's not that people have not tried. People do give it a try. Just that there is no right way to scale this. Everybody— Like we always say, everybody likes documentation, but nobody wants to do documentation.

13:41 And at Atlan, because this has been the hardest problem of context layer, and we've been at it for years, from starting from making it easy for you to write low-code, no-code rules-based automation, to driving automation, to using... Last year, we launched trigger-based AI inputs to help you generate this. But now I'm super excited to present context agents. Thank you to all our customers who've been working with us to shape context agents. Now let me make it real for all of you. What you're seeing is Atlan's native context agents, which help you accelerate the problem, the gaps, and fill the voids in your context layer.

14:26 What you're seeing is we've created curated set of collections of assets, and we've made this knowing the fact that these are the assets your agents will actually need. And this is based on existing information from your data layer. One of the favorite collections for our customers has been popular SQL assets, which is a reflection of all the assets that like the business is already using. So what I opened is a collection of, collection of popular SQL assets. Based on this, it shows that my context has hundred assets. The coverage for metadata looks like eight percent. And we have three things that we could do. We could run enrichment on descriptions, README, SQL intelligence.

15:08 And what it means is, it's as easy as clicking a button. What this means is our agents will traverse through the data map that we already have, goes through schema, column names, upstream lineage, downstream lineage, look at every query history that exists, l-look at your existing business concepts, existing metadata, and generate them. So for me, right now, let's just trigger this. So while this is happening, while it's going through traversing through this, let me open the asset which already has something. What you're seeing right now is the same table, just that the language now makes sense. It's no longer PLN ID.

15:44 It talks about exactly what descriptions exist so that when the agent tries to traverse, it does not do guesswork. And we didn't just stop here. We in fact went one more step. I'm sure a lot of you have so many stale Confluence documents that nobody gets updated. Now this README gets automatically updated based on existing data layer. In fact, one thing that I really like here is how teams use data. This has been amazing to just see how different teams have been using data.

16:12 And this could be super helpful for the agents when to decide what, what to pick up, what not to pick up, but from a business point of view, not just from a SQL point of view. And like I said, we didn't just stop here. We went o-one more step. Intelligence. This is something most people miss out on. It's not that pre-LLM or pre-talk to data, teams have not been using data. We've been using data, whether it's by ad hoc data queries, your BI layers, all your BIs are sharing queries. What we've done with context agents is we've reversed engineered every SQL question and given a list of questions that are getting actually asked.

16:51 So what this gives is, this gives another test set, eval set, and we look at some of this of how we are gonna use this to actually build an agent later. And we didn't stop with business questions. We looked at joins. We all know AI is not good at writing joins right now, so this makes it easy for it, ground it so that the reasoning gets better. We looked at filters so that it does not make like does not have problems doing operands. And we looked at what other columns are connected to this. So all your historical business questions that data has been used is actually the biggest IP of companies right now.

17:28 So let's go next. Now, we talked about how we used context agents to leverage on top of the data layer. But with large language models, something else has changed. These models now are able to process your unstructured data, which has never been used before. They've been just lying around. Nobody has been able to use them, tap into them. What you're seeing right now is what we call knowledge folders. Think about them. This is like the gold layer of unstructured data because you could have hundreds of data lying in your raw storage. This is the final layer, and this has primary documentation like your actual SOPs, your PDF files that will actually determine the SOPs.

18:07 I'm just gonna open one of them. I'm sure you folks will be able to relate to this. This is like a refund reference. I'm sure you can imagine how the AI agent right now, when we, when we go through this building the process, how it could use some of this to actually answer the questions that Prukalpa was talking about in that scenario. Now, how do we use it? So apart from this populous SQL assets, if you guys remember, we, we used this to actually generate a ton of context for the context layer.

18:34 We also have an agent that runs through all your data assets, helps you select the right knowledge folders and knowledge files, and generate not just a data graph, but your business graphs. So for the sake of demo, we've already done this, so let me go through one of them. What we've moved on to now is we moved on from a data graph, now we've moved on to a business graph. What you're seeing is a business concept of subscription. This might be... This document, this content of this might be lying around hundreds of document, the knowledge folders that we talked about.

19:07 Our context agents crawls through all of that to help you create this and auto manage this. So it help you generate this. Again, the description, what this could use. So the next time the agent wants to run, it knows the exact difference of what subscription looks like. Because just like business, we don't talk in the language of PLN, ACT, we talk in the language of business, and this is what it grounds any agent on. So we went through a journey of data graph traversal, so let's, let's also go through a journey of maybe business graph traversal. What you're seeing is subscriptions.

19:42 This is the entity that we created, which is determined by, which is created by context agents. Let me click. It realized that, it figured out that, hey, the subscription is actually built by an entity called invoice. Maybe let me expand a little bit more. It talked about that, hey, this is one... This is a different concept that applies to this. And maybe, and business concepts are not just about entities, these are also about metrics. So it's able to figure out the right metrics that are calculated, which are related to invoice.

20:10 Your agent does not need to do another calculated formula and, and try to reason and hallucinate on building the right formulas, it can just ground itself on some of these metrics. So we saw the data graph, we saw the business graph, but for the context layer, we brought both of them together. So not only our context agents creates your data graph or your business graph, with linked assets, we're able to bring all of them together. So not only this is the business concept of subscription, but this links together with all your existing assets in your data layer that actually references a subscription business concept. So let me go back.

20:49 Let me go back for a quick summary.

Varun Banka

20:52 But before I go to a summary, we... I want to also have a big shout-out to all our customers who joined us in an accelerated program which helped us shape a lot of context agents. At Atlan, we always build with our customers, and I'm so happy and excited to have Adriana with us to talk here about our journey, and I'm joined with my amazing colleague, Nandini, to take over from me to have a quick conversation with Adriana on her experience. Nandini.

Nandini Tyagi

21:15 Awesome. Thanks, Varun. I'm thrilled to welcome Adriana from Engine, who have been our design partners and have been leveraging Context Agent Studio. Adriana, so glad you're here. Let's dive in. Okay. So tell us about Engine and what were you trying to accomplish? Why did this matter to you?

Adrianna Clark

21:35 Yeah. So I'm a senior data governance analyst here at Engine. We are an all-in-one modern, modern travel platform. We help companies book and manage hotel, flights, and rental cars all in one place, and we've heavily been heavily investing in AI to help drive efficiency across our organization. We wanna always ensure that our customers have the best product. We are currently exploring Atlan as the nervous system for how AI operates within our data ecosystem, and we understand that AI is only as good as the context that it has access to.

22:08 So a big focus for us has been making sure that we build something that is accurate, a scalable context layer with well-governed data that already sits inside of Atlan. At the moment, our data teams are already using Atlan's MCP server. We're getting a ton of value there.

Nandini Tyagi

22:22 Got it. Makes sense. When you first started running the agents, what were you expecting? And then what actually happened?

Adrianna Clark

22:28 Yeah. So honestly, I was not sure what to expect. We were a few days out from launching Atlan. I had a ton of work to do trying to coordinate training. We were rolling out to the entire business. I expressed this to Charlotte, who is our customer success manager, and she mentioned the Context Agent Studio. I still was skeptical about the level of effort, but once I got in and started using it it was already pulling out our most used assets, and it had collections ready for me. And so it was very efficient, and I was shocked by how fast it was. So it was a very good experience.

Nandini Tyagi

23:00 Amazing. Amazing. Thanks for sharing that. And when you actually looked at the AI-generated output, I know it can bring a lot of feelings, what surprised you?

Adrianna Clark

23:09 So I was definitely nervous going in.

Nandini Tyagi

23:12 Yeah.

Adrianna Clark

23:12 I thought that there might be, like, some gaps in the definitions or SQL. But once I went through it, it was very accurate. I think one thing that really stood out to me was the ability to take some of our most popular business questions that people were already asking about our data. They wanted answers to this information and you know, this is the type of context that we need for our agents, and it definitely enabled a sense of trust for me in the Context Agent Studio.

Nandini Tyagi

23:38 Makes sense. Makes sense. Thanks for sharing. And as I'm thinking of wrapping up, stepping back, if we zoom out what did this actually unlock for you at Engine? Like, what's now possible that wasn't possible earlier?

Adrianna Clark

23:52 So I think for years we've been taught that in order to document, document human tribal knowledge, it has to be a very manual process. You need your data stewards, you need IT, you need the business. You have to have all this buy-in. It has to be hand-in-hand. And what I experienced with Context Agent is that you can now lead with technology so you don't have to lead with the business. I was able to run this entire exercise on my own. And at Engine, we really do not have the capacity across our business verticals to have a data steward to sit down and go through, you know, manual hours to really complete this exercise.

24:29 So I think that was the big shift, is you know, the old mode— old model really assumes that you have the headcount, you have the people, you have the capacity. But when you don't, you know, this really makes it possible to to put a solution in place.

Nandini Tyagi

24:45 Makes sense. I know in one of our one-on-ones you mentioned that it's having your own AI team. And always I laugh when I think of that. Thank you so much for sharing all of that. It was great having you here. Shout out to everyone who participated in XData. I'm seeing lots of really good comments. I'm gonna pass it back to Varun. Thank you so much for joining us, Adriana. Thank you.

Varun Banka

25:02 Thanks, Nandini, and thanks, Adriana. And we were equally worried when we launched this with some of our early customers, and we were like, "Hey, we'll— will we need, like, human in the loop set up for this?" But we were just surprised the fact that nobody wanted to override any of these descriptions. And I know for folks who want to hear more, reach out to, reach out to Atlan. We'll be happy to showcase what, what it's able to do. Now, but how does that happen? Why does it make it special? What's, what's happening that it's able to traverse through so much context? We believe that this is what we believe in the concept of context compounding.

25:37 Step one, we talked about how we connect to your underlying data layer to bring existing context, harness existing context, process that, and bring that into Atlan. Once that happens, our agents goes deeply through across a data source. If your, if your data is coming from Salesforce, we understand why Salesforce data was moved from Salesforce to your underlying system of analytics. We understand what's happening in your BI layer. Why was that as a dashboard that in use? Understand what the what business is actually asking these questions on those columns. Understand why those filters are happening. The agent understands that, and then it helps us create everything that we talked about.

26:13 Once it does that, we get, we get another unlock. Because once you have that, we are able to create metrics, KPIs, and actually create one level more of insight that we're able to create. And once you have that, we're able to unlock not just the technical graph, we're able to unlock the business graph itself. With the power of both structured data layer and unstructured data layer, we were able to bring all this together, and it compounded. So a quick recap before we close out on our phase one of Activate. One, we talked about our connectors, which go through and help you build the data layer.

26:49 Second, we looked at how context agents are filling the void that's always been there to actually get started with context layer. Third, we just didn't look at the data graph, but we also looked at the business graph and how context agents, with the power of knowledge folders, we were able to leverage and create these business entities, metrics, and ontology.

Varun Banka

27:07 Okay, let's go to phase two of this. Let's go back to our customer support agent that we were talking about. While we have done some work on the knowledge part for Maya, but we are still missing two things: expertise, how does it decide what tone, how to remedy, what should be the escalation? And it's missing out on the tools. How does it actually go to the data layer? How does it actually query? What does it need to do? It does not have the tools. So for this, we started working again with some of our amazing customers, and we call them our context pioneers.

27:46 So thank you to each one of each one of you to help us shape this and help us solve this problem. And with that, announcing Context Engineering Studio. And without saying a lot of words, I'm just gonna pass it to my amazing colleague, Anirudh, to take us through and make it real and help us create a trusted AI agent using context layer. Anirudh.

Anirudh Agarwal

28:08 All right. Thank you so much, Varun. So firstly, I'd like to say I'm an AI builder, which means that I love building agents. So what we're gonna do today as the first part of this demo is actually show you how you can leverage the Atlan context layer to build an agent in an IDE like Cursor. So let me share my screen. So what you see here is my Cursor workspace. In my Cursor workspace, we basically, I have integrated two MCP servers. The first MCP server is Atlan's MCP server, and the second MCP server is the agent framework that I'm using here, which is Claude Managed Agents.

28:46 Now, all I need to do is paste in the prompt that I want to build. Now, what immediately starts happening is that because of all your enterprise context that lives in Atlan, all of that can be leveraged to now build this contact center agent that we are building called Maya. It understands, and it knows that it needs to go and look at all of our tables. It helps you search for that. It helps you search for all of your knowledge files, which are all your SOPs. It teaches the agent what to do and in what situation. It also can build all of your skills.

29:18 Think of skills as mental models or capabilities that you need your agent to have to be able to do any operation. It builds things like semantic models. So semantic models are essentially files that allow your agent to understand your database. It allows your agent to understand your tables. It does simulations. It runs an evaluation to identify how good the context that you're building for your agent is. And then finally, it builds a soul file, which gives your agent an identity, and then it also goes ahead and deploys it to your agent framework. Now, what's happening here is that we're now searching through all of the different assets.

29:55 It's found something like Customer 360, which could be super useful for a customer support use case. It found something called contact center interactions, that it can historically go back and look at all of the different interactions that that user might have had, had had with that customer support agent. It also found all of our business concepts. It understood things like service reliability. It understood things like carrier interaction. All of that automatically gets bootstrapped and leveraged, making my life as a builder really, really easy. Let me show you what I did backstage right before the de-demo, is actually build an agent completely through it.

30:33 So as you can see, what we are doing, what the agent is automatically doing, is that it's finding all of our business concepts. It's finding all of our SOPs that we've defined and stored in Atlan that Varun just showed. Also, what it's doing is it's actually generating us all of the skill files that we need. So think about skill files such as the voice and tone of the agent. How do we route escalations? How do we manage refunds? All of them get automatically created from the context that we have in Atlan, as well as all of our semantic models also automatically get defined and created. And what this leads to is a completely deployed agent.

31:10 So let's show you what the deployed agent actually looks like. What we've used here is Claude Managed Agents to actually do the deployment, but you can also use any other agent framework, whether it's Google's ADK, whether it's LangGraph. You can easily build on top of them through all of these different tools that Atlan has. So as you can see here, our agent has a system prompt that was generated through the context that Atlan already creates in the context layer. We created all the tools that the agent needs, as well as all the skills that the agent needs to be able to complete this operation.

31:41 Now, before we even go into our demo further, the second thing that we wanted to show was, okay, we showed you how you can actually leverage Atlan to build an agent. But what also really matters is how the quality of that agent is. So what we have here on the left is an agent that's powered primarily on the raw context, in the raw files, the raw SQL tables, and on the right, something that's powered by Atlan's context layer. So you can see what the agent's trying to do. First, whenever a question is asked, similar to, you know, my son's burger had cheese despite his da-dairy allergies and the kids' meals were missing.

32:20 What the agent immediately starts doing is it starts looking at the ontology. It identifies the right tables. It also identifies the severity of what has happened, tags the right ID to it, and then uses that to trigger the right SOP and the right response to it. So when you compare these two responses side by side, what you realize is that the Atlan's context agent actually gives much more contextual answers. The first thing, it actually got the purse— the the, the son's name right. It also found the correct amount that it needed to refund. Also, it also follows the tone of the business. For MechContext, we want to make sure that, you know, we follow up.

33:00 We we say that we wanna text you back in two weeks or in two hours. And all of that comes because this is powered by Atlan's context layer. Now let's jump into the product and show you a little bit about how Atlan's Context Engineering Studio works. Now, if you're, if you're someone who doesn't like using IDEs or doesn't like using CLIs, we also offer you to do the same workflow within Atlan itself. So you can define the same natural language prompt, and we can build this agent through a context repository. Now, you might be wondering, what's a context repository? Let me show you one. So a context repository is nothing...

33:38 Is similar to how a, you have a GitHub repository for code. A context repository is a repository for context. Now, what a context repository has is everything that your agent needs to be able to do its task. Like we just saw, it has all of your semantic models. It has all of your skills that you need to do an operation. It And it has something called a SOUL.md file. So let's see what a SOUL.md file is. It's what gives Maya the unique identity of a MechContext contact center agent. W-what's the skill here? We have something called a loop closing skill.

34:14 A loop closing skill is what allowed it to do the follow-up in the first place, even after the issue was resolved, because that's what's unique about our contact or contact center and how we run our customer operations. It has all of our semantic models, which allow the agent to understand which columns to query, which tables to query. Now, all of this comes into our context layer. Finally, imagine Maya when she joined a new job. She was not first put into doing all of these calls. What Atlan offers is a way to simulate.

34:48 Just like Maya would go through training, she would go through mock calls with her seniors, with her managers, Atlan runs simulations of different scenarios so that you know how your agent is performing even before you deploy it to end customers. This gives builders a sense of confidence of how the agents are performing in, in a synthetic simulated environment. The reason we are able to do this so well is because Atlan has a deep contextual understanding of both your data and business graph, and all of that is leveraged to be able to create all of these scenarios for simulations.

35:23 One of the core beliefs that we have in Atlan is of openness, interoperability, and that's where a context repo becomes super useful. We have something called a lakehouse, which Varun will talk further about. We also have our Atlan MCP, which we showed you, which is compatible with, or with ChatGPT, with Claude, with Gemini, with any of your coding tools, also with all of the major AI frameworks. We also can help you deploy your semantic models to things like Snowflake Cortex Analyst or to a Databricks Unity Catalog. And we're building support for you to be able to sync these files to GitHub and your S3 lo-locations as well.

36:05 Now, for any agent operating in the real world, the, the learning does not stop at them, you know, doing their job. It comes through daily feedback that comes through their managers, through their peers. That's why Atlan has observability. What we do in observability is that every single trace that the agent is actually performing is captured here. And once it's captured, it creates a feedback loop for you to continue to improve your context layer. What we also offer is a way for you to deep dive into the trace to understand what the agent was doing, what parts of the context layer it's touching, what tools it's using.

36:44 And for anyone debugging an agent, it can also look at and understand what the reasoning trace was. Now, looking at hundreds of thousands of traces is really, really hard. What Atlan does is that it actually watches over every single one of your traces and identifies patterns that you can easily go and fix to improve your context.

37:04 So, for example, just like Maya would have sometimes associated the wrong tier to a customer and leading to a bad refund, similarly, if your agent does the same operation, Atlan's observability identifies them and gives you a way to improve that improve the context so that it doesn't repeat, building a self-improvement flow that can help you improve the quality of your context as more interactions happen. So that was all for the demo. I'll throw it back to Varun for the next bit. Thank you.

Varun Banka

37:37 Thank you for doing this. So what Anirudh talked about was how do we use the context layer to accelerate and build trusted agents. And we realize there are people in both spectrums, right? There are people who are building these agents in GitHub repos, so they need, they need... Or in their own IDs or in cloud code, they need just extensibility, portability. You brought You saw connecting to Atlan's MCP to actually drive that. So if you're a builder, you love that, you can do that. If you're someone who likes more of the cloud cowork world, you can also take in the Atlan's experience what you saw, to actually help you build those trusted agents.

38:13 Now, not only that, we help you create... Help you version this, understand how each of your context is upgrading so that you know how your skills are evolving. What was it before that? What did the agent right now talk to? So this is context. Context Studio is our IDE for both builders and both for operators.

Varun Banka

38:34 Now, as part of the next step, I have Jesse, who've joined us, who've been using— with Elastic, has been using similar extensibility capabilities to drive use cases at Elastic. And I'm gonna pass on to Prukalpa back to have this amazing conversation with Jesse. Jesse, welcome aboard at Activate.

Jessie Buelteman

38:51 Thanks, Varun. Thanks, Prukalpa.

Prukalpa Sankar

38:53 Hi, Jesse. Thank you so much for joining us today. I just wanna start by saying thank you. Context Engineering Studio was built as a labor of love, but it was honestly built because of partners like you that allowed us to experiment and work with you. These are hard problems. How do we take skills and tacit knowledge and expertise that humans have inside their organizations, and how do we, how do we actually translate it into a way that AI understands? And so really, really grateful for the partnership. Welcome to Jesse, I had a couple questions that I wanted to walk you through. Set the scene for us, right?

39:28 Like, tell us a little bit about the problem that you're trying to solve at Elastic.

Jessie Buelteman

39:34 Yeah. First, thanks for having me, Prukalpa, and working with the team has been, been a great partnership as we continue to evolve and try to drive these capabilities. So appreciate the team support there. So what, what we're trying to do at Elastic is enable talk-to-your-data use cases specifically in our finance organization. Right? Just given the nature of, of the data, the accurate— the responses need to be very accurate to, to make sure that we're able to provide good responses and not have that come back to the data teams to, to self-resolve or, or tell them why it's not accurate. So that's a key component.

40:11 And then additionally, like, reporting at Elastic is democratized across analytics teams. So any approach we build kinda has to be a repeatable and extensible across the broader organization. And as we look at that, right, we, we firmly believe that we need to start in the data layer. So the data layer's like the context foundation that's gonna make these raw facts really interpretable and usable by any machine without any human intuition. So we have a focus effort to get our data layer kind of up to par so we can move on and build out robust semantic models.

40:52 And that, that's just work that we're doing retroactively to get that up to speed and then moving forward, kinda how do we... Like, we wanna define those standards for what good context is and then afford— and enforce that before promotion to production. So moving forward, right, we have that ability to, to kinda streamline our process and go build semantics without going back constantly and build out our context, context layer. So all that, you know, that, that will give us the, the foundation for the semantic layer. And then that's helpful because all that context now lives in the data layer where it belongs. So we're no longer hard coding prompts or instructions. Those become... Are brittle.

41:39 They're hard to maintain. They're siloed, so you could create the same instruction and prompt. So really we wanna make sure that's available so when we build the semantic models that it's, it's across that level.

Prukalpa Sankar

41:50 Got it.

Jessie Buelteman

41:51 So as we do that, right, all that information gets there. So how do we think about creating the remes— robust semantic model, and, and what do we need to achieve that? Then it comes into, we need a solution that's open and interoperable, right? Our assets, they, they live across multiple tools. We don't want to necessarily have a lock-in, so we need a solution that can kinda work with us and, and scale as technology evolves use cases pivot. And so that's kinda our approach and where we're, where we're tackling this.

Prukalpa Sankar

42:25 Awesome. Okay. So it sounds like how do I create... How do I drive accuracy is one problem. The other is how do I make it open and portable, and how do I make sure it's, it's built for change in this world that's evolving a lot? How are you thinking about this framework internally? Like, what does... Like, what does the framework look like for building this? At, at Elastic Yeah, it's...

Jessie Buelteman

42:50 So first context engineering, like that— I would say that's our prerequisite, so we kind of talked about that. That would be the prerequisite to go into this. And then we'd probably break it down into, like, four different phases, starting with, with build. So we want one place that takes all of this context and it's not just one dimension. It's not just dbt or you're looking at your BI assets, right? You're able to scrape all that information, have all that underlying information to build that, that semantic layer. 'Cause we want all of it to build these robust models that you don't have to keep reiterating and, and have multiple versions to maybe answer multiple use cases.

43:27 So the build part is, is really having all that underlying information, and then the validation kind of as mentioned earlier, right? That, that's critical because when we, when we want to ship something, we want to ship it with confidence and know, "Hey, when we give this to our business users, if you ask this question, there's a very good accuracy response that, that you're gonna get." And it... Or let us know, "Hey, we built this model. We ask it this question. Here's the, here's the gap. Here's how you address it to get to that, you know, that threshold that, that you're looking for." And then finally on the governance side when we want...

44:07 We have a very good structure in place, and we don't want to build from scratch, so... Or re-reinvent the wheel. W-we already have our GitHub and dbt structured. This has co-ownership, so when we ship, we want to follow that same review approval process, that it just naturally fits in what we're doing for SQL models. We want to kind of implement the same for semantic models. And then the, the last component would be, like, the automated refresh. Obviously things change daily. New assets come in. Questions might evolve. So how do we get that prompt or that notification or just to automatically scrape additional information that would come in to keep that semantic model fresh?

Prukalpa Sankar

44:49 Gotcha. That makes sense. Yeah, I always like to think of this like context as a living compounding system. Like, underlying systems change, business changes, everything changes, and so how do you keep it alive in this, in this new world? I have one last question for you, Jesse. What does getting this right unlock at Elastic? Why does this matter?

Jessie Buelteman

45:07 Yeah. It... K-kind of just hinging on the, on some of the points above—... But again, the validation, right? So it is very critical because just the users across our organization that would be leveraging this semantic model, that it is accurate. So and maybe taking that a step further, right? We'd almost want to... We ship a solid semantic model, and then you get a response. Even in that response, if, you know, prompting it to say, what, what's the, the reasoning that the that the agent went through to derive that answer? Were there any assumptions that were made? What were the data quality signals, right?

45:48 All that information's available, so I want, one, I get the response, and then tell me why I should trust it. So that's really important, and, and the reason for that is it provides the users with self-serve capabilities with confidence. So it's not like, "Hey, I asked it a question," and then it comes to the data teams to validate that. So that's one component that I think will just alleviate the work on, on the data teams to constantly and provide that self-serve capability. And then on the governance following that same in-infrastructure, right? The, the reason that's super important, we'll be able to ship faster.

46:21 You just think from a development standpoint, we have that structure in place. We want something that kind of will, will fit into that. And we're able to ship and move things quicker and open it up for more people to use. So the more people that can use it, they already have their own review. You know, a marketing model might be different than a sales model, so the the proper people that are reviewing that code approval just follow that same infrastructure. And it just will eliminate development time and be able to ship things faster.

Prukalpa Sankar

46:51 Okay. And the, the last component, Prukalpa, real quick, would be on the consumption side.

Jessie Buelteman

46:55 As we build these, and then we want to make that available, right? Are we able to ship that in a place that's generally available for all the users, whether it's our data warehouse or GitHub, right? We have multiple personas, so any team, any tool, same answer is what we want to try to drive. And that, that's really a c-component to unlock this.

Prukalpa Sankar

47:14 Got it. I think that's a mic drop moment, Jesse. Any team, any tool, same answer. So with that, thank you for joining us at Activate, and I will hand it back to Varun for the last part of today's Activate.

Varun Banka

47:25 Thanks, Prukalpa. Thanks, Jesse, for joining us today. While the context window is okay, but based on my time clock, the time window is running out, so we have five minutes.

Varun Banka

47:35 So what I'm gonna do is hopefully rush through some of this, but feel free to add your questions there. Okay, the last part of this is around portability. If you remember the block that we talked about, the middle block is how do we make it extensible? And we want to talk about open context ecosystem. One of the common things I hear from all of our customers is the fact that everybody wants to be an agent layer. But the biggest problem is context engineering is happening in silos. And I'm sure when I'm saying this there is something going on in everybody's head that, "Hey, we've seen this play out with data."

48:04 There was BI sprawl, and the same thing is happening with AI, like there's agent sprawl. So what, what we believe is all the context needs to be interoperable irrespective of where agent gets built. And for that, we focus on interoperability. I'm gonna talk about three things. One, app framework. This is our app tooling which makes it easy for our partners, ecosystems, or even our customers to actually build applications and solutions on top of our context layer. We've been working with some amazing partners in the last six months. We have two amazing partners who, who are gonna join us via video, and we'll play the video for them. Second, we...

48:39 Anirudh talked about portability on Context Lakehouse. When we started this journey, we realized that we had to reimagine the infrastructure that was built for context layer. We had to learn from something that has already worked and scaled, and I'm sure all of you have used data lake houses, warehouses. So we took the same storage separation of compute, we have taken the same open interoperable file formats like Iceberg, and you can bring your own compute to leverage every context that you see in Atlan in your favorite Iceberg-compatible compute. Third, it needs portability. Builders are building everywhere. They have their own harness, they're building their own tooling.

49:13 We want to make sure we provide all the interfaces, whether it's SQL, API, MCPs, and we are soon launching file systems as well. So thank you to each and every one of our partners to helping us shape the marketplace. We have two partners today, Cyera and Immuta. Cyera was gonna we are gonna play a video around how they used their data classifying engine to actually classify the columns that I talked about. And Immuta, that they're gonna talk about how they use their dynamic policy enforcement to actually make sure the data is secured in any downstream tool.

Cyera

49:43 AI agents don't know which data is sensitive. They don't know who should have access. They don't know what compliance obligations attach to a column before they use it. This is a key part of context that's missing as we deploy AI. Cyera's classifications flow directly into Atlan, attached to the exact assets where humans and AI already work. Here on this Social Security number column, customer US SSN from Cyera, now living inside Atlan as a structured tag, wired into every downstream policy and AI workflow. At the column level, agents now inherit sensitivity context automatically. But zoom out to the database and the picture deepens. Security issues from Cyera surface here.

50:25 Credit card data stored in plain text, overly permissive access paths. With the compliance frameworks in scope, PCI, SOC two, and agent reasoning over this data doesn't have to infer risk. We're excited to bring Cyera's data classification context into Atlan's enterprise context layer to help build agents that are policy-aware. And this integration is live on Atlan's App Marketplace, built via the App Framework. The framework helped us build a first-class integration quickly and easily with standardized APIs and SDKs.

Immuta

50:58 Hello, everyone. I'm excited to be here to discuss the new Atlan Immuta request for access integration. So as you can see, I'm logged logged into Atlan here. This is my context layer to understand all data that's at my fingertips. I'm an analyst that needs to do some analysis on patient data, so I'm able to find this patients table in Snowflake. It looks perfect to me. It's got all these columns that I need. So I'm gonna go ahead and try to query it in Snowflake. So now I've left Atlan, I'm in Snowflake trying to run a query, and you can see I cannot. So lucky for me, we have this Immuta integration turned on.

51:32 That's why this button is here this Request Access button. So if I press this button, it's going to direct me out to Immuta to actually request access. And these request forms are completely configurable. I've got some questions in here that I configured. I also set up who does the approving for this particular asset. So I'm just gonna say research is my read— my reason, and then I love this integration a lot. I just wanted to put something silly in there just to prove that these forms are completely customizable. You can also add data use agreements that users can agree to. So I'm gonna go ahead and submit this request.

52:12 And when these r-requests get submitted, you can see we have this pending request here, that will fire off notifications in Slack and Teams to the appropriate folks that are assigned to do this approving or not. So let me go ahead and open up one of those people which happens to be me. So if I go ahead and refresh this screen, we should see Ruby's request for access. So you can see her answers to the questions. I can see what data she's requesting access to and if she'll be approved or not. If I approve it, there's ways to set rules to block people from getting access through Immuta as well.

52:51 I can see the data use agreement that she agreed to. I happen... So you can see the reviewers here. There's actually two sets of reviewers, someone from country United States and someone in the Department of the Office of the CIO. These can also be groups or permissions. You can assign these reviewers many different ways. I happen to meet both these criteria, so when I go to do this I'm gonna be able to do it for both. I'm gonna be able to approve this request holistically. So another cool thing that we do is Immuta monitors all the historical approvals. And as you can see, twenty-three similar users have been granted access and nobody d-denied.

53:30 Similar users is based on the attributes and groups of the requester. So this is saying low risk and to go ahead and approve. You can see you could also temporarily approve or deny. So I'm just gonna go ahead and submit this determination. Okay, so now Ruby's approved, and this is where we get to the aha moment of the demo, if you will. This does not create a ticket where some admin now had to go into Snowflake and add Ruby to something. In fact, Immuta is in Snowflake and pushing policies down into Snowflake, native policies in Snowflake. And this could be Databricks, Redshift, BigQuery, whatever data platform you're using.

54:08 I just happen to be demoing with Snowflake here. So if Ruby re-queries this patient table, she immediately now has access to it. So that's pretty exciting. We can see how we went from discovery at the context layer all the way down to being able to actually query data. How does this actually work? It's through the the Atlan App Framework. So we integrated this. So if you go to Workflows, you can see if you search for Immuta, you can just turn this on, and that's what gave us this very first-class button. That's why we were able to put this first-class button in the Atlan app. But Okay, now let's switching gears for a second.

54:55 Let's take this demo to a new level. Let's start talking about agents. Everyone's interested in how we, we manage agentic access, and how does the Atlan context layer plus Immuta fit into that story? So I'm gonna actually open something else now. I'm gonna go into Claude and Claude is a. And my face went away there. But Claude, it, it could be Claude or it could be anything. I just happen to be using Claude for this demo whatever you're using for your text to SQL interface. But my first question's gonna be, "Hey, I need to ask a question about customer orders. Do we have relevant data to support that?"

55:33 And what this is gonna do is it's gonna reach out to Atlan over MCP and, and figure out if there's data available. So I'm gonna go ahead and allow it to do this. And it says, "Hey, look, you know, this is what I found in Atlan. We've got a customer orders fact table and a shipping rates lookup table." I'm like, "Okay, this seems like what I need," so I'm gonna go ahead and ask a more detailed question of this data. And this is actually where Immuta comes into the picture.

56:05 We have a capability called agentic data access, where when a user asks a question of data Immuta can look up their access and actually vend a role or a group, depending on the data platform, that represents that user's access on the fly. And so this allows you to have your business users come through these AI tools to ask questions of data without having, have— with, with zero standing permissions in the data platform. You can simply define policies in Immuta, and we will vend that access on the fly for the agent scoped to what that user is allowed to see.

56:43 So this user that I'm running this query with is allowed to see this table, but as you can see, customer name's redacted. And it's saying, "Hey, customer name's masked. Do you wanna see it unmasked?" And I could say something like, "Yes, I need to see customer names. Can we, we request a masking exception?" And so this again is gonna go through that request workflow, but instead of getting to it through like discovering the data in the Atlan UI, we actually discovered it through the AI LLM interacting with the Atlan context and Immuta knowing that that customer name is masked. So I'm gonna go ahead and request a masking exception.

57:30 So this is gonna allow me to see that masked data in the clear. And you can see this is a separate request form. And in case, in this case, there's actually a rule here that is granting this user access for one hour automatically because I chose break glass fraud investigation as my intent. So again, this is updating the policies and that vended role that was given to the AI agent through Claude via Immuta will see that update, and we'll be able to see this redacted column here, customer name, in the clear. So we could say, "Looks like I was automatically approved. Can we see that query again?"

58:20 And then this will just go ahead and run, and you'll see the the data in the clear. Because again, Immuta is updating those policies on the fly and vending that access to the agent on behalf of the user here. So as this is running, just to recap you have Atlan as your context layer. It understands what data is available. We have Immuta as the policy layer, understanding what data is available, but potentially not for that particular user. We blend these two things together.

58:52 We're able to do these access requests on the fly and, you know, that, the, that context, that, that data context from Atlan and the policy context from Immuta is allow, allows us to do this very intuitive workflow, AI workflow, and you can see now, we can see the users in the clear. So great. That's, that's the demonstration. Love to chat more with folks that are interested. Again, a lot of this made possible with the Atlan app framework was, which was a very powerful way for us to integrate that button directly into Atlan. But thank you for watching.

Varun Banka

59:25 Context should be open and interoperable. We talked about lakehouse, and I'm just amazed to see the kind of reads and how everybody has plugged our lakehouse into their agents, workflows, MCP tools. We've had eight billion reads in the last ninety days and growing exponentially every day. And it's built primarily for the world where agents are gonna be the primary consumers of context. Let me just give you a sneak peek of what this feels like. All this means is I'm on a Databricks instance, and Databricks is one of the Iceberg compatible computes. Whether this is Snowflake, Brama, you could use any of them.

1:00:05 And now as soon as you connect the lakehouse, you see every context that we saw on the UI, the MCP used, everything is available for you to do, run SQL queries on. This is same like how you run your SQL queries on your data, you can now run SQL queries across that. You can see some of the skill artifacts that Anirudh showed already here. Let's go to next part of our portability, MCP servers. You saw Anirudh talking about how he used Atlan's MCP server embedded inside Cursor. At Atlan, we've always believed to take Atlan to where our users are.

1:00:38 We, in pre-LLM world, we built Chrome integrations to take our take our Atlan to every interface where humans have data ware. We realize now our users have changed. There are agents. There are places where agents are getting built. So we are bringing and activating Atlan to every AI interface that is available, where people are building agents using data and driving their operational outcomes. In fact, just yesterday, we have our MCP servers now directly available in the marketplace on Cursor and Visual Studio Code, and you can use MCPs across all of the logos that you see right now.

1:01:14 We brought in the same learnings within Atlan as well, and we are seeing some amazing kind of questions that we were never able to answer before for our users. And we are dogfooding our own context lakehouse to actually power some of this. The kind of accuracy evaluation that we are expecting and we are seeing with some of our customers is amazing. So thank you to each and every customers who've helped us shape this. Context is just not king, but context is your IP. So with this, I wanna just thank each and every one of you who've joined us across globally, joined us in this journey. Reach out to us, ask us questions. I saw...

1:01:47 Thank you for all the questions. Happy to help. Let's build this context layer together and solve the Maya's problem. Cheers folks. See you next time at Activate.