The Enterprise Context Layer Architecture in Action

Daniel Bergquist

0:00 All right. Howdy, everybody. We'll go ahead and get started. I have a, a short presentation for you, and then we'll jump into how Atlan can, can work with the, the context repository. Or or with, with context and help you guys have co-have context for your AI, your AI use cases.

0:38 So one of the first things we like to go over is we, we've learned as we've been working with our customers that our customers a lot of times all kind of hit the same stages as you're, as you're building out context within your environments. So the first stage being we've got to catalog and govern. So how do we go and find our data? A lot of times and a lot of our customers the way it ends up working is our, our customers don't actually have a great way of finding the data across their environments.

1:04 At my last company I was at, we had over forty-six thousand databases, and let me tell you, finding them was a lot of times a tap on somebody's shoulder like, "Which, which table am I supposed to be using?" And so what Atlan wants to do is help you. First, we wanna help you catalog and govern your, your tables so that they're findable and they're searchable in a consistent way. And then how do we make your data AI-ready? And we'll go through some of the attributes of what AI-ready means here in a moment.

1:29 But how do we make your data searchable not just by humans, but by an actual, you know, an actual agent that needs to understand and find the right data across your enterprise? From there, how do we make sure that data is contextually ready so we can then present that to an agent, and that agent can use that, that, that context within its environment? And then how do we build it out? How do we make more of them? Because in, in historic— if history repeats itself, once you build one thing that's really good, everybody wants you to build ten more of them.

1:59 And so how do we then scale those, those agents out so that they're more usable and, and that we can make more of them faster? And that's why we wanna make— that's what Atlan does, is to make a one unified brain of context to help expose that for all the agentic use cases that your companies might need.

Daniel Bergquist

2:16 And so what is, what is context? What is context to Atlan? Well, context is a com-combination of knowledge, expertise, and norms. And so when you look Let's say we're, we're talking about like a, a, you know, a, a, we call it our McContext burger. Let's say we're talking about a burger franchise, and you have a franchise owner, and the franchise owner asks a simple question like, "Why is our drive time— our drive-through time up this week?" That, that seems really easy, but at the end of the day, when an agent's, when an agent is asked that question, if the agent doesn't know the definitions to each component, like what is drive-through time?

2:50 What does this week mean to you? Your agent's going to quite literally make it up. And once that starts making it up, although sometimes it's right, if it's not consistent, that's not, that's not a helpful, that's not a helpful agent. So when you think about knowledge, well, what does drive-through time mean? Well, it means average drive-through seconds, not a point-of-sale number in finance. Or this week, well, this week is Monday through Sunday, starting at eleven fifty-nine PM in a rolling day window. Expertise, how do you handle things? What's the diagnostic process? How do we investigate root causes?

3:23 And in norms, what do, what do we understand about who the, who the store manager, what he can see, what type of metrics are the most important? Or even like think about if you need to process refunds. How do you process a refund? What's an acceptable version? And these are all the things that we wanna make sure are defined so that your agent can consistently go and have a, a single source of truth for context to go and refer to that data and that information and, and answer the questions correctly every single time.

Daniel Bergquist

3:51 So we talked about it for just a moment, but what is AI-ready data? Well, AI-ready data is data that's described, that's trusted, that's traceable, so we know the lineage, we know where it came from and where it's going. It's governed, arbitrated, and portable. So these six components toge-together can make it, can make us have the ability to have data that is very much usable by an agentic use case. Without, without some of these components, that's where you start to have kind of that, that that part where your agent's gonna have to guess.

4:24 You know, I had an experience where it was one of the customers that we were setting up some, some context for, and they had built this really cool agent internally to answer a lot of their, their, you know, executive quarterly report-type questions. Well, there's a certain set of metrics that within, in, internally that they were actually purchasing from an external company. And so those metrics, how those numbers were actually formulated, they didn't know because they bought them. Well, the CEO goes and asks the agent, "Hey, tell me about, tell me about where this metric came from." And that agent was so helpful and so wrong.

5:01 And so the next meeting, the, the CEO went and explaining how this number was made up, and it turns out one of the, one of the people in the room was like, "Well, well a-actually, we, we bought that from an, an external vendor. We don't know where it came from." So part of our goal when we came in to help them was to build a skill for their agent as part of the, the Atlan context that actually referred to the sources that, the sources that were vendor-managed.

5:27 And that way, if anybody asked that specific question, anytime that was asked, the, the agent would respond with, "Well, you need to reach out to these people because that came from an external source." So less, less awkward moments had in meetings after that.

5:41 From there, let's look at how Atlan's actually gonna do it, and this is kinda gonna be the flow of our demo today. So the first thing we've gotta do is we've gotta actually connect to your... And mine the context from your, from your different systems across your enterprise. So we need to connect to your data warehouses. We need to connect to your source systems, understand the ETL, how it's flowing from source to data warehouse, and then how it's being consumed at that reporting and visualization layer.

6:09 That's gonna help us go and build out the map that that helps us understand what your business— how your data is actually flowing across your business and how that AI-ready data, then how your business processes then layer on top of that. That'll then help us to create the semantic relationships and build skills that can be used as part of your agents through your context activ— through activating your context. Something very important to note is across this entire process, governance is crucial to it. So we have all of the governance for context. We have, we have lineage capabilities.

6:43 We keep versions, and we also have approval workflows to ensure that your context doesn't change just by anybody. That way, there's still management, there's still approvals, and you can keep this, this shared knowledge brain unified across your environment.

Daniel Bergquist

7:00 All right. From there, I think it's time to jump over to Atlan. So I'm gonna slide right on over. So this is Atlan. And when you're first building your system, like we mentioned, we wanna first go in and mine that context. So we'll go to our connector marketplace, and in our marketplace, we have, we have connections to well over a hundred different data sources where you can go and search, and we want to natively connect to as many sources as we can.

7:30 We do, of course, have ways to, to pull in, you know, not necessarily native connections, but but that's that's through our API or through other sources like open lineage and other ways of doing it. But specifically, let's check out our Snowflake, our Snowflake connectors. So our Snowflake connector is where we tell our customers to start with. And the first, the first connection we would do is connect to the Snowflake assets. Now, an asset to Atlan is more of like a structural object within Snowflake. So your tables, your columns, your, your views and your processes, things like that. We need to understand those, so we go, and we crawl those and pull them into Atlan.

8:10 Each of them will then get their own asset profile. The next thing we need to do is to mine those, those assets and mine those relationships. So our miner workflow then goes through and will pull out the query history and the account usage tables from your data or from your Snowflake environment. So that way, we can actually create the lineage and create the understanding of how your— of how those, those transformations are flowing across your system. And I'll show the, I'll show you that here in just a moment. But these workflows are all schedulable.

8:43 So you will, you, you put them on a schedule, and then those can be scheduled up to, you know, once a day, once an hour. Just depends on how you, how you want to integrate Atlan with Snowflake and how often you're really making metadata changes. Once we've gone and made those connections, and from here, we would also, you know, before, before, before we have what I'm about to show you in our lineage, you would need to connect to your other systems. So if you have DBT or you have Fivetran or SQL Server sources, we connect to each of those and run these, run these crawlers. From there, we'll then get our asset profile.

9:18 And this is a way that we can ensure that our data is AI-ready because this, this has the trust signals that will show that this is a, what this asset is, giving it descriptions. We'll look at the data quality that's associated to the asset. We define owners. We can even showcase the usage. And then we also have tags and information that we can pull directly from Snowflake. And then with Snowflake, for example we do a reverse sync tagging. So we can connect from Snowflake to Atlan and then Atlan to Snowflake.

9:51 So if the tag exists in Snowflake, when you tag an asset, another Snowflake asset in Atlan, the next time that workflow runs, that Snowflake tag will propagate back to Snowflake as well. So we do have that reverse sync capability. We also have the ability to propagate these tags throughout throughout your environment. So for example, if you have need to make sure that certain, you know, GDPR tags are propagated out to the actual reports that are connected to the Snowflake asset, we can actually propagate those through lineage, and we can do that in bulk. We can do that individually here, like I'm showing.

10:26 But this way, if you need to pull a report about all the tags that are associ— or all the assets that might be associated to a particular tag or attribute, you'll actually be able to do that and pull that information directly from Atlan. We also wanna make sure that we have, we have additional described described terms from our AI, for our AI-ready data, where we can associate busi— associate business glossary terms to each asset. Now, I'll show in a moment how we actually help you do that because we have, we have the ability to help enrich data for you to make, to make life a little easier.

11:02 And we can actually associate terms to the individual assets as part of our enrichment process. But then additional terms can be added, so that way you have your, your consumers know all the, the, the definitions that the business is expecting them to know about a given asset. And then, of course, we need to make sure that that data is trusted. And so even whether, whether a human wrote this information and described this or we used Atlan to enrich the data, we wanna make sure that a human or someone has come in and verified and certified this data so that we know that this data is ready to be used by people and by agents.

11:38 And then additional details can also be added, where we have timestamps and other really, really cool things from Atlan. So let's take a look at lineage. So part of what we also are capable— What, what we do is when we run those workflows, we also get the lineage built out for each of those assets. So So at this point, we've connected to your, your systems, and then we've actually will go and automatically create the lineage so that you can see lineage all the way from source out to your, out to your reporting layer. This can come in handy 'cause this can be this can be visualized through Atlan itself.

12:16 This can be visualized through our MCP integrations into many, many tools like Claude or, or Copilot Studio, et cetera. And then we can get that column-level view and see how each column flows across your systems. We also will Because we're, we're getting When we run that, that minor workflow, we're also getting those processes. So we can see how those transformations are actually created, and we'll see the SQL behind them. Other components that Atlan makes sure to, again, build trust and build those, those AI-ready data and make your, your data searchable is part of what we also will do is create the usage and intelligence where we'll actually go and we pull those, the query history.

13:01 We'll help reverse engineer the the questions or the, the SQL that's being asked on these tables and present those present those on the asset itself. So when it's time to find context for an agentic use case, and we wanna use Atlan, Atlan to do that, we'll be able to go and, depending on your use case, go and choose the assets that are most relevant, that are already answering the questions that you might be using in a BI report or that might be being queried by people across your organization. We'll also We'll also go and find the common joins and filters that are associated to those assets as well.

13:40 And then from there what I wanna go next is to how Atlan will help enrich data.

Daniel Bergquist

13:47 In my experience in my experience, people don't necessarily enjoy writing descriptions and writing readmes and doing kind of the, that, that, that governance and documentation work. Where Atlan came in is we, we decided to help with that. So what we've created is our context agents. These agents are So these agents, befo-before we run them, what happens is after we make the connections to Atlan, Atlan will bring those collections together, and it will look for related assets, and it's gonna put those into collections on its own. Now, you can also create collections, create your own collections by through tags and through custom metadata.

14:31 And so what that ends up doing is it gives us the ability to use our, our context agents to then go and run information and build out your descriptions. This is table and column-level descriptions. It also helps run that SQL intelligence that I showed earlier. And it will also help you to enrich and build out very verbose readmes. So what that looks like is first we look— we wanna make sure that we understand the plan. So what we're gonna give you as part of that readme enrichment is we give you the ability to, to bring in your own templates and then review them as well.

15:05 So you can choose our own, our template that we, that we create as part of Atlan, but you can also bring in custom templates f-to make sure that that or to that fit the needs of, of your use cases across your enterprises. From here, then you can select what percentage of this collection's asset to enrich. So how much do you actually want to expose? And then and then you can also give custom instructions to our agents.

15:35 So think of it as you can engineer a prompt to help understand the industry, what strengths, how to handle certain like certain attributes or what attributes that you wanna have part of your descriptions or your readmes, and you can have those as part of the custom instructions for these agents. And what does that look like? Well, let me show you an example of an asset that was fully enriched by Atlan. And I promise I didn't type anything in here. This was all one hundred percent enriched by Atlan on its own.

16:08 So what we did was first Atlan, we ran the descriptions, and we were able to enrich these descriptions where we have where from the table-level description, but also the column-level descriptions. And these are more than just your, your kind of run-of-the-mill descriptions. Since we understand your your, your data across your enterprise and we know how that data flows from source all the way out to destination, we're building these descriptions based on how the data actually runs and why these columns are important. So these get much better than just building a description based on the character type and the column name.

16:45 We also looked at your business glossary, and we were able to go and find related glossary terms and automatically apply them to the, to the table itself. So in this, in this particular in this particular example, we have an adverse action decision table, and then we were able to connect an adverse action notice glossary term directly to this table to help, to help, to help the, the, to help users or AI agents understand what this data actually means and what is associated to this table.

17:17 The next thing we did was we also developed a, a, the readme, and this is and this is a, a very verbose readme, and we've actually, we've gotten some feedback because sometimes the readmes are really long. Now, the, the good part about that is it gives detail, but sometimes humans are like: "Why did you write it so long?" Well, the intention of these, these readmes is not necessarily just for a human.

17:40 The intention is that this is making your data AI-ready, and this is giving an AI, a-an AI agent the ability to go through and understand all the different components that it needs to understand to make the right choices and help build a context, help build a context window for your agent. That way, it knows the right tables to use, and it knows how to build out those details. So from here, we were able to put in a purpose to understand what's the purpose of this table.

18:06 The grain and keys, what type of data is in the rows and is in the, is in this table what metrics are associated to this table and used downstream, what different conventions and gotchas you might run into about the table, what, what coverage, so where does this da-data fit?

18:25 And then one of my favorite components that we do is because we know your lineage and we know how this data is being used across your enterprise, we we also find the key columns, so we can locate the, the most important columns of a given table and why they actually matter to the table itself. And this is, this is so crucial 'cause this will help to build out the, the semantic models and then help enrich that information for you. And then we'll also identify those limitations and known constraints, as well as identify the FAQs, the frequently asked questions regarding these tables.

19:04 What we've learned is that for the most part, we've, we've now enriched well over a million different a million different assets. And what we've learned is that customers are loving this, loving this component because they're— we're getting between... We're getting r-right around a ninety percent acceptance rate on all of our generated descriptions and our generated our generated readmes. And now, why does, why does this matter?

Daniel Bergquist

19:33 Because having that trusted data, having that AI-ready data helps you prepare for the next component, which is actually building out an agentic use case. So one of the, one of the next components that Atlan has, we call it our Context Engineering Studio. And so from here, this is, this is something we are actively working on, but what Context Engineering Studio does is it enables, it enables you to describe an agentic use case that you're looking to, to build. So whether that's a talk to data use case or a contact center agent, what is the agent that you're looking to build? And then how, how are...

20:11 And then what we'll do is we'll help give you the information and run through that information to to build you a repository that binds that context that we've built in Atlan, so that your agent now has the true, the, the, the true knowledge that is associated to your enterprise that it can refer to each time it, it has to take an action, and whether that's an interaction with the user or it's an automated process, it knows the exact business knowledge that it needs to know to answer questions or perform its duties properly. So what that looks like is first it starts out in kind of a chat build.

20:46 So I, I was, you know, in, in reference to our earlier slide where we we were asking about the drive-through time, I asked for a, a context repository from a context burger operations analyst. So what Atlan did was it looked across my data sources and my data assets, and it started to find informa— find data that would be associated to this, this use case of building this operations analyst. From there, it went and found those sources, and it starts to build out a repository. And the repository looks a lot like... Well, it looks just like— It looks like this.

21:23 And so a repository is full of semantic models where we're pulling the information from your glossary and from your different data sources to build those dimensions and facts for each of those semantic models for this tables. But then we also take into account building skills that the agent needs to know. And so think of the skills like, like how do you... How would you train a brand-new employee? What data would you give it? Let's say you have standard operating procedures, or you have policies that it needs to know.

21:52 You can tell the, tell Atlan about those policies or add those in as PDF files, or we call them knowledge files and knowledge folders, where where then Atlan can take those, ingest them, and then start writing skills for how your agent needs to handle certain problems. Like, for example, this skill is about how to handle drive-through performance. So this is where it will actually run and give and, and know what metrics need to be produced when a when someone were to ask the question of, "What is my drive-through?" You know, "What is my... Why is my drive-through time down this week?" Well, first we need to understand what drive-through is.

22:27 So we have a skill that's related to drive-through drive-through performance analysis to better help to better help our agent be able to provide the information, in this case, to a franchisee that might need to understand the component of that. From here, we also wanna make sure that you can test these context repositories within Atlan. So we build out the ability to have a simulated test questions. We will also allow you to build in your own connect collections.

22:56 So if you have specific questions that you're expecting this agent to be able to answer, we can run those tests, so we can help you bind that context and bootstrap it together, but also test and go through the testing process all within Atlan. From there, after you're done testing, we need to make this context available, and this is out referencing to that, kind of that level four. How do you, how do you make this scale? Well, you make it scale by being able to port this context to any of the, any of the applications or harnesses that might be running your agent.

23:29 So whether that be Snowflake, whether we're connecting to OpenAI, Claude or, or Gemini, we wanna be able to have this context as a universal context layer to provide, to provide a bounded context window for any of your, of your agentic use cases across, across your enterprise. And then we also what we also wanna make sure is because we have the governance and we have this layer, so, like, for example, if we go into the glossary, and you have a term like customer. Let's say, let's say, you know, six months in, the, the definition of customer changes or gets modified, and that's something that might be impactful for for your context repository.

24:17 What we'll do is we will notate that and allow you to version this information based on based on, based on those updates. So because we've got those layers and those tracks, those traces into, into your, your context, when changes happen or when changes are facilitated, we're able to notate those and allow you to actually re-regenerate and rebuild this repository, and then push the new version out to your agent so that your agent can actually can actually have that, that relevant new information.

Daniel Bergquist

24:49 And now let's do a quick example of of what of what this looks like in practice. So let me refresh this. So, for example, this is this is regarding that drive-through time. What what, what would it look like, you know, what's the difference between running an agent with, with Atlan context versus without Atlan context? So in this instance, we need to understand the why. Why is my drive-through time up this week? Well, what's happening? So what we're gonna do is we, we run this. We're gonna watch on this side. This is without— this is an agent that doesn't have Atlan context. And on this side, we have Atlan context.

25:26 So when we looked at this side, the first thing we're doing is it, it goes straight to Snowflake, and it starts querying data based on ba-based on the Snowflake information. But what's happening is the, the agent is having to assume information. And anytime an agent assumes information is, is a time that you're probably gonna get some sort of hallu-hallucination. But it's not quite sure who the, who, who it's talking to, so it's assuming that this is correct. It's not entirely sure that it finds multiple different different metrics, so it's not quite sure which metric to actually report on. And then it doesn't have a benchmark or a baseline.

26:05 And so that baseline is something it even tells us that it's gonna be inventing. So it does look through, and it does find that, yes, there is, over the last four weeks, you've had a elevated drive-through time. I, I don't really know why, though. I can't confidently tell you why that's happening. Versus in Atlan, the first thing it does is it loads the persona, which is pers— it can be persona aware, and it loads, oh, well, we've got Marcus, Marcus J. Williams. He's a franchise operator, and he has fourteen thous— southeast stores.

26:33 It goes and looks up the glossary term for drive-through time, continues to find the related, the relations of from drive-through time for combo revenues, operators, and diagnostic playbook skill that it runs to then understand how to actually diagnose the why behind this question. And then it goes through additional models and skills that it runs to actually get the true answer to why is this happening. From there, what he's able to determine is what the agent's able to determine is based on the information, it turns out a, a, a new product was launched. And so because this, this product requires two two lanes, the single lane stores are actually getting an increase of drive-through time.

27:15 And so it's, it showcases the difference in the drive-through time benchmarks, and then it explains the different, the different window times, and it even gives a suggestion to, well, what we need to do is start doing a prep time additional prep for those single stores to make sure that we don't have that bottleneck anymore. And so with that, it can even do an estimate using some of the skills it's run, an estimate to see, to, to showcase the different revenue that might be at risk as well.

27:45 And so the, the idea is that with Atlan, you should be able to have the context and have the information available to you to actually to show, to show your— to give context window and context information to your agent so that you're not just scoped to answering the what questions, but the whys and the how questions, so that your business context gets layered in with your, with your data and your knowledge, and you're able to actually go and, and make these and, and make your agent smarter and make them more useful and most importantly, more consistent across your organizations.

Daniel Bergquist

28:23 All right. I wanna make sure we have some time for questions. I don't think I see any questions in the chat, but I'll, I'll give you guys a few minutes to look through to, to ask any questions you might have, and I can dive into those. We have, we have some time available to us. Is there anything anything that you'd like me to double dive to dive deeper into?

29:09 Any questions you might have or things, things that you all saw that might be interesting or something that you might want me to, to give a little bit more detail? We have a few minutes of extra time, so I wanna make sure that if there's any questions or any de-deep dives that you'd like me to, to dive into, I can do that for you.

30:01 All right. Got a question. So where does the Atlan model stored? Is it indexed from Snowflake during the ingestion? I'll, I let me break down that question. So if you're talking about our context agents models— Those are, those are models that we use.

30:20 We have a a couple different models we use. We'll use Claude, we'll use we'll use OpenAI, Azure's OpenAI. Those are, those are only gonna be looking at. They'll be stored, they'll be stored in the same, like, VPC that the data is that, that your metadata is stored in. So we'll leverage those models to then those those agents will leverage the metadata we have within your store, and we'll, we-we're only making we're only making, we're only enriching data based on your actual metadata that's within Atlan. We're not looking and we're not using public, public data. We're just making those enrichments based on the the data that's stored in your tenant.

30:59 And then from there we don't train or do any of that work on the models. We don't use any of that data to train. If you're talking about, like, the next component, so our Atlan models are only with, you know, our Atlan agents are being used directly within Atlan versus, like— Sorry. Oh, my bad. Okay. Oh, like, okay, I think I'm understanding. So, like, you might— I think you're referring to, like, how does it know how does the LLM models, during the context-layer run, how does the LLM understand what data models are necessary? Gotcha. So there's a few different ways. So we, we call it our context activation.

31:49 So you can activate the context. Think of your harness. You've got, like, if you've got Claude, for example, and Claude needs And you have an agent, and Claude is your harness. Your harness is gonna— You, you're gonna build that around an Atlan, or you're gonna build that around a particular use case. Your harness needs to know first where to get that context from, and that context can be delivered to Claude through a number of different ways. Most of our consumers will use our MCP connection, or they'll use our API connection, and they will reference the context first.

32:19 That context then delivers them these models and the skills and, and, and sets those to the agent, where the agent will then go from there after determining the context, so getting the definitions, getting these, the, the, the models and the skills from Atlan, will then use that in turn to pull from to then go to, like, Snowflake and pull the actual data from Snowflake to go and finish answering the question. So if you had a question like like, "What is my, like, what's my drive-through time?" Atlan's going to give it the It's gonna go to Atlan first to load the actual context.

32:53 And then from there, it'll get the models loaded that it needs to pull from in Snowflake, and then it will call Snowflake after calling Atlan to go and actually pull the relevant data information. 'Cause Atlan, we don't have your data. And so from there, we'll go and connect. We'll get the model from Atlan, and then we'll load, we'll pull the data from Snowflake. So think of it as another tool call to Snowflake after getting that context. Hopefully that, that got close to answering your question. Oh, yes, I got a cool. Thank you. Any other questions? Aw, thanks. Any other questions that folks have?

34:03 Well, all righty. I don't I don't see a lot of questions coming in. I don't wanna I don't wanna have you guys, you know, I don't want you guys to be stuck staring at me. So if there's not any questions, we can, we can wrap up a little early. I really appreciate you guys for, for joining, and hopefully we can get some, get, get you to join some, some of our next sessions or, or get to talk to you more more at a, at an, a enterprise level where we can actually come in and do some more customized demos for you. All righty, I'm gonna stop sharing and... All right. Thanks, everybody.