War of the Data Catalogs: Age of AI

Austin Kronz

0:00 All right. Welcome, everybody. I am so excited for today's session. We have two amazing people to talk about the state of the enterprise data catalog market, where it is, where it's going, what's happening with metadata. Is metadata bigger than data today? Really excited for the group. A couple housekeeping tips. This is gonna be an ask-me-anything type session at the end, so please use the chat feature, use the question and answer feature in Zoom, and we will start curating questions for the end for Jayesh and Prukalpa, our guests. But this is an Atlan webinar, and we know we need to get the audience involved early. So, quick question for the group as people are rolling in.

0:37 A little wrong answers only here. What will AI do for your organization? Let's hear from the group and get, get a little warm-up in before we dive into this amazing session. Answers should come in live here too, but the chat is already, already blowing up with great responses. "Take my job," the winky face of confidence there. Awesome. Yeah, this is great. Keep the wrong answers rolling in for what AI will do for your organization, but let's not waste any more time. We have such a jam-packed day today and I'm really excited for what we have for you.

1:20 So we're gonna start with Jayesh, Forrester lead analyst, to go through the AI native enterprise data catalog landscape. We're gonna quickly roll into co-founder of Atlan, Prukalpa, to talk about the future of AI native data catalogs. And what you all came for, a chance to pick these amazing people's brains and understand, you know, what your specific questions are and, and hear from them and their perspectives in all of the customers and conversations they're having with data governance and data catalog companies buying data catalog. So without further ado, let me jump right in and hand it over to Jayesh. Jayesh, thank you for joining us today.

Jayesh Chaurasia

1:59 Thank you, Austin, and the whole Atlan team for having me here. My section or the topic that I'm covering for just fifteen, twenty minutes is how do you navigate this AI native EDC landscape, which is quickly growing in in the market today. My name is Jayesh Chaurasia. I cover master data management, data governance, data quality, data observability, pretty much everything around data management-related topic, and enterprise data catalog is definitely one of that. So first thing, quick agenda for my section is that I'll quickly touch on the purpose and the importance of enterprise data catalog. What are some market trends that are leading up to this?

2:43 Some key considerations for the individual and the organization who is looking forward to evaluate their enterprise data catalog, and then some recommendations. For purpose and importance, we'll start with the d-data point that you can see here. When we asked what are the challenges for achieving data objective to the organization some of the prominent responses were around data silo poor data collaboration, weak processes, and I'm just highlighting the ones that are usually associated with enterprise data catalog and how EDC can solve those challenges. Next was, again, specific on AI, something that just Austin asked you. What are the barriers for the AI adoption in your organization?

3:35 The prominent one, which is not a, you know, a, a, a tough response, is like data privacy and security, trust in AI governance and risk. So again, these are some of the prominent one that I think EDC can help you solve. But before we go any further again, a poll to keep you engaged. At what stage is your organization's investment in generative AI efforts?

Austin Kronz

4:08 Should be able to answer live in the Zoom poll. So we'll give this a little bit of time. Oops. Should start to see some results coming in as people select.

Jayesh Chaurasia

4:24 I almost tried to vote, and it says host and panelists are not allowed. That's, that's not helpful.

Austin Kronz

4:30 It's truly, it's really an unbiased poll. Awesome. There we go. So I think promising, promising results here. Looking like a little less than half, about forty-eight percent, are already piloting small projects with gen AI. The lowest percentage, not yet investing in, in generative AI, which makes sense. And, and impress... Actually, I think impressive, I don't know about you, Jayesh, but seventeen percent already actively investing in large scale gen gen AI initiatives, so.

Jayesh Chaurasia

5:03 Yeah, it's, it's great. And it... This is reflective of the market today. Most of the companies are piloting more than actively doing a lot of this thing because, first of all, it's expensive and all those concerns that we talked about are taking priority. So yeah, great. Let's begin to to see some of the prominent use cases of enterprise data catalog and data governance. You know, companies... Why are companies using these tools? And of course, they are trying to solve those challenges, improve the data quality and reliability, modernize their data governance tools work on improvement of data lineage so that you...

5:43 They know where the data started and where it is ending up, those data governance processes, and also the improving the literacy in their organization. Following on the same path, like it's, it's good that organizations are actually prioritizing their resources, prioritizing their spending. The dollars are going into data governance activities. Here you can see that sixty-eight percent of the organizations say that they are going to increase the spending in On data governance and you know, data... And, and I'm assuming, and not assuming, but kind of saying that data governance data, enterprise data catalog is fitting within data governance purview here. So by default, the investment in the enterprise data catalog is also increasing.

6:34 And this is the last slide on data where what are companies doing with enterprise data catalog specifically is that fifty-seven percent of them are currently using it today, and thirty-three percent say they are going to implement it in the next twelve months. So another poll on the same thing. What is your organization doing? Are you guys using it today or are you planning to implement it in the next twelve months?

Austin Kronz

7:11 At this point, Jayesh, would you be surprised if there's someone not using a data I, I was about to say that.

Jayesh Chaurasia

7:17 Like, I, I should not see a lot of no plans for implementation because that would just not be good for, for people to be here. Okay. We are in good, good company.

Austin Kronz

7:28 Yeah. Or they came, or they came to the right webinar, right?

Jayesh Chaurasia

7:30 Yeah.

Austin Kronz

7:31 Amazing. Yeah. There we go. It's about fifty-fifty using versus planning to implement though, so pretty, pretty interesting results so far.

Jayesh Chaurasia

7:40 Yep. Fantastic. And it's a little bit different than the result that I showed, which is, you know, more people using than thinking, which is great that you are exploring it today and you know, looking forward to learn more about data catalogs so that you can make a better decision. So when I say enterprise data catalog, and I have already said EDC and e-enterprise data catalog a couple of times today in the presentation, what do I mean specifically? I just wanna clarify that before we go any further. Is that. Oh, I see. Okay. So certain things, someone playing behind the scene.

8:19 But what I see I'm, I'm categorizing enterprise data catalog as the catalog of catalogs market.

Jayesh Chaurasia

8:27 I'm saying this because there are some platform-driven catalogs like Unity Catalog, as I have here in the in the platform catalog bucket, is that Unity Catalog used to play in the purview of just Databricks environment. Now they have open sourced it, so you can look into other data sources as well, but you have to customize to do that. Dataplex is another example from Google. So again, Dataplex is within the purview of the GCP environment. They can... They don't involve other data sources too much. But when I talk about enterprise data catalog, I'm focusing on the catalogs that can go across data sources.

9:08 It could be the traditional data sources of SAP and Salesforce, the, the operational data sources, places where you do ETL, ELT pipeline management data storage and compute layers of Snowflake and Databricks as well. Those catalogs too. So if you are maintaining your metadata in those enterprise and/or platform catalogs, you can bring that metadata into these type of catalog of catalogs type of EDC tool. And then some end user application, Power BI, Looker, Tableau, things like that, what, where users act-active actively and actually perform their task.

9:48 So now that we understand what, what am I referring to when I'm saying EDC, let's look at some of the market trends in that in that environment, in that zone. First of all, this shouldn't be a new or mind-blowing news to anyone that data and complexity around data is increasing. And you know, let's take an example of Netflix. Netflix gets petabyte or maybe zettabyte of zettabyte level of information, the volume of data every day, and that data is not static. It's not being ingested in batch. It's real-time streaming data set in variety of different formats. It's structured, semi-structured, unstructured. There are comments, there are people rating stuff.

10:38 And then the... It needs to be very accurate, consistent, and trustworthy in order to take different step. When, when I'm engaging with Netflix, I want to see similar movie recommendations so that it makes sense for me to watch rather than just scrolling, scrolling, which usually end up happening anyway. But so that data and complexity is increasing as a market trend. And EDC market or the future EDCs need to be... And for some reason I'm not able to go to the next slide. Oh, it went two slides ahead. Okay. So here it is. The next slide was the, on the future of enterprise data catalog. What does future EDCs look like? Or what would they have?

11:28 First, they need to be AI powered, no big news, but they need to not only handle data, but AI as well along with it. It needs to integrate with governance tools so that you are maintaining that risk along with playing with the data. It needs to enable self-service, which is one of the prominent tasks most of the organizations are trying to do today. And it should enable you to build data products or think in the, in the mindset of data products rather than just data asset or data as, as just a fuel. You know, most of the time we say data is a fuel for current organization, but let's make it bit more than that.

12:13 Let's make it an end product which will drive success. So how do you evaluate what... How you know these are the future trends and markets? What do you need to have or consider when you are evaluating these solutions? Of course, first of all, I would like you to use Forrester Wave.

Jayesh Chaurasia

12:33 I just completed this enterprise data catalog evaluation last month, and I want you to evaluate not only the current capabilities, what the product offers you today, but also the strategy capabilities. Where is their strategy going? And as you can see in, in our Forrester Wave, we have considered both of those aspects. And there were twelve different vendors, they ranked differently. There were twenty-two different capabilities on which they... We I evaluated them on. And those capabilities, those twenty-two capabilities are listed here.

13:08 And while we go through these capabilities, I am also going to highlight those future aspects and how those future aspect or future capability is going to affect or are going to be relevant for each of these criteria. Starting with AI powered. For AI powered enterprise data catalog, I could probably just highlight each and every section here because AI is being used in almost everything. But some of the prominent one that I feel is more important and, and AI has a deep impact is the metadata management, automated curation is there automated discovery, profiling, governance and risk compliance related stuff, data privacy and security.

13:57 These are the places where I see most of the impact from AI automation. And in the strategy side, you need to look at the organization or the vendors who has innovation in, in as the mindset around whenever they are inno-innovating, they are keeping AI as the center. And they are also seeing adoption around AI poweredness. So when they are offering this AI solution, are customers also using it? And what are the enablers they, these vendors are providing for you to adopt and and, and use AI capabilities more and more?

14:37 On the data plus AI side, and this is one of the prominent predictions from Forrester, that most of the data and AI governance is going to happen on the single platform. For, for those, again, metadata management is important, but what what is new is here the data lineage. You want to see not only how the data moves from one place to another and ends up with in the hands of the consumer, but also how AI models are being built. What are the data that is being used in an AI model? What are the outputs? Where this output is going eventually? What are the impacts of that output?

15:13 You know, the data privacy and security, all those, all those trust factors. So data lineage, governance, risk management. And in terms of strategy, you need to look at the vendors who have this vision of combining data and AI together and also have a roadmap that will enable this. It shouldn't just be in their vision, but also in should be reflected in their roadmap. For governance integration, of course again, data lineage is one part of is a big part of data governance and governance risk and compliance. But the other section is on data quality and observability.

15:50 A vendor that offers or in has great partnership with data quality and observability platform should be the focus of your evaluation. They should enable monitoring and alerting capabilities, privacy and security features as well. And in the strategy side, there should be multiple partners who support them for the governance, for privacy and security. There, there are some companies who are also partnering with identity management, MDM vendors. So you look for those vendors who do have a broader market partner ecosystem. Along with that, they are also supporting the services. They are offering supporting services and and, and ways in which you can enable the governance integration. Next we have self-service.

16:39 For self-service, of course the knowledge comes from metadata. So you're providing that business knowledge using metadata to your end user so that they can self-serve themselves. You're enabling through workflows and task management, collaboration capabilities. But two main features that I want to... I have already highlighted here is the user interface. It matters a lot to your end users. Look for user interface and user experience that can enable self-service, that excites them to continuously come to the platform and leverage that platform again and again. And then there has to be a marketplace or an exchange platform from where they can actually search for the data products or the data assets that they are looking for.

17:23 In the strategy side, again, the vendor should be innovative enough to work on those UI cases, marketplaces, and enabling self-service capabilities and offer some supporting services around data quality and integrating with other tools so that you are you see similar self-service capabilities from... And, and without losing the the, the flow when, when a user is coming to your platform. Last but not the least is the data product.

17:53 I evaluated specifically the vendors who enable you to create data products and what are the ways in which you can do that how you can automate it, what are the integration capabilities they offer with other systems so that you can create that data product holistically on their platform. User interface, user experience is again very important to create a data product or, and share data product across your organization using marketplace and exchange board. Data quality and observability is a key characteristic of any data product. So when you are creating data products, you are going to look at that by default. Last in the strategy side is supporting services and offering.

18:34 Again, you want to enable and share and integrate your data product and your services with as many people and as many prod platforms as possible to make it more effective.

Jayesh Chaurasia

18:48 Going towards the last section of my presentation is the recommendation, what you need to do next. You have the the capabilities and things to look for, you know what... How to evaluate them, so what you will do next. First, you need to evaluate yourself, your maturity, and your needs. And based on that maturity and the need, see which tool support your maturity today, but also has a feature and those future capabilities to support your needs and maturity as you grow. It is also cost effective, so it, it most of the vendors have different ways in which they cost.

19:33 So look for that cost effectiveness and the ways in which they are costing they, they have their cost model set up, and select the one that works well for you. And lastly, improve your program not in one day. You know, don't try to boil the ocean in one day. Try to improve incrementally and continuously as we do with any other data management or just data projects. So that's where my talk ends. Here are some resources from Forrester that can help you, again, with your evaluation as well as your data governance and data quality of initiatives. There are some thought leadership papers and, and things like that. I think next is Prukalpa. Thank you.

Austin Kronz

20:19 Amazing. Thank you so much, Jayesh. Really practical insights, and I think people can start applying that today. We're gonna switch gears just a little bit. I'm gonna stop sharing for one second and we are gonna turn it over to the co-founder of Atlan, Prukalpa, to talk a little bit more about the future of AI native data catalogs. So... And just a quick reminder, keep participating in the chat, keep participating in the question and answer, and we'll have plenty of time at the end to talk to Jayesh and Prukalpa and get some live responses. So with that, I'll turn it over to you, Prukalpa.

Prukalpa Sankar

20:52 Awesome. Thanks, Jayesh. Thanks, Austin. So, well, heavy topic future of AI native data catalogs. The way I was thinking about structuring this was I wanted to start with looking back, how we got here and our backstory because every time that you think about looking forward, you need to kind of go back and think about the, what got us here. Second, I wanna spend some time on why traditional governance fails and our platform principles as we've built and architected Atlan. From there talk about the top three barriers to operationalizing AI that we're seeing as we work with data leaders across the ecosystem. And fourth, looking forward our hypothesis on the future of AI native catalogs.

21:39 And I specifically call out hypothesis on the future of AI native data catalogs. One of the things that we're trying to stay away from is a lot of the buzz that that basically... If, if you go, if you walk into a data conference today, like every booth has the word AI somewhere in it. And what we're going to try and do is keep this real and talk about what we're seeing and what we believe the future's gonna look like and what that means for data teams and how we adapt to that future. So getting started with our backstory, some of you may know this, but at Atlan, we actually started as a data team ourselves.

22:18 My co-founder, Varun, and I founded a company called SocialCops and our mission was to go solve these critical world problems, like national-level healthcare and poverty alleviation and we wanted to use data science for social good. And we were working on these projects which, on the outside seemed like pretty cool projects. At one point, we were dealing with data for five hundred million Indian citizens and billions of pixels of satellite imagery a day. Very dreamlike projects at the time. On the inside, not cool or fun at all. It was just chaos every single day. This is actually a picture from our office back in the day.

22:59 We literally had set up a room with bunk beds and we had, like, eight bunk beds in office because as a data team, we would wake up with, like, a fire drill. Like, I remember, there was, like, a phase in my life where for about six months I think I hardly left the office, 'cause I would wake up with a fire drill, and then you'd somehow put it out that day, and then you'd sleep at 3:00 AM, and then you'll wake up with another fire drill, right?

23:22 Like, a number on a dashboard is broken, analyst quit and, you know, they left with all the knowledge on our project and, you know, just, like, all kinds of things that would break on a, on a daily basis. And so we actually got started with this project internally, where we realized we couldn't scale like that, and we started this project. We called it the assembly line at the time, and we said, "Our data team, we need to find a way for it to work more effectively."

23:46 So what you're seeing on the screen, we ran this, this project called the design sprint, and we came up with all these, like, sticky notes of all the areas where our process was broken, and we said, "Can we find a way to make our team work more effectively?" And then we did what most logical, you know, data teams would do. We went to try and buy a solution. And we failed three times setting up a traditional data catalog as a data team ourselves. The first time, we we actually I think built maybe one of the most powerful search engines that existed. Like, it was, like, this amazing sort of, like, NLP-powered search.

24:27 Like, you know, based on exactly what we searched in the search term, it would not just, like, search through your data assets, it would give you the actual results. Like, these are the ten doctors that charge more than this dollars, for example would show up. And then we realized that it was... It demoed really well but it was really hard to, like, get up and running, 'cause putting in the metadata was too manual— And we, you know, like, we literally never got it up and running beyond the demo. So we failed. Second time we tried to buy and this time we...

24:59 Like, we were a modern data team, and we, you know, we worked on the cloud, we were using Slack every day. We wanted all these, like... User experience mattered to us. We, we couldn't drive adoption top-down, and nothing that we saw at the time could help us solve that problem. You know, and, and I'll, I... You know, I have a longer blog on this. I'm not going to go through each one of these, these... You know, the third time we were semi-successful. And the fourth time the first version of Atlan was born. We built Atlan for ourselves. We never meant to open it up as software for other teams.

25:35 We actually made our team six times more agile in that time, ran about 200 data projects, and then realized, "Can we go help all these other teams around the world with this?" That's really how Atlan was born. Today we, you know, we work with, you know, amazing data teams around the world across 40 industries, you know, in over 20 countries. You know, the largest auto manufacturer in the world, the largest media company in the world. And it's been, like, you know, you know, we always said we wanna go power the best, amazing data teams around the world. And it's been an absolute privilege.

26:10 Customers often come to us after failing to make their traditional catalogs work, and they tell us three reasons why they failed. The first, they talk about this catalog ghost town problem. Like, they go like, "We never got up and running with our project. We could... It took us years to integrate with our metadata systems." You know, like, "We never got our data documented. It was too manual," right? And they, they use keywords like that. Second, no business adoption. "The user experience was too confusing. We couldn't, you know, like, it... We just never got our end users to use it." And third, they talk about these words like future ready.

26:53 So they'd say, you know, like, "We, we were going to the cloud, and we needed something modern." And they'd say, you know AI ready you know, new kinds of compliance and workflows for that. And we've realized that the things that helped us succeed as a data team back in the day are, are some of those principles that have helped us figure out how to help our customers become successful. And those come down to what we believe are the two fundamental truths in data. The first, in data teams, diversity is not a bug, it's a feature. Data teams will always be diverse.

27:31 Analysts, engineers, scientists, business, all of these people will have their own tooling preferences, their skill sets, their own DNA in the way that they work. This diversity is something that you need to build for as a feature. Second, and we live in a world that is changing constantly. Two years ago, if somebody had mentioned the word vector databases as an integration point, most of us would have been like, "Huh?" And today it's very much a reality. And I guarantee you, two years later, we're probably gonna be talking about something else. This is maybe one of the fastest changing ecosystems that we live in.

28:08 And we built Atlan accepting that diversity and change are the only realities in data teams. As we think about each of these diverse ecosystems, diverse people, diverse tools, and diverse team structures, orgs, technology needs to adapt to this instead of organizations adapting to technology. And that's meant four core principles. First, automate wherever possible. Human resource time is the most expensive time in a company. Make it as automate... Anything that can be automated should be automated. Our most recent version of this was Atlan AI. Today about 70% of Atlan AI recommendations are fully accepted by a human.

28:56 And we've, you know, pushed the boundary of what AI can do in partnership with humans. As you think about going back to that problem of the catalog ghost town, how do we reduce the manual effort that it takes to get a data catalog project up and running? Second, adoption. We believe that change management is really hard, and so don't do change management unless you really need to do change management. So instead of getting a user to come to a tool, take technology to the users. And what that's meant is we call it active metadata. We've taken Atlan and embedded it in tools that people already use. This has meant tooling like in engineering workflows.

29:38 We integrate directly into GitHub. We have data contracts that integrate and embed context into data assets at source. And on the other hand, for a business user when they're in a dashboard and they say, "Can I trust this dashboard?" A business user does not want to jump into a different tool and search for their login that they've most likely forgotten. They just want context where they are and when they need it, and we serve them context right in those tools driving up adoption. Third configurable and personalized. Every data team is different. And, you know, I think the best way to think about this, we've made everything Atlan fundamentally configurable and fundamentally personalized.

30:21 So if, you know, the question we asked was: If Netflix can have, you know, completely different views for you and me, why can't we do that for end users inside the organization? I think the way Mark Kidwell from Autodesk described this is, you know, one of our customers, like, they said, "We wanted a tool that's 80% lightweight and DIY, so we can get up and running, and 20% configurable because we know that we'll have our own unique needs." And fourth I think this is the most critical part to the AI-first world, Fundamentally open and extensible. We are moving to a world where metadata itself is becoming big data.

31:02 You know, we're managing assets like, you know, millions and millions of assets. Fundamentally everything in Atlan is open. Today we have over 400 million metadata updates happening on a monthly basis via SDKs, APIs developer first approach. Metadata in into Atlan, metadata out from Atlan is all fundamentally open and accessible. All of this up till 18 months ago, and then everything changed. Sam Altman, the CEO of OpenAI, recently wrote a, a blog post and he said, "We entered the intelligence age." Technology brought us from the Stone Age to the Agriculture Age to the Industrial Age, and now we're entering the Intelligence Age. And that started this big AI internet moment, right?

31:52 Every board is calling every CEO and saying, "What is our AI strategy?" They're all calling their data leaders and saying, "What is our AI strategy?" And the inside joke is that everyone's ready for AI except your data. And every data leader I've been speaking to has started asking me, "How do I get AI ready?" I have a hot take. "How do I get AI ready?" Is the wrong question to ask. In five years not many of us know what the world's gonna look like, and so the right question to ask is... Here's what's gonna happen.

32:29 There's gonna be a lot of change, and there's going to be a lot of new diversity, lots of new kinds of personas, like prompt engineers maybe, like new roles of an analyst. Like, we don't fully know what is gonna change, but things are gonna change. How do I support that diversity? Everything's going to change quickly. How do I keep evolving to this rapid change? As we talk to these data leaders, they tell us three barriers to operationalizing AI.

Prukalpa Sankar

32:57 First, specific AI over generic AI. How do I make AI that's relevant for my business? For example, if I'm an investment management firm, TAM means Total Addressable Market. It could mean, like, 100 things on the open internet, but in my company it means Total Addressable Market. How do I teach AI that? Second data access for AI accessibility. How do I control access based on who's asking the question? For example, if I have a chatbot for my HR team, it's okay to use payroll data. Probably don't wanna use it for anything else. Third trusting a response over getting a response.

33:40 It might be fun to, you know, do some research on the internet on ChatGPT but when I'm making a business critical decision, I really want to trust the answer. And what's been most interesting across all of this has been this realization across the ecosystem that the things that we thought were going to be difficult, which was actually the models and the LLMs, seem to be the easier part of the problem, and the thing that we thought were was gonna be easy, which was the data, is actually the really hard part. What... How do we create AI-ready data? And we think about this as Jayesh talked about this concept of data products.

34:23 What does it mean to create a AI-ready data product? A AI-ready data product is discoverable, it's self-describing, it's trustworthy, it's transparent, it's secure, it's governed. It helps answer all of those three questions that I talked about in the previous slide. What that means for us, we started out with this simple idea to empower the humans of data, these diverse people to do their life's best work and take on these big challenges, and now we're at this place where, you know, there's a new era of AI. There's a whole new set of knowledge workers including these bots and copilots and autonomous agents, and our mission must evolve.

35:05 And the way we're thinking about our mission is how do we, you know, help the humans of data and their AI counterparts do more together? What does that mean? Three key questions we're pondering about. We believe the humans of data are going to be key to creating AI-ready data, and the question we're asking is: Can we be the home for them? We're thinking about this in two core components. The first automate wherever possible for the AI data supply chain, unstructured data, automated model lineage, and explainability. How do we automate as much of the AI data supply chain as possible? And second, AI stewardship.

35:47 We think of this as how do we create iterative trust loops between humans and AI? And how do we ensure that humans of data can work closely to improve AI responses as we create more and more AI-ready data and trust in the responses around AI? The second key hypothesis, there's a lot of thought talk around AI ethics and AI governance. Here's the thing. Policy around AI ethics and AI governance is a human collaboration and decision-making problem, not a technology problem. And so the problem we are going after is how do we create an amazing collaboration hub for AI policy?

36:33 We launched something called Policy and Transparency Center some time ago, and that's designed for human collaboration around how we think about governance and policy. And second, we're looking to say, "Okay, how do we now create measures for ethical AI?" The same way. Today in Atlan you have something called a data product score, which is automated. The second node that we're building is what is a similar ethical AI score that we can build around all of the AI models and applications that organizations are building across. And the third and most important, trust. Trust is going to be key in operationalizing diverse AI use cases and applications.

37:24 How do we create a unified trust engine? We believe that the applications of AI are going to be universal and cross-system. There's going to be different kinds of copilots across the ecosystem. There's going to be different kinds of tooling. There's going to be vertical AI tooling. There's going to be internal tooling, bots, agents that get deployed. All of these diverse applications are going to need a common trust layer and we're thinking about this as an open metadata exchange fundamentally built on open platform principles where we are able to open and power all of these copilots and ecosystems with that final question: Can I trust the answer that I'm getting?

38:07 And so with that I'm going to end actually with a quote from a recent blog from Sam. And you know, he said, you know, "Many, many years ago, if a lamplighter could see the world today, he would think the prosperity around him was unimaginable." And, you know, if we could fast-forward 100 years from today, we would feel the same way. We are in an unprecedented time and entering a different age of what's possible with technology. And so the real question for all of us to ask is, in all of these times, how do we adapt to the diversity and change that's coming?

38:46 And so with that I guess I'll hand it back to Austin and, you know, we'll open up the floor for questions.

Austin Kronz

38:53 Yeah. Amazing. Thank you, Prukalpa. I, I think a lot of common threads in both of those sessions, right, around trust and diversity of humans and technology and data. And I think, you know, bringing this back to we're, we're talking about war of catalogs in a new era, you know, it's a different time. Jayesh, maybe I'll come back to you for a first question. And just, you know, the timing of the Forrester Wave, I think is really important, but why now? Why different than six months ago versus six months from now? You know, how do you envision that, that changing or, or why is this such an important time for the enterprise data catalog?

Jayesh Chaurasia

39:33 Absolutely. Thank you. That's a great question. The simple answer would be because we are supposed to refresh our wave every 18 months, so I did that. But the, the deeper answer is it was a, an absolutely amazing time to do that. People were in the frenzy of AI, generative AI, in the whole 2023. In the beginning of 2024, realization has, has started like, "Oh we, we missed the boat on data management. We did not think about our data. We don't know what is the governance look like. What does the quality of our data look like? We just went after AI."

40:15 And so enterprises with the realization, they came back to the basics, and the basics are let's find out where data reside. Who is the owner of it? What is the business context around it? And naturally, it's the enterprise data catalog that can give you all that holistic picture. So we, we felt like, okay, this is a great time to just, just refresh that wave, as well as provide the information around what is going on in this market. And which was another reason why I I shifted the focus of our EDC this year from more technical or more business to technical plus business, where not only...

40:56 And, and you see that from the persona also, where data engineers are being asked to not only do what they used to do in the technical realm, but also understand what is the business requirement. The data governance and data quality is also shifting towards left, so they have to think about those policies and rules too. So that was the reason, and maybe six months after, it would be too late.

Austin Kronz

41:20 Makes sense. Yeah, I think that's great. AI seemed to get the conversation about governance going again. Or as my colleague liked to say, you know, making governance not cool again, but cool for the first time. Prukalpa, I'm curious, you know, you're working with a lot of customers who are now in the midst of implementing generative AI. What are some of the blockers, the biggest blockers that you're seeing when you're talking to customers?

Prukalpa Sankar

41:45 Yeah. I think a large part of sort of like the, the few barriers that I covered I think the thing that stood out for me the most, I think I spoke to a CDO the other day and, you know, he was like, "We have 1,000 AI use cases in our roadmap. And we don't know how to start 'cause we don't have the data ready for any of it." And, and I think what's, what's been really interesting has been the things that people thought were going to be difficult, like we have to go learn all these like modules and LLMs and all of these things.

42:15 Actually, the interface points for those things are actually relatively simple or simpler than most people thought. But getting the organization data to be at a point where we can actually use any of this in production is... Like, I, I think there was a stat the other day around even the amount of dollars that's being spent on AI right now, and if you think... And it was broken between production versus experimental. And I think about, like, 90% of all the money that's being spent is on experimental stuff over production stuff.

42:47 And I think the real challenge is going to be, as we think about our organizations and compliance and, you know, trust and all of these things, how do we cross the barrier between a very cool experiment that demos well to something we can actually take into production?

Austin Kronz

43:05 Yeah, makes sense. So if I could drill into that a little bit, there's been a lot of talk of AI, there's a lot of questions in the chat, in the comments of, you know, we're hearing AI in all of the sessions that we're attending. Jayesh, maybe I'll, I'll start with you. What does AI-ready catalog mean? Or is there a difference between catalogs utilizing AI and then catalogs helping with building gen AI?

Jayesh Chaurasia

43:32 Yeah. I think there, there was one question in the chat from Eric. He said, like, "I'm sick of hearing about AI." Like, it's too much. Every buzz, buzzword involves AI. And I agree with that that it is true. We are incorporating AI in almost every conversation. But to some extent it's required as well because it is improving the capabilities. So he did ask, like, what are the actual benefits that AI is bringing in terms of capabilities on maybe EDC platform or in any other platform that you're using. So talking about, you know, let's, let's take Atlan's example itself, like how Atlan has brought generative AI to build your definition of the product that you bring.

44:19 So metadata, getting technical metadata into your catalog, simple enough. A lot of companies can do that. But what happens when you have to write and type all of the definition, the business context, the relationships and you know, build that knowledge graph that we talk about, it used to take a lot of time. I remember when I was in the practitioner role, we were sick of doing that because we were init— we initially started with a platform then thought what... It, it can be done on the Excel spreadsheet. We started doing an Excel spreadsheet, ten different versions, ten different people working on different time zones and different timelines, did not work out well.

44:59 So this is where generative AI is helping you do that. Traditional AI, it's profiling the data. It's classifying the data into different buckets so that you understand what is the sensitive data, what is something that you can share. Based on the usage metadata also, you can enable or tag a certain user as the owner of particular data. This is where one of our client was struggling just to find who's the owner of particular data. If the company-wide policy would be whoever used the most is the owner, guess what? You found the owner very quickly using... So this is where traditional AI plus generative AI is not on...

45:41 And I just gave couple of examples, but at every as I said, for AI-powered slide, that almost every capability, those eighteen current capabilities that I have, they all use AI in some way or form, whether it is metadata management, even in discovery, you, you are discovering data using AI so that relevant data is discovered at relevant time. There are governance policies that you, that are onboarded on you know, on a regular basis. Then there is collaboration features. So by default, whenever someone tags you or. And, and data quality, I completely forgot about data quality. Data quality is the biggest place where AI is being used.

46:24 You are, you're getting insight into not only the completeness, accuracy and consistency, but also how to fix it. What are the possible causes? What can you actually do? Lineage. I'm amazed by Atlan's lineage, how you guys are actually bringing, you know, all those data quality components, the governance, the metadata, overlaying it on top of the lineage. But along with that, making it easy to do the root cause analysis, impact analysis very quickly using AI so that you, you can converse it with your lineage to ask certain questions. So AI, yes, it's the buzzword, but it is still being... It, it is impacting the platforms enough.

Austin Kronz

47:06 Great. Yeah, that's really helpful.

Jayesh Chaurasia

47:07 So I forgot... I'm sorry, I forgot the question altogether. I'm answering something else.

Austin Kronz

47:11 I, I think you covered it. I think you covered it.

Jayesh Chaurasia

47:13 Okay. Okay.

Austin Kronz

47:13 It was, it was more of how does AI kind of manifest itself in actual use cases, right? And it's... And you gave a lot of great examples of where AI is impacting the functionality of enterprise data catalogs as well as, you know, supporting AI use cases in the organization. So I think you, you covered it perfectly. So that was great. There's a couple questions. I'll combine the question from... We have a question from Blair, and then there was a question earlier from Mohit, and they were... They're both around data governance tooling. Jayesh, I think you covered this a bit in the report, but you mentioned things like governance integrations.

47:48 How do we see this relationship between an enterprise data catalog or a governance tool? Are they the same, different? What does integration mean? Can you talk a little bit more about that?

Jayesh Chaurasia

47:58 Yeah. Quickly it... The, the market is converging on, on that front. We are seeing more and more vendors who used to offer enterprise data catalog becoming a governance vendor or a governance vendor becoming an enterprise data catalog vendor too. So the... And the reason for that is EDCs used to find the data, present it, index it, and, and that's where it is supposed to end. But for the client, that's not where they want it to end.

48:24 They want to not only find the data, but also now operationalize it with that governance features, bring those privacy and security concerns, the policy management layer, the collaboration layer, all of that in one place, the workflows, so that they can now not only find the data, but then operationalize it, use it, create a data product, find that on their marketplace and, and go from there. So more and more, around probably almost all the vendors will offer ADC plus governance together.

Austin Kronz

48:57 Makes sense. Prukalpa, but kinda similar, similar type question, but governance for AI and governance for, for data, is one an extension of the other? Are they, are they different? Or what do... How do organizations have to think about these things, data governance versus AI governance, or should they look at them that way?

Prukalpa Sankar

49:18 Yeah. I mean, I think not to like digress from here, but like I think there's a quick... There's a general, general what do we mean by governance, like, definition challenge statement that the ecosystem has to deal with. But I think principally, I think the way to think about this is assets in your data and AI ecosystem, right? What used to... Like, dashboards used to be assets before, because humans used to go read on a dashboard to understand something and make a decision. We'll move from that into more autonomous systems. People used to search before. In the future, people will chat, and so the artifact will change of how people will use it, right?

50:03 But the use case for an end, like it's to solve this Like, what will not change is it's this business problem that needs to be solved. It's this decision that needs to be made, right? Like, that's not going to change. And so I think the way to think about this as you, as we think about governance, is that a lot of the same problem statements of the underlying data ecosystem are going to apply to the applications of AI inside the org. And so the same way that a dashboard or an analytics asset started to become...

50:34 So if you think about the world, it it used to be data governance, then it became data and analytics governance so we added analytics assets into it. Now it's, you know, data and AI governance, so we'll add AI assets to it. These are just fundamentally, if we think about the journey that we are taking towards, you know, really using data inside an organization, we're just moving further and further ahead in that journey by creating and, and that evolution. And so the fundamental governance principles should apply across all of these assets across an ecosystem.

Austin Kronz

51:09 Makes sense. So, so you're saying our acronyms are getting longer. It was DG, now it's DNA G, DNA IG. I don't... I think we already have DAIG, yeah... I think DA is already taken.

Prukalpa Sankar

51:19 There's like some, yeah. We'll, we'll keep adding the and, and, and. We're, but, you know, we're great at acronyms and, you know, probably creating a business class study around those, too.

Austin Kronz

51:28 Fantastic. Fantastic. Yeah, Jayesh, we might need some help getting that, that word out there for our new acronym, but we'll, we'll see how that goes. Cool.

Jayesh Chaurasia

51:35 Oh, yeah. With, with our new service, we are also struggling with some of our acronyms and how to publicize this more with data analytics AI.

Austin Kronz

51:44 Yeah. Makes sense. So with, with all of this changes happening, Jayesh, how can... Or what should companies that are evaluating or are thinking about evaluating data catalogs, what types of, I would say, traits in a vendor should they be looking for? I mean, we're, we're just saying that six months ago we were having a different conversation. Six months from now, we have no idea something we've never even heard of could come up. So, like, how do you evaluate whether a company can be adaptive enough in the enterprise data catalog market?

Jayesh Chaurasia

52:16 So, so the space is absolutely, you know, every vendor is changing their tactic depending upon what's going on in the market. No one was thinking of generative AI before 2022. The entire 2022 to 2023 is when vendors started to implement it. So there, there, there won't be a simple answer. Like, not every vendor would change their tactic so quickly. Like, as, as I talked about Atlan, Atlan has has incorporated generative AI capabilities in multiple factors wherever required, but some other vendors were not as agile. And of course, it's not their fault because they, they can't do it.

52:56 Everyone can do it in, in some time, but it's the different you know, they have so many enterprise clients that they can't just create a pilot project and do it. Some clients are not ready to accept it. I still speak with so many clients who are, have this mandate of not using any generative AI capabilities within their platform. Even if it is offered, they turn it off. "No, we don't want to use it right now because we have been asked by our security teams to not... To, to, to stay off that thing."

53:26 So for, for evaluation, I think looking at, as I said these Forrester reports or other evaluative research would help you definitely to, to see what's going on in that vendor market. But along with that, the strategy piece. See where the vision of that particular vendor is. How have they performed it on their roadmap? When they promise something, they are going to bring certain things within next six months, did they bring it? Did they solve your problem? Speak with their customers. That's a, like, something that we usually do when, when we go and buy certain product, we look at the reviews. For enterprise data catalogs, do...

54:05 Or, or any other tool that you select, do the same thing. Speak with people. I see the chat is filled with people who want to talk and share their stories. Engage with them and ask what, what they have found helpful or not. And I'm not saying this for Atlan, for other vendors as well. They, all of you are in this space where you are competing with each other, but you're also offering some really great capabilities, and everyone is there to, to do this business, just in different ways.

Austin Kronz

54:37 Well, I think we have time for maybe one, one or two more questions. There's a question, really interesting question talking about evaluating data catalogs based on where a company's maturity is. Now, I think I have a lot of follow-up questions on that, like what maturity? Data maturity, governance maturity, culture. But, you know, how do you approach, "We're at square one today. How do we go from square one to, you know, five out of five?" And does that affect an evaluation, or what should you be looking for? Are there foundational capabilities versus nice-to-haves? Maybe talk a little bit more about that.

Jayesh Chaurasia

55:12 Yeah, so we have multiple frameworks at Forrester for evaluating the maturity. We have data governance maturity framework, we have total data and analytics maturity framework. Now we are also developing an AI maturity framework so that you can you can assess yourself on different things. But the core concept would be just to evaluate yourself on the, on the strategy itself. What is your data strategy? And that would incorporate all of these different data management plus the use, analytics plus AI, so how you are managing your data and how you're using that data. And based on that, you can then use, you know, any of these tools for, for wherever you stand.

55:55 But at least think of two years in the future. Not too much, because you, you never know what will change, as Prukalpa was saying. But at least you can still gauge a little bit for two years, and if something mind-boggling doesn't happen, like generative AI, you would still be in a good place with that particular vendor.

Austin Kronz

56:17 Great. Well, I'll, I'll ask last question, and then we can wrap up. And I'm sure, Jayesh, your inbox will be full. Prukalpa, I'm sure your inbox is always full with questions following this, so, so do catch up with us. But, you know, what is, what is true today that isn't going to be true going forward when it comes to, to data catalogs? Or maybe, you know, what is most surprising in some of the research, and, and, Prukalpa, in some of the customers that you've been talking to, and the challenges that they bring up? Prukalpa, I can start with you, and then Jayesh, we'll, we'll wrap with, with yours.

Prukalpa Sankar

56:52 Why don't you... Jayesh, why don't you go first, and then I'll go.

Jayesh Chaurasia

56:55 Sure. Yeah. For data, like something that is true today is as I said, the, it's going to be more than data. It's going to be data analytics and AI. AI is definitely something that not all the vendors have brought into this purview, so they need to work on that, and that would be the next. Something that won't be part of this is going to be that not using AI. At every capability, at every sense, you are going to use AI. Just try to, you know, enable that. Try to accept it, like, "Okay, I want to." And it is it will help you.

57:34 Yes, there are some, some privacy security concern, but I think vendors are also very cautious of that. So discuss that with the vendor to see what would be the perfect balance, and, and just enable it.

Prukalpa Sankar

57:49 Yeah. I would agree with that. I think in terms of, I think the way we're thinking about this a lot is whatever happens in the data and AI ecosystem. So, A, for for the first time, like actually we as a company stayed away from this word machine learning and AI for a very long time. Like, you know, like, we were like, "This is just a buzzword." Like we're using, like... You know, people are, like, building rule engines and calling it AI. So it was only last year when we, like, actually launched Atlan AI with like real AI capabilities that we started using the word AI. And it's real. Like, this stuff is real.

58:18 Like, it's finally real. It's not a buzzword anymore. It's, it actually works. It actually helps. We're seeing that, and it's going to happen. And we're thinking about it a lot as like it's not going to be a separate application, it's going to be sprinkled across every part of the workflow, and everything that a human does today will probably have a counterpart which is an AI coworker. So that's first. The second in that ecosystem we're thinking about this a lot as like a control collaboration plane.

58:50 So the, the data and AI plane will need a control collaboration layer across all of that diversity that'll get created, and we're thinking a lot about, like, how do we build that collaboration control plane that can act as that trust engine? So yeah.

Austin Kronz

59:05 Amazing. Yeah. Well, thank you both. I know it's been a, a very busy hour. I want to thank the audience for, for joining today. Prukalpa, thank you. Jayesh, thank you so much. I'm gonna share my screen again very quickly. There we go. If you're interested in checking out the report, Jayesh and his team have invested a ton of time, and there's such great depth in this research. Please do check out the Forrester Wave for Enterprise Data Catalogs. And if you're looking to connect with the Atlan team, we'll be in Las Vegas for dbt Coalesce, so please meet us out there and say hello.

59:35 And with that, I will just thank you all, and have a good week. Thanks, everyone.

Prukalpa Sankar

59:41 Thanks, everyone.