Why traditional data catalogs fail

Atlan presenter

0:05 I think before I answer that question, it's really important to get on the same page of what the landscape of data governance really entails. So the way that I like to think about data governance is, and this is a very simplified perspective, but the very simplified perspective that I see is it is the connective tissue between data producers on one side in an organization and data consumers on the other side. Very simplistically, data producers want to guarantee trustworthy self-service data, and very simplistically, data consumers basically want to trust and use the right data.

0:46 Beyond this paradigm of enabling users inside an organization, data governance often tends to be the foundation for enabling several business and data initiatives, right? So right from data mesh to data compliance to cloud migration to, in the recent times, AI governance inside an organization. I think this framing is really important to context set and understand why traditional governance fails, why we failed those three times in implementing a traditional program. So we brought together some data from data teams that failed with the traditional data governance, so they moved to a modern approach. And we found that 40% of initiatives fail because of the lack of adoption and business engagement.

1:36 This makes so much sense to me because traditional governance, in some ways, you're trying to solve a silos problem by adding one more tool to the silos. So just think about this. I am a business user. I am on a dashboard, and I say, "Can I trust this data?" At that point, the last thing I want to do at that point is jump from that tool into another tool, which could mean my catalog, search for my login, which I've most likely forgotten then log in, find my dashboard, look at the lineage, and figure out if I want to trust it or not, right?

2:11 As an end user, I just want context where I am and when I need it. Second reason, manual. Like, 20% of teams said that it was just too much burden on people to curate metadata. We've had, you know, leaders say it's taken 18 months to get up and running, and this was fine in a world where you could deploy hundreds of data stewards inside your organization, but in today's world, and honestly business environment, most fast-moving agile team just don't have the kind of resource or time to invest in. And finally, 40% of teams said that the traditional solution didn't set them up for the future. They mentioned paradigms like data mesh, data products, data contracts.

2:56 They talked about moving to the cloud. And traditional governance is closed. Traditional catalogs are closed, and, you know, they don't necessarily allow you to actually enable these new paradigms that data teams are moving towards. So how do we get this right? We think about this in three pillars. The first, going from siloed to collaborative. How about instead of when I'm in the dashboard and say, "Can I trust this data?" Instead of forcing me to move to another tool, I get the context right where I am and when I need it.

3:33 Something that we do at Atlan and we think about a lot with active metadata is taking your context into the workflows of people and teams. For example, we have a browser extension that sits on top of the tools that end users use on a daily basis. This could be BI tools like, you know, Tableau or Looker. This could be, you know, data warehouses like Snowflake or Databricks, and it's bringing context into the daily workflows of people and, and teams in the places that they're using it. How about, you know, when we're collaborating and communicating on platforms like Slack?

4:05 Every data team typically has this channel where, you know, it's called data gurus or something like that, and people say, "What do we mean by this metric?" You know, "How do we, you know, think about, like, revenue?" And how about if instead of having to, like, move from there into another tool, right then and there you have a bot that's answering these questions, right? So how do you seamlessly integrate into daily workflows, be it developer workflows, GitHub, dbt, Jira, to communication workflows that people are already using? I think about this a lot with, you know, only do change management when you really need to do change management.

4:40 If it's possible for technology to solve the problem, why don't you just let technology solve the problem instead of having people change manage their workflows? The second pillar is going from manual to automated flows. This means embedding documentation in data producer workflows. You'll be hearing from Autodesk later today, but for example, they actually have added a P to their ELTP, or their ELT process, which is ELT followed by publish. They leverage dbt and Atlan together as a part of their dbt developer workflow. They're actually creating documentation standards. Their developers are shipping documentation. They're publishing that into an overall ecosystem to power their data mesh. Second, AI.

5:30 Talk of the town, we launched AI, Atlan AI, and we just some time ago, we're already seeing a 55% acceptance rate of Atlan AI suggestions, which means over 50% of the work that, that was previously done by humans are actually being taken over by AI. And finally, you know, things like, you know, bulk updates. So think what, you know, Zapier can do. How do you do bulk rule-based updates? Tide, as an example, used that for GDPR tagging, and they took what they estimated to be a 50-day-long process and turned it into something that was five hours.

6:04 And so how do, you know, take what was previously a relatively long manual process to set up, which meant that your catalogs were never really updated and your catalogs were ne-never really— Sailed and enriched and leverage automation to get you to a point where your end users can get all the value that they need. And the third pillar, going from closed to open by design and extensible. We have fundamentally rethought the way metadata should flow in your ecosystem, right? So if you think about this concept of the single source of truth across your entire data estate, the thing about your single source of truth is it's not just used for giving context to end users.

6:49 If you know that, you know, these are my dashboards that get used the most, and they get used at 10:00 a.m. On a Monday morning by 90% of my users, you could actually auto-scale up your data warehouse compute workloads at nine thirty and ensure that your pipeline gets updated at nine forty-five, right? How do you actually leverage this context across your ecosystem, across the diversity of your tools to leverage and create a better ecosystem? So what that means in Atlan is we've opened up, you know, all metadata change events through event versioned queue. You know, we have... Everything's open via open APIs, webhooks, you know, package marketplaces.

7:31 And, you know, the way we've started to see data teams leverage this as they future-proof themselves for these new paradigms is, you'll hear from Elastic again later today, but for example, they're actually doing proactive pipeline health and failure alerting. So when there's a ETL pipeline failure, they're actually taking that and turning that into Slack messages that are getting auto-published to the right people. They are publishing this directly in the dashboard, the downstream dashboard leveraging lineage, ensuring that the end user that's probably using the pipe— the, the final data that's updated through that pipeline already knows, red alert, "Hey, don't use this today, it's probably not updated," right?

8:10 And, like, we're seeing use cases like, you know, automated legal notification workflows based on metadata change events happening in a data lake or warehouse. We're seeing automated data deletion workflows based on what data actually gets used and not isn't used inside an organization. Data observability alerting, and just tons of use cases where by opening up this metadata, you, you create a significantly more business value inside your organization for your governance initiatives. So in summary, as we think about rethinking data governance, you know, shout out to Re:Govern, we think about moving from a passive approach to a truly active approach. What does that mean?

8:53 Going from siloed to collaborative, going from manual to automated, and going from closed to fundamentally open and extensible.