Key features to look for in a data catalog

Mihir Modi

0:05 What are the key component features? Now, I would say these are the four pillars any data catalog should have, and I'm gonna explain you why do we need that. First of all, search and discovery, right? If you are starting your journey with a catalog, and as a user if you think you need an intuitive search engine, a Google-like s-search engine which will quickly kind of, you know, tell you that, "Hey, here is my data asset related to, let's say, impression," or, "Here is my data asset related to, you know, any particular category," right? So efficient search, metadata extraction is one of the key components.

0:40 Gone are those days where you kind of, you know, take a, a lot of time extracting that metadata from your databases, whether it's Redshift, Snowflake, or any particular database, right? So, and your search and discovery aligns with your glossary, right? Someone should be able to understand what my key KPIs are, and that's where a glossary kind of plays a key important role. You group together all your business users, come up with those metric definitions, and then kind of, you know, put it into your... Right? How do I kind of preview my data, right?

1:10 So as a user, once you log in to a catalog, you should be having an ability to search data or, or see at least a sample record, right? And that's how I, I... In, in my next slide, I would say, explain you why we think, you know, we need a one data platform basically, right? So we covered search and discovery. Let, let's move on to data access and governance, right? Now, as Steven, you mentioned, right, there are different personas, and our approach was that how can I make a tailored experience for each and every business users, right? So the idea was that some...

1:42 When a business user logs in, he shouldn't be kind of looking at all the assets which are not, you know, required for them, right? Should be looking at the only the top 10, 20 assets, which are highly curated assets for a business user, right? That was a kind of, you know, tailored experience which we kind of thought of, and that's where the data access and governance kind of, you know, plays into a role, right? How can I easily classify my data? Now, I might have some sensitive data, IP address, email address coming from different data sets.

2:12 How can I kind of, you know, tag those assets and put a masking policy on top of it, right? So these are some of the, like, kind of, I would say, basic feature. And on top of it, with respect to data access request, let's say I don't have access to a, a, a data asset. And if I am curious to know more about it, your governance platform should be able to route a user to kind of, you know, navigate and then raise that access request and quickly turn around and get you the access, whatever is required, right?

2:39 The third critical component is lineage and quality, which I was also discussing in my panel discussion, is that you want to understand how my data translates from raw to my staging layer to my, you know, transform layer, and what are the different transformations happening during this lineage process, right? Like, how, how is the data traveling basically from my source to my target, right? And aligned with that, building that trust factor, you know, is critical, right? And how do I build trust, right? So we have data observability platforms which can alert on a particular asset that, hey, my data is stale, right? My data is not getting refreshed.

3:21 My volume, there is a sudden spike in, let's say, there's an NFL game, and there's a sudden spike in volume. I want to know as a user that, hey, why do I get a sudden spike, right? So those are, you know... A-and from, if you think from the operation perspective, identifying why, what went wrong, you know, identifying, hey, what can be the root cause for these kind of data issues, right? And getting to know in the catalog itself that, hey, I can trust this data. I have a trusted score for this asset. Everything is kind of checked up on. This data is fresh. This data has, you know, the expected volume.

3:58 I have all my business dashboards up to running date, right? So that gives us kind of, you know, trust factor. So, and the last, not but the least, but it's integration, right? That's one of the key component. With the modern data stack, there are so many data tools, right? You have different databases. You have your Looker and Tableau. You have your ETL tools. You have your ELT tools like dbt, you know, which plays a vital role. How do I kind of interconnect these data sets into one central repository which will truly enable your data democratization, right?

4:28 So that, so those are the, like, kind of, I would say, the f-four core fundamental features which everybody should look for a catalog. But having said that, these are not just the only factors which we should drive to for looking at a catalog, right?