Evaluating a modern data catalog in 10 minutes

Mihir Modi

0:06 First step, what do we do? We try to identify user problems, talk to them, do some surveys, go through our internal documents and Slack channels and different interactions with the business. That's how it prompted us that we need a data catalog, right? Next step was you as a company need to define your own requirements, basically, right? You need to come up with your fundamental three to four core values and try to see if those core values are aligning with your data catalog partner, right? So that's how we started. We defined our requirements, and you'll see in the next slide what were the core components of our data catalog, basically.

0:43 We started identifying common theme across different business groups, right? So we reached out to individual business teams, identified their pain points, and grouped together and defined a theme out of it which is commonly coming, right? And that's where we thought that, hey, there is one particular theme which is coming out, and what might be the best features to first target for, basically, right? And that's how I'm going to now show that what are the key component features? Now, I would say these are the four pillars any data catalog should have. First of all, search and discovery, right?

1:14 If, if you are starting your journey with a catalog, and as a user, if you think you need an intuitive search engine, a Google-like search engine, which will quickly tell you that, hey, here is my data asset related to, let's say, impression, or here is my data asset related to any particular category, right? So efficient search. Metadata extraction is one of the key components. Gone are those days where you take a, a lot of time extracting that metadata from your databases, whether it's Redshift, Snowflake, or any particular database, right? So your search and discovery aligns with your glossary, right? How do I preview my data, right?

1:52 So as a user, once you log in to a catalog, you should be having an ability to search data or see at least a sample record, right? Let's move on to data access and governance, right? Now, as Steven, you mentioned, there are different personas, and our approach was that how can I make a tailored experience for each and every business users? So the idea was that some— when a business user logs in, he should be looking at the only the top 10, 20 assets, which are highly curated assets for a business user. And that's where the data access and governance plays a role. How can I easily classify my data?

2:28 I might have some sensitive data. How can I tag those assets and put a masking policy on top of it, right? And on top of it, with respect to data access request, let's say I don't have access to a, a, a data asset, and if I am curious to know more about it, your governance platform should be able to route a user to, you know, navigate and then raise that access request and quickly turn around and get you the access, whatever is required, right? The third critical component is lineage and quality.

2:55 You want to understand how my data translates from raw to my staging layer to my transform layer, and what are the different transformations happening during this lineage process. Like how, how is the data traveling basically from my source to my target? And aligned with that, building that trust factor is critical. And how do I build trust? So we have data observability platforms which can alert on a particular asset that, hey, my data is stale. I want to know as a user that, hey, why do I get a sudden spike?

3:26 If you think from the operation perspective, identifying why, what went wrong, identifying, hey, what can be the root cause for these kind of data issues, and getting to know in the catalog itself that, hey, I can trust this data. I have a trusted score for this asset. Everything is checked upon. This data is fresh. So that gives us trust factor. And the last, not but the least, but it's integration, right? That's one of the key component. With the modern data stack, there are so many data tools, right? You have different databases. You have your Looker and Tableau. You have your ETL tools. You have your ELT tools like DBT, which plays a vital role.

4:03 How do I interconnect these datasets into one central repository which will truly enable your data democratization, right? So those are like, I would say, the four core fundamental features which everybody should look for a catalog. But having said that, these are not just the only factors which we should drive to for looking at a catalog, right? There are a lot of factors which a modern data catalog offers, and why do we need it? So let me begin with one of our core fundamental feature which we were looking for and strategic move was having a one-stop data platform.

4:36 As a user, you come, you discover your asset, you understand what that asset talks about with respect to the metadata which has been put into the catalog. You understand key KPIs and metrics right from your glossary. You understand about your domain. What top questions does it answer? How do I really calculate this particular metric? And then you navigate your... To Looker dashboards right within the catalog, right? And on top of it, you have a insights, you have a query tool capability. So now once you understand your asset, you also are hooked up onto query right within the platform. You need not go to a SQL workbench or a a different ID tool for querying, right?

5:14 That was one of the differentiator which we was looking for, and Atlan clear stood out in terms of having that one data platform. The second thing is partnership. I would say out of all these four, partnership and one-stop data platform were key core components why we chose Atlan. We being at Fox, we wanted someone to, you know, partner with and also allow us to shape that future of data governance. And Atlan has been fantastic so far. It... There has been collaboration of ideas between both the teams, how do we solve common business problems, and how can we be innovative, right? And Atlan, it is, it built on open AI, API architecture, right?

5:50 That clearly allows us to have an extensibility across all our data stacks. I think those were two key game changers. The two key components of what a modern data catalog offers is truly activating the metadata, right? So I would say metadata is the new gold, but it also depends upon how do you activate that, right? If you are having your dashboards being broken, people still don't trust your data, you don't know on a real-time basis that, hey, how do I... Why is my data stale, right? That itself tells that, hey, you need to activate your metadata, right?

6:22 Right from your raw data pipelines, if something is breaking, an alert should quickly go to your BI dashboard or the rest of the data ecosystem and notify a user before he finds that, hey, there is some problem, we are dealing with it, right? And one thing which kind of stands out with this, all these modern data catalog 3.0 is bidirectional flow of metadata, right? So you understand your data asset from one, from catalog perspective. But let's say as a user, I log into Looker or Tableau, right? I want to understand what this dashboard is talking about. What are the key metrics it serves to, right?

6:56 That's why that bidirectional flow of metadata happens when you truly enable active metadata. And last but not least, at Fox, we wanted like a simplified governance approach. It shouldn't be like stressed upon that, hey, you have to follow this process, right? It should be seamless, it should be smooth enough, and part of an integrational journey across all the data stacks, across all the data teams. I think Atlan and these modern data catalog tools would allow you to do that thing. Next is now we looking at that evaluation process. We studied what are the different technical and business requirements. We distribute RFP to vendors.

7:31 So we distributed RFP to different vendors, started evaluating working with different vendors, and restricted to only partners which were key to partner with us and align on our core values and principles. So we invited three to four vendors for demo. We did extensive POC with our use cases, and then we requested for reference calls. And then we started preparing an evaluation scorecard. So here is a sample kind of evaluation scorecard. It will give you some glimpse of it, right? So on, on the left-hand side, you see different features which I've listed down. These are the high-level features which we thought of, and we decided to do a ranking system, right?

8:09 We started giving weights to e-each of our category and started ranking based on our POC and our use case, our kind of interaction with the customer vendor, and we came up with a score. Again, score is not the only evaluator here, but it will give you some glimpse that, hey, which vendor tool stand out against the others, basically. The, the weight allows you to differentiate different data catalog partners across the industry. And for us, what I thought and some of my team members thought was that, hey, if we pick top two, top three features which we're looking for, and what are those key features, right? So data discovery, as I was explaining, becomes really important.

8:48 That's why we kind of assigned 20% weightage to it, right? If we solve these two problems of having a governance platform and data discovery, I think we, we have won, I would say, half the battle, right? Definitely, rest of the integration end plays a also key role, but these are the two differentiators. So if you look at some of the points which I've mentioned that, hey, can I have a support for my business glossary, right? Can I do some basic sampling or running my SQL queries onto my platform, right? How efficient is my natural language search? Am I getting lost or no?

9:22 I-I'm getting a platform wherein I can quickly find and discover my assets, right? So that kind of made us to align and assign a higher weightage as compared to others, which kind of played a key differentiator.