Supercharged automation to map your data estate

Matt Madden

0:08 Everyone, and welcome to the latest and greatest Atlan Activate: Supercharge Automation to Map Your Data Estate. My name is Matt Madden, and I will be your host for today. So the theme for today is really about supercharged automation. So when you think about automation, it's really about the power of reducing manual processes and tasks in your day-to-day work life. So to start it off, we're going to actually have a poll, get your guys' thoughts around how much time you're spending with manual data tasks. So it looks like a lot of people are spending between three and four hours. Second place is more than four and one to two.

0:52 So a lot of manual tasks going on, and these could be things like data collection, simple approvals, updating information in a document, or even just simply copy and pasting. So we're here today to really talk to you about how Atlan helps reduce these types of things. So with that, I'm gonna turn it over to the Atlan co-founder, Varun.

Varun Banka

1:15 Thanks a lot, Matt, and I see sixty people right now. Thanks everybody for tuning in. Like Matt said, Activate is our flagship online event where we bring the best of the world from active metadata to all of you. And today's, today's agenda. What, what, so what we have for all of you today, we're gonna start with the activation stack. The primary thing that we'll go through with a bunch of amazing demos for all of you is automation and activation, what does that look like with Atlan, and an amazing surprise at the end that folks can't miss. And in case we have time, we'll do some live Q&A.

1:57 Otherwise, like Matt said feel free to add your questions on chat. We'll take it from there. So on that note, let's get started. Now, before we got started today, 'cause building this and what we wanna present to all of you, we started asking some questions to Google or the new ChatGPT Three, and started seeing if we can get creative with some prompts. I think the first one was we started asking, "Hey, what's the..." And I'm just gonna hold this here. What does active metadata mean via the new world of AI? Some good rhyming there.

2:41 And I'm sure if I show you all my prompt history, it's gonna be ages, but but instead of focusing on the chat prompts, let's, let's jump to today's agenda. Okay. So at Atlan, what, what's our mission? Our mission is to help humans or data people like you do more together, and it very well fits with the whole agenda that we have for all of you today. But before we start with activation and look at humans of data, we thought, let's look at some other, other teams that we have in all our companies. So we looked at two teams.

3:18 We looked at a sales team, we looked at a GitHub team or an engineering team that's driven by GitHub, and a sales team who's home in Salesforce. And like— And we looked at what does the activation stack look like for these teams. On the left you have an activation stack for Salesforce built on accounts, funnels, app exchange, core CRM. You can do now visualizations, communication, a lot build a lot of reports, do forecasting. Similarly, how GitHub has empowered engineers, so the primary unit is a repo, but they have version control built in, pull request, issue projects, and together they activate their functions. Where the Salesforce activates the sales teams and GitHub activates engineering teams.

4:06 So what does activation mean from a metadata perspective? So here it comes. What does metadata activation stacks looks like? So let's— So there are too many, too much things happening on the slide, so we will— I will help you navigate through this slide. We will start from the left, which is you have the active metadata platform. Below you have the data estate. I'm sure the data estate is familiar to all of you who have joined us here. You have the databases, warehouses and lakes, the business intelligence, the pipelines, your notebooks, your Kafka queues, all the documentations in Confluence. A lot of, lot of now teams use a lot of APIs, queries.

4:48 They save a lot of queries. Data quality metrics, incidents, observability and infrastructure. Together they create data estate. And we believe it's not just about rows and columns. We believe together they form our data estate. And top of it is the Atlan's active metadata platform, built using some common prin— pillars, and we'll touch upon some of them today as part of the demo. You have the marketplace, the metadata marketplace, playbooks, open APIs, integrations, SQL parsing to mine a ton of observability, lineage data. That's creates the metadata platform. If you go right, you have the workspace.

5:31 So the workspace, Atlan workspace, which the end users use is built on top of the Atlan metadata platform, where you have— we power things like data discovery, data lineage, business glossary, SQL insights, reporting dashboards, and a lot of them that you're seeing on the screen. And together, data teams use both the platform and the workspace to drive ton of use cases. And again, today we'll be talking about some of them whether this is cost optimization use cases or data teams reducing the service overhead, moving to a product mindset, reducing the number of tickets or doing root cause analysis. Or doing access control, decentralized governance.

6:12 While in the next fifty minutes, we won't be able to cover all parts of this activation, but I'm sure we'll touch upon most of it during the next forty, forty-five minutes. And in case anything on the screen intrigues you, feel free to reach out to us. We'll be happy to answer more questions async. So let's get started. So let's directly jump to the demo.

Varun Banka

6:39 The first part of the demo is around the Metadata Marketplace. So Metadata Marketplace is the collection of packages that helps us to extract, mine, transform, create relationships across your data estates. So just like could, could give you a reference example, in the world of Python, there are four hundred K Python packages that are available. The world of NPM, which the JavaScript developers use, or GitHub, where there are amazing apps and actions on top of GitHub. AppExchange on top of Salesforce, where there are a lot of tools that are built up on the core activation sales platform to drive use cases.

7:19 Similarly, Atlan has built a similar Metadata Marketplace that today powers a lot of metadata use case for a lot of our customers. So we've opened three kinds of packages right now on our marketplace. Number one, connectors. Connectors are straightforward connectors, goes, goes to your sources, touches the information schema, looks at all the logs, and extracts metadata to create technical metadata on the platform. Miners, which are more— which goes through your operational metadata, goes through query logs, extracts popularity, drives lineage, and those kind of cost use cases on the platform, and we'll touch upon some of them today as well.

8:02 And utility, which data team does not need a Swiss knife, so if you make mistakes, utility is there to help you in that process. So without, in fact, spending more time on this, let's actually see what the marketplace looks like in real life. I'm gonna just share my screen, take over from Matt, and we'll go from there. What you're seeing right now is the Atlan workspace. If you looked at the stack, we have the platform, but this is the workspace that the end users use. So for the sake of demo, let's create a new workflow. We'll go to We'll go to the marketplace and see how that— all the packages look like.

8:40 So like I said, so let's— Whether this is Snowflake, your warehouse, these are packages, DIY connectors, where you can, as long as you have the credentials, we can extract all the technical governance metadata that exists in your warehouse. Or you love Databricks. We in fact recently built a native integration with Unity Catalog. So if you're u-looking to drive lineage via Databricks Atlan is Unity Catalog ready. Or so th-these are just from a source perspective, but what if if you want to bring, create cross source lineage?

9:17 So we have native integration with Fivetran, where we can look at the APIs and create end-to-end lineage on Atlan out of the box, self-serve, as long as you have your credentials to go through the next step. And apart from the modern data world, we've started investing in some legacy sources like the Oracle SAP HANAs of the world. While a lot of our customers use Databricks and Snowflake, but all of them have one or two Oracle databases, Postgres databases where they store their actually transactional data.

9:52 Now, beyond these connectors, you have the miners, which helps you to out of the box bring in your ops query metadata, lineage metadata, popularity metadata to drive a lot of data activate data use cases on top of Atlan. Again, self-serve, you can decide to do a connection. You can decide what kind of extraction method you want to drive out of, w-uh, what, what, at what timeline you wanna start with do some pre-flight checks. We've made it very easy for data teams to set some of this up. Now, once you've set some of this up, how does this actually look like when you look at from an Atlan perspective?

10:35 So let's go to assets, which is your data marketplace, which brings in assets across assets and metadata across your data estate. What we are seeing right now is a list of all your assets across the different sources that you've set up. So for the sake of this demo, I'm just gonna look at what have you gotten in from a Snowflake perspective. So what you're seeing is obviously you know now which database, which schema, this table lands at, what's the actual model. So if you're using dbt as a transformation layer, you get to see actually the model that is driving this. You, you get to know when was the last model run. Is it stale?

11:12 Is it not stale? What's the configuration? What's been materialized? You can get to see classifications in case you are capturing these tags at Snowflake or dbt. You can bring some of those in. So this is from a business metadata, technical metadata. But we also touched upon miner, where we are able to capture all your queries. So not only that, but out of the box, we'll be able to bring usage capabilities. So on this, you're seeing that there have been sixty-five queries that have been running by ran by three users in the last thirty days. This is how the number of queries have changed. The users is changing. When was this last queried?

11:48 Who should be the person that I might reach out to in case I wanna know something? Is it stale? How fresh is this? When was this— When was the last row that got updated from this? Again, out of the box, without any effort, all of this is available automatically or automatically on Atlan. And you can check out here as well. This is your companion sidebar, where you can see the rich metadata about these assets. But we took one step ahead. We don't think about just usage, cost. Everybody cares about cost now, especially how the next year is gonna be. All data teams want to improve their cost warehouse cost, teams cost.

12:29 So what you can see is who are the top users that have been using this, which have been the recent users that have been using it, what are some of the slow queries that you want to optimize, change your warehouses, what have been some of the expensive queries in form of credits that have been— actually been running that you can change the warehouse for or tune your workloads, and the total compute cost per asset. While to understand every query cost from a dollar perspective, we'll have to do a PhD because I think— I don't think a lot of people...

13:01 It's very difficult to know how, how Snowflake charges per query, but you get to see your total credits for that one particular asset for the last thirty days that's been happening. So technical metadata, tr-transformational metadata, users metadata, cost metadata, everything available via the metadata marketplace. Now, this is about metadata, but what about lineage? So let's look at lineage. So you get to see upstream, downstream. So we able to bring in end-to-end lineage right from your transactional database to your operational warehouse, to your BI, Business Intelligence lab. We are able to bring this lineage information right here on Atlan. You see a list right now, but let's see a better view for this.

13:51 So as soon as I switched, it starts showing a view where you have the fact orders table. This is coming from these views. There is a dbt model that must be transforming this taking in. This table is also linked to a warehouse to a BI lab, which is Tableau. I can expand that, go more granular, look at the exact dashboard that is running this. We also integrate with dbt metrics. So in case you've started using it, you can start seeing how these metrics, revenue customer metrics are derived from it. You can go upstream as well in case these are coming from a, a transactional database.

14:27 So right from your Postgres, let me zoom out a bit. Right from a transactional database, your different layers of your warehouses, your BI lab, be able to bring this end-to-end lineage on Atlan. Now, now we're able to see at a one asset level. We just looked at fact orders, and I'm sure we can't go through fifteen K assets together today. So let's look at a reporting center where if you're an admin, if you are a data leader, you want to look at a high level what your data estate looks like.

14:59 We have a curated dashboard that's available where you can see across your data estate how many SQL assets exist, what's the workload across your dbt, how many processes the pipeline is running, what's the asset enrichment that looks like, who's being— who are the people that are helping it to enrich and documenting what's been the updates. Some of the rich, very rich metadata that's available at a thirty thousand feet for data teams. Now, so we saw from the power of metadata connectors of how they were able to bring all kinds of metadata and create that asset three-sixty profile.

15:39 Now, in fact, some of the popularity metadata, some of the data teams have started even using it for cost optimization. We had a customer that used a lot of popularity lineage data to actually save two hundred K annually on their warehouse spend. They're able to optimize spends based on unused asset. They're able to clean it up, reduce their compliance exposure, not just cost, but reduce the kind of assets that are actually exposed that are possible exposure to PII, able to fine-tune some upstream, clean up downstream with dashboards that are not getting used. So together, able to drive use cases based on this metadata. Moving on.

Varun Banka

16:20 Let's go to the second part of today's demo, which is playbooks. Now, as we've, we've been working with data teams, one common theme all of them are talking about is they want to automate metadata at scale, right? And we've seen, seen things like this on the right, right? Like all other teams, marketing teams, sales teams are already using similar kind of things, whether it's via Zapier, IFTTs of the world, where if you get a lead, do X, do Y. And when we started doing discovery with our data teams, we got tons of use cases. All data teams wanted to drive use cases like this, but they may not have bandwidth.

17:01 They didn't have engineering folks who can help them to actually write scripts because engineering folks are busy in fixing pipelines or doing requests, and data teams wanted to solve some of this. So introducing Atlan playbooks, which is the first low-code, no-code metadata automation, similar power like Zapier, but applies on your data estate. So, so I'm sure we, we can't go through all of these use cases today, so we've picked three amazing use cases for this demo. We'll start with the first one, which is, which, which is asset deprecation. So let's, let's go to the demo screen, and then we'll talk about the use case here.

17:42 So let's say you want to deprecate assets which have not been queried, which have been not been queried in the last three, three sixty-five days, and you want to deprecate them, add a certificate, and later decide whether you want to actually deprecate it in your physical lab. So it's almost like a retention policy or a deprecation policy. So let's see what we can do here. So we have filtered all our assets into Snowflake. Let's say I want to find all assets which have been last queried before twenty twenty-one, forty-first December. As soon as I switch that, looks like it reduces the scope.

18:20 Suddenly we have less assets, which likes a lot of unused assets, which is business as usual for a lot of data teams, right? Now, if you just quickly hover on this filter, let's convert this filter to a playbook. So for the sake of th-this demo, I'm gonna... I'm gonna call it Snowflake Deprecation Playbook. This— Let's do a quick create. As soon as I did that, it automatically landed up into the playbook section in the governance screen, where it automatically applied some of these filters for you, saying, "Hey, if connector is Snowflake and last query is before this," it gives you a preview that there are fourteen k actually assets that have not been queried.

19:11 And you can see a quick preview in case maybe you wanna change the rules, do something, and you can then take action. So for the sake of this demo, what we are gonna do is, for all the assets which have been not used, maybe for last three sixty-five days, let's add a certificate deprecated so that folks know that they should check before using some of this because this might be stale. So as soon as you do this, it's as simple as that. We just did some filtering, clicked the button, it was able to create, and we can decide to schedule this. Now you can decide when you wanna schedule.

19:50 What this will mean is, you can set up not this as a one-time thing, but you can ensure that every time there's an asset which crosses this window of staleness or unusage, you can keep marking them as deprecated, right? And later decide whether you actually want to clean it up in your warehouse or decide not to clean it up because it was actually relevant. So that's, that's the first playbook, which is one of the most common playbooks that we've been seeing, especially in the new macro world, where everybody wants to save cost. Let's go to the second one, which is change and add ownership. So for this, let's take a Salesforce example.

20:31 So I'm just gonna clear all the filters. We will switch to Salesforce. As soon as I switch to Salesforce, the whole discovery experience switches to a Salesforce experience. Now I can see only Salesforce assets, which are these objects, fields, organizations that have been set up. And one common use case is I want to tag all of them as the owners for all of this is the revenue ops team in the company. And I want to make sure if there's any asset that gets set up on Salesforce, it's always tagged as owners, as rev ops.

21:04 This is to ensure people don't reach out, people reach out to the right people in case they have— they need any clarity. So in this case, let's do the same. We'll convert this again, filter into playbook. Let's time, this time let's call it Salesforce Ownership Playbook. As soon as we do this, again, we've made it easy. All the things are set up here. You can see a quick preview. But in this case, let's do Owners, and maybe we have a rev ops group here. Yes, we do. So we can set this up again, decide to schedule it, or just run once. And in future, we are bringing more actions.

21:47 We're bringing— where you can share this on an email, share it on Slack, create a Jira issue if something goes wrong. If your popularity data or if your any distinct data got missing, you can create a Jira issue. So tons of use cases coming up with the action framework going forward. Let's switch to now the third, third one, which is what we call active trust management. How do we become proactive and ensure if someone uses a data asset, it's highly trusted? And one metadata that plays a big role is the profiling metrics. This could be your distinct, your mix, your minimum, your maximum about a column.

22:27 So we took a different approach for profiling, and we leverage our playbook framework to drive profiling for data teams. So in fact, we built a special playbook called the Profiling Playbook. So let's try this out as well. Who doesn't like an icon? So emoji, so I'm just gonna add an emoji. Let's select. I want to run it for that little demo. So now you don't want to maybe profile for all your assets, all your table and views, because, one, you wanna save cost. You don't— Maybe all of them are not getting used. You can use popularity metadata to only do profiling for only your top assets.

23:14 Or maybe if, if you have a certification framework, you can say, "Hey, I want to profile, but I want to only profile assets which are verified, because those are the assets that my team cares about." Right? So I want to make sure they are trusted, the right metrics are available, so when they do next time they do a join, they know which are the right joins to do. They— If they are using a column for a data science model, they, they should know which is the right column, or they should use it, or they should not. So again, you see a preview, like looks like only five assets are available. You see an action.

23:47 Now you don't see all the action, but you see profiling. You can again decide there is a collection of metrics that you can decide. There are some advanced metrics as well. Again, the power is in your hands. As data teams you can decide what metrics is relevant for your team, what elements is available for these assets. Again, if you wanna schedule this, you can schedule this, or if you just want to run once to get some sample data available, sample metrics, you can set this up as well. And once you run some of this, so some of this are already running. So let's go to reporting center.

24:29 Again, we've made it easy again for you to always look at it thirty thousand feet. So as you can see here, there are three active playbooks. One has been scheduled. These many assets are getting updated. What have been the number of playbook runs? Are there— You can do observability in case some of the playbooks are failing. Who are the users that are using playbooks? Again, all the rich metadata, aggregated level of metadata available for you to have a thirty thousand feet control across your data estate. So in the, in the last five minutes, we created three playbooks.

25:02 We created a scalable deprecation policy playbook, which will help us deprecate assets on a regular basis, not as a one-time project. We created an ownership playbook to make sure ownership is always assigned with every new asset that might land. And third, we created an active trust playbook, which is you are able to profile your top assets to make sure when somebody's using a column for a data science model, when somebody's using the col-column to drive new insights, they have these metrics available right there in context when they're driving their use case. Let's go to the next part, the third, the, the third part of the demo.

25:43 So while we thought about low-code, no-code, data teams are always an edge case. There are enough data teams that want to build. They don't want to be limited by low-code, no-code. They want to build their own use cases. And for that, we took a next step. We were like, "Hey, in the world of access management, teams are building these use cases, whether these are Okta's, when somebody logs in, your teams are driving ton... IT teams are driving ton of use cases. Or if you are an e-commerce company, Shopify events, payment events, Salesforce events, those are driving tons of use cases for sales teams.

26:19 Observability, incident management are more, more and more straightforward, the kind of use cases SRE and cloud teams are driving. But what about metadata management?

Varun Banka

26:28 How do we drive production-grade, scalable, event-driven use cases in the world of metadata? So for that, I'm just gonna pass it to Rohan, who's gonna take over this part of the demo.

Rohan

26:43 Awesome. Thanks, Varun. Before we dive into use cases, active metadata use cases EventBridge use cases, let's, let's talk a little bit about EventBridge, what EventBridge is. So it's a service by AWS in which basically what happens is you can throw in a bunch of events, and it easily allows you to consume and build use cases on top of that. Right? Now, l-let's imagine what that can mean for something like Atlan. So imagine someone raises a announcement on top of Atlan, a table, and you want to create a Jira ticket out of it, right?

27:18 And one of the use cases that we heard from our customers was Atlan was sort of becoming a black box that someone changed the ownership of a table, but no one got to know about it. So now with this integration, what you can do is as soon as a ownership changes, you can send a notification on Slack. So I'm going to show you a lot of demos. The possibilities are infinite. But let's see. Before we jump into that, let's see how the setup happens. So you just go to your Atlan account, go to admin, and like you set up your Slack and Jira integrations, it's really quick.

27:53 You just enter your AWS account ID, AWS region, and it's done. Like, it's just a super simple two click, two click process. And after that, you'll start getting all the Atlan metadata events in your EventBridge account, and you'll be able to drive use cases on top of it. Okay. So today I have two... Plan two awesome demos for you. So I'm going to jump into the lineage real estate that we have built. So here you can see the data is flowing from Postgres to Snowflake via Fivetran, and then there's some dbt transformation happening and finally landing into a dbt view, Snowflake view, right?

28:33 And so for today's magic trick, what I'm going to do is I'm going to mark this cu-customer name as PII because I know it has some PII data, and apply a masking policy inside Snowflake. So before doing that, let's see how the data looks like today. So I'm just going to go to a table in a click. In Atlan you can easily see the sample of your data. So let's see the sample. I can see it's storing some customer names which shouldn't be visible to everyone.

29:03 So what I'll do is I'll, I'll just mark it as PII at the transactional store level so that after that what Atlan will do is it'll automatically detect that it has some downstream lineage, and it'll automatically mark all those downstream lineage assets as PII. So let's go here. And you can see there's a suggestion popping up. We'll, we'll talk a little bit more about it. But this seems relevant. I'll mark this as PII. Now, behind the scenes, what will start happening is Atlan will take this journey and start tagging all the downstream assets as PII.

29:39 So it'll tag this in the Snowflake table, the Snowflake view, and the final Snowflake view that we're going to use. So while that happens, I, I'll just quickly show for the technical folks who are present here how the event looks like, which you get and you consume and drive use cases. So you can see this, the whole payload. You can see the operation type is classification add. You can see what exactly changed. And to increase the developer experience, what we also do is we send you the full asset JSON so that you don't have to make another API call to just fetch the asset, and you can just utilize this event. Awesome.

30:18 Let's, let's go back to lineage. Let's see if the propagation actually worked. So now what I'll do is I'll just check out this column real quick. And you can see the PII just showed up here. And it, it also tells you where it's propagating from. So it says propagated from name. Now, now let's check out did, did the masking actually got applied? So I'll again hit the sample, and let's see. Here you go. So you don't get to see the customer names. And so, so I'll tell you behind the scenes what we did.

30:52 We consumed the event that I showed you here, and using that, we made a API call to Snowflake and applied masking on it. So if I go to Snowflake, currently I can see there are customer names, and if I just run this query again, you'll see the customer names are also masked in Snowflake, which is great because now if someone uses this in, let's say, a Jupyter Notebook or a BI dashboard, they won't get to see the customer names. So this was one of the use cases that we wanted to show you today. The other one which I wanted to show you is really popular is getting notifications.

31:30 Like, as soon as something changes in Atlan, I want to get notified, right? So let's say Andrew just left our org, and instead of him, I'll tag Mark as the owner, and let's see what happens. Owners have been updated. Now if I go to my Slack workspace, you'll see Rohan updated owners to Mark. And if you, let's say, change the description. Let's say, this table no more, no more contains all all product data. It excludes APAC customers. So you update the description, and you'll see a notification instantly popping up in your Slack. Yeah, this is it for today. Turn back to you.

Varun Banka

32:12 Thanks, Rohan. That is quick. Okay, we, we'll get back to where we were. So what Rohan talked about was the power of events, right? Which is infinite use cases we're really excited about. I think some of the customers have started using this as well as part of a beta program. They started using it for their driving use cases. I think one of the customers that are in Fortune One Hundred tech company, they wanted to make sure schema changes are notified to different teams.

32:40 On one hand, they want to notify for security team to make sure they know if there's a new column to evaluate, to audit if it has sensitive data, to triage, or to have an audit trail. Similarly, they want to notify for engineering team if there's any downstream impact or if there's any, any changes that have existed. So super excited for this. Please feel free to reach out to your nearest customer success person. We'll be happy to learn more about your use cases and make you part of our amazing beta program. So what does current and future look like?

33:14 So Atlan Open APIs, already a lot of our customers have started using it to drive the programmatic use cases. AWS EventBridge production grade, leveraging the world of your own cloud to drive event-based use cases at scale. SDKs, who does not love them? So we have a Java SDK in transit right now, which is in alpha. Python SDK coming up next quarter. Webhooks for customers who want to quickly just without investing in EventBridge, do some POCs, drive some event-based use cases. Again, that's coming up next quarter. And developer portal to make sure we make it easy for all of you to leverage our leverage our APIs, use our SDKs to drive your own use cases.

34:03 Super excited by the kind of things folks will— people are gonna come up with.

Varun Banka

34:08 So that ends this part of the demo. But if you look at, if you look at this use case, there is still something missing here. While we were able to notify the security team or the engineering team, but the whole world is moving to left. And I don't mean the left and right in the world of political world, but where data assets are getting produced. Which is how do we move governance to when the data asset is produced? So let's take— we'll take maybe an example of in the world of dbt.

34:42 Where you come into a dbt model, make some changes, and I'm sure there are thousands of times it has happened that unknowingly you've impacted lives of so many people because there were tons of dashboards, tens of derived things that were dependent on your change, and you didn't know about that. So what we've done is we've brought the power of active metadata right to your GitHub. So let's take— let's look at this example. So in this example, we've opened up a pull request. Let's see what we've changed. There's an existing fact orders dbt model. We've just changed some conditionality, and I'm like, "Okay, this is not gonna impact anyone. I'm just doing a small change.

35:23 I don't need to reach out to anyone. I can work in my own silo." Let's do Create pull request. Now, as soon as we create a pull request, now what's happening is now we are running a GitHub action. So apart from merging being blocked for me we're running some GitHub actions and trying to get all downstream assets which are getting impact. So it'll find out. So here you go. Voila. So you have Before you can decide to merge or go to review, you can look at every asset that going to get impacted by the change that you've made on this code.

36:01 You can reach out to the owners, collaborate, find out, notify them, understand if if this change is okay. You can drive all these use cases here. And if you want to know more about the assets, I'm sure there's hundreds of times that have happened that I had no clue that this dashboard or this, or this data sources are getting used. I can quickly come back to Atlan again, look at what the asset looks like, who's using it, whether it's used, unused. I can, again, self-serve, take a lot of these calls as, as a data engineer or as an analyst engineer. So let's go back again.

36:40 So this is about how do we make, make sure, and I'm sure all of us believe that prevention is always better than cure. So on that note, let's move to the last part of today's demo, Trident.

Varun Banka

36:58 So Trident is our investment. This is our way to make metadata enrichment fun, engaging. We, we give— pop up a lot of suggestions for you and help you take a lot of these enrichment decisions via our suggestions. In fact, from the time we rolled out, Trident has already contributed one-third of all description updates on Atlan. And looking at this impact, we have started now expanded this to owners, terms, and classifications. So today we use a basic rule-based engine find duplicate assets recommend if there have been duplicate description or owners. And every time customers used it, they were like, "Hey, can you use AI/ML?" Right? Why don't...

37:47 I'm sure everybody's favorite word the acronyms AI/ML. They're like, "Okay, why don't we use it?" And we wanted to make sure, like, we when we actually do it, we actually use leverage AI/ML. Unlike, unlike a lot of people who say that in the space, they're using AI/ML. And this is, in fact, one of our company values, which is we want to never be satisfied. We are okay changing directions. We are okay changing plan. We want to strive to get better and better every day. So on that note, and this is the surprise drop today, introducing Trident AI. The powers of Trident, powered by what we started this— today's discussion with, GPT-3.

38:36 Disclaimer, this is still in development and still is in experimental stage, but the kind of use cases we see going forward that this can help. We all have this order ID table that nobody wants to document, right? So can you ask description for column names? We all have a great business glossary, right? There is tons of SaaS, whether if you're a SaaS company, you, you have things like MRR. If you are a supply chain company, you have different kind of terms on supply chain. README, everybody loves reading README. Who likes to write README? And I'm sure many more.

39:13 And I'm sure as I've been speaking, a lot of you have already started thinking about use cases that this can power. So we are excited to hear more from all of you. If there are more use cases you think this can power in the world of metadata, please reach out to your neighborhood customer success person or reach out to our team. We'll be more than happy to hear all these use cases and see how we can leverage. Do we have a demo for it? Oh, yeah, we do have a demo for it. So let's, let's jump, let's jump to the demo. So let's go to Glossary. We are seeing customer success. I'm seeing customer churn.

39:55 If I'm going on the right, you see Trident giving some suggestion for this now, but maybe you don't like the suggestion. You want to check with our new introduction, our Trident AI. Let's see. Now, as soon as you do this, you have a quick recommendation, and you can apply this, and enrichment is done. So let's see. Maybe let's try health score. So we are gonna have a dis-discussion. Let's see. Let's see the kind of recommendation. The AI pops up. We Every day, we are surprised as well with these kind of descriptions and suggestions that are coming up. So let me just apply.

40:44 So when we started using this, we started getting creative, and we're like, "Hey, can we push this to README as well? What does the prompt looks like if you have to actually drive this to README?" So while we saw a lot of description, so here is what you have in README. You have a quick one, one suggestion. Looks like because this is empty. And obviously, some of these APIs are cached right now. That's why it's not going and loading but you get a full markdown. And folks who are interested in the prompts that we are asking, happy to share the kind of prompts, but it helps you to build READMEs like this.

41:23 Okay, let me also try this. Let's apply. Let's save. In less than a minute, two minute, we have documentation and rich documentation across our stack. And maybe let's do one column since... If this sounds unreal or if this feel like this is actually unreal, don't worry, we, we, we feel the same when we started experimenting with this, so you're not alone. So let's see order ID while Trident give a description. Let's ask latitude, longitude. I'm gonna do some...

42:16 I think I can keep going on for the whole session today. Okay, let's see. Oh, this is the five plan ID column. So it actually pulls up this recommendation for this. Okay, I think I need to stop now. So on that note— So we have Trident AI, feel free to share across more use cases that you folks have. Be as crazy as you can and we'll try our best to bring that to the world of metadata. On that note, that's a wrap from a demo perspective. We can take in any questions, anything maybe that has not been answered, or anything the folks are curious about.

Matt Madden

43:02 Hey, Varun. Yeah, so it looks like most of the questions are being answered in the chat in Q&A, but there was one that just popped up around: Is it possible to train the AI with recognizing typical abbreviations used in your specific company?

Varun Banka

43:19 Yeah. So right now we are leveraging the Generate GPT-3 as to recommend some of this. So all of this recommendation you see is direct via GPT-3. It's not necessarily trained on anything, I'm sure, but that's a great idea, actually, around this. It just— we have to figure out what this means from a data-sharing perspective from an... If there are specific abbreviations that the data teams have that they don't wanna share across. But yes, as I think we evolve and maybe do a custom training across all the metadata, that's a great direction to go into. Thanks for asking the question.

Matt Madden

43:56 Yeah, and this was actually answered in the Q&A, but I thought maybe you could talk a little bit about it as well. You, you showed the cost optimization— Yeah... In focusing on Snowflake. Are there any other data warehouses that we're looking at?

Varun Banka

44:10 Yeah, we're going to BigQuery next. So BigQuery is next for us in the radar. They'll look— we have a bunch of customers who are superly eagerly waiting for us to take every investment that we have done with Snowflake with BigQuery, so BigQuery is next.

Matt Madden

44:26 Great. And kind of following on to that there was a question around the asset deprecation. Can this be dynamic dynamic filter for all assets unused in an X amount of days?

Varun Banka

44:40 Could you expand maybe if you want to unmute, like, what does dynamic mean here? If maybe the person is on the call, you want them to unmute.

Attendee

44:51 Yeah. Thank you, Matt. So my question was from the perspective that we showed, right? That if a particular asset is not used before thirtieth October, twenty twenty-one, and then we can mark it as stale.

Varun Banka

45:03 Yeah.

Attendee

45:03 Instead, I create dynamic rule saying just mark any asset older than one year as stale, right? So I don't need to create these kind of rules. Would that be possible? I think the answer was yes.

Varun Banka

45:15 Yeah. Yeah. You can create, you can create any kind of rules, any kind of dynamic rules on the system. So think of this— So, and this is how we're thinking about this, right? On one hand, we are creating an asset three sixty. We are bringing metadata from all your data set, whether this is operational metadata, warehouse metadata, your data quality metrics. We're bringing that all and creating an asset three sixty first, right? From metadata activation. And then we are building these layers on top of it. So you can create any if and else rules on top of it.

Attendee

45:47 Makes sense. Makes sense. Thank you. Yeah, I was looking for the exact thing. If three sixty rules are possible, that's great. Thank you.

Varun Banka

45:54 Do you have any more questions, Matt, that we can take in?

Matt Madden

45:57 I think that was pretty much it. All the other questions have been answered, so just wanna thank everybody for taking the time out of their day to join us today. Let me just share the my screen one last time, give you guys some more information. Anything you're interested in learning more about today, if you are a customer, I would suggest reaching out to your customer success rep. If not, you know, somebody from our sales team will definitely be in, in touch with you to discuss your needs, your data curiosity, and, and much, much more.

46:30 If you're also interested in learning more about Atlan or following us ar-around different new, new updates to features we have new weekly newsletters. You can check out our product tour or request a demo to our sales team. So that concludes our Atlan Activate event, and we wanna thank everybody again for taking the time. Thank you to Varun and to Rohan for your expert demoing. Was really appreciated and have a great day everyone.

Varun Banka

47:00 Yeah. Thanks everybody for tuning in.