Supercharged automations for the modern data stack

Austin Kronz

0:05 Hi, everyone. Welcome to this edition of Atlan Activate. I am super excited for what we have for you today. The Atlan Activate sessions are becoming, I think, a fan favorite as well as a personal favorite of mine because it really is the embodiment of bringing the modern data stack together and, and showing what's capable with these awesome tools and technologies. Much like any Atlan session, we wanna start off with a poll and get the audience involved as much as we can. We're gonna be talking a lot about data governance and this concept of shifting governance to the left and moving it across the data stack.

0:44 So why not start out with what does data governance mean to you? I think everyone has a unique angle, almost, you know, what pops into your head first, and we'll, we'll take a couple minutes there before I, I hand it off to our awesome panel today. Yeah, we already got some coming in. Yeah, lowering company risk, data policies. Yep. Security. Awesome. I see some in the chat too. Feel free to put them in the Slido too. I see data enablement in the chat. Organization. Yeah, security being dominant here. A lot of, like, standardization requirements, compliance. Awesome. Yeah, security and access control definitely being some of the dominant ones, and I have...

1:31 I'm positive we will touch on these throughout the session. So thank you for that. Feel free to keep them coming. Well, without further ado then, I will move on to our awesome panel for today. Like I said, we're really happy to be joined by our friends and our partners across the modern data stack to, to show you what's possible whenever we really join together. So with that being said, I'd like to introduce Varun Banka, co-founder of Atlan. We have Niraj Vora. He's the Lead Sales Engineer for Fivetran. Paul Gancz, Partner Solution Architect at Snowflake, and Colin Thornburg, a Solutions Architect at dbt Labs. So everything everything covered across the data stack right now.

2:19 With that, I'll hand it over to Varun to go over some of our agenda for today 'cause we have a jam-packed session. Thanks, everyone. See you at the end.

Varun Banka

2:30 Thanks, Austin. I'm, yes, super excited to have everybody on board Atlan Activate, our flagship event to bring the best from the metadata world to data teams globally. So what, what do we have today? We'll be talking about governance moving to left, what does governance looks like in the new data estate. We look at our partners with amazing friends from across the board who's gonna talk about same. We'll, we'll do a quick demo, and depending on how much time do we have, we will do some live Q&A. In the meanwhile, if you have any questions, feel free to just put it on chat. There are folks who can help answer while, while we do this.

3:10 So without further ado, let's get started. Okay, what are you seeing on your screen right now? Three big gray boxes, I assume, with three big questions. Why, who, how? And no, we are not playing Jeopardy today. We'll be going to talk about data teams. Now, all great companies have a common foundational value that gets attached, that's ingrained in the ethos, that's ingrained in the DNA. To give you example, this would be Apple has designed user experience as part of the ethos, as part of the DNA. Amazon has customer service, customer care as part of their DNA. At Atlan, we care about being problem first.

3:56 And when we were a data team ourselves, we used to always ask ourselves these three questions: Why do we do what we do? Who are the stakeholders involved? Who are the personas? And how do we get there? Now, most often than not, how was the easy answer? And in fact, when now we have started working with data teams across the world, a lot of this is just a variation of what you see on the screen. I'm sure a lot of you, for your data teams will see similar kind of stack, some variations, some flavors of this.

4:27 There are source systems, some pipelines moving the data to a data cloud or compute, transformation metrics engine, which finally leads to analysis and output. But this is not where it stops. This is where most people miss out on. But data teams also have diverse set of users, diverse set of humans of data. You have data engineers, scientists, and a special shout out to now the prompt engineers. And last but not the least, why? What is the use case? What's the data project that folks are running? What are the domains?

5:02 Now, looking at the diversity of data state, diversity of humans of data, and diversity of use case, every data team needs a control plane, or what we at Atlan call home for data teams. Now, before I start talking about a bit about Atlan, let's look at what does the previous generation of control plane look like. So traditionally, the approach with data governance has been shift down, where there was no metadata available across the stack. People used to hire tons of armies of people, data stewards, to manage and govern data, and there was no automation built in, into these systems.

5:49 But in the last eighteen months, something has changed, and that's what we are here to talk about. There's an approach around shifting left. Now, the shift left philosophy has already come into the world of software delivery. It has already come into the world of security. Now, what does it mean from a aspect of governance? It's a practice of how do we move documentation, governance, standards, and testing closer to the data asset creation process rather than the consumption process. Now, what does this truly mean?

6:25 If you look at the same diagram that we started this with, this means that how can we get as data moves from your source, Fivetran moves data from your source systems to a data cloud, or as compute powers a lot of analytics and data science use cases, as dbt runs transformation and creates the metric layer, how do we capture metadata signals and drive automation via the control plane? Now, instead of me talking about this today, we have amazing partners with us, so I'm gonna just hand over maybe let's, let's start with left, like this diagram says, we will start with the world of Fivetran with replication.

7:08 We'll move to the world of data cloud, and then we'll move into the world of dbt. So to get started with, I'm just gonna pass it over to Niraj from Fivetran to take it over.

Niraj Vora

7:20 Hey, everyone. I'm Niraj Vora. I'm the lead sales engineer at Fivetran. So let's talk about the modern data stack and, you know, how Fivetran can help, you know, with the modern data stack and the governance, et cetera, and so on. So the modern data stack traditionally is thought of as simple infrastructure in that moves the data from left to right. Left has all your data sources. Then you have your data pipelines, destinations, and then finally you have these BI tools. You may throw in like a reverse ETL or transformation, you know, that brings the loop back. But then how are these modern data needs being met?

7:57 They're met more and more by enterprise business with a cloud data platform where data is ingested from a wide variety of on-prem or cloud databases, events, applications, files, et cetera more into one or more cloud services, which can then handle these different types of workloads they need to run. Data is moved in, modeled in within one's cloud data platform. And again, you know, we need to make sure that all of this is happening in real time without compromising security or compliance, et cetera, and so on. And that's what the Fivetran's vision is, to basically support and power this cloud data platform. We have developed an automated data management platform.

8:42 We can ingest, you know, all of your data from your most popular enterprise databases, SaaS applications, biggest operational and analytical workloads from on-prem, cloud, data warehouses, lakes, and more. Now, as we move the data in order for you to achieve your business initiatives like, you know, like centralizing data to power your business analytics, et cetera, and so on, our platform will basically support your business where you're looking to democratize that data so that more and more folks within the business have access to the data, thereby, you know, you increase the data users without increasing the governance risks or the costs.

9:25 So we connect to your data sources, and then we move that data into your cloud destinations or targets. We are baking in as much automation, reliability, and scale as possible along the way to truly allow you to let go of all the work associated with building and managing those data pipelines. At the core of our automation is, as you guessed, automation. So throughout this entire ELT process, we have basically automated the most time-consuming tasks for data engineers and built in this massive amount of automation so that you do not have to worry about, you know, moving your data, but basically allows you to, you know, scale your applications, et cetera, and so on.

10:11 So let's talk about how we scale our approach. We have about three hundred fully managed sources, which means that with few clicks and with no code, you're basically ingesting your data from majority of all the data sources within your organizations, you know, be it applications, events, files, database, data lakes, cloud warehouses, et cetera, and so on. For sources we don't support, we actually have an by request out of, you know, a connector, a light connector, which we can build it for you, or we can even, you know, write a custom connector, which can basically pull data automatically from nearly every source.

10:53 Now, throughout your extract and load process, you know, you can then select your application type depending on what kind of access, speed, security you need, and then let all the automation and reliability which we have baked in take over from your automated data updates to automated schema applications like your DDL changes, your schema drifts, et cetera. We will even, you know, load destinations with efficient writes so that you save on all those compute cost. Now, all of this is done with zero code, so you do not have, you know, write, manage any code.

11:30 Now, as we are moving the data, we are also automatically detecting for any PIIs, so which means that you can mask or hash your sensitive information. And then you will pull out all your metadata to automatically plug into a governance catalog tools like Atlan so that you can then secure a dat-data end to end. Now, once the data is ready in your data warehouses, you know, that's where you have transformations, and we also have those automations and scalability built in using with our integration with dbt. Again, it's automated no-code transformation that runs when the data is loaded, either using our pre-built data models, or you can write your own using our visual code extension.

12:15 And then for organizations, you know, which are looking for additional data movements, we have maximum security built in as part of our bo-out-of-the-box integration, which allows you to you know, data movements to really meet the needs of your cloud data platform. Now, while this is happening, you know, we also have this Fivetran's metadata APIs which provides enhanced visibility into what the data is coming from, who accessed it, what changed, or what changes occurred in pipeline, you know, which can send informations to your data catalog so now your data stewards have visibility of all the data being loaded.

12:54 So again, as Arun said earlier, right, with modern day's data stacks, the tagging is moving to the left. So as Fivetran moves data, Fivetran knows each and every row. Fivetran is well-positioned to, you know, tag the data right there when ingesting the data into your data warehouse like Snowflake. And then the modern governance tools like Atlan can then import those tags from your Snowflake warehouse and enable all those discovery use cases. So with governed data movement, it's all about increasing access to data without compromising on security and protection. So with data democratization models that enable self-service analytics, you know, that requires a cultural shift across the enterprise.

13:40 So as your data is in flight from your source to destination, Fivetran automates the governance so that you know your data, protect your data, and then control who accesses your data. So you can share your data by scaling access and empowering data producers to own their own pipelines, but still retain the oversight on what data is being loaded. You can plug in Fivetran into your wider data governance strategy by integrating with your existing catalog and governance tools like Atlan. And then ensure data producers and consumers have access to everything they need and basically control, you know, what they can see or what they cannot see.

14:24 You already know that your data is strategic assets, which means that, you know, you need to manage it with the same level of care you lend to your other strategic assets like your financial or physical assets. So da-data governance that increases the data visibility and control helps you protect your most valuable assets. For example, like, you know, logging and auditing capabilities, you know, it helps with audit and controls and compliance reporting, protecting your data and organizations from misuse and risks. Increased data visibility also ensures that your policies can be applied to the data that needs protection.

15:04 For example, you know, identifying and blocking any PII data so that your data is safe, compliant and accurate for downstream usage. And then finally, being able to see the end-to-end data journey, which gives your team better insights into data pipelines, and then you can identify breaking points or any potential upstream breaking points, which helps you reduce the amount of time that the team needs to, you know, spend troubleshooting and fixing those pipelines. All this ensures that the data flows are reliable and producing high quality and trustworthy data to drive your analytical decision-making and, you know, make operational ef-efficiencies.

15:47 So Fivetran's data platform automates the governance of pipelines, you know, for ease of use, but understanding what data is being loaded, who is loading the data, increase the visibility of data in flight by sharing metadata with your data catalog so you always know where your data came from and how it is being used. Secure your PII and sensitive data. Increase the sensitive or rather identify the sensitive data and prevent PII from ever entering into your system or encrypt sensitive data so that this ensures that your users can only query and use safe data. And then scale authorized users to be data-driven.

16:33 Set up like granular role-based access controls and automated user provisioning, which allows you to, you know, set and scale user access to the data. Also use Fivetran's APIs to bulk load users without any human error. So I'll pause here and hand it back to Varun. And if you're interested in, you know, learning more about the automated data movement and governance capabilities, you know, please reach out to us. I'll put our information in the chat.

Varun Banka

17:01 Thanks, Niraj, and we're equally excited about the governance roadmap and how Fivetran is especially the metadata API, we were really happy to integrate with it and looking forward to the roadmap. Paul, passing you to next.

Paul Gancz

17:15 Thanks, Varun. I'm honored to be here with you and our other great partners, Fivetran and dbt. So I'm the Partner Solutions Architect at Snowflake, supporting our data governance and data security partners like Atlan. And for those of you who don't really know Snowflake or know much about it, Snowflake Data Cloud's a global self-managed network across all the major clouds that supports all of your workloads against all of your data, including external data from your customers and partners, as well as third-party suppliers. It's made up of the platform, as well as the content and applications that are directly available to you.

17:56 We see our customers moving all kinds of data through tools like Fivetran to ingest the data into Snowflake, including highly sensitive regulated data and all kinds of workloads against that data in the Snowflake Data Cloud. All possible because of the unique architecture that allows all workloads to run against the same data with their own compute resources and to easily share live data across all clouds and regions. And those workloads can run SQL, Python, Java, Scala, JavaScript directly on those Snowflake compute resources. So you don't need separate compute for those workloads.

18:43 As customer adoption's grown, we've seen the world's most demanding organizations using Snowflake to mobilize their data and eliminating silos and breaking down barriers, all while maintaining really high standards of compliance and governance. So some of the most common data governance challenges we see across our customers, and I assume many of you are seeing some of these same challenges, are first, data's isolated into silos, stored in multiple storage locations, serving different analytic systems and workloads that are spread across locations. So to know what your data is and how it's being used, you have to eliminate as many silos as possible, both inside and outside your organization. Second, managing access to data is complicated and complex.

19:32 Since data's scattered and siloed both inside and outside the org, to easily and effective, effectively manage your data, you really need centralized visibility and control of the environment. And finally, security and governance policies tend to be custom-built for each workload, analytical system, and storage location, making them inflexible and contributing to proliferation of policies. To secure and govern your data and to really have a good federated data governance story, you need unified policies that can be consistently applied across different workloads, clouds, and geographic regions that correspond to all the evolving regulations. So how does Snowflake, more importantly our customers, think about data governance?

20:21 First, it's really important to know your data. Need to understand exactly what I have, including whether any data is sensitive or private, and I need to be able to audit how that data's being used and how it's moving through the system, as well as how changes to that data might affect downstream processes. Once I know the data, I can protect it. Here it's really important to make sure it, it... To make sure that all controls can be defined once, applied consistently across all the Snowflake data and users, regardless of how they access it, in an efficient, scalable manner.

20:57 And then, of course, once I understand the data, I can confidently ensure it's properly secure, I can unlock it and allow all of my organization's various constituencies, the data engineers, data analysts, data scientists, all the people that Varun talked about earlier, and all those personas, to access it. All this is also really important if you want to monetize your data, make it available to users at external organizations. Snowflake includes a number of native data governance capabilities to make it easier for our customers and partners to know, protect, and connect their data so they can be connected to their ecosystem.

21:43 For knowing your data, we natively support object tags, which can be used in a variety of ways, including tagging Snowflake users and resources for cost governance and for tagging sensitive data elements. Snowflake also includes native sensitive data classification using using those same Snowflake tags. You'll see how Atlan works with Snowflake tags in the demo a little later, which I'm really looking forward to seeing with you.

22:13 Snowflake also gives you the ability, through our access history view in the account usage schema, to easily tell which data attributes are being accessed and by who, both to determine where the popular, useful data is, as well as to audit who's querying the sensitive data or private data. This view can also be used to track data lineage as it moves through the system, and we have views to be able to tell you what impact or what dependencies a change to an object will have on other downstream objects.

22:51 For protecting your data, you have a number of native capabilities, including row access policies, which is the ability to limit the rows returned based on a user's role and other conditions; the ability to dynamically, at runtime, mask sensitive data elements; support for external tokenization so that data can always be stored in its encrypted or tokenized form; support for masking based not only on the user's role and access levels, but also on other data such as the user's location or privacy preferences. And we also support tag-based masking, which allows you to automatically mask particular data elements based on how they're tagged.

23:38 So you could create a masking policy, for example, for all customer emails, assign it to a tag named category, the value of customer email, and then all columns with that tag will automatically have their policy applied. And with all of these capabilities, Snowflake provides the building blocks for governing and protecting your data. And then, of course, it's really important to be able to use these same controls as you connect your ecosystem, including getting data from or making data available to other internal or external organizations. So with that, I'll turn it over, back over to Varun to introduce dbt Labs.

Varun Banka

24:22 Awesome. Thanks, Paul. Yeah, I think especially with object tagging and access control, we're equally excited to integrate really well, and our customers are looking forward as well with this. So next we have... Colin from dbt Labs. I'm just gonna pass over Thank you, Varun and Atlan.

Colin Thornburg

24:40 Really appreciate the the partnership along with the other vendors, Fivetran and Snowflake. So I'm Colin Thornburg, solution architect here at dbt Labs, gonna be talking to you for the next 10, 15 minutes on enabling data governance in dbt. So first, just a quick primer on dbt for those that might be unfamiliar with the technology. So the easiest way to think about dbt is that it is the transformation component of the extract load transformation prod, process. So the T in ELT. So dbt's strength is taking your data in its raw form after the data has been ingested into a cloud data platform like Snowflake, and you could use a tool to do that like Fivetran.

25:29 So once Fivetran loads your source data, dbt is gonna take the baton and transform that data into a consumable format for your downstream BI and reporting tools and even your data science application. So an important concept of dbt is that all of that transformation logic that you build is actually pushed down to your cloud data warehouse like a Snowflake. And this is really beneficial for two primary reasons. So from a speed aspect, right? Because that data is staying in Snowflake and the data is not moving into our platform, we can tap into the awesome compute capabilities that Snowflake offers, right? You don't have to devalue your investment in Snowflake. We wanna take advantage of that.

26:18 I noticed at the beginning, security is a, is a big concern for a lot of folks, and that is one of the big benefits of using dbt with Snowflake. Again, that data is not coming into our servers for processing, so it stays in your secure Snowflake environment, so the risk of your, your data actually being compromised is, is really minimized. Moving on here so what is the problem that dbt is actually solving for from a governance aspect or perspective? So although customers can benefit from cloud data warehouse's scalability and compute power, their environment can still look something like what you see on the screen, right?

26:59 Due to, you know, long unmanageable stored procedures, custom SQL, maybe using Python notebooks or even GUI-based tools with restricted access where, you know, maybe only a handful of folks in the organization actually have the, the, the understanding of, you know, all the WYSIWYG objects that you need to drag into the screen. So in these scenarios, those data transformations are often untested, they're error-prone, they're difficult to modify. And what does this do? It leads to slow responsiveness to business needs.

27:34 So as organizations seek to adopt more modern or decentralized, maybe think of a data mesh type approach when you've got something that looks like this on the screen, these approaches can become a governance nightmare and pose significant barriers to effective data management. So how does dbt come into the picture and, and solve for some of these challenges? It's important, I think, to take a, a step back here and understand kind of two baseline principles of dbt. So build like developers build applications, so SQL first. SQL is the language of all databases. So by leveraging a common language between data analysts and engineers, more people can contribute and collaborate when building these data pipelines.

28:23 To add on, someone can really be effective in dbt on day one if they just know fundamental SQL. So if you can create a basic select statement, you know dbt. And this is because dbt, it abstracts a lot of the more complex parts of SQL, such as writing DDL or DML statements. Dbt does all of that for you under the hood. And second, we believe in adopting best practices from the software engineering space. So concepts like testing, version control, documentation in CICD and bringing those into the data space. So how do these concepts that we just talked about in the last slide enable better data governance?

29:09 So dbt standardizes data transformations across your organization for consistent and reliable results, facilitating that collaboration among data teams. Robust testing frameworks, framework includes automated testing to ensure data accuracy and quick error detection. We also have adoption of CICD from the software engineering space. So dbt integrates with all of the major Git providers for seamless version control to move pipelines from development to QA to production seamlessly and efficiently.

29:46 Dbt automates dependency management, so freeing your data practitioners to actually focus on the business logic, not understanding which SQL statement needs to run before this stored procedure. It frees up your data engineers again to focus on just the business logic and the value that it's providing. And finally, all of the modular SQL is visible in the IDE, improving that developer experience and hosted documentation for end users and external auditors can also be leveraged.

30:21 So let's start to double-click on a few things here on these governance aspects and see how dbt and Atlan work together to make governance more accessible So as mentioned earlier, all of the transformation steps that you do in dbt are automatically documented and hosted for you, so there is no setup actually needed. All of it's fully automated. Data engineers as well as end users can leverage the documentation to have more intelligent conversations to enable faster pipeline development. So think about your BI developers that are out there, they just need a few columns of data for that additional analysis.

30:58 Now they have access to this documentation at their fingertips, they can relay that information fast to those data engineers or actually contribute to that pipeline themselves. But dbt's integration with Atlan allows for all of that metadata and the tags that your data engineers apply in dbt, that can actually be directly visible as well in Atlan's UI, which you'll see here in a bit in the demo. Dbt also has automated continuous integration capabilities. And so what does that mean?

31:35 It means that dbt is actually aware of the changes that you make in development, and it only builds or materializes those transformation steps that are in scope, which then can be used to validate the data and make sure it's accurate prior to actually merging that code into a production or a main or master branch. So there is no need to actually run your entire pipeline when someone makes a simple change, especially when you're going through that QA process, which saves you, ultimately, time and money. We also have robust job scheduling and alerting capabilities.

32:13 Taking this one step further, because Atlan integrates all of your data assets, including your ingestion tools, BI dashboards, spreadsheets, data science application, an organization can have full visibility into that entire data ecosystem, not just what's in your cloud data warehouse, that dbt fitting on top of it, and show how each of those changes in the pipeline actually impacts those downstream processes. Important to call out the ability as well in Atlan to track column-level lineage. So if someone is making a change in a specific column in the data warehouse, there is that instant visibility into the assets and the users that are actually impacted by a change.

33:00 Quickly transitioning here into the dbt semantic layer. You will also hear this call called the metrics layer. And what does this mean from a data governance perspective? So the metrics layer is a place to build centralized logic for all of your critical metrics. So no matter what downstream tool's being used, whether that's a commercial BI tool, even Excel, data science use cases, your end users are able to query and slice and dice that data in a very consistent manner.

33:31 So an example, let's say, you wanna look at something like monthly recurring revenue, and based on the downstream tool being used, you might have users arrive at all sorts of different answers depending on the filters that have been applied or the data that's being used. But the goal of the semantic layer is to improve that consistency and the trust, precision of your metrics. And of course, these metrics are tightly integrated with Atlan, so you get that full end-to-end visibility, which you can see here on the screen. And finally, to wrap up, I wanted to give you all just kind of a quick, quick peek into the future of dbt.

34:10 So data governance and scalability are a huge focus for our team this year. So starting with dbt constraints, which will serve as a building block for enabling a more seamless multi-project experience. If this is something that you see as valuable, just know that we are having a live demo of this in March. I'll put the event link in the chat right when I'm finished here, but it's a great chance to see what this dbt constraints is all about and get valuable feedback from our community. You can also find this information in our dbt Slack community.

34:48 But think of dbt constraints as a defense mechanism to ensure that when data is exchanged, it remains consistent and reliable. This is something crucial when you start to think about data contracts or data products in your organization. Another big point of emphasis for our product team is the support for the multi-project experience. This is something that is possible in dbt today, but we want to make this experience as seamless as possible. So the mono repo approach simply is not all that scalable, and a big focus for us is gonna be enabling dbt at scale for organizations, so you can adopt that decentralized or data mesh style approaches to building your pipelines.

35:36 And lastly I talked about the semantic layer and the metrics layer in the previous slide, but a big part of this year is broadening the support in our partner integrations, specifically in that business intelligence space, so that downstream tools can leverage the power of the semantic layer and get their metrics in a good, consistent way. And so with that I am going to hand it back over to Varun to take us home with a demo.

Varun Banka

36:07 Thanks, Colin. And yeah, we've, we have a lot of customers, data teams who started leveraging a lot of embedded documentation with dbt, documenting in dbt, pushing in Atlan, and we have a great example as well today. So after praying to Lord Murphy, I'm gonna share my screen, and we'll go from there. So how do we go about today's demo? So this will be around how do we take data governance from reactive to proactive. And I think everybody has teed up really well. They've already talked about a bunch of things, so we just wanna straight into this. We've broken this down into four different themes.

36:39 We will look at each theme and then come back and go back again to our demo. So let's start with the first one, which is how do we do root cause analysis, impact analysis, and how do we embed documentation closer to the left. So I'm just gonna switch my screen. You should be able to see the Atlan workspace. I'm not going to ask the famous Zoom, Zoom question that you can see my screen. I'm just gonna jump in directly. So let's get quickly started. So first, across the data estate, how do you get first all the metadata, signals, tags, metadata, API, everything that we talked about, how do we get this?

37:15 So DIY, we have these— We have metadata marketplace. You can click some of this, no code, get your credentials, test some, get your network credentials right, test some authentication, and we will bring all the metadata from all of the data estate to create the data inventory. So whether this is, whether this is Snowflake, this is Fivetran or dbt. In case you use dbt Core or dbt Cloud, we support both of them, and we bring those metadata in. Now, as soon as you get that up and running, what you get is a data inventory.

37:51 Now, today, for the focus of this demo, we are not gonna focus a lot on discovery, we are focused a lot on g-governance. So let's go into maybe one of my favorite tables. I'm just gonna search sales. Looks like there is a business term, there is a relational table. So let me just filter it via some of the most popular assets, so I just find the right relevant assets. I found the fact sales. I'm just gonna open that up. For folks who are new to the Atlan workspace, what you see right now is an what we call an asset three-sixty. This is a fact, fact sales table in a schema in Snowflake.

38:29 You can imagine this, this is just like your LinkedIn profile. Your LinkedIn profile has everything about you, your lineage, where do you work, what has been the work, what are skills, all of those. Similarly, this asset profile has everything about this asset. It has a quick summary, who the owners are, data dictionary, if customers, if data teams have enabled sample data, if a sample data, readmes, all of those are available. And your favorite companion sidebar, which is rich metadata, which we'll come back to it in the data part of the demo. But for now, let's jump to lineage.

39:04 You would have seen as part of the presentation today, all of us had this diagram which source moves data to pipeline. All of this had this block diagram. And one thing we were always curious about Atlan— in Atlan is how do we truly represent this data estate? Can we make it real? So I'm gonna just expand. What you see right now is fact orders table. There are two downstream Sigma Sigma dashboards that are linked to it. There are upstream some table. Let's, let's start expanding upstream. Looks like this table, it's materialized by a dbt model, which moves into fact sales. This is coming from four different tables. Let's keep expanding. Let's keep going upstream.

39:45 It comes from the customers table. This comes from the relational table. So if I'm going to zoom out, this is the diagram that all of us started with, which is you have your relational OLTP, you have Fivetran. This is the Fivetran URL. You can even go in in case you wanna go debug what's happening with the pipeline, what's moving, what's not. You wanna optimize it, you can easily go into that. This is moving into your warehouse, similarly, creating a new table, which is more of a staging table. And then you have a dimension table, which then dbt model takes over. These are the queries that run. It materializes this into a fact order table.

40:24 And not only creating the end-to-end lineage relationship, you also get to see operational metadata. If you're running dbt cloud, what is the last success rate? How much time it took to run this? And in case you're more curious, you want to look at what's happening in dbt, you want to optimize it, you can easily open door. So we do a easy handover to whatever the data estate is. Now, let's take a... Let's take one of the use cases. Let's talk about somebody wants to change some column name or something in this model, which is named Customers.

41:01 And we were talking about, like the theme of today's demo, about how do we move governance to the left. So we'll go to where the code resides, which is GitHub. We have just one sample PR that is open here. Looks like this analytics engineer is looking to make some change. And thanks to integration between GitHub actions and Atlan, this analytics engineer is able to see all the downstream impact across the data real estate if this— if they make this change, across their Snowflake table, across the other Snowflake tables that exist. Now, how is this possible? We have— You can go to today to the GitHub marketplace. You have the Atlan dbt action available.

41:43 You can set this up. And if there are any changes, thanks to the whole lineage of metadata that exist here, we're able to bring that governance right to GitHub closest to code. Not only that, right? Let's take one step forward. As Atlan, our goal is how do we keep taking the power of governance closer to the tools that people use? And one of favorite tools for everybody is spreadsheet. So let's go to one example of Google Sheet where we also have an Atlan integration with Google Sheets where you can just decide and download lineage.

42:20 So I'm just gonna click import lineage because maybe I want to share the list of tables that are going to get impacted. I want to share it with the team as a list. So let me just close this. Let me find the same table that we are searching for. Let me just download this lineage. What this does, does is it brings all the downstream assets that will get impacted if you make a change here, right here. So what you can see is all the assets are available. You can share this with your colleagues share this as a list with any of your teammates.

43:00 But what we'll do is let's send a warning that there is a new column added. And who does not like Sheets? So I'm just gonna copy this because all four assets are gonna get impacted. Similarly, I'm gonna send this warning to everybody. And again, all I need to do is go back to this Atlan extension again and say, "Push to Atlan." You will soon hopefully see a green— big green check that will ensure that your metadata is synced. So right from impact analysis, making collaborative, love the power of Sheets, sync the better data back to Atlan.

43:46 Now, when you go back to the same lineage table, if you'll start seeing these announcements, red check mark, which means that there's something wrong with these tables. So you will start seeing announcements, any warnings that have existed in these tables or somebody has marked in this lineage graph. Now, let's talk about the second part of this, of how do you embed— how do you bring documentation closer to the left? We looked at lineage, we looked at impact analysis. Now let's talk about documentation. So Colin already talked about this. If you're using dbt, you can start documenting metadata like...

44:23 So if you see a big, big green tick, this is our version of the Twitter blue tick, which recognizes that this this asset is verified. You can start verifying these assets, put owners right when the model is getting materialized. You don't need to wait on Atlan. So right when the asset is getting created, these documentation will get pushed to Atlan. Okay, so let's, let's go back— Let's go back to where we started.

Varun Banka

44:50 So we talked about impact analysis, embedded documentation. Let's go to the second part of the demo, which is access control via tags. We believe at Atlan believe that tags is going to be one of the most important metadata in the data estate. We, we heard from everybody here, right from Fivetran's roadmap to bringing tags as the data gets replicated. Snowflake has invested heavily on data tagging. Paul was talking about deep access control, masking everything is built in. So let's, let's go to that demo. So we're gonna start with Snowflake. If you see, we have just set up the Snowflake environment where we've created a confidential tag with some allowed values.

45:28 We've added these tags to three assets in the Salesforce schema. And just for verification, we can see that these assets have been attached with this tag. Now, with the recent upgrade integration that we built with Snowflake, what we have done is we brought we brought in classifications tagged there on Snowflake into Atlan. So I've moved into the governance section of Atlan. You can see a list of classification and tags that are available there. But if you see in confidential, there is a u-unique thing here, which is it says it's synced from Snowflake. If I click there, I can look at the same three assets that are tagged on Snowflake.

46:09 So you don't need to tag them on Atlan. As long as they are getting produced, tagged, this tagging curation is available to all data consumers. You can look at what, what is happening, you can enable, disable them, and you can manage them. Now, let's take one step forward. Let's go back to the same, same lineage graph. If you go back to the same lineage graph, and let's look at maybe column level lineage. So I'm sure there's a customer table, so there must be some PII around customer name. Now, as soon as, one, we click that, one, it gives end-to-end visibility at column-level lineage.

46:43 But one thing that we do unique at Atlan, if you see customer name, it's tagged as PII. But if I look at the downstream customer name, it says PII, which has been propagated via customer name. Via the lineage relationship, we move the move— we help propagate these tags right from your left to the right. What that means is if I'm looking at fact sales, we've also— what we've done is we've also set up a masking policy on Snowflake. So what we have done is if there's an asset which has been marked as PII, let's mask that data. So let's see that example, how that works. So I'm just clicking fact sales again.

47:22 I see a number of rows. Let's see a quick preview. We'll find that customer, customer ID column. This should ideally be masked. So this means I can quickly copy this, download this, share this as analyst or scientist without worrying about my PII. Okay, so let's go back to the... So we looked at impact analysis, we looked at access control via tags. Now let's talk about proactive collaboration and extensibility. So let's go back to maybe the same table. Let's find the, find the name column. If I open this up, this opens up the companion sidebar, which has rich metadata across from your data estate.

48:11 It has popularity when these columns was getting used dbt operational metadata. We have out-of-the-box profiling, which is your statistics, basic statistics that might be relevant, adds more trust to the system. And what I see is also Slack and Teams. So maybe let's take an example of quick around Teams. So if you have your favorite communication tools, we create proactive collaboration on top of it. So for the sake of demo, I'm just gonna remove a full stop. I'm sure data consumers will hate me for this, but I'm just gonna remove a full stop for this and go back to the Microsoft Teams.

48:47 And if you have channels built in, you will have direct proactive monitoring updates on any metadata change. You Folks can subscribe to specific metadata changes that they're looking for. So that's around proactive monitoring. But what about you don't like this flow, and you want to extend this, b-build your own tooling, workflows around this? Atlan is the only metadata platform that's available natively with AWS Cloud. So we directly integrate with EventBridge, where you can capture all the metadata change events and build your own production-grade workflows, hooks with any metadata change events. And what this means, go to your favorite AWS console, find partner events. You will see the big Atlan logo. You can set this up.

49:32 There's rich documentation, or you can reach out to your nearest customer success person from Atlan. We'll be happy to help you get this set up. Now, let's, let's end this with maybe a thirty-thousand view of the whole data estate. I think I saw somebody talking about GDPR as well on the chat. What we've also done is, for proactive monitoring and reporting, we've set up these dashboards, which gives thirty thousand view of across the data estates, what are the assets, how they are distributed, what's the tag distribution, what's the asset enrichment looks like, who are the owners. And we recently started rolling out users and co-cost dashboards as well, looking at all the popularity data.

50:08 Which are the popular assets you We are giving nudges for folks to review assets. We've started seeing, in fact, a lot of data teams started to clean up their data real estate where— and in fact, a lot of them are going complete Marie Kondo on this, where they're removing, cleaning, and deprecating assets to reduce their data exposure and save cost in some scenarios. So impact analysis, access control via tags, collaboration extensibility, and proactive reporting. That is the four parts of the demo theme.

Varun Banka

50:36 So we have five— I see on the clock we have five minutes left, so we have one last thing for all of you. We talked about moving from shift down to shift left, but there's another movement that started to happen, which is how do you shift up? Of how do you democratize the benefits of artificial intelligence across a data real estate? I was listening to a great quote recently by the NVIDIA CEO, which talks about that this could be the whole GPT-3 LLM movement could be the almost the iPhone movement for the world of software. So as Atlan, we wanna be early on it. What does— what do we mean by shift up?

51:21 We believe that artificial intelligence will impact all the data tools across the stack. And in fact, we made an early investment. So last year, we rolled out Trident, which is our internal assistant in the Atlan universe, which helps our data teams enrich, suggest, nudge for documentation. In fact, till as of now, this assistant has contributed to one-third of all description updates that have happened on Atlan. But we wanted to take this a step forward. We wanna feel like twenty-thirty by twenty-thirty. So we, we are continuously experimenting of how do we bring AI across the data stack for governance. So let's— we have four or five demos.

52:07 I know we have four minutes, so just hold on. We will start with three. So okay, I'm just gonna jump back to the, to the product again. Now we are going to go into the glossary section of the product, which is the more, which connects the business world to the physical world, the semantic world to the physical world. So let's open up some business terms. Let's look at supplier supplier invoice. If you look If you look at on the right, you will see a description. This is our internal rule-based engine, which suggests some description that already might have been set up. If you like it, great, apply it.

52:45 But if you don't like it, you can go and ask the Trident AI. Trident AI goes, generates a description. Looks like it generated a description. I can apply that. Boom, the enrichment is done. Let's look at readmes. Who does not like to read a readme when they land on a data asset or a business term to understand what's happening? But very few of us wants to actually write it. So let's see if we can get Trident AI to write some of these readmes. As soon as we click this, we send the right context to the LLM models that are available, and we bring readme, generate rich markdown with some links on the product.

53:27 Now, while I was doing this, some of you might be wondering, do we need to do one at a time? Because maybe I have a ton of business glossary, I have ton of data assets, hundreds of data assets, and I want to just unleash this AI, unleash this Trident. So we're gonna— At Atlan, we also invested in concept of playbooks, which is rule-based automation engine, which helps you do bulk automation to do enrichment and governance. One common theme we've seen with customers as they onboard with Atlan, they want to start with governing the most popular assets. So let's take a simple example. We've set up this for this playbook.

54:07 They want to find all assets where maybe the number of queries that are running in the last thirty days is greater than ten. Number of users that are using this are greater than one. They see that there are sixteen assets that are matched. They want to update them as verified, the blue tick or the green tick I talked about. But now we've also opened up to unleash Trident.

54:28 What that means What this will do as soon as I update is, and you can set it up on a recurring basis, it'll regularly find all popular assets, and if it does not find a description, it'll start enriching them or verifying them because they are popular, and you wanna make sure they are verified for the assets. A few other examples similarly in the world of SQL. Let's start with this happens a lot in SQL. A lot of people make a lot of syntax errors, issues. Not everybody is equipped with SQL. If I, if I run this right now, this shows an error.

55:06 Again, what I can do, try it and just ask this saying to fix the query. And if the AI lords are doing okay, they would do that. So what had came out is it fixed the query, told us what needs to be fixed, added the right queries, and now if I run this, the data is available. So democratizing knowledge, democratizing access to data, improving SQL. Let's take one more example. Let's say this is a complex query. A colleague has sent this to me. I have no clue what this does because I want to make edits, but I want to quickly understand this. So let's first ask Trident to explain this.

55:50 What it does is in English language, natural language, it explains the complex query. If I want to now want to change this, maybe I can add some more queries, change this, and run this. This shows actual queries to your warehouse, Snowflake, and gets you the data. Now let's, let's look at maybe some natural English prompts. I don't know SQL. I want to just... This is what I need. Very simple. I want to just find all orders made before twenty-twenty-one. Simple English statement. Let's see if it can just complete and get, get the answer. It gets the data. It gets— builds the SQL query, and looks like AI has an error as well.

56:33 Did not complete it. So let's go to the next example. So let's take one of the most complex ones, which is you have multiple joins, you have invoices, you have customers, and you want to join them. You know what you want to get to, but you don't know how to write a complex join. Again, you have AI to the rescue. You have the SQL query. You run this. You will get the data. Now, a lot of this in beta. Feel free to reach out for especially customers who are trying to get early access. Would love to help you folks and get a lot of feedback around these models. Now, just take a pause.

57:18 I know we are off time. Super excited to see how data teams move from reactive to proactive, and I'm just passing over back to Austin to wrap this up.

Austin Kronz

57:30 Awesome. Thanks, Varun. Talk about a jam-packed session. Some of that was extremely cool even for, for me, seeing some of the latest things that we can offer. Just big thank you to our partners and friends from Fivetran, dbt, and Snowflake, and thank you to the audience for taking time to listen. We have a couple things on screen I'd love to just run by. If you don't already follow our newsletter, if you don't follow us on LinkedIn, feel free to do that. With that thanks again, and we will see you next time. Thanks, everyone.