Data contracts demo: governance as code

Srini Raghavan

0:03 It's very interesting when the common currency and language leads us to collaborate with data products and domains. And moving away from the rigidness of the governance structure to a very refreshing, adaptable governance model based on the people, as we saw through the transparency centers and the workflows. We're now moving to another exciting topic, which is about embedding data governance process in the daily workflows.

0:41 At Atlan, we think about this as building an embedded declarative way of governance, or simply call it data governance as code. Let's delve into understanding how something like data contracts can help us in that journey. When you talk about contracts, there are two personas that immediately come to our mind: the producers and the consumers. We saw the data product owners with data products. We start the stewardship, the AI stewardship, with the concept of the policies and workflows.

1:23 Let's invite our enthusiastic data engineers who are working with code every day in, day out and see how something like data contract as an embeddable governance model be used in the world. But before we get into this, let's spend a couple of minutes trying to understand what is the problem statement.

Srini Raghavan

1:44 Why do we even need this whole idea of data contracts? I love this picture on the right. Look at these data producers, data consumers, multiple-threaded discussions, but they just don't meet each other at all, resulting in what you see on the left, siloed conversations. What gets worse is if there are incidents that happens, imagine never getting that reported. But now we have a policy center, that's not gonna be the case. And the other part is change management. You are more often than not caught off-guard without understanding why or what something changed that resulted in your downstream impacts.

2:26 Taking these as, as problems, we always apply a first principle approach on building anything into our world of Atlan.

Srini Raghavan

2:35 We broke this into three stages of standardizing, organizing, and enforcing. So let's understand each of these a little bit more detail. When we say standardizing, the contract specification needs to be something that people understand already in some shape or form. It's not a brand new way of introducing yet another tool or language for us to learn. Take that one step forward and embed and bring the context of the assets that we are already dealing with. Great. Data engineers love the world of CI/CD. How about embedding that contract into that very choice of tools that we are familiar with using it on a day-to-day basis? Remember, it's about adopting governance to what we already know.

3:26 And all of this would now maybe build a different picture of the consumer and the producers being more aligned. Everything we do in a data catalog starts with categorization and classification. For our search and discovery to be effective, if you can organize the estate in a manner that makes it easy and intuitive for people to find what they want, isn't that great? And what we do with contracts is bring that context into our data estate assets. So you can now go and search and say, "Hey, get me all those which don't have a contract," or, "Get me which has a contract." It automatically helps build trust.

4:05 And finally, if you take all this and don't put it into action, it's not really valuable, is it? And that's where the enforcement bit comes in. And here we have also done a very innovative thing. Given how we are having an open platform, open architecture in place, starting ground up, we have been able to think about a provision to have data quality checks that can be brought in natively embedded into the system, and this can open up in the future for anything that can bring quality checks.

4:40 And of course, one of the things about enforce that putting into action is to make sure that you are preventative in nature, so that downstream impacts are cut up to a certain extent. All right. This is all good, but how do we do this? Let's talk about these three steps, do it yourself, how to create a contract, how to validate a contract, and how to update this within Atlan.

Srini Raghavan

5:06 Let's jump right away on how do we do that. We'll go back to our favorite data products. I'm gonna play the role of a data engineer tasked with the job of building a pipeline for the ML engineers to do some kind of predictive analysis on products. So I'm in the product three sixty, part of the Instacart domain. And as I get into this product, I notice that there are two codes that are exposed for me as a consumer to go and build my pipeline. This is great. I'm looking at a Snowflake table, Postgres table. But one thing that caught my attention when I was looking at this table is this definition of a contract.

5:47 Curious, I go in and see what this is like. The moment there is a contract definition, it starts giving me a certain level of confidence, okay, that this is something that I can actually take and use. You will see as data engineers the very familiar YAML format. What we start off, keep it very simple. We are able to bring in the context of what the source as well, like the name, the data type of the column. But then we are able to augment this by adding some checks and constraints. This particular column needs to have this constraint of three characters and maximum ten characters.

6:23 We also bring our enrichment into the contracts, so you know who the owners are, what tags are used, and so on. And last but not the least, we also bring in the data quality checks in. You see how intuitive this is, a YAML format, very familiar, very easy to use. We don't stop there. If you've been attentive enough to notice the V4, we've also br-brought in version control day one. This is something that, again, data engineers would love to see and compare these versions across systems, very similar to how we do this in our code. And as you can see, the contracts themselves have gone through iterations.

7:03 They've been in draft version and finally moved to verified version. I'm almost tempted to say the contract is now signed and bled at this point of time. But this is all great. What about those less tech-savvy, less YAMLs having folks who would like to understand this more in a document format?

Srini Raghavan

7:24 Enter again our AI to rescue. Using Atlan Copilot AI, what we have done is we've taken the whole YAML as input, fed it into our AI system, and generate this wonderful-looking document which captures the intent of the contract the same as you saw in the YAML file. So we have all the schema details. We have the metrics. We go ahead, talk about the SLAs, SLOs, everything that you need to eventually define this lovely contract. Wonderful. So I'm sold on this. I'm also going back to my products, and I notice, hey, it'd be nice if I can have this contract also for my transactional system where it all starts.

8:07 Remember some of the folks talking about shifting to the left and so on? If I can have a contract right there for this source, how nice would that be?

Srini Raghavan

8:16 So how easy is it to create these contracts? Let's jump into the table that we were looking at. And if I go to the Contracts tab here, you will see how the do-it-yourself really plays into action. We start off, the first thing you notice is an Atlan CLI. Oh, yes. For the first time ever, we're super excited to bring to you the Atlan CLI that engineers would just love to use. We have a developer portal which captures all the details, and all it takes is simply to go ahead and run this curl command, download the binaries in your local workspace. Okay, so let's see how this gets into action.

9:00 I'm going to bring in my favorite VS Code. I'm sure all the data engineers are gonna be very excited looking at this. I've installed the Atlan binaries already. Typo. And there you go. You have the Atlan CLI right there. We also have an environment variable to capture the API keys. This is required to authenticate yourself and connect. And we have a config file which tells you which instance to connect to, where do I need to push my Atlan code and for what Atlan contract and for what data asset. We keep this very interesting. We also have a help section, the documentation right there for each of the different commands that we have.

9:43 So we will do this. For this particular demo I'm going to pick up the contract generation for the one we just saw. It's as simple as just typing this single command. We have a contract generated right there. So you see the context of the asset is already baked into the contract to start with, and all the different details that we need in terms of the constraints, in terms of checks, all of this gets in. Now, you can now take this to the next step and add more details and enforce this contract as much as you want to. You'll notice that I've also integrated this with my Git.

10:21 So what I'm gonna do is first draft of the contract is something that I'm gonna push into my scratchpad you know staging repository. Now, once I do that, I will now go back to the Git world, and let's understand what happens in the repository.

Srini Raghavan

10:47 You will notice that a new change has been put. So I'm gonna open up the pull request and see what has come in. Okay. There is a new contract that's been coming in. I create a pull request. If everything looks good, I'm gonna merge this into the main branch. And what we have also done is we've taken the capability for Git workflows and created an action that is going to take this particular contract that we have just pushed, and it's gonna push this directly into Atlan. Remember the D-DIY steps. We created the contract.

11:28 We were able to look at and look at all the confirmation of what this contract is about, and then we are now able to push this contract directly into Atlan. This is the step where all the magic happens. It seems to have done it. Let's indeed confirm this, that the contract has come in. So what you're gonna see is the very first version that we so seamlessly created through CLI. We're able to push this and notice all of the content specific to this particular source has come in. It is as simple as a pie, and I meant eating the pie, not baking it.

Srini Raghavan

12:10 Okay, so I'm gonna go back to our slides and to wrap it all up, wanted to tell you how we could take governance really and embed into the very code, the very workflows that we deal with on a day-to-day basis. This ability to embed is going to be so powerful. It's gonna really make some of these enforcements, some of these actions valuable as we proceed in our data governance journey. And the best part, it builds the trust of data.