Data Lineage Explained: A 10-min Guide to Understanding the Importance of Tracking Your Data's Journey

Last updated on: March 24th, 2023, Published on: March 24th, 2023

Data lineage explained - Atlan

Share this article

Data lineage is the metadata that provides you with detailed information about the flow of data from its origin to its destination.

Some of this metadata is readily available in different data sources and destinations. Still, for the most part, it has to be derived from ETL scripts, SQL queries, stored procedures, and explicitly-defined dependencies in workflow engines, among other places.

Data lineage is important in solving many data engineering problems around visibility, trust, developer experience, data quality, and development agility.

Like genealogy provides information about the familial makeup of humans one generation after the other, data lineage includes information about how the data has been transformed throughout its journey from source to destination.

Table of content

  1. What is data lineage?
  2. Types of data lineage
  3. What are the benefits of data lineage?
  4. How do you generate data lineage?
  5. Data lineage: Related reads

Data lineage example


Here are some examples of how data lineage helps data practitioners make better use of the data at their disposal.

Visibility into data lineage helps data consumers in a business understand the origin and movement of data throughout its lifecycle, including:

  • How the data has reached them
  • Specifically how different entities have broken up and merged
  • How columns have been transformed
  • How data has been anonymized, and how it has been cleaned.


Video showing the impact of automated and intuitive data lineage visualization. Source: Atlan

In this article, we will discuss the basic concepts related to data lineage, different types of data lineage, and the areas where having data lineage helps data teams the most.

Toward the end of the article, we will also discuss some tips for deciding on a data lineage tool. Let’s start with the basics.

What is data lineage?

Data lineage is an all-encompassing term for all the data flow metadata from your data pipelines, workflow engines, and ETL scripts.



Data engineering processes, right from the capture of source data to the consumption layer, are packaged with a lot of useful operational metadata. In many cases, this metadata is used to enrich the understanding of data processes at your organization, especially for data quality and reliability.

Where do we get lineage metadata?


There are many places where an organization can get lineage metadata. The most valuable and reliable source of lineage metadata is the scripts responsible for moving data from one layer to another, i.e., ETL scripts.

These ETL scripts can be pure SQL or in a programming language like Python, Scala, Go, etc. In addition, you can also use your workflow engines, like Airflow, Luigi, or Argo, to extract the lineage metadata.

Moreover, many data teams enrich metadata by adding comments and business context to data assets. These comments can be stored in various places, such as the data catalog. There are many other ways of storing and extracting data lineage, most of which are based on the type of information they hold.

Let’s look at the different types of data lineage in the next section.

Types of data lineage

The types of data lineage discussed in this section are table-level and column-level data lineage. Additionally, the section explains the difference between business data lineage, which provides business context, and technical data lineage, which provides detailed insight into the processes by which data is consumed.

Table-level vs. column-level data lineage


The main difference between table-level and column-level is not the type of lineage but the granularity of it.

Data lineage, like any other metadata, comes in various granularities. The granularity depends on the multiple systems part of your data infrastructure and the level of logging and observability set on those systems.

As most data stored in data warehouses and lakes can be represented in a tabular form, there are broadly two granularities to work with - table-level and column-level.

What is table-level data lineage?


Table-level lineage translates to the lineage metadata of a relational database or a data warehouse table. This metadata will tell you about the mapping of one table to another but won’t get into finer details about the mapping of various columns.

For instance, table-level lineage will tell you about the intermediate tables that were created during the cleansing and transformation of a table and how it got further transformed to create a dimensional table.

Table-level lineage in DataHub.

Table-level lineage in DataHub. Source: DataHub Documentation

What is column-level lineage?


Column-level lineage, on the other hand, will tell you how specific columns of every table have changed. It will tell you how data types have changed, how the precision has changed, and how new columns have been created by merging columns from different tables, among other things.

Column-Level Lineage in Atlan.

Column-Level Lineage in Atlan. Source: Atlan Product Screenshot

Business vs. technical lineage


The difference between business data lineage and technical data lineage is the same as the difference between business metadata and technical metadata.

Business data lineage provides business context through lineage. It could be comments, data classifications, justifications for data masking, notes for consumers, and more. Business data lineage can come in various forms. You can annotate your queries, create separate documents, or add notes somewhere accessible to other teams.

On the other hand, technical data lineage is primarily meant for engineers and technical analysts but is consumable by everyone in the organization.

The technical data lineage provides you an end-to-end, detailed insight into the processes by which the data has reached where a team consumes it. If you use a tool like dbt or Airflow for end-to-end workflows in your organization, you’ll be able to extract the technical data lineage very easily.

Depending on your use case, you can classify data lineage based on other parameters as well. Here’s an article that explores popular ways of classifying data lineage.

What are the benefits of data lineage?

The following are the benefits of data lineage:

  • Trust and transparency in data practices
  • Data quality and reliability
  • Data and application debugging
  • Data security and compliance with regulations

Trust and transparency in data practices


Teams in the organization consume data at different points in the data’s journey. Data lineage enables trust and transparency in an organization by allowing the teams to take a peek into the journey the data has taken from the source layer to the consumption layer, making the data movement, change, and transformation more visible.

Without consolidated & automatically generated data lineage, teams are left with no choice but to go through individual SQL scripts, data pipeline workflows, detailed documentation (which might be out of date), and test suites. This means that teams would have to put a lot of effort into ensuring they can trust the data and the processes by which it arrives.

Having data lineage, however, doesn’t require teams to be super vigilant about the data all the time. Data lineage builds trust and transparency by providing you with consolidated and organized information about the data’s journey. This goes hand in hand with quality and reliability - another side of the same coin - which we’ll discuss in the next section.

Data quality and reliability


Data quality works at a finer level than what was discussed in the previous section. Rather than just ensuring the visibility of workflows, data quality ensures that the data in the individual data assets are reliable and up to the mark. Data lineage helps organizations achieve this reliability.

Throughout data’s journey, it has to be moved around, reshaped, transformed, enriched, backfilled, and so on. Many of these activities are often lossy in nature, i.e., data is lost in the process. Most of the time, this loss is intentional, for instance, when you’re trying to truncate strings, split a column, adjust the precision of a date and time column, etc.

Even though this is intentional, it can result in bugs and errors. Therefore, it is very important to have guardrails, and checks and balances, to ensure that no valuable data is unintentionally lost during this process. As data lineage gives you end-to-end pipeline visibility, you can readily identify if and when something has gone wrong, which brings us to the next section, which discusses how data lineage helps with debugging.

Data and application debugging


Having extensive data and application tests is a good practice, but still, there are times when bugs and issues slip through the cracks. In those situations, it is vital to have all the tools that can help you debug at your disposal.

Representation of how data lineage can give you visibility of things that need to be fixed.

Representation of how data lineage can give you visibility of things that need to be fixed. Source: Atlan

Data lineage provides a new layer of information that is not usually available in standalone data quality, data profiling, and testing suites.

Having data lineage visualization while debugging is one of the most important features for data and analytics engineers, as they can track down issues faster with increased visibility into the data’s journey.

Having data lineage for this purpose is of the most value when the testing suites are integrated tightly with data lineage, i.e., when you can have the lineage information of a data asset along with its test results and issues in one place.

Data security and compliance with regulations


One of the up-and-coming use cases of data lineage is that it can provide organizations with a view into how data privacy and security are handled at different stages of the data’s journey. This is essential for compliance and audit purposes.

Representation of enforcing compliance by propagation governance policies via lineage.

Representation of enforcing compliance by propagation governance policies via lineage. Source: Atlan

Organizations need to know the answers to queries like when their data was masked, where it was virtualized, and have all the data privacy and security issues taken care of before delivering the data to a clean room, for instance.

This is especially critical when dealing with PII (personally identifiable information) or PHI (personal health information). Not complying with government standards can lead organizations to pay hefty fines that can definitely be avoided.

How do you generate data lineage?

To fully leverage the benefits of data lineage, it’s best to use a tool that connects across the data stack and automatically generates lineage metadata. You should look at the following capabilities while evaluating a data lineage tool:

  • Automated data lineage
  • Manual enrichment and editing of data lineage
  • Advanced SQL parsing for inferring data lineage
  • Granular, column-level data lineage
  • Support for a variety of data sources for extracting data lineage

Check this guide that lists 14 questions to ask during your search for the right data lineage tool to fully assess its depth (number of unique sources supported), breadth (number of fields or objects supported for each source), and utility (ability to power insights and actions across diverse data personas).

You can also check out our coverage of open-source data lineage tools, where we talk about tools like Tokern, Egeria, Pachyderm, OpenLineage, and TrueFacet.

Conclusion

This article took you through the basics of data lineage, why it is needed, and the value it brings to an organization. It also discussed the various types of data lineage.

We also dived a bit deeper into the numerous benefits of having data lineage in place for your organization, as it helps make the data more trustworthy, the data process more reliable, more compliant, and easy to develop.

In the end, the article also talked about various tips for evaluating data lineage tools. Do check out our other resources on data lineage for further learning and exploration.

Share this article

[Website env: production]