Data Lineage in Amundsen
Amundsen is an open source metadata management platform that was initially developed by Lyft engineering in response to their data discovery challenges. However, you can also use it to explore the data lineage of different data sources. There are several tools to capture data lineage from a data source, such as dbt and OpenLineage. This guide will help you set up data lineage for your data sources using Amundsen.
How does dbt help in setting up lineage?
dbt allows you to build, modularize and templatize your models and create more efficient transformation pipelines for your data warehouses and analytics systems. One of the added advantages of dbt is that it captures the flow of data based both on the models and the database metadata. To make use of the data flow metadata, you can use Amundsen’s data lineage capabilities.
Why is data lineage in Amundsen important?
A good data lineage gives you the easiest and earliest route to backtrack to the origins of data. It gives you a clear picture of how the data was created and how it has evolved through its lifecycle. Be it open source tools like Amundsen or commerical tools, it's important to look for best-of-breed data lineage capabilities to ensure easy spotting of data quality errors, quick root-cause analysis, impact analysis, policy propagation through lineage etc.
Prerequisites for setting up Data Lineage in Amundsen
- Amundsen GitHub repository
- Sample dbt data as metadata and lineage source — GitHub gists for catalog and manifest files
- Configuration file to enable table and column lineage in Amundsen — GitHub gist
- Docker and Docker Compose to build and run Amundsen’s images locally after the changes
Set up Amundsen
Clone the GitHub repository
Clone the official Amundsen Git repository. The easiest way to deploy Amundsen is to use the
docker-amundsen.yml file, which deploys the Amundsen with the default neo4j backend with the pre-built images fetched from Docker Hub. This integration with dbt requires some minor code changes, so you’d have to build the code, create your own Docker images, and deploy. To do that, you should use the
docker-amundsen-local.yml file. This is the ideal setup for developers.
git [email protected]:amundsen-io/amundsen.git
For a detailed understanding, please read our set-up guide
Enable Lineage in Amundsen
Go to the cloned directory and then to the following subdirectory:
This is where the frontend configuration resides. Out of the several configuration files, you need to replace
config-default.ts with the file stored on this link. Alternatively, you can make the change yourself by toggling
true for both
Build and Deploy
Now that you have enabled data lineage in Amundsen, you’ll need to build the frontend again. Although you can build the frontend individually, building it with the
docker-amundsen-local.yml is a much cleaner method, as it builds everything that you need to deploy Amundsen:
docker-compose -f docker-amundsen-local.yml build
Once the build is done, you’re ready to run Amundsen using the following command:
docker-compose -f docker-amundsen-local.yml up -d
Give it a minute or so to fire up and check the status of all the containers using the
docker ps command. If everything’s good, move to the next step.
Set up dbt
When you set up and run dbt with a source system, dbt creates a
manifest.json file in the
target directory. It also creates a
catalog.json file in the same directory after you run the
dbt docs generate command. These files contain all the metadata and lineage information Amundsen needs.
Amundsen comes with a dbt extractor, using which you can extract data from dbt after ingestion (loading) into Amundsen. Now, you can either install dbt, connect a source, create models, etc. to generate the lineage data, or you can use the following sample dbt files in the Amundsen examples to set up basic data lineage in Amundsen:
Alternatively, you can install dbt initialize a new dbt project, as follows:
python3 -m venv dbt-env dbt-env/bin/activate pip3 install dbt dbt init sample_dbt_project
Once you do that, you’ll need to configure a source in your
profiles.yml file. When your dbt models, seeds, and so on are ready, you can finally run the
dbt run command to create the entities in the source and populate the
manifest.json file. Run the
dbt docs generate file to create a
catalog.json file. You can access both these files on the following path —
Extract and Load Metadata
To load your custom generated data or the dbt sample data into Amundsen, you’ll utilize the sample dbt loader script provided in the Amundsen examples. This script does the following things:
- Uses Amundsen’s dbt extractor to get the metadata from the
- Populate the table search index in Elasticsearch based on the newly ingested data
Assuming that you’ve already either pointed Amundsen to the correct JSON files or copied them to the default sample data location, you can run the
sample_dbt_loader.py script to load the metadata into Amundsen, as shown in the image below:
You can ignore the Elasticsearch warnings. If you go to the UI, you’ll be able to see your dbt tables in Amundsen.
Navigating the Lineage UI
After enabling lineage for Amundsen, notice how an
Upstream column and a
Lineage tab has appeared in the UI. Search for your latest uploaded metadata and navigate to one of the tables. Depending on the placement of your table in the data flow, you’ll either see an
Upstream column, a
Downstream column, or both. In the following example,
fact_third_party_performance, there’s one
Navigate to the
Upstream tab to find out that
fact_catalog_returns is the upstream table for
fact_third_party_performance, which means to say that
fact_third_party_performance depends on
fact_catalog_returns, as shown in the image below:
Clicking on the
Lineage tab on the top-right corner will take you to the following screen, where you will see a visual representation of the lineage, as shown in the image below:
Real-life data sources will have much more complicated lineage graphs. The lineage graph above is the simplest example of data lineage.
In April 2021, Amundsen announced improvements to data lineage with native support for table and column level ingestion and storage. In the August community meeting, you can find more about Alvin, which integrates with Amundsen to provide a more comprehensive data lineage solution.
In this article, we discussed steps to set up data lineage capabilities in Amundsen using dbt.
Looking for an automated data lineage solution?See the demo