Snowflake Metadata Management: Importance, Challenges, and Identifying The Right Platform

What is Snowflake? #

Snowflake’s platform enables a wide variety of workloads and applications on any cloud, including data warehouses, data lakes, data pipelines, and collaboration as well as business intelligence, data science, and data analytics applications.

Quoting Snowflake’s website:

Snowflake is a fully managed service that’s simple to use but can power a near-unlimited number of concurrent workloads. Snowflake is your solution for data warehousing, data lakes, data engineering, data science, data application development, and securely sharing and consuming shared data.

Snowflake stands out because it decouples both storage and compute. This means you can spin up and down machines on demand based on the analytics workload.

Snowflake is a cloud-agnostic platform that can distribute data across regions as well as across cloud providers such as AWS S3, Azure, and GCP. Some of the customers of Snowflake include Dropbox, Doordash, Hubspot, Adobe, and Fitbit.

Learn more - Best cloud data warehouse solutions: A comparison and evaluation guide

Snowflake Architecture diagram. Source: Snowflake

Importance of Snowflake metadata management #

The following points make metadata management in Snowflake a critical practice:

1. Faster access to insights
2. End-to-end visibility
3. Improved data quality and trust
4. Improved operational efficiency
5. Compliance and regulations
6. Improve ROI on data

1. Faster access to insights #

Metadata management in Snowflake makes it easy to search, filter, and find data assets by various criteria.

2. End-to-end visibility #

Metadata gives you complete visibility into the lifecycle of a data asset — its source, where it is used, who uses it, the transformations on the data, etc.

3. Improved data quality and trust #

Metadata helps evaluate, understand and trust the data for its relevancy and fit for use.

4. Improved operational efficiency #

Metadata enables engineers to better design workflow automation — crawling, ingestion, and ETL. Metadata not only helps surface data issues but also helps resolve them by enabling root-cause analysis.

5. Compliance and regulations #

Metadata management helps audit the implementation of regulatory policies to meet compliance standards (GDPR, HIPPAA).

6. Improved ROI on data #

Improved productivity and operational efficiency help create more ROI from data management. The self-serve nature of modern metadata management increases the opportunities to extract more value from the data assets.

Snowflake metadata management: Storage and access #

Snowflake stores all the metadata in a global, unified solution called the Data Cloud.

Snowflake automatically creates metadata for data residing both externally (S3, Azure, GCP) and internally (within Snowflake), stores it as a key-value pair (dictionary), and makes it available via the Information Schema.

The Information Schema provides metadata that can be broadly classified into:

Views: Provide metadata about the database itself — Schema, views, tables, columns, file types, referential constraints, and usage/access privilege.

Table functions: Provide metadata about historical information and usage — Database usage and refresh history, query history, policy history, and warehouse load and metering history.

Challenges in Snowflake metadata management #

Even though Snowflake exposes metadata via INFORMATION_SCHEMA, the only way to access the schema/metadata is by writing SQL queries. This means the metadata is not accessible to everyone, say a typical business user.

As the definition of what is considered metadata is expanding, metadata exposed by Snowflake might be very limiting. If you consider bringing in custom metadata from ETL logs, quality checks, and pipeline error alerts then you might need a dedicated metadata management solution to tap into the full potential of metadata.

Even though Snowflake restricts access to sensitive data through DAC and RBAC, the complex requirements of modern data teams require more granular and automated ways to classify PIIs and propagate them downstream.

A Guide to Building a Business Case for a Data Catalog

Download free ebook

Atlan: Active metadata management for Snowflake #

The crux of active metadata is its openness and interoperability. Atlan makes it possible to effortlessly move metadata across your data stack — data lakes, warehouses, BI tools, pipelines, ETL — and helps embed rich context on the tools you are already familiar with.

Always on: Active metadata is a framework that continuously listens, collects, and processes metadata from ETL logs, SQL query history, quality metrics, and usage statistics. Atlan’s open API infrastructure allows you to expand the scope of what is traditionally considered to be metadata. This opens up a whole lot of metadata use cases such as discovery, lineage, observability, and monitoring.

Intelligent and action-oriented: Atlan automates building and updating lineage by parsing SQL queries. Use the API to build your own bots to update asset descriptions; classify, tag, and propagate sensitive assets such as PII, HIPAA, and GDPR; alert end users on stale and anomalous data assets.

Learn more - Active metadata: The foundation of modern data experiences

A framework for an active metadata platform. Source: Atlan

Snowflake Metadata Management with Atlan

Snowflake metadata management use cases #

Managing metadata of your Snowflake data assets powers the following use cases:

1. Data cataloging
2. Visualizing data lineage
3. Documentation in a data dictionary
4. Operationalizing data governance
5. Fostering collaboration over data
6. Implementing DataOps activities

#1 Data cataloging #

Metadata adds value to your Snowflake data assets by giving structure and meaning and thereby making identification and discovery easier. Metadata is the engine behind the rich search experience. It enables the users to find the right data faster, creating a positive user experience that directly improves trust and user adoption of data catalogs.

Snowflake metadata use case: Data discovery through data catalogs. Source: Atlan

#2 Visualizing data lineage #

Data lineage leverages Snowflake metadata to track and visualize the journey of your data assets— from ingestion to BI dashboards. Lineage enables data to be discovered at all points within its lifecycle. Data lineage provides the visibility needed to trace and troubleshoot data quality issues and fix broken pipelines using root cause analysis.

Snowflake metadata use case: Track the data from its source to the BI dashboard through Lineage. Source: Atlan

#3 Documentation in a data dictionary #

Snowflake Data Cloud provides a rich source of metadata. A data dictionary uses metadata to describe the data. A data dictionary is a documentation for all data assets in snowflake. It provides information like table names, table descriptions, relationships; column names and their referential constraints; data types, classifications, data profiling, and SQL queries attached to the data asset.

Snowflake metadata use case: Documentation for your database through a data dictionary. Source: Atlan

#4 Operationalizing data governance #

Metadata provides the framework for automating data governance and compliance for the Snowflake Data Cloud. Atlan helps automate classifying sensitive (PII) and private user data. The classification is then propagated both downstream and upstream using data lineage. Atlan also helps establish data access controls at an individual user and group/role level.

Snowflake metadata use case: Automate classification and access control through data governance. Source: Atlan

#5 Fostering collaboration over data #

No more frustration searching for table context across applications. Link Slack threads and Jira discussions directly to the data asset in contention — let everyone be on the same page. Collaboration establishes trust, encourages participation which directly drives adoption, and encourages finding new opportunities to extract value from data.

Snowflake metadata use case: Embedded collaboration with data assets and team members in the tools you are familiar with. Source: Atlan

#6 Implementing DataOps activities #

Snowflake metadata helps DataOps engineers to design and architect data pipelines and manage data quality. Atlan uses metadata to help detect and surface data anomalies and alert downstream business users. Orchestration and ETL logs as metadata are used to optimize and fix data flow issues. Provenance as metadata is used to find, retire and archive unused data sources and thereby reducing storage and compute costs.

Atlan active metadata management use cases. Source: Atlan

Case Study: Atlan + Snowflake — Metadata Management At Wework

4 types of metadata supported by Atlan #

1. Technical metadata #

It is metadata about the data assets in the Snowflake Data Cloud. It describes schemas, tables, columns, size, data types, relationships, classification, referential integrity, etc.

2. Operational metadata #

It gives information about transformations, SQL queries, ETL logs, pipeline error notifications, and data quality audit results.

3. Business metadata #

It is a collection of business terms, definitions, KPIs, and metrics that helps associate and understand the physical data assets linked with them.

4. Social metadata #

It is metadata generated by the users of data, which includes: Chat messages, tasks, tickets, notes, READMEs, upvotes, verifications, and shared SQL queries

Learn more - Types of metadata: How each helps with faster data discovery and better insights

Bring your own custom metadata from ETL, pipeline orchestration and observability tools. Source: Atlan

Atlan: A Snowflake Ready Technology Partner for metadata management #

Atlan is the first data catalog and metadata management solution approved by the Snowflake Ready Technology Validation Program.

Atlan is more than a metadata management and data cataloging tool. It is built by data engineers to solve the evolving needs of modern data teams which include faster discovery, transparent data flow, robust governance, and collaboration built on open infrastructure and an easy-to-use user interface.

The deep integration and the open API enable Atlan to solve other modern metadata use cases such as DataOps, workflow management, and pipeline automation.

Atlan has been named a leader in The Forrester Wave™: Enterprise Data Catalogs for DataOps, Q2 2022.

The report states,

“Atlan is the tool of choice for DataOps and data product deployment. Atlan’s vision is to create frictionless data product deployment through a single metadata and data automation platform.”