Your Airflow DAG renamed a column six weeks ago, and the AI agent answering revenue questions never found out. Airflow is where most enterprise pipelines already run, and since Airflow 2.7.0 it ships a built-in provider, apache-airflow-providers-openlineage, that emits OpenLineage run events for every DAG and task without a separate package to maintain. According to Astronomer’s State of Airflow 2026 survey, 32% of Airflow users now run GenAI or MLOps workloads in production, so the orchestrator producing your lineage events is increasingly the same one feeding your AI agents.
That lineage does more for an agent than it ever did for a dashboard. A human analyst who sees a suspicious number can pause and ask questions; an agent that retrieves a stale definition just answers, confidently and wrong. Atlan ingests the OpenLineage events your Airflow DAGs already produce into a governed Enterprise Data Graph, so the column-level lineage an engineer built for compliance reporting becomes the same signal an agent checks before it answers.
- Every Airflow task lifecycle transition (start, success, failure) can emit a standard, tool-agnostic lineage event
- Facets carry schema, column lineage, SQL, and quality signals alongside each event, not bolted on after
- An agent that can query this graph knows not just what a table contains, but where it came from and whether that’s still true
| What it is | Automatically capturing OpenLineage run events for every Airflow DAG and task via the built-in provider |
|---|---|
| Governing standard | OpenLineage, a graduate project of the LF AI & Data Foundation |
| Airflow integration | apache-airflow-providers-openlineage, built in since Airflow 2.7.0 (AIP-53) |
| Core event types | START, COMPLETE, FAIL, and ABORT run events, each carrying facets |
| Key facets for AI grounding | schema, columnLineage, dataQualityMetrics, sql, sourceCodeLocation |
| Adoption signal | 32% of Airflow users run GenAI or MLOps workloads in production (Astronomer, 2026) |
What is OpenLineage and why does Airflow produce most of its events?
Permalink to “What is OpenLineage and why does Airflow produce most of its events?”OpenLineage answers a narrow question with an outsized consequence: which job touched this dataset, when, and what did it do to it. The specification defines a Run Event as the observed state of a job execution, carrying an eventType (START, COMPLETE, FAIL, or ABORT), an eventTime, a unique run ID, a job identified by namespace and name, and the input and output datasets involved. Every sender in the ecosystem, whether it’s Airflow, Spark, or dbt, emits the same event shape: one format any consumer can parse without a custom adapter per producer.
OpenLineage graduated from the LF AI & Data Foundation in 2023, with a steering committee drawing from Airflow, Snowflake, dbt, and Microsoft. That governance model matters for context infrastructure because no single vendor controls the format an agent’s context layer depends on.
Why Airflow specifically dominates the event volume
Permalink to “Why Airflow specifically dominates the event volume”Airflow remains the most widely used open source orchestrator in production, and unlike a bolt-on agent deployed separately, the provider is already installed wherever Airflow runs. Every DAG and task transition becomes an event candidate by default. That combination, the largest installed base plus zero-config emission, is why Airflow produces more OpenLineage events in the wild than Spark or dbt integrations, even though those tools support the spec too.
What changed when the integration became a native provider
Permalink to “What changed when the integration became a native provider”Before Airflow 2.7, lineage capture ran through an external openlineage-airflow package that depended on Airflow’s internals and broke whenever those changed. AIP-53 moved that logic into a first-class provider, tested alongside core releases and versioned independently, so a fix ships without waiting on an Airflow point release. The Astronomer survey also found 44% of Airflow users pair it with dbt, so most estates already run two of the three major producers side by side. Lineage that survives an orchestrator upgrade is lineage an agent can depend on across quarters, not one release cycle.
How does the Airflow OpenLineage provider actually capture lineage?
Permalink to “How does the Airflow OpenLineage provider actually capture lineage?”The provider works as an AirflowPlugin that registers an OpenLineageListener, which Airflow calls directly when a DAG or a TaskInstance starts, completes, or fails. For DAG-level events, the listener runs inside the Airflow scheduler; for task-level events, it runs on the worker executing that task, which keeps the emission close to where the actual data movement happens rather than centralizing it somewhere lineage data has to be reconstructed after the fact.
What happens between a task starting and an event being emitted
Permalink to “What happens between a task starting and an event being emitted”When a TaskInstance listener fires, it first constructs the event’s identity: a unique run ID and an event timestamp. It then works through extraction precedence: operators with native get_openlineage_facets_* methods are used first, since the operator author knows its own inputs and outputs best; operators without native support fall back to the provider’s built-in extractors for common patterns (BigQuery, Snowflake, S3, and others); SQL-based operators without either fall back further to automatic SQL parsing, which reads the query text to infer table-level and, where possible, column-level lineage. Teams get useful lineage on day one and improve its precision incrementally, without every operator needing hand-instrumentation first.
Reading the DAG as a graph, not a schedule
Permalink to “Reading the DAG as a graph, not a schedule”Once events flow, the DAG stops looking like a schedule and starts looking like a knowledge graph: jobs are nodes, datasets are nodes, and each run event is an edge carrying a timestamp and a set of facets. That structure is what makes impact analysis tractable before a change ships. Before changing a source table, a team can traverse forward through every downstream job the change will touch, the same traversal an agent needs to run backward when a wrong answer needs root cause analysis instead of a guess.
Which OpenLineage facets carry the context AI agents need?
Permalink to “Which OpenLineage facets carry the context AI agents need?”A run event without facets tells you almost nothing beyond “something ran.” Facets are where the useful detail lives, and they attach to three entities: the Run, the Job, and the Dataset, each independently versioned so a new facet type never breaks an older consumer.
| Facet | Attaches to | What it captures | Why it matters for agents |
|---|---|---|---|
| schema | Dataset | Column names and types at the time of the run | Tells an agent what fields existed when a decision was made, not just what exists now |
| columnLineage | Dataset | Which upstream columns fed each output column | Lets an agent trace a specific number to its exact source column, not just the table |
| dataQualityMetrics | Input dataset | Row counts, null rates, distinct values at run time | Flags whether the data an agent is about to cite passed quality checks that day |
| sql | Job | The query text a task executed | Lets a human (or another agent) audit exactly what transformation produced a dataset |
| sourceCodeLocation | Job | Where the job’s source code lives | Ties a lineage edge back to the actual DAG file and commit that produced it |
Why column-level lineage matters more than table-level for agents
Permalink to “Why column-level lineage matters more than table-level for agents”Table-level lineage tells you “this report depends on that table.” Column-level lineage tells you which specific field produced the number an agent is about to state, the difference between “somewhere in this pipeline” and a root cause an engineer can act on in minutes. Organizations with strong business glossaries already have definitions; the columnLineage facet connects them to the exact transformation step where they were last applied correctly, or silently stopped being applied at all.
Custom facets close the last gap
Permalink to “Custom facets close the last gap”Because facets are extensible, a team can define a custom facet, a policy classification or a data steward assignment, without waiting on a spec revision. Airflow’s custom_run_facets configuration lets any team inject organization-specific metadata directly, which matters for types of metadata AI agents need beyond what the standard spec anticipated: ownership, sensitivity tags, or a certification flag an agent should check before citing a table.
Why does pipeline lineage matter for grounding AI agents, not just dashboards?
Permalink to “Why does pipeline lineage matter for grounding AI agents, not just dashboards?”A dashboard has a person looking at it. If a number looks wrong, someone asks a question before it reaches a decision. An AI agent skips that checkpoint entirely: it retrieves context, reasons over it, and answers, with no human in the loop to catch the moment a definition went stale or a source table silently changed shape.
Freshness: how agents know a definition is still true
Permalink to “Freshness: how agents know a definition is still true”An Airflow DAG that refreshes a metric daily is a freshness signal in disguise. When that DAG’s OpenLineage events flow into a context layer, an agent can check not just “does this table exist” but “was it refreshed on schedule, and did the refresh succeed,” the same context freshness discipline that keeps an agent from confidently citing a number three runs out of date.
Provenance: proving where an answer actually came from
Permalink to “Provenance: proving where an answer actually came from”Provenance is the part regulators increasingly ask about directly. Fivetran’s 2026 Agentic AI Readiness Index found 42% of data professionals name missing lineage the top barrier to reliable agentic AI in production, and Gartner projects 60% of AI projects will be abandoned through 2026 for lack of AI-ready data. Airflow-instrumented lineage closes that gap without ripping out existing pipelines, because the events already exist; the work is connecting them to where agents actually query.
Blast radius: what a broken DAG means for every agent downstream
Permalink to “Blast radius: what a broken DAG means for every agent downstream”When a source table’s schema changes and an Airflow task fails, the OpenLineage graph already shows every job and dataset downstream of that failure. Without it, a team discovers the blast radius by watching which dashboards break, hours or days after the fact. With it, impact analysis runs as a graph traversal before the change ships, and an agent querying that context layer can be told a dataset is degraded the moment the failure lands, not after it has answered on bad data.
What breaks when Airflow lineage stops at the warehouse boundary?
Permalink to “What breaks when Airflow lineage stops at the warehouse boundary?”Instrumenting Airflow is necessary, not sufficient. Four gaps recur even in teams running OpenLineage cleanly.
Custom operators without native or extractor support. A bespoke API-calling operator with no get_openlineage_facets method, no matching built-in extractor, and no SQL to parse produces no automatic events for that step, often the newest and least-documented part of the pipeline.
Backfills and reruns muddy the timeline. A backfill emits the same job name with a different logical date, and without careful facet design, a three-year-old backfill can look identical to this morning’s run. An agent checking “was this refreshed today” needs the actual event time, not just an entry in the graph.
Unstructured and semantic sources sit outside the graph. OpenLineage tracks structured datasets well; it says nothing about the glossary term that defines what a column means or the policy document that governs who can see it. A pipeline can be perfectly instrumented and an agent can still retrieve the wrong definition, because that definition was never a tracked dataset in the first place.
It stops at the warehouse and never reaches the agent. Plenty of teams have excellent column-level coverage into their BI layer and nothing connecting that graph to the context store an agent actually queries at inference time. The events exist; the wiring to the agent’s retrieval path does not, the single most common reason teams with “good lineage” still can’t explain a wrong agent answer.
None of these are reasons to skip OpenLineage. They’re the reasons instrumenting Airflow is the starting line, not the finish line, for context an agent can actually rely on.
How Atlan turns Airflow’s OpenLineage events into governed AI context
Permalink to “How Atlan turns Airflow’s OpenLineage events into governed AI context”The challenge
Permalink to “The challenge”Most enterprises already run Airflow, and a growing share already have the OpenLineage provider enabled somewhere in their estate. The events exist. What’s usually missing is a place for them to live alongside business definitions, quality signals, and ownership, so an agent gets one coherent answer instead of a graph in one tool and a glossary in another.
The approach
Permalink to “The approach”Atlan ingests OpenLineage events directly from Airflow, importing DAGs, tasks, runs, and facets as native assets in the Enterprise Data Graph, the same graph carrying column-level lineage from Snowflake, BigQuery, and Databricks. A schema facet from an Airflow task and a certified glossary term end up as connected nodes, queryable from the same place. The MCP Server exposes that graph to agents directly: a single call returns a column’s provenance, quality score, and owner, sourced from the run that produced it.
The outcome
Permalink to “The outcome”When a source table’s schema shifts and a task fails, that failure event propagates through the graph immediately, and every enterprise memory an agent draws from can reflect the degraded status before it answers on top of it, not after. Atlan AI Labs research shows a 38%+ query accuracy uplift when agents operate on enriched metadata with lineage rather than raw schema alone, a gain that compounds when it comes from the orchestrator already running your pipelines, not a parallel system built for AI alone.
The 38%+ figure is from Atlan’s own research.
Why the orchestration layer is the cheapest place to build AI trust
Permalink to “Why the orchestration layer is the cheapest place to build AI trust”Every other layer in the stack, the semantic layer, the vector store, the agent’s own memory, depends on knowing where the data came from and whether it’s still current. Airflow already produces that signal on every scheduled run, for free. Teams that get the most agent reliability per dollar aren’t building a separate lineage system for AI; they’re connecting the events their orchestrator already emits to the context layer their agents query, and treating a broken DAG as a signal that should reach the agent before it reaches a wrong answer.
That’s a smaller lift than it sounds, and it starts with lineage you already have. See how Atlan connects your existing Airflow OpenLineage events straight into the context every agent in your organization queries.
FAQs about Airflow and OpenLineage
Permalink to “FAQs about Airflow and OpenLineage”1. What is OpenLineage?
Permalink to “1. What is OpenLineage?”OpenLineage is an open specification for collecting lineage metadata from data pipelines, governed as a graduate project of the LF AI & Data Foundation. It defines a standard format for run events and facets, so any tool that emits or consumes the spec can interoperate without custom code per pair of systems.
2. How does Airflow implement OpenLineage natively?
Permalink to “2. How does Airflow implement OpenLineage natively?”Since Airflow 2.7.0, the apache-airflow-providers-openlineage package ships as a built-in provider under AIP-53. It registers an AirflowPlugin that runs an OpenLineageListener, which fires on DAG and task lifecycle events and emits START, COMPLETE, and FAIL run events without requiring a separately maintained external package.
3. What is a facet in the OpenLineage spec?
Permalink to “3. What is a facet in the OpenLineage spec?”A facet is an atomic, named unit of metadata attached to a run, job, or dataset entity in an OpenLineage event. Standard facets include schema, columnLineage, dataQualityMetrics, sql, and sourceCodeLocation, and any team can define custom facets without breaking the base event format.
4. Why does Airflow produce more OpenLineage events than other orchestrators?
Permalink to “4. Why does Airflow produce more OpenLineage events than other orchestrators?”Airflow is the most widely used open source workflow orchestrator, and its provider emits an event for every DAG and task lifecycle transition by default, with no separate agent to deploy. That combination of installed base and zero-config emission makes it the largest single source of OpenLineage events in production today.
5. Do I need to instrument every custom operator manually?
Permalink to “5. Do I need to instrument every custom operator manually?”No. The provider checks whether an operator implements native get_openlineage_facets methods first, falls back to its library of built-in extractors for common operators, and applies automatic SQL parsing for SQL-based tasks. Manual instrumentation is only needed for operators outside all three paths.
6. Why does AI agent grounding need pipeline lineage, not just data quality checks?
Permalink to “6. Why does AI agent grounding need pipeline lineage, not just data quality checks?”A quality check tells you a table passed its tests today. Lineage tells an agent where a number came from, which DAG produced it, and whether the definition changed upstream, which is what lets a wrong answer be traced to a root cause instead of dismissed as a model problem.
7. Does adopting OpenLineage require replacing existing Airflow DAGs?
Permalink to “7. Does adopting OpenLineage require replacing existing Airflow DAGs?”No. The provider instruments existing DAGs and operators as they run; there is no DAG rewrite required. Teams enable the provider, configure a transport for events, and lineage capture starts on the next scheduled run.
Sources
Permalink to “Sources”- OpenLineage Specification, OpenLineage/OpenLineage (GitHub)
- AIP-53 OpenLineage in Airflow, Apache Software Foundation
- OpenLineage Airflow Integration Structure, Apache Airflow Documentation
- LF AI & Data Foundation Announces Graduation of OpenLineage Project (2023)
- State of Airflow 2026, Astronomer
- Lack of AI-Ready Data Puts AI Projects at Risk, Gartner (2025)
- The 2026 Agentic AI Readiness Index, Fivetran (2026)
- Simplify Your Airflow Data Lineage With OpenLineage, Atlan
