OpenLineage solved the transport problem for lineage: one event format, any instrumented engine, no per-platform integration to write. DataHub implements it as a backend, and does so in two places rather than one. Atlan ingests metadata from DataHub and OpenMetadata into a governed context layer, then adds what a lineage standard deliberately leaves out: certified definitions, ownership, and policy that reach an agent at query time. Lineage tells you where a number came from. It does not tell you what the number means.
How does DataHub support OpenLineage?
OpenLineage defines a common event format for jobs, runs, and datasets, so any instrumented engine can publish lineage without a platform-specific integration. DataHub implements support primarily through a dedicated REST endpoint and a Spark Event Listener plugin.
The REST endpoint
This is a generic HTTP receiver for any OpenLineage producer, from Dagster to Trino to a custom emitter. It carries the eventType, run, job, inputs, outputs, and any attached facets. DataHub’s OpenLineage documentation specifies a single POST target for incoming events:
POST GMS_SERVER_HOST:GMS_PORT/openapi/openlineage/api/v1/lineage
For Airflow, the transport block follows the Apache Airflow OpenLineage provider configuration and points at the same path with an API key for authentication. What those Airflow OpenLineage events carry once they land is a narrower set than most teams assume.
The Spark Event Listener plugin
This is a Java agent that attaches to a Spark session and emits enriched events directly. It listens for application start and end events plus SQL execution events, creating pipelines and tasks.
- Distribution: published as
io.acryl:acryl-spark-lineagefor Scala 2.12 and 2.13. - Runtime requirement: the agent jar targets Java 17 bytecode, so Java 8 and 11 clusters are not supported.
- Emission options: REST by default, with Kafka, file, and S3 emitters also available.
DataHub’s Spark lineage documentation lists capabilities the generic endpoint does not offer:
- PathSpec support: maps HDFS, S3, GCS, and Azure paths onto table-level dataset URNs.
- Transformation types: captures OpenLineage’s column lineage facet transformation types and maps them to DataHub’s fine-grained lineage.
- Cross-platform instances: resolves a platform instance per upstream connection when one job reads from several accounts or hosts.
- Patch emission: appends rather than overwrites existing lineage edges on a dataset.
That last one matters more than it looks. Overwrite semantics are how a lineage graph quietly loses edges a different producer wrote, which is why lineage parsed from SQL beats hand-mapped lineage on completeness.
What does DataHub’s OpenLineage support include?
Support covers tracking at both the table and column levels. It captures dataset facets, job lifecycle events, and run states. Here is what gets captured and mapped onto DataHub’s entity graph:
- Jobs to DataJobs: each OpenLineage job becomes a task entity in DataHub.
- Runs to DataFlows: pipeline-level entities are created with cluster and environment attributes applied.
- Datasets to Dataset URNs: inputs and outputs resolve to dataset entities using OpenLineage’s naming conventions.
- Facets to aspects: schema and column lineage facets populate DataHub’s schema and fine-grained lineage aspects, a narrow slice of the metadata types an agent draws on.
Column-level lineage is enabled by default, and lineage applies across platforms, so a graph can run from raw source systems through orchestration and into BI dashboards, the raw material for an enterprise data graph. That breadth is what makes lineage useful for impact analysis and root-cause work, and it is the same substrate an agent leans on when it has to justify a number, which is the job lineage does for AI.
Known limitations
DataHub’s OpenLineage page notes that full PathSpec support is not yet available on the OpenLineage endpoint, though it is present in the DataHub Cloud Spark plugin.
DataHub also recommends its native Spark and Airflow plugins over the generic endpoint for those two engines. Its metadata standards documentation explains why: the Airflow plugin’s schema-aware SQL parsing produces more accurate column-level lineage than the OpenLineage extractors on their own.
The limitation that outlasts every version bump is structural. OpenLineage has no domain facet, so classification cannot ride along with a payload. DataHub works around it with DATAHUB_OPENLINEAGE_DOMAINS set at the endpoint, which applies the listed domain URNs to every DataFlow and DataJob created there, replacing any existing assignment including one set by hand in the UI. Domain is a property of the endpoint, not of the event. Making that kind of guarantee explicit is what data contracts for AI are for.
When is DataHub the right choice?
DataHub fits when a team needs an open-core, API-first metadata platform handling scale across millions of assets. It works well when:
- You have engineering-led teams: the open-source edition means running and upgrading Kubernetes, a relational database, Elasticsearch or OpenSearch, and Kafka. On a Kafka-native architecture where metadata changes publish as events to many subscribers instantly, DataHub is a natural fit.
- Your focus is pipeline and technical metadata: the event model and GraphQL APIs suit core engineering workflows, tracking ingestion pipelines and mapping technical impact analysis, which is where metadata management for AI starts. That is metadata management, and it is a different discipline from semantic understanding.
- Your AI scope is narrow and predefined: DataHub ships purpose-built agents for specific fixed jobs. Where agent requirements are tightly confined, a general-purpose context substrate may be more than you need.
What is the catch?
Self-hosting is free to license and not free to run. Most enterprise governance controls sit in paid DataHub Cloud rather than the open-source edition, and business-user surfaces are Cloud-only, so adoption often stalls inside the data team before any business context reaches the people who need it. Running it yourself is a real cost, and the criteria for building or buying metadata tooling price it honestly, as does the TCO of building, buying or bundling a context layer.
How does Atlan extend lineage into governed context?
OpenLineage solves transport. It does not handle what happens to lineage once it lands. DataHub’s OpenLineage support is metadata infrastructure for a platform team tracking pipelines, assets, and technical lineage, and the audience is primarily engineers.
Atlan is a managed AI context platform. It governs metadata, engineers context from SQL, pipelines, and BI semantics, and delivers it to agents and business users, which is what an enterprise context layer does. That is a different altitude from a data catalog, and a different structure from a knowledge graph.
Atlan treats lineage as one input into a governed context layer rather than the output itself:
- Ingests metadata from open-source catalogs including DataHub and OpenMetadata, so teams can layer governance and context on top during a migration.
- Parses actual SQL, pipeline definitions, and BI semantics across connectors covering modern, legacy, and SaaS systems, rather than mapping lineage by hand.
- Compounds definitions, metrics, and entities from real usage into an active business graph agents can reason across, rather than a glossary with fixed relationship types, closer to a knowledge graph than a data catalog. Active Ontology is the structure that holds it, sitting on a metadata knowledge graph.
- Builds eval suites from real dashboards and queries in Context Engineering Studio, testing whether an agent answers correctly before it reaches production, because evaluating context is not the same as testing data quality.
- Delivers governed context to Cortex, Claude, Cursor, and Agentspace over MCP and A2A, and lets agents write what they learn back into a shared context repository. Choosing between MCP and A2A, or between MCP and a plain API, depends on whether the agent needs to discover the tool or just call it.
Many teams run both. DataHub OpenLineage support captures pipeline metadata for the platform team, while Atlan serves as the governed context layer for business users and agents. That pattern works because Atlan reads DataHub’s output rather than competing for the same ingestion path, and because serving lineage over MCP turns those edges into something an agent can query.
Moving forward with DataHub and OpenLineage
DataHub OpenLineage support does what it sets out to do. The REST endpoint accepts standard events from any instrumented producer, and the Spark plugin adds depth where the generic path is thin. If what you need is impact analysis and pipeline observability, it is sufficient, and the self-hosting cost is a known quantity you can plan around.
If you want lineage to help agents answer business questions, or to support people who do not write SQL, lineage is not enough. It tells your agent where a number came from, not what the number means, whether the definition changed last quarter, or which team owns it. Those are semantic and governance questions, and no lineage standard answers them. Answering them is why AI agents need an enterprise context layer, and getting lineage there first is what makes it AI-ready.
Weigh the event-driven power of DataHub against what your agents and your business users will actually ask for. Most estates end up needing both, and the interesting question is which layer owns the definition when the two disagree.
FAQs about DataHub OpenLineage support
1. Does DataHub support OpenLineage natively?
Yes. DataHub accepts OpenLineage events natively through a REST endpoint and translates them into its internal metadata model, where they appear in the UI and APIs alongside natively ingested assets. No intermediate collector is required. DataHub also ships a Spark Event Listener plugin that builds on OpenLineage’s Spark integration with additional capabilities.
2. What is the DataHub OpenLineage endpoint URL?
The endpoint is a POST to GMS_SERVER_HOST:GMS_PORT/openapi/openlineage/api/v1/lineage, with the OpenLineage message supplied in the request body as JSON. For Airflow, the transport configuration points at /openapi/openlineage/ as the base URL with api/v1/lineage as the endpoint, authenticated with an API key.
3. Does DataHub capture column-level lineage from OpenLineage events?
Yes, and it is enabled by default. The DATAHUB_OPENLINEAGE_CAPTURE_COLUMN_LEVEL_LINEAGE property defaults to true on the REST endpoint, and the equivalent spark.datahub.captureColumnLevelLineage setting defaults to true on the Spark plugin. Both can be disabled, which is sometimes done for performance on very large datasets.
4. Should I use the OpenLineage endpoint or DataHub native plugins?
DataHub’s own documentation recommends the native Spark Lineage and Airflow plugins over the generic endpoint for those two engines, citing tighter integration. The Airflow plugin uses schema-aware SQL parsing to produce more accurate column-level lineage than the OpenLineage extractors alone. The generic endpoint is the better fit for producers with no dedicated DataHub integration.
5. Can OpenLineage events carry domain or ownership classification into DataHub?
Not on the event itself. OpenLineage has no domain facet, so classification cannot travel with the payload. DataHub works around this by letting you set DATAHUB_OPENLINEAGE_DOMAINS at the endpoint, applying the listed domain URNs to every DataFlow and DataJob created from events on that endpoint. This replaces any existing domain assignment, including one set manually in the UI.
6. What are the known limitations of DataHub OpenLineage support?
Full PathSpec support is not available on the OpenLineage REST endpoint, though it is present in the DataHub Cloud Spark plugin. The generic path also lacks the transformation-type mapping and patch behavior the Spark listener provides. Separately, OpenLineage carries lineage and schema but no business semantics, so definitions, ownership and policy still have to come from elsewhere.
7. Can an OpenLineage-based lineage graph make AI agents more accurate?
Lineage helps but is not sufficient on its own. It tells an agent where data came from, which improves grounding and makes impact analysis possible. It does not encode what a field means in business terms, whether that meaning has drifted, or which definition is canonical when several exist. Agent accuracy depends on that semantic layer sitting above lineage, delivered at query time rather than documented after the fact.
