Key Takeaways
- The Precisely Data Integrity Suite supports OpenLineage, unifying lineage from Airflow, Spark, dbt, Dagster, and more into a single trusted catalog without custom connectors.
- OpenLineage enables operational lineage, a continuously updated view of what happened during pipeline execution, alongside static lineage from native integrations.
- Lineage stays complete: placeholder assets preserve edges for undiscovered datasets, and replayed events never corrupt catalog state.
Data lineage is only valuable if it reflects the reality of your data environment. Today’s enterprises rely on a mix of orchestration tools, transformation engines, cloud data platforms, and analytics services.
While these tools all move data, they don’t always share lineage information in a consistent way. The result is fragmented visibility that makes it harder to understand where data came from, how it changed, and what downstream systems may be affected when something goes wrong.
At Precisely, we believe your data catalog should adapt to your technology ecosystem, not the other way around. That’s why we’re investing in open standards like OpenLineage, making it easier to bring lineage from the tools you already use into the Precisely Data Integrity Suite.
How OpenLineage Extends an Open, Interoperable Platform
Open, interoperable platforms give you the flexibility to use the tools that best fit your business. That’s why the Precisely Data Integrity Suite supports OpenLineage, the open standard for sharing lineage information across modern data ecosystems. By supporting OpenLineage, the Data Integrity Suite can receive lineage events from any compatible orchestrator and incorporate them into a single, trusted lineage graph.
This extends lineage beyond the platforms we support through native connectors, making it easier to unify lineage across heterogeneous environments. Whether your organization uses Airflow, Spark, dbt, Dagster, Flink, Trino, or another OpenLineage-compatible platform, runtime lineage can flow into the same catalog alongside lineage harvested through ournative metadata integrations.
If you already use one of these tools, you’re only a configuration change away from capturing operational lineage in the Precisely Data Integrity Suite.
| Tool | OpenLineage Support |
| Dagster | Built-in via openlineage-dagster |
| Apache Airflow | Built-in via apache-airflow-providers-openlineage |
| dbt | Built-in via dbt-core OpenLineage integration |
| Apache Spark | OpenLineage Spark integration (automatic column lineage) |
| Apache Flink | OpenLineage Flink integration |
| Trino / Starburst | OpenLineage Trino integration |
Supporting OpenLineage is another example of Precisely’s commitment to openness and interoperability. As your technology stack evolves, your lineage can evolve with it, without requiring a custom integration for every new tool you adopt.
How Static Lineage and Operational Lineage Work Together
Beyond expanding interoperability, OpenLineage also enables operational lineage, giving you a continuously updated view of what happens as pipelines execute.
The Data Integrity Suite connects to supported data sources and extracts metadata about pipelines, views, stored procedures, and relationships to build static lineage. Because this approach derives lineage from metadata rather than execution, it requires no instrumentation of production workloads. It provides an efficient, non-invasive way to understand how data is designed to move across your environment.
But today’s data environments are increasingly dynamic. Pipelines run continuously. Transformations change throughout the day. Jobs are retried, modified, and redeployed. Metadata alone doesn’t always capture what actually occurred during execution.
That’s where operational lineage comes in. Instead of periodically harvesting metadata, the Data Integrity Suite can receive lineage events as workloads execute. These runtime events describe the actual reads, writes, and transformations that occurred during a pipeline run, creating lineage grounded in real execution rather than design-time definitions.
Operational lineage complements static lineage. Together, they provide a more complete understanding of how data moves throughout your organization.
| Static Lineage | Operational Lineage |
| Derived from metadata | Derived from runtime events |
| Shows how data is designed to move | Shows how data actually moved |
| Harvested from source definitions | Captured continuously as workloads execute |
| Design-time visibility | Runtime visibility |
By combining both perspectives, your organization gains greater confidence when performing impact analysis, troubleshooting production issues, supporting governance initiatives, and demonstrating compliance.
Connecting Your Open Lineage Producers
How Do You Connect an Orchestrator to the Precisely Data Integrity Suite?
Connecting an OpenLineage producer requires only a few configuration steps. Configure your producer to send events to the Data Integrity Suite API using your workspace credentials.
Endpoint: POST /v2/catalog/lineage
Authentication: API key or bearer token from your workspace credentials
| Region | Value |
| US | https://api.cloud.precisely.com |
| EU | https://api.eu1.cloud.precisely.com |
| GB | https://api.gb1.cloud.precisely.com |
| AU | https://api.au1.cloud.precisely.com |
openlineage.yml example:
Once configured, lineage begins flowing into the Data Integrity Suite after the next successful pipeline execution.
What happens after an event arrives?
Every OpenLineage event is acknowledged immediately and processed asynchronously, ensuring pipeline execution isn’t delayed while lineage is updated.
After a pipeline run completes, you get:
- Searchable, browsable TransformationJob assets for every pipeline run
- Lineage edges connecting source and target datasets
- Full column-level lineage with transformation labels
- Placeholder assets that upgrade to fully enriched assets when discovery runs
Because the platform processes the OpenLineage standard, the same ingestion pipeline works across multiple orchestration tools without requiring separate integrations for each one.
How OpenLineage Maps into the Data Catalog
The Catalog Concept Mapping
| OpenLineage Concept | Catalog Concept |
| Job (namespace + name) | A Transformation Job asset, searchable and browsable |
| Run (unique run ID) | Tracked for audit |
| Dataset (namespace + name) | An existing catalog asset, or a placeholder |
| Input → Output edge | A lineage relation |
| Facets | Asset properties: schema, ownership, data quality, docs |
What Happens When a Dataset Hasn’t Been Discovered Yet?
Pipelines often run before formal data source discovery completes. Rather than dropping lineage edges, the catalog creates placeholder assets — fully navigable catalog entries with provenance metadata from the event.
When discovery later harvests metadata from the source system, those placeholders are enriched automatically without rebuilding the lineage graph.
This approach ensures organizations can begin capturing operational lineage immediately while continuing to expand their metadata catalog over time.
Column-Level Lineage
Operational lineage doesn’t stop at the dataset level. When supported by the producer, the Data Integrity Suite can also capture detailed column-level lineage. Many OpenLineage producers, including Spark and dbt, emit detailed column-level lineage automatically.
When available, the Data Integrity Suite captures not only source and target columns, but also the transformation context behind every relationship, including transformation type, masking information, execution details, and producer metadata.
Each relationship links to a TransformationJob asset, allowing data teams to understand not only where data originated, but also how it was transformed before reaching downstream systems.
Transformation Job: Full Transformation Context
Each column lineage relation links to a Transformation Job asset that captures:
| Property | What IT Tells You |
| Name | The pipeline that produced this column mapping |
| Type / Subtype | Transformation category (e.g., AGGREGATION / SUM, IDENTITY, TRANSFORMATION) |
| Column Masked | Whether the source value was masked or anonymized |
| Run ID | The specific run that generated this lineage |
| Namespace | The orchestrator environment (e.g., dagster-prod) |
| Event Time | When the pipeline run completed |
| Producer | Which tool emitted the event |
Built for Reliable Production Workloads
Operational lineage must be reliable enough for production environments. The Data Integrity Suite processes OpenLineage events idempotently, meaning replayed or duplicate events never create duplicate lineage relationships or overwrite existing metadata.
The platform also handles partial events gracefully. Dataset-level lineage remains available while unresolved column mappings are retried after additional metadata becomes available through discovery.
These capabilities allow organizations to trust operational lineage even in environments where retries, deployments, and replayed jobs are part of normal operations.
Any tool that emits standard OpenLineage RunEvent payloads to an HTTP endpoint will work.
Summary
| Capability | Detail |
| ✓ Zero-connector integration | Any OpenLineage-compatible tool connects with a URL and a token |
| ✓ Dataset lineage | Automatic lineage relations from every COMPLETE pipeline event |
| ✓ Column lineage | Field-level lineage with transformation type, subtype, description, and masking context |
| ✓ Placeholder assets | Lineage is complete from day one, even before discovery runs |
| ✓ Metadata enrichment | Schema, ownership, data source, and documentation from OpenLineage facets |
| ✓ Safe retries | Duplicate or replayed events never corrupt catalog state |
| ✓ TransformationJob assets | Full provenance trail of what transformed each column and when |
Bringing Static and Operational Lineage Together
No single approach captures every aspect of data movement.
Static lineage provides the design-time view of how data is intended to flow across systems. Operational lineage validates what actually happened during execution.
Together, they provide a richer, more complete understanding of your data landscape.
By supporting OpenLineage, the Precisely Data Integrity Suite extends lineage beyond native integrations while adding continuously updated runtime visibility from the tools your teams already use. The result is a unified lineage graph that combines metadata-driven lineage with operational insights, giving data teams greater confidence in the accuracy, completeness, and trustworthiness of their data.
As organizations continue to modernize their data ecosystems, open standards like OpenLineage help ensure lineage evolves alongside them, without sacrificing interoperability or visibility.
Learn more about OpenLineage and explore the growing ecosystem of compatible tools and integrations.
_____________________________________________________________________
Frequently Asked Questions
Q. Does OpenLineage work with my existing orchestrator?
A. If your orchestrator is Airflow, Spark, dbt, Dagster, Flink, or Trino/Starburst, built-in or mature community support is available. Configuration is a single YAML change pointing to the Precisely API endpoint. If your tool is not on this list, any tool that emits standard OpenLineage RunEvent payloads over HTTP will also work without modification.
Q. What happens if a dataset hasn’t been discovered yet?
A. The catalog creates a placeholder asset with provenance metadata from the event, keeping lineage edges intact. When discovery runs later, the placeholder is automatically enriched with full metadata. No lineage needs to be rebuilt.
Q. Is dataset-level lineage still available when column-level lineage is captured?
A. Yes. When column-level lineage is resolvable, dataset-level lineage is automatically inferred by rollup so both views are available in the catalog UI. There are no duplicate edges in the graph.
Q. What happens if an event is re-sent or replayed?
A. Nothing changes in the catalog. Events are processed idempotently — re-sending the same event does not create duplicate lineage relations, re-create Transformation Job assets, or overwrite existing metadata.
