OpenLineage: add Persistence Layer SPI by iting0321 · Pull Request #4826 · apache/polaris

iting0321 · 2026-06-18T11:13:49Z

Description

This PR adds the persistence lineage contract and runtime delegation needed for future OpenLineage storage implementations.
The link for Persistence Contract.

It defines the dataset, dataset-edge, and column-edge payloads used by persistence backends, adds a LineagePersistence SPI, and wires lineage ingest/query calls through the runtime service while keeping lineage disabled unless explicitly configured.

Changes

Add persistence-oriented lineage records:
- LineageDataset
- LineageEdge
- LineageColumnEdge
- LineageFieldReference
- LineageIngestRequest
Add LineagePersistence with methods for:
- upsertDatasets
- replaceDatasetEdges
- upsertColumnEdges
- loadLineage
Add DisabledLineagePersistence as the default placeholder implementation.
Wire DefaultLineageService to delegate ingest/query operations to LineagePersistence.
Add unit tests for lineage persistence and service delegation.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Copilot

Pull request overview

This PR introduces a new lineage persistence SPI in polaris-core and adds a runtime-service implementation that gates lineage operations behind config + a realm feature flag, with a default “disabled” persistence implementation and unit tests.

Changes:

Added core lineage request/response models plus a LineagePersistence SPI and LineageService boundary.
Added runtime wiring (DefaultLineageService) and a default placeholder persistence (DisabledLineagePersistence) with tests.
Added a new realm feature flag (ENABLE_LINEAGE) and documentation/config reference entries for lineage-related properties.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_lineage.md	Adds a new config-section page for `polaris.lineage.*` properties.
site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md	Documents the new `ENABLE_LINEAGE` realm feature flag.
runtime/service/src/test/java/org/apache/polaris/service/lineage/DisabledLineagePersistenceTest.java	Verifies the default disabled persistence throws as expected.
runtime/service/src/test/java/org/apache/polaris/service/lineage/DefaultLineageServiceTest.java	Tests enablement gating and delegation behavior in `DefaultLineageService`.
runtime/service/src/main/java/org/apache/polaris/service/lineage/LineageConfiguration.java	Introduces SmallRye config mapping for `polaris.lineage.*`.
runtime/service/src/main/java/org/apache/polaris/service/lineage/DisabledLineagePersistence.java	Provides a default CDI bean implementing `LineagePersistence` that throws when used.
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java	Implements `LineageService` and delegates ingest/query to `LineagePersistence` when enabled.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageService.java	Defines a core service boundary for lineage ingest/query operations.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageQueryRequest.java	Adds a normalized lineage query request model.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineagePersistence.java	Adds the persistence SPI contract for lineage backends.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageNodeType.java	Adds enum for node kinds in lineage graphs.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageNode.java	Adds lineage graph node model with optional field mappings.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageIngestRequest.java	Adds extracted ingest payload model for persistence.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageGraph.java	Adds normalized lineage graph response model.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageGranularity.java	Adds enum for dataset vs. column granularity.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageFieldReference.java	Adds dataset+field reference used for column-level edges.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageFieldMapping.java	Adds source→target field mapping model for responses.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageEdge.java	Adds dataset-level edge model.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageDirection.java	Adds enum for upstream/downstream/both queries.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageDataset.java	Adds dataset identity model (with optional Polaris entity id linkage).
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageData.java	Adds dataset metadata model returned in lineage responses.
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageColumnEdge.java	Adds column-level edge model.
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java	Adds new `FeatureConfiguration<Boolean> ENABLE_LINEAGE` flag.

… reference

flyingImer

IMHO, the layering is close to the correct arch. See my inline comments below

flyingImer · 2026-06-24T21:39:00Z

+import java.util.Objects;
+
+/** A field-level lineage relationship between two dataset columns. */
+public record LineageColumnEdge(LineageFieldReference source, LineageFieldReference target) {


I feel these data models should belong to extensions/ for SPI placement, tho I am fine to leave under core/, but currently slightly inclined to put under extensions/ with the rationale of lineage is somehow "optional" capability to Polaris being a Iceberg irc spec impl. To keep core/ lean and tight towards that vision, I suggest to place under extensions/

Thanks for your feedback!
I agreed with your point, I just moved these data models to extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/

flyingImer · 2026-06-24T21:40:08Z

+import org.apache.polaris.core.lineage.LineageService;
+
+@RequestScoped
+public class DefaultLineageService implements LineageService {


why there is a default service? IMHO, it should be an default SPI impl under extensions/

I agree. Keeping new features in distinct modules is allows for more flexibility and control in downstream projects and does not limit what Polaris can include by default.

Moved runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java to extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/ and renamed it to DefaultPolarisLineageHandler.java .

flyingImer · 2026-06-24T21:42:44Z

+package org.apache.polaris.core.lineage;
+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {


I see, the "Service" naming is confusing. At a glance, this is the right SPI placement.

2 cents on naming: PolarisLineage or PolarisLineageHandler or the convention. My rationale is that xxxService usually implies runtime, which should be under runtime/

Change to PolarisLineageHandler.

flyingImer · 2026-06-24T21:47:13Z

+ * dataset resolution. Persistence backends persist dataset nodes, dataset edges, and column edges,
+ * and load normalized lineage graphs.
+ */
+public interface LineagePersistence {


This is seems to be durable logic related, I would suggest to name to something similar to MetastoreManager, e.g., LineageStoreManager

given lineage default behavior should be no-op per last discussion, so it definitely does not belong to core/, but extensions/

Change to LineageStoreManager and move all the data models from polaris-core/src/main/java/org/apache/polaris/core/lineage/ to extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/

dimas-b · 2026-06-25T02:02:49Z

+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {
+  void ingest(LineageIngestRequest request);


This class is not use in polaris-core ATM. Will this class be used by other polaris-core classes in the future? What is the envisioned use case / call path? If not, I do not see a reason for it to be in polaris-core. I believe it should be in a new SPI module dedicated to the OL feature.

Thanks for your feedback! I moved all the data models toextensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/.

iting0321 added 9 commits May 18, 2026 13:18

add Core SPI and domain model

b9265f0

update the node type

d630a09

add lineage data field

c073aae

fix the DefaultLineageServiceTest.java for feature is dark coverage

b85e227

refactor LineagePersistence

c8bd140

remove jakarta.annotation.Nullable

a793842

add lineage feature configurations to documentation

00c432a

remove persistence-related model

bf4c5d9

add persistence spi

1007ce9

github-project-automation Bot added this to Basic Kanban Board Jun 18, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jun 18, 2026

iting0321 changed the title ~~OpenLineage: Persistence Layer SPI~~ OpenLineage: add Persistence Layer SPI Jun 20, 2026

change upsertDatasetEdges to replaceDatasetEdge

21edaad

iting0321 marked this pull request as ready for review June 22, 2026 04:26

Copilot AI review requested due to automatic review settings June 22, 2026 04:26

Copilot started reviewing on behalf of iting0321 June 22, 2026 04:27 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

iting0321 added 2 commits June 22, 2026 20:21

add check polaris.lineage.persistence.enabled before ingest/query

32d12b0

Add the generated polaris.lineage config section to the configuration…

b23b8fb

… reference

snazy mentioned this pull request Jun 22, 2026

OpenLineage: Add Persistence schema and JDBC models #4827

Open

6 tasks

flyingImer reviewed Jun 24, 2026

View reviewed changes

dimas-b reviewed Jun 25, 2026

View reviewed changes

iting0321 and others added 3 commits June 26, 2026 21:15

move lineage models to extensions module

42001a7

move lineage defaults into extension module and rename the files

8b47436

Merge branch 'main' into ol-persistence-spi

2a17bd1

iting0321 mentioned this pull request Jun 27, 2026

OpenLineage : Add core lineage service scaffolding #4705

Open

6 tasks

Uh oh!

Conversation

iting0321 commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flyingImer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iting0321 commented Jun 18, 2026 •

edited

Loading