Introduce multiple datasources by adutra · Pull Request #4812 · apache/polaris

adutra · 2026-06-17T16:58:16Z

This change enables multiple datasources in Polaris, with possibility of runtime activation of the desired datasource.

This is achieved by:

Introducing a new enabler property: polaris.persistence.relational.jdbc.datasource, which points to a named datasource to activate.
Declaring, at build time, one named datasource for each JDBC driver shipped with the application.
Introducing a new DataSourceActivator config interceptor that will, at runtime, change the active setting of the target datasource to true, and all others to false (active is a runtime setting, contrary to db-kind).

This change also makes the server ship with the H2 driver by default, since that's the pre-condition for an H2 datasource to be selected at runtime.

This change will further enable as future improvements:

Using to JDBC + H2 by default for server images, while still allowing users to switch to PostgreSQL without having to rebuild Polaris.
Enabling the persistence to "plug" into not one, but many datasources, e.g. a specific datasource for main persistence, and a separate datasource for metrics. This was outlined in Introduce DataSourceResolver for multi-datasource support in JDBC per… #3960 – but the machinery to make that happen was missing.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Enables selecting among multiple JDBC datasources at runtime for Polaris relational-jdbc persistence, adds an interceptor to activate the chosen datasource, and updates defaults/docs/tests accordingly (including shipping H2 by default).

Changes:

Introduce polaris.persistence.relational.jdbc.datasource and a ConfigSourceInterceptor to toggle quarkus.datasource.<name>.active.
Add named datasource defaults (H2 + PostgreSQL) and update Gradle dependencies/licenses to ship H2.
Update documentation and test profiles/resources to use named datasource configuration.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
site/content/in-dev/unreleased/metastores/relational-jdbc.md	Docs updated to configure named PostgreSQL datasource + new Polaris selector property
site/content/in-dev/unreleased/helm-chart/production.md	Helm bootstrap example updated to named PostgreSQL datasource env vars
site/content/in-dev/unreleased/helm-chart/persistence.md	Helm pool tuning example updated to named PostgreSQL datasource keys
site/content/in-dev/unreleased/helm-chart/dev.md	Helm dev bootstrap example updated to named PostgreSQL datasource env vars
site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_persistence_relational_jdbc.md	Config docs expanded + new datasource selector documented
site/content/in-dev/unreleased/admin-tool.md	Admin tool examples updated to named PostgreSQL datasource env vars
runtime/test-common/src/main/java/org/apache/polaris/test/commons/RelationalJdbcProfile.java	Test profile renamed to Postgres-specific profile (class name changed)
runtime/test-common/src/main/java/org/apache/polaris/test/commons/PostgresRelationalJdbcLifeCycleManagement.java	Postgres test resource config updated to named datasource + selector property
runtime/test-common/src/main/java/org/apache/polaris/test/commons/CockroachRelationalJdbcProfile.java	Cockroach test profile overrides simplified
runtime/test-common/src/main/java/org/apache/polaris/test/commons/CockroachRelationalJdbcLifeCycleManagement.java	Cockroach test resource config updated to named datasource keys + selector property
runtime/service/src/test/java/org/apache/polaris/service/it/PolicyServiceIntegrationTest.java	Integration test profile switched to H2 relational-jdbc profile
runtime/service/src/test/java/org/apache/polaris/service/events/listeners/inmemory/InMemoryBufferEventListenerTestBase.java	Tests updated to inject a named datasource
runtime/service/src/test/java/org/apache/polaris/service/events/listeners/inmemory/InMemoryBufferEventListenerIntegrationTest.java	Tests updated to inject a named datasource
runtime/service/src/test/java/org/apache/polaris/service/Profiles.java	Centralize H2 relational-jdbc test config and apply to multiple profiles
runtime/service/src/intTest/java/org/apache/polaris/service/it/relational/jdbc/JdbcViewFileIT.java	ITs updated to use Postgres-specific JDBC profile
runtime/service/src/intTest/java/org/apache/polaris/service/it/relational/jdbc/JdbcRestCatalogIT.java	ITs updated to use Postgres-specific JDBC profile
runtime/service/src/intTest/java/org/apache/polaris/service/it/relational/jdbc/JdbcPolicyServiceIT.java	ITs updated to use Postgres-specific JDBC profile
runtime/service/src/intTest/java/org/apache/polaris/service/it/relational/jdbc/JdbcManagementServiceIT.java	ITs updated to use Postgres-specific JDBC profile
runtime/service/src/intTest/java/org/apache/polaris/service/it/relational/jdbc/JdbcApplicationIT.java	ITs updated to use Postgres-specific JDBC profile
runtime/service/build.gradle.kts	Add H2 runtime/test dependency (excluding jts-core)
runtime/server/distribution/LICENSE	Add H2 driver licensing notice
runtime/server/build.gradle.kts	Ship H2 JDBC runtime dependency in server distribution
runtime/defaults/src/main/resources/application.properties	Define named datasources (h2/postgresql) + new selector property default
runtime/defaults/src/main/resources/application-test.properties	Test defaults updated for datasource activation + disable devservices
runtime/defaults/src/main/resources/application-it.properties	IT defaults updated + note about build-time properties
runtime/common/src/main/resources/META-INF/services/io.smallrye.config.ConfigSourceInterceptor	Register `DataSourceActivator` interceptor
runtime/common/src/main/java/org/apache/polaris/quarkus/common/config/jdbc/QuarkusRelationalJdbcConfiguration.java	Add `dataSource()` to config mapping + richer javadoc
runtime/common/src/main/java/org/apache/polaris/quarkus/common/config/jdbc/JdbcProducers.java	New producer selecting the configured named datasource
runtime/common/src/main/java/org/apache/polaris/quarkus/common/config/jdbc/DataSourceActivator.java	New interceptor toggling datasource `.active` based on Polaris config
runtime/admin/src/testFixtures/java/org/apache/polaris/admintool/AdminProfiles.java	Remove datasource activation overrides from admin tool test profiles
runtime/admin/src/main/resources/application.properties	Admin defaults updated to named PostgreSQL datasource + selector property
runtime/admin/build.gradle.kts	Align JDBC driver dependencies with Quarkus named datasource approach
persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/idempotency/RelationalJdbcIdempotencyStorePostgresIT.java	Update test config to implement new `dataSource()` method
persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/SimpleRelationalJdbcConfiguration.java	Update test config to implement new `dataSource()` method
persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/RelationalJdbcProductionReadinessChecksTest.java	Update readiness check test to point at new offending property
persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/MetricsReportPersistenceTest.java	Update test config to implement new `dataSource()` method
persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/JdbcGrantRecordsIdempotencyTest.java	Update test config to implement new `dataSource()` method
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/RelationalJdbcProductionReadinessChecks.java	Update readiness check offending property to new selector property
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/RelationalJdbcConfiguration.java	Add required `dataSource()` to persistence config interface
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java	Remove old datasource producer (moved to runtime/common)

Comments suppressed due to low confidence (1)

site/content/in-dev/unreleased/metastores/relational-jdbc.md:1

The property key quarkus.datasource.postgresql.jdbc.jdbc-url is inconsistent with the rest of this doc (which uses .jdbc.url) and with Quarkus datasource configuration keys (...jdbc.url). Update this entry to quarkus.datasource.postgresql.jdbc.url to avoid directing users to a non-functional configuration.

adutra · 2026-06-17T20:05:47Z


-  @Produces
-  @ApplicationScoped
-  static DatasourceOperations produceDatasourceOperations(


Moved to polaris-runtime-common.

This is probably for my information, why do we move it? Should this be part of JDBC module for better modularization?

I had a strong requirement for this move: the new producer method now needs to access io.quarkus.agroal.DataSource.DataSourceLiteral. I noticed that #3960 did the same move for similar reasons.

adutra · 2026-06-17T20:06:15Z

  Optional<String> databaseType();
+
+  /** The datasource name to use. Required. */
+  String dataSource();


Note: the datasource name is distinct from the database type.

adutra · 2026-06-17T20:08:03Z

+# Named datasources for relational persistence backends.
+# Polaris ships with many built-in datasources; by using named datasources, users can configure
+# at runtime which datasources to use for relational persistence backends.
+quarkus.datasource.postgresql.db-kind=postgresql


The admin module does not ship H2, at least for now.

We may want to include H2 later. There are pros and cons:

con: running the admin tool on an ephemeral database is meaningless

pro: running the admin tool on an ephemeral database allows users to play with the different commands without having to configure a database.

Not sure H2 helps - at least not as long it's only in-memory.

adutra · 2026-06-17T20:09:18Z

-public interface QuarkusRelationalJdbcConfiguration extends RelationalJdbcConfiguration {}
+public interface QuarkusRelationalJdbcConfiguration extends RelationalJdbcConfiguration {
+
+  /** The maximum number of retries before giving up the operation. */


We need to repeat the javadocs here for them to appear in the docs.

Maybe follow-up work in the docs tooling.

adutra · 2026-06-17T20:13:01Z

+* Maven group:artifact IDs: com.h2database:h2
+
+Project URL: https://www.h2database.com/
+License: Eclipse Public License 1.0 - https://www.eclipse.org/org/documents/epl-1.0/


I picked EDL 1.0 since it's category A, while Mozilla is category B. @jbonofre @snazy is that OK?

Slight correction: H2 is dual-licensed: EPL 1.0 and MPL 2.0 - both are category "B" and fine to be included in a binary convenience artifact.

🤦‍♂️ - sorry about the confusion.

adutra · 2026-06-17T20:13:24Z

  runtimeOnly(project(":polaris-relational-jdbc"))
  runtimeOnly("io.quarkus:quarkus-jdbc-postgresql")
+  runtimeOnly("io.quarkus:quarkus-jdbc-h2") {
+    exclude(group = "org.locationtech.jts", module = "jts-core")


This dependency is for geometry types; we don't need it.

adutra · 2026-06-17T20:16:32Z

-        Map.entry("quarkus.datasource.username", postgres.getUsername()),
-        Map.entry("quarkus.datasource.password", postgres.getPassword()),
-        Map.entry("quarkus.datasource.jdbc.initial-size", "10"),
-        // Configure metrics named datasource to use the same PostgreSQL instance


These "metric datasource" settings came as a surprise to me. I think it's an oversight; they were introduced by #3385 – but that PR does not implement separate datasources in its final form. Multiple datasources were attempted in #3960 (but it was closed without merging).

dimas-b · 2026-06-19T15:31:02Z

+      if (value == null
+          || value.getValue() == null
+          || active != Boolean.parseBoolean(value.getValue())) {
+        value = newConfigValue(value, Boolean.toString(active));


nit: why not do this unconditionally?

To avoid tampering with the configuration if not absolutely necessary.

Yes, but it defers any potential bug discovery 🤔

Do you think so? I can't really imagine a situation where this would be concerning.

it's a "nit" comment, not a big deal 🙂

dimas-b · 2026-06-19T15:35:36Z

-  @Inject Instance<DataSource> dataSource;
+  @Inject
+  @DataSource("h2")
+  Instance<AgroalDataSource> dataSource;


just wondering: why do we need to change the type here?

Heh, we don't, but if you keep DataSource, there is an annoying clash between java.sql.DataSource and io.quarkus.agroal.DataSource.

ah, ok 👍

dimas-b · 2026-06-19T15:42:36Z

 ## [Unreleased]

 ### Highlights
+- Polaris now supports dynamic datasource activation. Out of the box, two datasources are provided: `postgresql` and `h2`. The datasource to use can be selected at runtime by setting the `polaris.persistence.relational.jdbc.datasource` configuration property (the default is `postgresql`). This allows operators to switch between supported relational databases without rebuilding Polaris.


Helm-based deployments are not affected by this change, right?

However, the Admin Tool and plain Polaris Server binaries are affected, I guess 🤔 Should we call this out as a "breaking change"?.. or at least add an "upgrade notice"?

Hmm actually the Helm chart is affected – but the admin tool isn't (H2 is not available there). Let me rephrase this sentence.

Re: breaking change: the inclusion of a new artifact is not a breaking change, is it? Imho this PR is not a breaking change.

By "breaking" I mean old quarkus.* config properties that users might have in their scripts/deployment descriptors.

But I agree that an upgrade notice could be welcome, let me add that.

Hmm OK, in fact I agree that the configuration changes constitute a breaking change.

dimas-b · 2026-06-19T16:54:40Z

    - Names containing control (invisible) characters
    - Names with leading or trailing whitespace
    - Names containing any of these characters: <code>/\:*?"<>|#+`</code>
+- Due to the introduction of dynamic datasource activation, the default (PostgreSQL) datasource is now unused. If you had any custom configuration for that datasource, it should be migrated from `quarkus.datasource.*` to `quarkus.datasource.postgresql.*`. The same is valid for environment variables: `QUARKUS_DATASOURCE_*` should be replaced with `QUARKUS_DATASOURCE_POSTGRESQL_*`.


This is a reasonable notice to users, IMHO... However, given that the 1.6.0 release is supposed to happen pretty soon, should we defer merging this PR until after 1.6.0 is branched?

Yes, that's reasonable. We can defer this to 1.7.0, which would then include the complete switch to JDBC+H2.

Now that 1.6 is branches, I guess this is no longer a concern ;)

This change enables multiple datasources in Polaris, with possibility of **runtime** activation of the desired datasource. This is achieved by: 1) Introducing a new enabler property: `polaris.persistence.relational.jdbc.datasource`, which points to a named datasource to activate. 2) Declaring, **at build time**, one named datasource for each JDBC driver shipped with the application. 3) Introducing a new `DataSourceActivator` config interceptor that will, at runtime, change the `active` setting of the target datasource to `true`, and all others to `false` (`active` is a runtime setting, contrary to `db-kind`). This change also makes the server ship with the H2 driver by default, since that's the pre-condition for an H2 datasource to be selected at runtime. This change will further enable **as future improvements**: - Using to JDBC + H2 by default for server images, while still allowing users to switch to PostgreSQL *without having to rebuild Polaris*. - Enabling the persistence to "plug" into not one, but many datasources, e.g. a specific datasource for main persistence, and a separate datasource for metrics. This was outlined in apache#3960 – but the machinery to make that happen was missing.

snazy · 2026-06-26T11:44:14Z

-public interface QuarkusRelationalJdbcConfiguration extends RelationalJdbcConfiguration {}
+public interface QuarkusRelationalJdbcConfiguration extends RelationalJdbcConfiguration {
+
+  /** The maximum number of retries before giving up the operation. */


Maybe follow-up work in the docs tooling.

snazy · 2026-06-26T11:44:23Z

+# Named datasources for relational persistence backends.
+# Polaris ships with many built-in datasources; by using named datasources, users can configure
+# at runtime which datasources to use for relational persistence backends.
+quarkus.datasource.postgresql.db-kind=postgresql


Not sure H2 helps - at least not as long it's only in-memory.

snazy · 2026-06-26T11:49:11Z

+* Maven group:artifact IDs: com.h2database:h2
+
+Project URL: https://www.h2database.com/
+License: Eclipse Public License 1.0 - https://www.eclipse.org/org/documents/epl-1.0/


Slight correction: H2 is dual-licensed: EPL 1.0 and MPL 2.0 - both are category "B" and fine to be included in a binary convenience artifact.

snazy · 2026-06-26T11:51:16Z

+quarkus.datasource.postgresql.jdbc.username=<your-username>
+quarkus.datasource.postgresql.jdbc.password=<your-password>
+quarkus.datasource.postgresql.jdbc.jdbc-url=<jdbc-url-of-postgres>


Suggested change

quarkus.datasource.postgresql.jdbc.username=<your-username>

quarkus.datasource.postgresql.jdbc.password=<your-password>

quarkus.datasource.postgresql.jdbc.jdbc-url=<jdbc-url-of-postgres>

quarkus.datasource.postgresql.username

quarkus.datasource.postgresql.password

quarkus.datasource.postgresql.jdbc.url

snazy · 2026-06-26T11:51:58Z

    - Names containing control (invisible) characters
    - Names with leading or trailing whitespace
    - Names containing any of these characters: <code>/\:*?"<>|#+`</code>
+- Due to the introduction of dynamic datasource activation, the default (PostgreSQL) datasource is now unused. If you had any custom configuration for that datasource, it should be migrated from `quarkus.datasource.*` to `quarkus.datasource.postgresql.*`. The same is valid for environment variables: `QUARKUS_DATASOURCE_*` should be replaced with `QUARKUS_DATASOURCE_POSTGRESQL_*`.


Now that 1.6 is branches, I guess this is no longer a concern ;)

Copilot AI review requested due to automatic review settings June 17, 2026 16:58

github-project-automation Bot added this to Basic Kanban Board Jun 17, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jun 17, 2026

adutra force-pushed the dynamic-datasource branch from 43523ce to 345c62a Compare June 17, 2026 16:59

adutra changed the title ~~This change enables multiple datasources in Polaris, with possibility…~~ Introduce multiple datasources Jun 17, 2026

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Copilot started reviewing on behalf of adutra June 17, 2026 17:44 View session

adutra force-pushed the dynamic-datasource branch 3 times, most recently from bddc63d to 03f46e8 Compare June 17, 2026 20:04

adutra commented Jun 17, 2026

View reviewed changes

adutra force-pushed the dynamic-datasource branch 2 times, most recently from 394da9c to 8b924f4 Compare June 17, 2026 20:40

adutra mentioned this pull request Jun 17, 2026

Introduce DataSourceResolver for multi-datasource support in JDBC per… #3960

Closed

4 tasks

dimas-b previously approved these changes Jun 19, 2026

View reviewed changes

github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jun 19, 2026

adutra dismissed dimas-b’s stale review via 982d31a June 19, 2026 16:37

dimas-b previously approved these changes Jun 19, 2026

View reviewed changes

adutra added 5 commits June 26, 2026 12:06

Fix scripts

bb3cb70

CHANGELOG.md

b88e82f

rephrase changelog

14b5d27

more rephrase changelog

2fab4d2

adutra dismissed dimas-b’s stale review via 2fab4d2 June 26, 2026 10:07

adutra force-pushed the dynamic-datasource branch from 7156f77 to 2fab4d2 Compare June 26, 2026 10:07

snazy reviewed Jun 26, 2026

View reviewed changes

adutra added 2 commits June 26, 2026 15:39

fix bad conflict resolution

c5f6829

review

b9f7d31

snazy approved these changes Jun 26, 2026

View reviewed changes

Uh oh!

Conversation

adutra commented Jun 17, 2026

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment