Introduce multiple datasources

CHANGELOG.md

@@ -28,11 +28,13 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 ## [Unreleased]
 
 ### Highlights
+- Polaris now supports dynamic datasource activation. This allows operators to switch between supported relational databases without rebuilding Polaris. Currently, the Polaris server ships with two datasources: `postgresql` and `h2`. The admin tool, however, includes only `postgresql` for now. Out-of-the-box support for more datasources and JDBC drivers may be added in the future. The datasource to activate can be selected at runtime by setting the `polaris.persistence.relational.jdbc.datasource` configuration property (the default is `postgresql`).
 
 ### Upgrade notes
 - Event listeners are now executed on a dedicated executor. **This executor does not propagate the original request's CDI context**; listeners that were improperly relying on that should instead manage their own CDI request scope from now on. Furthermore, two new configuration options were introduced to configure the executor: 
   - `polaris.event-listener.executor.pool-size` configures the thread pool size.
   - `polaris.event-listener.executor.queue-size` configures the queue size for pending events when all threads are busy.
+- The Helm chart now includes a new `persistence.relationalJdbc.datasource` option to select the relational database to activate. The default is `postgresql`, which matches pre-existing behavior.
 
 ### Breaking changes
 - The `MaintenanceService.performMaintenance()` signature now requires an explicit `OptionalLong overrideRunId` argument to supersede the latest unfinished maintenance run.
@@ -42,6 +44,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
     - Names containing control (invisible) characters
     - Names with leading or trailing whitespace
     - Names containing any of these characters: <code>/\:*?"<>|#+`</code>
+- Due to the introduction of dynamic datasource activation, the default (PostgreSQL) datasource is now unused. If you had any custom configuration for that datasource, it should be migrated from `quarkus.datasource.*` to `quarkus.datasource.postgresql.*`. The same is valid for environment variables: `QUARKUS_DATASOURCE_*` should be replaced with `QUARKUS_DATASOURCE_POSTGRESQL_*`.
 
 ### New Features
 - Added GCS principal attribution for vended credentials (the GCP counterpart of AWS STS session tags). Set `GCS_PRINCIPAL_ATTRIBUTION_ENABLED=true` to activate; the feature flags `GCS_PRINCIPAL_ATTRIBUTION_WIF_AUDIENCE`, `GCS_PRINCIPAL_ATTRIBUTION_TOKEN_ISSUER`, and `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_FILE` are then required (a missing value is a fatal configuration error). Also requires a `gcpServiceAccount` on the catalog StorageConfiguration. When enabled, credential vending chains a catalog-signed JWT through a Workload Identity Federation token exchange and service-account impersonation, so the Polaris principal appears in GCS Data Access audit logs (`serviceAccountDelegationInfo.principalSubject`) for any client. `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_ID` sets the JWT `kid` for JWKS key rotation. Attribution is keyed per-principal in the credential cache; when disabled (default), GCP vending behaviour is unchanged.

helm/polaris/templates/_helpers.tpl

@@ -255,9 +255,9 @@ Prints an environment variable for a secret key reference.
 Prints database/persistence connection environment variables.
 */}}
 {{- define "polaris.persistenceEnv" -}}
-{{- include "polaris.secretToEnv" (list .Values.persistence.relationalJdbc.secret "username" "quarkus.datasource.username") -}}
-{{- include "polaris.secretToEnv" (list .Values.persistence.relationalJdbc.secret "password" "quarkus.datasource.password") -}}
-{{- include "polaris.secretToEnv" (list .Values.persistence.relationalJdbc.secret "jdbcUrl" "quarkus.datasource.jdbc.url") -}}
+{{- include "polaris.secretToEnv" (list .Values.persistence.relationalJdbc.secret "username" (printf "quarkus.datasource.%s.username" .Values.persistence.relationalJdbc.datasource)) -}}
+{{- include "polaris.secretToEnv" (list .Values.persistence.relationalJdbc.secret "password" (printf "quarkus.datasource.%s.password" .Values.persistence.relationalJdbc.datasource)) -}}
+{{- include "polaris.secretToEnv" (list .Values.persistence.relationalJdbc.secret "jdbcUrl" (printf "quarkus.datasource.%s.jdbc.url" .Values.persistence.relationalJdbc.datasource)) -}}
 {{- include "polaris.secretToEnv" (list .Values.persistence.nosql.secret "connectionString" "quarkus.mongodb.connection-string") -}}
 {{- end -}}
 

helm/polaris/templates/configmap.yaml

@@ -51,6 +51,9 @@ data:
 
     {{- /* Persistence */ -}}
     {{- $_ = set $map "polaris.persistence.type" .Values.persistence.type -}}
+    {{- if eq .Values.persistence.type "relational-jdbc" -}}
+    {{- $_ = set $map "polaris.persistence.relational.jdbc.datasource" .Values.persistence.relationalJdbc.datasource -}}
+    {{- end -}}
     {{- if eq .Values.persistence.type "nosql" -}}
     {{- $_ = set $map "polaris.persistence.nosql.backend" .Values.persistence.nosql.backend -}}
     {{- $_ = set $map "quarkus.mongodb.database" .Values.persistence.nosql.database -}}

helm/polaris/tests/configmap_test.yaml

@@ -122,9 +122,10 @@ tests:
 
   - it: should configure relational-jdbc persistence
     set:
-      persistence: { type: "relational-jdbc", relationalJdbc: { secret: { name: "polaris-persistence" } } }
+      persistence: { type: "relational-jdbc", relationalJdbc: { datasource: "postgresql", secret: { name: "polaris-persistence" } } }
     asserts:
       - matchRegex: { path: 'data["application.properties"]', pattern: "polaris.persistence.type=relational-jdbc" }
+      - matchRegex: { path: 'data["application.properties"]', pattern: "polaris.persistence.relational.jdbc.datasource=postgresql" }
 
   - it: should configure nosql persistence with default values
     set:

helm/polaris/tests/cronjob_maintenance_test.yaml

@@ -590,23 +590,23 @@ tests:
       - contains:
           path: spec.jobTemplate.spec.template.spec.containers[0].env
           content:
-            name: quarkus.datasource.username
+            name: quarkus.datasource.postgresql.username
             valueFrom:
               secretKeyRef:
                 name: my-jdbc-secret
                 key: u-key
       - contains:
           path: spec.jobTemplate.spec.template.spec.containers[0].env
           content:
-            name: quarkus.datasource.password
+            name: quarkus.datasource.postgresql.password
             valueFrom:
               secretKeyRef:
                 name: my-jdbc-secret
                 key: p-key
       - contains:
           path: spec.jobTemplate.spec.template.spec.containers[0].env
           content:
-            name: quarkus.datasource.jdbc.url
+            name: quarkus.datasource.postgresql.jdbc.url
             valueFrom:
               secretKeyRef:
                 name: my-jdbc-secret

helm/polaris/tests/deployment_test.yaml

@@ -1470,28 +1470,58 @@ tests:
   - it: should set relational-jdbc persistence environment variables
     template: deployment.yaml
     set:
-      persistence: { type: "relational-jdbc", relationalJdbc: { secret: { name: "polaris-persistence", username: "username", password: "password", jdbcUrl: "jdbcUrl" } } }
+      persistence: { type: "relational-jdbc", relationalJdbc: { datasource: "postgresql", secret: { name: "polaris-persistence", username: "username", password: "password", jdbcUrl: "jdbcUrl" } } }
     asserts:
       - contains:
           path: spec.template.spec.containers[0].env
           content:
-            name: quarkus.datasource.username
+            name: quarkus.datasource.postgresql.username
             valueFrom:
               secretKeyRef:
                 name: polaris-persistence
                 key: username
       - contains:
           path: spec.template.spec.containers[0].env
           content:
-            name: quarkus.datasource.password
+            name: quarkus.datasource.postgresql.password
             valueFrom:
               secretKeyRef:
                 name: polaris-persistence
                 key: password
       - contains:
           path: spec.template.spec.containers[0].env
           content:
-            name: quarkus.datasource.jdbc.url
+            name: quarkus.datasource.postgresql.jdbc.url
+            valueFrom:
+              secretKeyRef:
+                name: polaris-persistence
+                key: jdbcUrl
+
+  - it: should set relational-jdbc persistence environment variables for custom datasource name
+    template: deployment.yaml
+    set:
+      persistence: { type: "relational-jdbc", relationalJdbc: { datasource: "mydb", secret: { name: "polaris-persistence", username: "username", password: "password", jdbcUrl: "jdbcUrl" } } }
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: quarkus.datasource.mydb.username
+            valueFrom:
+              secretKeyRef:
+                name: polaris-persistence
+                key: username
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: quarkus.datasource.mydb.password
+            valueFrom:
+              secretKeyRef:
+                name: polaris-persistence
+                key: password
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: quarkus.datasource.mydb.jdbc.url
             valueFrom:
               secretKeyRef:
                 name: polaris-persistence

helm/polaris/values.schema.json

@@ -1157,6 +1157,10 @@
                 "relationalJdbc": {
                     "type": "object",
                     "properties": {
+                        "datasource": {
+                            "description": "The name of the Quarkus named datasource to use for relational persistence. Must match one of the built-in named datasources (h2, postgresql), or a custom datasource configured via extraEnv / extraConfig (in this case, you must use a custom Polaris server image including the custom datasource). Polaris will activate this datasource automatically.",
+                            "type": "string"
+                        },
                         "secret": {
                             "type": "object",
                             "properties": {

helm/polaris/values.yaml

@@ -922,6 +922,12 @@ persistence:
   type: in-memory  # relational-jdbc, nosql
   # The configuration for the relational-jdbc persistence manager.
   relationalJdbc:
+    # -- The name of the Quarkus named datasource to use for relational persistence.
+    # Must match one of the built-in named datasources (h2, postgresql), or a custom datasource
+    # configured via extraEnv / extraConfig (in this case, you must use a custom Polaris server
+    # image including the custom datasource). Polaris will activate this datasource automatically.
+    # @section -- Persistence
+    datasource: postgresql
     # The secret name to pull the database connection properties from.
     secret:
       # -- The secret name to pull database connection properties from

persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java

@@ -22,8 +22,6 @@
 
 import io.smallrye.common.annotation.Identifier;
 import jakarta.enterprise.context.ApplicationScoped;
-import jakarta.enterprise.inject.Instance;
-import jakarta.enterprise.inject.Produces;
 import jakarta.inject.Inject;
 import java.sql.SQLException;
 import java.time.Clock;
@@ -32,7 +30,6 @@
 import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.ConcurrentHashMap;
-import javax.sql.DataSource;
 import org.apache.polaris.core.PolarisCallContext;
 import org.apache.polaris.core.PolarisDiagnostics;
 import org.apache.polaris.core.config.BehaviorChangeConfiguration;
@@ -93,13 +90,6 @@ public class JdbcMetaStoreManagerFactory implements MetaStoreManagerFactory {
 
   protected JdbcMetaStoreManagerFactory() {}
 
-  @Produces
-  @ApplicationScoped
-  static DatasourceOperations produceDatasourceOperations(
-      Instance<DataSource> dataSource, RelationalJdbcConfiguration relationalJdbcConfiguration) {
-    return new DatasourceOperations(dataSource.get(), relationalJdbcConfiguration);
-  }
-
   protected PrincipalSecretsGenerator secretsGenerator(
       String realmId, @Nullable RootCredentialsSet rootCredentialsSet) {
     if (rootCredentialsSet != null) {

persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/RelationalJdbcConfiguration.java

@@ -21,18 +21,21 @@
 import java.util.Optional;
 
 public interface RelationalJdbcConfiguration {
-  // max retries before giving up
+  /** The maximum number of retries before giving up the operation. */
   Optional<Integer> maxRetries();
 
-  // max retry duration
+  /** The maximum retry duration in milliseconds. */
   Optional<Long> maxDurationInMs();
 
-  // initial delay
+  /** The initial retry delay. */
   Optional<Long> initialDelayInMs();
 
   /**
    * Explicitly configured database type. If not specified, the database type will be inferred from
    * the JDBC connection metadata. Supported values: "postgresql", "cockroachdb", "h2"
    */
   Optional<String> databaseType();
+
+  /** The datasource name to use. Required. */
+  String dataSource();
 }

persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/RelationalJdbcProductionReadinessChecks.java

@@ -40,7 +40,7 @@ public ProductionReadinessCheck checkRelationalJdbc(
       return ProductionReadinessCheck.of(
           ProductionReadinessCheck.Error.of(
               "The current persistence (jdbc:h2) is intended for tests only.",
-              "quarkus.datasource.jdbc.url"));
+              "polaris.persistence.relational.jdbc.datasource"));
     }
     return ProductionReadinessCheck.OK;
   }

persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/JdbcGrantRecordsIdempotencyTest.java

@@ -154,5 +154,10 @@ public Optional<Long> initialDelayInMs() {
     public Optional<String> databaseType() {
       return Optional.of("h2");
     }
+
+    @Override
+    public String dataSource() {
+      return "h2";
+    }
   }
 }

persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/MetricsReportPersistenceTest.java

@@ -202,5 +202,10 @@ public Optional<Long> initialDelayInMs() {
     public Optional<String> databaseType() {
       return Optional.empty();
     }
+
+    @Override
+    public String dataSource() {
+      return "h2";
+    }
   }
 }

persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/RelationalJdbcProductionReadinessChecksTest.java

@@ -72,7 +72,8 @@ void jdbcWithH2ReturnsWarning() {
             error -> {
               assertThat(error.message())
                   .isEqualTo("The current persistence (jdbc:h2) is intended for tests only.");
-              assertThat(error.offendingProperty()).isEqualTo("quarkus.datasource.jdbc.url");
+              assertThat(error.offendingProperty())
+                  .isEqualTo("polaris.persistence.relational.jdbc.datasource");
               assertThat(error.severe()).isFalse();
             });
   }

persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/SimpleRelationalJdbcConfiguration.java

@@ -56,4 +56,10 @@ public Optional<Long> initialDelayInMs() {
   public Optional<String> databaseType() {
     return Optional.of(databaseType);
   }
+
+  @Override
+  public String dataSource() {
+    // The datasource name is irrelevant for tests that do not leverage CDI
+    return "datasource1";
+  }
 }

persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/idempotency/RelationalJdbcIdempotencyStorePostgresIT.java

@@ -82,6 +82,11 @@ public Optional<Long> initialDelayInMs() {
           public Optional<String> databaseType() {
             return Optional.empty();
           }
+
+          @Override
+          public String dataSource() {
+            return "postgresql";
+          }
         };
     DatasourceOperations ops = new DatasourceOperations(dataSource, cfg);
     try (InputStream is =

runtime/admin/build.gradle.kts

@@ -33,9 +33,13 @@ dependencies {
 
   compileOnly("com.fasterxml.jackson.core:jackson-annotations")
 
+  implementation(enforcedPlatform(libs.quarkus.bom))
+
+  // JDBC persistence + backends
   runtimeOnly(project(":polaris-relational-jdbc"))
-  runtimeOnly("org.postgresql:postgresql")
+  runtimeOnly("io.quarkus:quarkus-jdbc-postgresql")
 
+  // NoSQL persistence + backends
   implementation(project(":polaris-persistence-nosql-api"))
   implementation(project(":polaris-persistence-nosql-maintenance-api"))
   runtimeOnly(project(":polaris-persistence-nosql-metastore"))
@@ -46,8 +50,6 @@ dependencies {
 
   runtimeOnly("io.quarkus:quarkus-mongodb-client")
 
-  implementation("io.quarkus:quarkus-jdbc-postgresql")
-  implementation(enforcedPlatform(libs.quarkus.bom))
   implementation("io.quarkus:quarkus-picocli")
   implementation("io.quarkus:quarkus-container-image-docker")
 

runtime/admin/src/main/resources/application.properties

@@ -31,7 +31,12 @@ quarkus.container-image.registry=docker.io
 quarkus.container-image.group=apache
 quarkus.container-image.name=polaris-admin-tool
 quarkus.container-image.additional-tags=latest
-quarkus.datasource.db-kind=postgresql
+
+# Named datasources for relational persistence backends.
+# Polaris ships with many built-in datasources; by using named datasources, users can configure
+# at runtime which datasources to use for relational persistence backends.
+quarkus.datasource.postgresql.db-kind=postgresql
+
 # if set to true it will try to start localstack at build and run time for the local environment
 # https://docs.quarkiverse.io/quarkus-amazon-services/dev/amazon-rds.html#_configuration_reference for more details
 quarkus.rds.devservices.enabled=false
@@ -41,12 +46,13 @@ quarkus.mongodb.devservices.enabled=false
 # ---- Runtime Configuration ----
 # Below are default values for properties that can be changed in runtime.
 
-# Available types:
+# Polaris persistence type. Available types:
 # - in-memory - InMemoryPolarisMetaStoreManagerFactory
 # - in-memory-atomic - InMemoryAtomicOperationMetaStoreManagerFactory
-# - relational-jdbc - JdbcMetaStoreManagerFactory
-# - nosql - NoSQL persistence backend, define the backend type via 'polaris.persistence.nosql.backend'
+# - nosql (beta) - NoSQL persistence backend, define the backend type via 'polaris.persistence.nosql.backend'
+# - relational-jdbc - JDBC persistence backend, define the JDBC datasource via 'polaris.persistence.relational.jdbc.datasource' and 'quarkus.datasource.<datasource-name>.*'
 polaris.persistence.type=relational-jdbc
+
 # Database backend for 'nosql' persistence-type
 # Available backends:
 # - InMemory - for testing purposes
@@ -55,6 +61,21 @@ polaris.persistence.type=relational-jdbc
 #     See https://quarkus.io/guides/mongodb#configuration-reference for details about these configurations.
 #polaris.persistence.nosql.backend=InMemory
 
+# Datasource to activate for the 'relational-jdbc' persistence type.
+# Built-in supported datasources:
+# - postgresql - activates the PostgreSQL datasource (default)
+#     Configure the necessary PostgreSQL properties starting with 'quarkus.datasource.postgresql.' in this file (see below).
+# See https://quarkus.io/guides/datasource#configuration-reference for details about these configurations.
+# Please do NOT set quarkus.datasource.active manually, Polaris will set it automatically based on
+# this property.
+polaris.persistence.relational.jdbc.datasource=postgresql
+
+# Postgres datasource configuration:
+quarkus.datasource.postgresql.devservices.enabled=false
+#quarkus.datasource.postgresql.jdbc.url=jdbc:postgresql://localhost:5432/my_database
+#quarkus.datasource.postgresql.username=<your username>
+#quarkus.datasource.postgresql.password=<your password>
+
 ## MongoDB version store specific configuration
 #quarkus.mongodb.database=polaris
 #quarkus.mongodb.metrics.enabled=true