(feat) Implement metrics rest api

CHANGELOG.md

@@ -42,8 +42,11 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
     - Names containing control (invisible) characters
     - Names with leading or trailing whitespace
     - Names containing any of these characters: <code>/\:*?"<>|#+`</code>
+- The `PolarisMetricsReporter` SPI (previously in `runtime/service`) has been replaced by `IcebergMetricsReporter` in `extensions/metrics-reports/spi`. Custom reporters must implement `org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter` and update the `@Identifier` annotation for CDI selection via `polaris.iceberg-metrics.reporting.type`. The method signature now includes a `receivedTimestamp` parameter of type `java.time.Instant`.
+- Metrics reporting now requires the `TABLE_READ_DATA` privilege on the target table for read (scan) metrics and `TABLE_WRITE_DATA` for write (commit) metrics.
 
 ### New Features
+- Added an **experimental (beta)** Table Metrics REST API (`/api/metrics-reports/v1/`) for querying Iceberg scan and commit metrics. **This API is a proof-of-concept; it is subject to breaking changes in any future release without prior notice and should not be used in production.** In this release the read path returns HTTP 501 until durable query backing is installed in a follow-up extension. Querying requires the new `TABLE_READ_METRICS` privilege on the target table.
 - Added GCS principal attribution for vended credentials (the GCP counterpart of AWS STS session tags). Set `GCS_PRINCIPAL_ATTRIBUTION_ENABLED=true` to activate; the feature flags `GCS_PRINCIPAL_ATTRIBUTION_WIF_AUDIENCE`, `GCS_PRINCIPAL_ATTRIBUTION_TOKEN_ISSUER`, and `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_FILE` are then required (a missing value is a fatal configuration error). Also requires a `gcpServiceAccount` on the catalog StorageConfiguration. When enabled, credential vending chains a catalog-signed JWT through a Workload Identity Federation token exchange and service-account impersonation, so the Polaris principal appears in GCS Data Access audit logs (`serviceAccountDelegationInfo.principalSubject`) for any client. `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_ID` sets the JWT `kid` for JWKS key rotation. Attribution is keyed per-principal in the credential cache; when disabled (default), GCP vending behaviour is unchanged.
 - Added `SESSION_NAME_FIELDS_IN_SUBSCOPED_CREDENTIAL` feature flag for AWS credential vending. Operators can now configure an ordered list of fields (`realm`, `catalog`, `namespace`, `table`, `principal`) to compose structured STS role session names (e.g. `p-acme-hr_catalog-employee-etl_writer`). Session names are sanitized and proportionally truncated to the AWS 64-character limit. When unset, existing `INCLUDE_PRINCIPAL_NAME_IN_SUBSCOPED_CREDENTIAL` behaviour is preserved.
 - Added `hostUsers` support in Helm chart.
@@ -121,9 +124,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 
 - The (Before/After)CommitViewEvent has been removed.
 - The (Before/After)CommitTableEvent has been removed.
-- The `PolarisMetricsReporter.reportMetric()` method signature has been extended to include a `receivedTimestamp` parameter of type `java.time.Instant`.
 - The `ExternalCatalogFactory.createCatalog()` and `createGenericCatalog()` method signatures have been extended to include a `catalogProperties` parameter of type `Map<String, String>` for passing through proxy and timeout settings to federated catalog HTTP clients.
-- Metrics reporting now requires the `TABLE_READ_DATA` privilege on the target table for read (scan) metrics and `TABLE_WRITE_DATA` for write (commit) metrics.
 - The `REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE` operation no longer requires the `PRINCIPAL_ROLE_MANAGE_GRANTS_FOR_GRANTEE` privilege. Only `CATALOG_ROLE_MANAGE_GRANTS_ON_SECURABLE` on the catalog role is now required, making revoke symmetric with assign. This allows catalog administrators to fully manage catalog role assignments without requiring elevated privileges on principal roles.
 
 ### New Features
@@ -141,7 +142,6 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 - Relaxed `client_id`, `client_secret` regex/pattern validation on reset endpoint call
 - Added support for S3-compatible storage that does not have KMS (use `kmsUavailable: true` in catalog storage configuration)
 - Added support for storage-scoped AWS credentials, allowing different AWS credentials to be configured per named storage. Enable with the `RESOLVE_CREDENTIALS_BY_STORAGE_NAME` feature flag (default: false). Storage names can be set explicitly via the `storageName` field on storage configuration, or inferred from the first allowed location's host.
-- Added support for persisting Iceberg metrics (ScanReport, CommitReport) to the database. Enable by setting `polaris.iceberg-metrics.reporting.type=persisting` in configuration. Metrics tables are included in the main JDBC schema.
 - Added setup options to Polaris CLI.
 - Added CockroachDB as a supported database for the relational JDBC persistence backend. Set `polaris.persistence.relational.jdbc.database-type` to `cockroachdb` to get started.
 

api/metrics-reports-service/build.gradle.kts

@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
+
+plugins {
+  alias(libs.plugins.openapi.generator)
+  id("polaris-client")
+  id("org.kordamp.gradle.jandex")
+}
+
+dependencies {
+  implementation(project(":polaris-core"))
+
+  compileOnly(platform(libs.jackson.bom))
+  compileOnly("com.fasterxml.jackson.core:jackson-annotations")
+
+  compileOnly(libs.jakarta.annotation.api)
+  compileOnly(libs.jakarta.inject.api)
+  compileOnly(libs.jakarta.validation.api)
+  compileOnly(libs.microprofile.fault.tolerance.api)
+  compileOnly(libs.swagger.annotations)
+
+  implementation(libs.jakarta.servlet.api)
+  implementation(libs.jakarta.ws.rs.api)
+
+  compileOnly(platform(libs.micrometer.bom))
+  compileOnly("io.micrometer:micrometer-core")
+
+  implementation(libs.slf4j.api)
+}
+
+val rootDir = rootProject.layout.projectDirectory
+val specsDir = rootDir.dir("spec")
+val templatesDir = rootDir.dir("server-templates")
+val generatedDir = project.layout.buildDirectory.dir("generated-openapi")
+val generatedOpenApiSrcDir = project.layout.buildDirectory.dir("generated-openapi/src/main/java")
+
+openApiGenerate {
+  inputSpec = provider { specsDir.file("metrics-reports-service.yml").asFile.absolutePath }
+  generatorName = "jaxrs-resteasy"
+  outputDir = provider { generatedDir.get().asFile.absolutePath }
+  apiPackage = "org.apache.polaris.service.metrics.api"
+  modelPackage = "org.apache.polaris.core.metrics.api.model"
+  ignoreFileOverride.set(provider { rootDir.file(".openapi-generator-ignore").asFile.absolutePath })
+  removeOperationIdPrefix.set(true)
+  templateDir.set(provider { templatesDir.asFile.absolutePath })
+  globalProperties.put("apis", "")
+  globalProperties.put("models", "")
+  globalProperties.put("apiDocs", "false")
+  globalProperties.put("modelTests", "false")
+  configOptions.put("openApiNullable", "false")
+  configOptions.put("useBeanValidation", "true")
+  configOptions.put("sourceFolder", "src/main/java")
+  configOptions.put("useJakartaEe", "true")
+  configOptions.put("generateBuilders", "true")
+  configOptions.put("generateConstructorWithAllArgs", "true")
+  configOptions.put("hideGenerationTimestamp", "true")
+  additionalProperties.put("apiNamePrefix", "Polaris")
+  additionalProperties.put("apiNameSuffix", "Api")
+  additionalProperties.put("metricsPrefix", "polaris")
+  serverVariables.put("basePath", "api/metrics-reports/v1")
+}
+
+listOf("sourcesJar", "compileJava", "processResources").forEach { task ->
+  tasks.named(task) { dependsOn("openApiGenerate") }
+}
+
+sourceSets { main { java { srcDir(generatedOpenApiSrcDir) } } }
+
+tasks.named<GenerateTask>("openApiGenerate") {
+  inputs.dir(templatesDir)
+  inputs.dir(specsDir)
+  actions.addFirst { delete { delete(generatedDir) } }
+}
+
+tasks.named("javadoc") { dependsOn("jandex") }

bom/build.gradle.kts

@@ -30,6 +30,7 @@ dependencies {
     api(project(":polaris-api-iceberg-service"))
     api(project(":polaris-api-management-model"))
     api(project(":polaris-api-management-service"))
+    api(project(":polaris-api-metrics-reports-service"))
 
     api(project(":polaris-container-spec-helper"))
     api(project(":polaris-azurite-testcontainer"))
@@ -99,6 +100,7 @@ dependencies {
     api(project(":polaris-extensions-federation-bigquery"))
     api(project(":polaris-extensions-federation-hadoop"))
     api(project(":polaris-extensions-federation-hive"))
+    api(project(":polaris-extensions-metrics-reports-spi"))
     api(project(":polaris-hms-testcontainer"))
 
     api(project(":polaris-spark-3.5_2.12"))
@@ -107,6 +109,7 @@ dependencies {
       api(project(":polaris-spark-3.5_2.13"))
       api(project(":polaris-spark-4.0_2.13"))
     }
+    api(project(":polaris-extensions-metrics-reports"))
 
     api(project(":polaris-admin"))
     api(project(":polaris-runtime-common"))

extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json

@@ -307,11 +307,12 @@
       { "itemId": 28, "name": "table-create",     "label": "Table Create",     "category": "CREATE", "impliedGrants": [ "table-list" ] },
       { "itemId": 29, "name": "table-drop",       "label": "Table Drop",       "category": "DELETE" },
       { "itemId": 30, "name": "table-list",       "label": "Table List",       "category": "READ" },
-      { "itemId": 31, "name": "table-data-read",  "label": "Table Data Read",  "category": "READ",   "impliedGrants": [ "table-list", "table-properties-read" ] },
+      { "itemId": 31, "name": "table-data-read",  "label": "Table Data Read",  "category": "READ",   "impliedGrants": [ "table-list", "table-properties-read", "table-metrics-read" ] },
       { "itemId": 32, "name": "table-data-write", "label": "Table Data Write", "category": "UPDATE",
         "impliedGrants": [
           "table-list",
           "table-data-read",
+          "table-metrics-read",
           "table-properties-read",
           "table-properties-set",
           "table-properties-remove",
@@ -338,6 +339,7 @@
           "table-create",
           "table-drop",
           "table-list",
+          "table-metrics-read",
           "table-properties-read",
           "table-properties-write",
           "table-properties-set",
@@ -423,6 +425,7 @@
           "table-statistics-remove"
         ]
       },
+      { "itemId": 70, "name": "table-metrics-read",           "label": "Table Metrics Read",           "category": "READ", "impliedGrants": [ "table-list" ] },
 
       { "itemId": 56,  "name": "view-create",           "label": "View Create",           "category": "CREATE", "impliedGrants": [ "view-list" ] },
       { "itemId": 57,  "name": "view-drop",             "label": "View Drop",             "category": "DELETE" },

extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java

@@ -86,6 +86,7 @@ enum ResolvedPathRooting {
   private static final String TABLE_WRITE_PROPERTIES = "table-properties-write";
   private static final String TABLE_READ_DATA = "table-data-read";
   private static final String TABLE_WRITE_DATA = "table-data-write";
+  private static final String TABLE_READ_METRICS = "table-metrics-read";
   private static final String TABLE_ATTACH_POLICY = "table-policy-attach";
   private static final String TABLE_DETACH_POLICY = "table-policy-detach";
   private static final String TABLE_ASSIGN_UUID = "table-uuid-assign";
@@ -254,6 +255,10 @@ enum ResolvedPathRooting {
         PolarisAuthorizableOperation.REPORT_READ_METRICS,
         new RangerPolarisOperationSemantics(
             toSet(TABLE_READ_DATA), null, ResolvedPathRooting.ROOT));
+    RBAC_SEMANTICS_BY_OPERATION.put(
+        PolarisAuthorizableOperation.LIST_TABLE_METRICS,
+        new RangerPolarisOperationSemantics(
+            toSet(TABLE_READ_METRICS), null, ResolvedPathRooting.ROOT));
     RBAC_SEMANTICS_BY_OPERATION.put(
         PolarisAuthorizableOperation.REPORT_WRITE_METRICS,
         new RangerPolarisOperationSemantics(

extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json

@@ -307,11 +307,12 @@
       { "itemId": 28, "name": "table-create",     "label": "Table Create",     "category": "CREATE", "impliedGrants": [ "table-list" ] },
       { "itemId": 29, "name": "table-drop",       "label": "Table Drop",       "category": "DELETE" },
       { "itemId": 30, "name": "table-list",       "label": "Table List",       "category": "READ" },
-      { "itemId": 31, "name": "table-data-read",  "label": "Table Data Read",  "category": "READ",   "impliedGrants": [ "table-list", "table-properties-read" ] },
+      { "itemId": 31, "name": "table-data-read",  "label": "Table Data Read",  "category": "READ",   "impliedGrants": [ "table-list", "table-properties-read", "table-metrics-read" ] },
       { "itemId": 32, "name": "table-data-write", "label": "Table Data Write", "category": "UPDATE",
         "impliedGrants": [
           "table-list",
           "table-data-read",
+          "table-metrics-read",
           "table-properties-read",
           "table-properties-set",
           "table-properties-remove",
@@ -338,6 +339,7 @@
           "table-create",
           "table-drop",
           "table-list",
+          "table-metrics-read",
           "table-properties-read",
           "table-properties-write",
           "table-properties-set",
@@ -423,6 +425,7 @@
           "table-statistics-remove"
         ]
       },
+      { "itemId": 70, "name": "table-metrics-read",           "label": "Table Metrics Read",           "category": "READ", "impliedGrants": [ "table-list" ] },
 
       { "itemId": 56,  "name": "view-create",           "label": "View Create",           "category": "CREATE", "impliedGrants": [ "view-list" ] },
       { "itemId": 57,  "name": "view-drop",             "label": "View Drop",             "category": "DELETE" },

extensions/metrics-reports/base/build.gradle.kts

@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+plugins {
+  id("polaris-server")
+  id("org.kordamp.gradle.jandex")
+}
+
+dependencies {
+  implementation(project(":polaris-core"))
+  implementation(project(":polaris-api-metrics-reports-service"))
+  implementation(project(":polaris-extensions-metrics-reports-spi"))
+
+  implementation(platform(libs.iceberg.bom))
+  implementation("org.apache.iceberg:iceberg-api")
+
+  implementation(libs.jakarta.enterprise.cdi.api)
+  implementation(libs.jakarta.inject.api)
+  implementation(libs.jakarta.ws.rs.api)
+  implementation(libs.guava)
+  implementation(libs.smallrye.common.annotation)
+  implementation(libs.slf4j.api)
+
+  compileOnly(libs.jspecify)
+
+  testImplementation(platform(libs.junit.bom))
+  testImplementation("org.junit.jupiter:junit-jupiter")
+  testImplementation(libs.assertj.core)
+  testImplementation(libs.mockito.core)
+
+  // Provides jakarta.ws.rs.ext.RuntimeDelegate needed to build Response objects in plain unit tests
+  testRuntimeOnly(enforcedPlatform(libs.quarkus.bom))
+  testRuntimeOnly("io.quarkus.resteasy.reactive:resteasy-reactive")
+}

runtime/service/src/main/java/org/apache/polaris/service/reporting/DefaultMetricsReporter.java → extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporter.java

@@ -16,37 +16,30 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.apache.polaris.service.reporting;
+package org.apache.polaris.extension.metrics.reports;
 
-import com.google.common.annotations.VisibleForTesting;
 import io.smallrye.common.annotation.Identifier;
 import jakarta.enterprise.context.ApplicationScoped;
 import java.time.Instant;
 import org.apache.iceberg.catalog.TableIdentifier;
 import org.apache.iceberg.metrics.MetricsReport;
+import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * Default implementation of {@link PolarisMetricsReporter} that logs metrics to the configured
- * logger.
+ * Log-only implementation of {@link IcebergMetricsReporter}.
  *
- * <p>This implementation is selected when {@code polaris.iceberg-metrics.reporting.type} is set to
- * {@code "default"} (the default value).
+ * <p>Selected when {@code polaris.iceberg-metrics.reporting.type} is set to {@code "log"}.
  *
- * <p>By default, logging is disabled. To enable metrics logging, set the logger level for {@code
- * org.apache.polaris.service.reporting} to {@code INFO} in your logging configuration.
- *
- * @see PolarisMetricsReporter
+ * <p>Logging is at INFO level. Enable it by setting the logger level for {@code
+ * org.apache.polaris.extension.metrics.reports} to {@code INFO}.
  */
 @ApplicationScoped
-@Identifier("default")
-public class DefaultMetricsReporter implements PolarisMetricsReporter {
-  private static final Logger LOGGER = LoggerFactory.getLogger(DefaultMetricsReporter.class);
-
-  private final ReportConsumer reportConsumer;
+@Identifier("log")
+public class LoggingMetricsReporter implements IcebergMetricsReporter {
+  private static final Logger LOGGER = LoggerFactory.getLogger(LoggingMetricsReporter.class);
 
-  /** Functional interface for consuming metrics reports with timestamp. */
   @FunctionalInterface
   interface ReportConsumer {
     void accept(
@@ -58,15 +51,15 @@ void accept(
         Instant receivedTimestamp);
   }
 
-  /** Creates a new DefaultMetricsReporter that logs metrics to the class logger. */
-  public DefaultMetricsReporter() {
+  private final ReportConsumer reportConsumer;
+
+  public LoggingMetricsReporter() {
     this(
         (catalogName, catalogId, table, tableId, metricsReport, receivedTimestamp) ->
             LOGGER.info("{}.{} (ts={}): {}", catalogName, table, receivedTimestamp, metricsReport));
   }
 
-  @VisibleForTesting
-  DefaultMetricsReporter(ReportConsumer reportConsumer) {
+  LoggingMetricsReporter(ReportConsumer reportConsumer) {
     this.reportConsumer = reportConsumer;
   }
 
@@ -80,4 +73,9 @@ public void reportMetric(
       Instant receivedTimestamp) {
     reportConsumer.accept(catalogName, catalogId, table, tableId, metricsReport, receivedTimestamp);
   }
+
+  /** Backward-compatible alias for {@code "log"}; redirects the legacy {@code "default"} type. */
+  @ApplicationScoped
+  @Identifier("default")
+  public static class DefaultMetricsReporter extends LoggingMetricsReporter {}
 }