diff --git a/CHANGELOG.md b/CHANGELOG.md index 67c7898fedd..3b97755b5b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,8 +42,11 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti - Names containing control (invisible) characters - Names with leading or trailing whitespace - Names containing any of these characters: /\:*?"<>|#+` +- The `PolarisMetricsReporter` SPI (previously in `runtime/service`) has been replaced by `IcebergMetricsReporter` in `extensions/metrics-reports/spi`. Custom reporters must implement `org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter` and update the `@Identifier` annotation for CDI selection via `polaris.iceberg-metrics.reporting.type`. The method signature now includes a `receivedTimestamp` parameter of type `java.time.Instant`. +- Metrics reporting now requires the `TABLE_READ_DATA` privilege on the target table for read (scan) metrics and `TABLE_WRITE_DATA` for write (commit) metrics. ### New Features +- Added an **experimental (beta)** Table Metrics REST API (`/api/metrics-reports/v1/`) for querying Iceberg scan and commit metrics. **This API is a proof-of-concept; it is subject to breaking changes in any future release without prior notice and should not be used in production.** In this release the read path returns HTTP 501 until durable query backing is installed in a follow-up extension. Querying requires the new `TABLE_READ_METRICS` privilege on the target table. - Added GCS principal attribution for vended credentials (the GCP counterpart of AWS STS session tags). Set `GCS_PRINCIPAL_ATTRIBUTION_ENABLED=true` to activate; the feature flags `GCS_PRINCIPAL_ATTRIBUTION_WIF_AUDIENCE`, `GCS_PRINCIPAL_ATTRIBUTION_TOKEN_ISSUER`, and `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_FILE` are then required (a missing value is a fatal configuration error). Also requires a `gcpServiceAccount` on the catalog StorageConfiguration. When enabled, credential vending chains a catalog-signed JWT through a Workload Identity Federation token exchange and service-account impersonation, so the Polaris principal appears in GCS Data Access audit logs (`serviceAccountDelegationInfo.principalSubject`) for any client. `GCS_PRINCIPAL_ATTRIBUTION_SIGNING_KEY_ID` sets the JWT `kid` for JWKS key rotation. Attribution is keyed per-principal in the credential cache; when disabled (default), GCP vending behaviour is unchanged. - Added `SESSION_NAME_FIELDS_IN_SUBSCOPED_CREDENTIAL` feature flag for AWS credential vending. Operators can now configure an ordered list of fields (`realm`, `catalog`, `namespace`, `table`, `principal`) to compose structured STS role session names (e.g. `p-acme-hr_catalog-employee-etl_writer`). Session names are sanitized and proportionally truncated to the AWS 64-character limit. When unset, existing `INCLUDE_PRINCIPAL_NAME_IN_SUBSCOPED_CREDENTIAL` behaviour is preserved. - Added `hostUsers` support in Helm chart. @@ -121,9 +124,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti - The (Before/After)CommitViewEvent has been removed. - The (Before/After)CommitTableEvent has been removed. -- The `PolarisMetricsReporter.reportMetric()` method signature has been extended to include a `receivedTimestamp` parameter of type `java.time.Instant`. - The `ExternalCatalogFactory.createCatalog()` and `createGenericCatalog()` method signatures have been extended to include a `catalogProperties` parameter of type `Map` for passing through proxy and timeout settings to federated catalog HTTP clients. -- Metrics reporting now requires the `TABLE_READ_DATA` privilege on the target table for read (scan) metrics and `TABLE_WRITE_DATA` for write (commit) metrics. - The `REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE` operation no longer requires the `PRINCIPAL_ROLE_MANAGE_GRANTS_FOR_GRANTEE` privilege. Only `CATALOG_ROLE_MANAGE_GRANTS_ON_SECURABLE` on the catalog role is now required, making revoke symmetric with assign. This allows catalog administrators to fully manage catalog role assignments without requiring elevated privileges on principal roles. ### New Features @@ -141,7 +142,6 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti - Relaxed `client_id`, `client_secret` regex/pattern validation on reset endpoint call - Added support for S3-compatible storage that does not have KMS (use `kmsUavailable: true` in catalog storage configuration) - Added support for storage-scoped AWS credentials, allowing different AWS credentials to be configured per named storage. Enable with the `RESOLVE_CREDENTIALS_BY_STORAGE_NAME` feature flag (default: false). Storage names can be set explicitly via the `storageName` field on storage configuration, or inferred from the first allowed location's host. -- Added support for persisting Iceberg metrics (ScanReport, CommitReport) to the database. Enable by setting `polaris.iceberg-metrics.reporting.type=persisting` in configuration. Metrics tables are included in the main JDBC schema. - Added setup options to Polaris CLI. - Added CockroachDB as a supported database for the relational JDBC persistence backend. Set `polaris.persistence.relational.jdbc.database-type` to `cockroachdb` to get started. diff --git a/api/metrics-reports-service/build.gradle.kts b/api/metrics-reports-service/build.gradle.kts new file mode 100644 index 00000000000..ce80aceb238 --- /dev/null +++ b/api/metrics-reports-service/build.gradle.kts @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.openapitools.generator.gradle.plugin.tasks.GenerateTask + +plugins { + alias(libs.plugins.openapi.generator) + id("polaris-client") + id("org.kordamp.gradle.jandex") +} + +dependencies { + implementation(project(":polaris-core")) + + compileOnly(platform(libs.jackson.bom)) + compileOnly("com.fasterxml.jackson.core:jackson-annotations") + + compileOnly(libs.jakarta.annotation.api) + compileOnly(libs.jakarta.inject.api) + compileOnly(libs.jakarta.validation.api) + compileOnly(libs.microprofile.fault.tolerance.api) + compileOnly(libs.swagger.annotations) + + implementation(libs.jakarta.servlet.api) + implementation(libs.jakarta.ws.rs.api) + + compileOnly(platform(libs.micrometer.bom)) + compileOnly("io.micrometer:micrometer-core") + + implementation(libs.slf4j.api) +} + +val rootDir = rootProject.layout.projectDirectory +val specsDir = rootDir.dir("spec") +val templatesDir = rootDir.dir("server-templates") +val generatedDir = project.layout.buildDirectory.dir("generated-openapi") +val generatedOpenApiSrcDir = project.layout.buildDirectory.dir("generated-openapi/src/main/java") + +openApiGenerate { + inputSpec = provider { specsDir.file("metrics-reports-service.yml").asFile.absolutePath } + generatorName = "jaxrs-resteasy" + outputDir = provider { generatedDir.get().asFile.absolutePath } + apiPackage = "org.apache.polaris.service.metrics.api" + modelPackage = "org.apache.polaris.core.metrics.api.model" + ignoreFileOverride.set(provider { rootDir.file(".openapi-generator-ignore").asFile.absolutePath }) + removeOperationIdPrefix.set(true) + templateDir.set(provider { templatesDir.asFile.absolutePath }) + globalProperties.put("apis", "") + globalProperties.put("models", "") + globalProperties.put("apiDocs", "false") + globalProperties.put("modelTests", "false") + configOptions.put("openApiNullable", "false") + configOptions.put("useBeanValidation", "true") + configOptions.put("sourceFolder", "src/main/java") + configOptions.put("useJakartaEe", "true") + configOptions.put("generateBuilders", "true") + configOptions.put("generateConstructorWithAllArgs", "true") + configOptions.put("hideGenerationTimestamp", "true") + additionalProperties.put("apiNamePrefix", "Polaris") + additionalProperties.put("apiNameSuffix", "Api") + additionalProperties.put("metricsPrefix", "polaris") + serverVariables.put("basePath", "api/metrics-reports/v1") +} + +listOf("sourcesJar", "compileJava", "processResources").forEach { task -> + tasks.named(task) { dependsOn("openApiGenerate") } +} + +sourceSets { main { java { srcDir(generatedOpenApiSrcDir) } } } + +tasks.named("openApiGenerate") { + inputs.dir(templatesDir) + inputs.dir(specsDir) + actions.addFirst { delete { delete(generatedDir) } } +} + +tasks.named("javadoc") { dependsOn("jandex") } diff --git a/bom/build.gradle.kts b/bom/build.gradle.kts index 11705713b38..857c23bc4bc 100644 --- a/bom/build.gradle.kts +++ b/bom/build.gradle.kts @@ -30,6 +30,7 @@ dependencies { api(project(":polaris-api-iceberg-service")) api(project(":polaris-api-management-model")) api(project(":polaris-api-management-service")) + api(project(":polaris-api-metrics-reports-service")) api(project(":polaris-container-spec-helper")) api(project(":polaris-azurite-testcontainer")) @@ -99,6 +100,7 @@ dependencies { api(project(":polaris-extensions-federation-bigquery")) api(project(":polaris-extensions-federation-hadoop")) api(project(":polaris-extensions-federation-hive")) + api(project(":polaris-extensions-metrics-reports-spi")) api(project(":polaris-hms-testcontainer")) api(project(":polaris-spark-3.5_2.12")) @@ -107,6 +109,7 @@ dependencies { api(project(":polaris-spark-3.5_2.13")) api(project(":polaris-spark-4.0_2.13")) } + api(project(":polaris-extensions-metrics-reports")) api(project(":polaris-admin")) api(project(":polaris-runtime-common")) diff --git a/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json b/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json index 701746f589b..057c5317b80 100644 --- a/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json +++ b/extensions/auth/ranger/src/intTest/resources/authz_it_tests/dev_polaris.json @@ -307,11 +307,12 @@ { "itemId": 28, "name": "table-create", "label": "Table Create", "category": "CREATE", "impliedGrants": [ "table-list" ] }, { "itemId": 29, "name": "table-drop", "label": "Table Drop", "category": "DELETE" }, { "itemId": 30, "name": "table-list", "label": "Table List", "category": "READ" }, - { "itemId": 31, "name": "table-data-read", "label": "Table Data Read", "category": "READ", "impliedGrants": [ "table-list", "table-properties-read" ] }, + { "itemId": 31, "name": "table-data-read", "label": "Table Data Read", "category": "READ", "impliedGrants": [ "table-list", "table-properties-read", "table-metrics-read" ] }, { "itemId": 32, "name": "table-data-write", "label": "Table Data Write", "category": "UPDATE", "impliedGrants": [ "table-list", "table-data-read", + "table-metrics-read", "table-properties-read", "table-properties-set", "table-properties-remove", @@ -338,6 +339,7 @@ "table-create", "table-drop", "table-list", + "table-metrics-read", "table-properties-read", "table-properties-write", "table-properties-set", @@ -423,6 +425,7 @@ "table-statistics-remove" ] }, + { "itemId": 70, "name": "table-metrics-read", "label": "Table Metrics Read", "category": "READ", "impliedGrants": [ "table-list" ] }, { "itemId": 56, "name": "view-create", "label": "View Create", "category": "CREATE", "impliedGrants": [ "view-list" ] }, { "itemId": 57, "name": "view-drop", "label": "View Drop", "category": "DELETE" }, diff --git a/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java b/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java index f8436d9024b..56eded02324 100644 --- a/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java +++ b/extensions/auth/ranger/src/main/java/org/apache/polaris/extension/auth/ranger/RangerPolarisOperationSemantics.java @@ -86,6 +86,7 @@ enum ResolvedPathRooting { private static final String TABLE_WRITE_PROPERTIES = "table-properties-write"; private static final String TABLE_READ_DATA = "table-data-read"; private static final String TABLE_WRITE_DATA = "table-data-write"; + private static final String TABLE_READ_METRICS = "table-metrics-read"; private static final String TABLE_ATTACH_POLICY = "table-policy-attach"; private static final String TABLE_DETACH_POLICY = "table-policy-detach"; private static final String TABLE_ASSIGN_UUID = "table-uuid-assign"; @@ -254,6 +255,10 @@ enum ResolvedPathRooting { PolarisAuthorizableOperation.REPORT_READ_METRICS, new RangerPolarisOperationSemantics( toSet(TABLE_READ_DATA), null, ResolvedPathRooting.ROOT)); + RBAC_SEMANTICS_BY_OPERATION.put( + PolarisAuthorizableOperation.LIST_TABLE_METRICS, + new RangerPolarisOperationSemantics( + toSet(TABLE_READ_METRICS), null, ResolvedPathRooting.ROOT)); RBAC_SEMANTICS_BY_OPERATION.put( PolarisAuthorizableOperation.REPORT_WRITE_METRICS, new RangerPolarisOperationSemantics( diff --git a/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json b/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json index f55a3286595..a7b255fdaf2 100644 --- a/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json +++ b/extensions/auth/ranger/src/test/resources/authz_tests/dev_polaris.json @@ -307,11 +307,12 @@ { "itemId": 28, "name": "table-create", "label": "Table Create", "category": "CREATE", "impliedGrants": [ "table-list" ] }, { "itemId": 29, "name": "table-drop", "label": "Table Drop", "category": "DELETE" }, { "itemId": 30, "name": "table-list", "label": "Table List", "category": "READ" }, - { "itemId": 31, "name": "table-data-read", "label": "Table Data Read", "category": "READ", "impliedGrants": [ "table-list", "table-properties-read" ] }, + { "itemId": 31, "name": "table-data-read", "label": "Table Data Read", "category": "READ", "impliedGrants": [ "table-list", "table-properties-read", "table-metrics-read" ] }, { "itemId": 32, "name": "table-data-write", "label": "Table Data Write", "category": "UPDATE", "impliedGrants": [ "table-list", "table-data-read", + "table-metrics-read", "table-properties-read", "table-properties-set", "table-properties-remove", @@ -338,6 +339,7 @@ "table-create", "table-drop", "table-list", + "table-metrics-read", "table-properties-read", "table-properties-write", "table-properties-set", @@ -423,6 +425,7 @@ "table-statistics-remove" ] }, + { "itemId": 70, "name": "table-metrics-read", "label": "Table Metrics Read", "category": "READ", "impliedGrants": [ "table-list" ] }, { "itemId": 56, "name": "view-create", "label": "View Create", "category": "CREATE", "impliedGrants": [ "view-list" ] }, { "itemId": 57, "name": "view-drop", "label": "View Drop", "category": "DELETE" }, diff --git a/extensions/metrics-reports/base/build.gradle.kts b/extensions/metrics-reports/base/build.gradle.kts new file mode 100644 index 00000000000..906697f7175 --- /dev/null +++ b/extensions/metrics-reports/base/build.gradle.kts @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +plugins { + id("polaris-server") + id("org.kordamp.gradle.jandex") +} + +dependencies { + implementation(project(":polaris-core")) + implementation(project(":polaris-api-metrics-reports-service")) + implementation(project(":polaris-extensions-metrics-reports-spi")) + + implementation(platform(libs.iceberg.bom)) + implementation("org.apache.iceberg:iceberg-api") + + implementation(libs.jakarta.enterprise.cdi.api) + implementation(libs.jakarta.inject.api) + implementation(libs.jakarta.ws.rs.api) + implementation(libs.guava) + implementation(libs.smallrye.common.annotation) + implementation(libs.slf4j.api) + + compileOnly(libs.jspecify) + + testImplementation(platform(libs.junit.bom)) + testImplementation("org.junit.jupiter:junit-jupiter") + testImplementation(libs.assertj.core) + testImplementation(libs.mockito.core) + + // Provides jakarta.ws.rs.ext.RuntimeDelegate needed to build Response objects in plain unit tests + testRuntimeOnly(enforcedPlatform(libs.quarkus.bom)) + testRuntimeOnly("io.quarkus.resteasy.reactive:resteasy-reactive") +} diff --git a/runtime/service/src/main/java/org/apache/polaris/service/reporting/DefaultMetricsReporter.java b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporter.java similarity index 66% rename from runtime/service/src/main/java/org/apache/polaris/service/reporting/DefaultMetricsReporter.java rename to extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporter.java index e58f740c3c0..f1d699226db 100644 --- a/runtime/service/src/main/java/org/apache/polaris/service/reporting/DefaultMetricsReporter.java +++ b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporter.java @@ -16,37 +16,30 @@ * specific language governing permissions and limitations * under the License. */ -package org.apache.polaris.service.reporting; +package org.apache.polaris.extension.metrics.reports; -import com.google.common.annotations.VisibleForTesting; import io.smallrye.common.annotation.Identifier; import jakarta.enterprise.context.ApplicationScoped; import java.time.Instant; import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.metrics.MetricsReport; +import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter; import org.slf4j.Logger; import org.slf4j.LoggerFactory; /** - * Default implementation of {@link PolarisMetricsReporter} that logs metrics to the configured - * logger. + * Log-only implementation of {@link IcebergMetricsReporter}. * - *

This implementation is selected when {@code polaris.iceberg-metrics.reporting.type} is set to - * {@code "default"} (the default value). + *

Selected when {@code polaris.iceberg-metrics.reporting.type} is set to {@code "log"}. * - *

By default, logging is disabled. To enable metrics logging, set the logger level for {@code - * org.apache.polaris.service.reporting} to {@code INFO} in your logging configuration. - * - * @see PolarisMetricsReporter + *

Logging is at INFO level. Enable it by setting the logger level for {@code + * org.apache.polaris.extension.metrics.reports} to {@code INFO}. */ @ApplicationScoped -@Identifier("default") -public class DefaultMetricsReporter implements PolarisMetricsReporter { - private static final Logger LOGGER = LoggerFactory.getLogger(DefaultMetricsReporter.class); - - private final ReportConsumer reportConsumer; +@Identifier("log") +public class LoggingMetricsReporter implements IcebergMetricsReporter { + private static final Logger LOGGER = LoggerFactory.getLogger(LoggingMetricsReporter.class); - /** Functional interface for consuming metrics reports with timestamp. */ @FunctionalInterface interface ReportConsumer { void accept( @@ -58,15 +51,15 @@ void accept( Instant receivedTimestamp); } - /** Creates a new DefaultMetricsReporter that logs metrics to the class logger. */ - public DefaultMetricsReporter() { + private final ReportConsumer reportConsumer; + + public LoggingMetricsReporter() { this( (catalogName, catalogId, table, tableId, metricsReport, receivedTimestamp) -> LOGGER.info("{}.{} (ts={}): {}", catalogName, table, receivedTimestamp, metricsReport)); } - @VisibleForTesting - DefaultMetricsReporter(ReportConsumer reportConsumer) { + LoggingMetricsReporter(ReportConsumer reportConsumer) { this.reportConsumer = reportConsumer; } @@ -80,4 +73,9 @@ public void reportMetric( Instant receivedTimestamp) { reportConsumer.accept(catalogName, catalogId, table, tableId, metricsReport, receivedTimestamp); } + + /** Backward-compatible alias for {@code "log"}; redirects the legacy {@code "default"} type. */ + @ApplicationScoped + @Identifier("default") + public static class DefaultMetricsReporter extends LoggingMetricsReporter {} } diff --git a/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/MetricsReportsService.java b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/MetricsReportsService.java new file mode 100644 index 00000000000..120e0aebe56 --- /dev/null +++ b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/MetricsReportsService.java @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.extension.metrics.reports; + +import com.google.common.annotations.Beta; +import jakarta.enterprise.context.RequestScoped; +import jakarta.inject.Inject; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import java.util.Arrays; +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.exceptions.NotFoundException; +import org.apache.polaris.core.auth.PolarisAuthorizableOperation; +import org.apache.polaris.core.auth.PolarisAuthorizer; +import org.apache.polaris.core.auth.PolarisPrincipal; +import org.apache.polaris.core.catalog.PolarisCatalogHelpers; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.core.entity.PolarisEntitySubType; +import org.apache.polaris.core.entity.PolarisEntityType; +import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper; +import org.apache.polaris.core.persistence.resolver.PolarisResolutionManifest; +import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory; +import org.apache.polaris.core.persistence.resolver.ResolvedPathKey; +import org.apache.polaris.core.persistence.resolver.ResolverPath; +import org.apache.polaris.core.persistence.resolver.ResolverStatus; +import org.apache.polaris.core.rest.NamespaceUtils; +import org.apache.polaris.service.metrics.api.PolarisCatalogsApiService; +import org.jspecify.annotations.NonNull; + +/** + * Service implementation for the Metrics Reports API. + * + *

Resolves catalog/namespace/table names to internal IDs and performs authorization. In this + * release the read path returns HTTP 501 Not Implemented; durable query backing is provided by the + * {@code polaris-extensions-metrics-reports-jdbc} extension in a follow-up release. + * + *

TODO (#4756): wire durable metrics query provider when the extension is installed. + */ +@Beta +@RequestScoped +public class MetricsReportsService implements PolarisCatalogsApiService { + + private final PolarisAuthorizer authorizer; + private final PolarisPrincipal polarisPrincipal; + private final ResolutionManifestFactory resolutionManifestFactory; + + @Inject + public MetricsReportsService( + @NonNull PolarisAuthorizer authorizer, + @NonNull PolarisPrincipal polarisPrincipal, + @NonNull ResolutionManifestFactory resolutionManifestFactory) { + this.authorizer = authorizer; + this.polarisPrincipal = polarisPrincipal; + this.resolutionManifestFactory = resolutionManifestFactory; + } + + @Override + public Response listTableMetrics( + String catalogName, + String namespace, + String table, + String metricType, + String pageToken, + Integer pageSize, + Long snapshotId, + String principalName, + Long timestampFrom, + Long timestampTo, + RealmContext realmContext, + SecurityContext securityContext) { + + Namespace ns = decodeNamespace(namespace); + TableIdentifier identifier = TableIdentifier.of(ns, table); + + resolveAndAuthorizeTableMetrics(catalogName, identifier); + + return Response.status(Response.Status.NOT_IMPLEMENTED) + .entity( + "Durable metrics query is not available in this deployment. " + + "Install the polaris-extensions-metrics-reports-jdbc extension to enable it.") + .build(); + } + + private void resolveAndAuthorizeTableMetrics(String catalogName, TableIdentifier identifier) { + PolarisResolutionManifest manifest = + resolutionManifestFactory.createResolutionManifest(polarisPrincipal, catalogName); + manifest.addPassthroughPath( + new ResolverPath( + Arrays.asList(identifier.namespace().levels()), PolarisEntityType.NAMESPACE)); + manifest.addPassthroughPath( + new ResolverPath( + PolarisCatalogHelpers.tableIdentifierToList(identifier), PolarisEntityType.TABLE_LIKE)); + ResolverStatus status = manifest.resolveAll(); + + if (status.getStatus() == ResolverStatus.StatusEnum.ENTITY_COULD_NOT_BE_RESOLVED) { + throw new NotFoundException( + "TopLevelEntity of type %s does not exist: %s", + status.getFailedToResolvedEntityType(), status.getFailedToResolvedEntityName()); + } + if (status.getStatus() == ResolverStatus.StatusEnum.PATH_COULD_NOT_BE_FULLY_RESOLVED) { + throw new NotFoundException("Table not found: %s", identifier); + } + + PolarisResolvedPathWrapper tableWrapper = + manifest.getResolvedPath( + ResolvedPathKey.ofTableLike(identifier), PolarisEntitySubType.ANY_SUBTYPE, true); + + if (tableWrapper == null) { + throw new NotFoundException("Table not found: %s", identifier); + } + + authorizer.authorizeOrThrow( + polarisPrincipal, + manifest.getAllActivatedCatalogRoleAndPrincipalRoles(), + PolarisAuthorizableOperation.LIST_TABLE_METRICS, + tableWrapper, + null); + } + + private static Namespace decodeNamespace(String encodedNamespace) { + if (encodedNamespace == null || encodedNamespace.isEmpty()) { + throw new IllegalArgumentException("namespace must not be empty"); + } + return NamespaceUtils.splitNamespace( + encodedNamespace, NamespaceUtils.DEFAULT_NAMESPACE_SEPARATOR); + } +} diff --git a/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/NoOpMetricsReporter.java b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/NoOpMetricsReporter.java new file mode 100644 index 00000000000..7fa4fcab12f --- /dev/null +++ b/extensions/metrics-reports/base/src/main/java/org/apache/polaris/extension/metrics/reports/NoOpMetricsReporter.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.extension.metrics.reports; + +import io.smallrye.common.annotation.Identifier; +import jakarta.enterprise.context.ApplicationScoped; +import java.time.Instant; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.metrics.MetricsReport; +import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter; + +/** + * No-op implementation of {@link IcebergMetricsReporter} that silently discards all metrics. + * + *

Selected when {@code polaris.iceberg-metrics.reporting.type} is set to {@code "no-op"}. + */ +@ApplicationScoped +@Identifier("no-op") +public class NoOpMetricsReporter implements IcebergMetricsReporter { + + @Override + public void reportMetric( + String catalogName, + long catalogId, + TableIdentifier table, + long tableId, + MetricsReport metricsReport, + Instant receivedTimestamp) {} +} diff --git a/runtime/service/src/test/java/org/apache/polaris/service/reporting/DefaultMetricsReporterTest.java b/extensions/metrics-reports/base/src/test/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporterTest.java similarity index 85% rename from runtime/service/src/test/java/org/apache/polaris/service/reporting/DefaultMetricsReporterTest.java rename to extensions/metrics-reports/base/src/test/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporterTest.java index 2240f41c921..cc47c55d9fd 100644 --- a/runtime/service/src/test/java/org/apache/polaris/service/reporting/DefaultMetricsReporterTest.java +++ b/extensions/metrics-reports/base/src/test/java/org/apache/polaris/extension/metrics/reports/LoggingMetricsReporterTest.java @@ -16,7 +16,7 @@ * specific language governing permissions and limitations * under the License. */ -package org.apache.polaris.service.reporting; +package org.apache.polaris.extension.metrics.reports; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verify; @@ -26,13 +26,13 @@ import org.apache.iceberg.metrics.MetricsReport; import org.junit.jupiter.api.Test; -public class DefaultMetricsReporterTest { +public class LoggingMetricsReporterTest { @Test void testLogging() { - DefaultMetricsReporter.ReportConsumer mockConsumer = - mock(DefaultMetricsReporter.ReportConsumer.class); - DefaultMetricsReporter reporter = new DefaultMetricsReporter(mockConsumer); + LoggingMetricsReporter.ReportConsumer mockConsumer = + mock(LoggingMetricsReporter.ReportConsumer.class); + LoggingMetricsReporter reporter = new LoggingMetricsReporter(mockConsumer); String warehouse = "testWarehouse"; long catalogId = 12345L; TableIdentifier table = TableIdentifier.of("testNamespace", "testTable"); diff --git a/extensions/metrics-reports/base/src/test/java/org/apache/polaris/extension/metrics/reports/MetricsReportsServiceTest.java b/extensions/metrics-reports/base/src/test/java/org/apache/polaris/extension/metrics/reports/MetricsReportsServiceTest.java new file mode 100644 index 00000000000..a1b636525e8 --- /dev/null +++ b/extensions/metrics-reports/base/src/test/java/org/apache/polaris/extension/metrics/reports/MetricsReportsServiceTest.java @@ -0,0 +1,245 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.extension.metrics.reports; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.ArgumentMatchers.isNull; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import jakarta.ws.rs.ForbiddenException; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import java.util.List; +import java.util.Set; +import org.apache.iceberg.exceptions.NotFoundException; +import org.apache.polaris.core.auth.PolarisAuthorizableOperation; +import org.apache.polaris.core.auth.PolarisAuthorizer; +import org.apache.polaris.core.auth.PolarisPrincipal; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.core.entity.PolarisEntitySubType; +import org.apache.polaris.core.entity.PolarisEntityType; +import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper; +import org.apache.polaris.core.persistence.resolver.PolarisResolutionManifest; +import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory; +import org.apache.polaris.core.persistence.resolver.ResolvedPathKey; +import org.apache.polaris.core.persistence.resolver.ResolverPath; +import org.apache.polaris.core.persistence.resolver.ResolverStatus; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +/** + * Unit tests for {@link MetricsReportsService}. + * + *

The read path currently returns 501 Not Implemented pending the durable extension (#4756). + * These tests cover authorization, resolution error paths, and input validation. + */ +class MetricsReportsServiceTest { + + private static final String CATALOG = "test-catalog"; + private static final String NAMESPACE = "dbschema"; + private static final String TABLE = "events"; + + private PolarisAuthorizer authorizer; + private PolarisResolutionManifest manifest; + private PolarisPrincipal principal; + private ResolutionManifestFactory factory; + private MetricsReportsService service; + private RealmContext realmContext; + private SecurityContext securityContext; + + @BeforeEach + void setUp() { + authorizer = mock(PolarisAuthorizer.class); + principal = mock(PolarisPrincipal.class); + + PolarisResolvedPathWrapper tableWrapper = mock(PolarisResolvedPathWrapper.class); + manifest = mock(PolarisResolutionManifest.class); + factory = mock(ResolutionManifestFactory.class); + realmContext = mock(RealmContext.class); + securityContext = mock(SecurityContext.class); + + when(manifest.resolveAll()).thenReturn(new ResolverStatus(ResolverStatus.StatusEnum.SUCCESS)); + when(manifest.getResolvedPath( + any(ResolvedPathKey.class), eq(PolarisEntitySubType.ANY_SUBTYPE), eq(true))) + .thenReturn(tableWrapper); + when(manifest.getAllActivatedCatalogRoleAndPrincipalRoles()).thenReturn(Set.of()); + when(factory.createResolutionManifest(eq(principal), eq(CATALOG))).thenReturn(manifest); + doNothing() + .when(authorizer) + .authorizeOrThrow( + any(PolarisPrincipal.class), + any(Set.class), + any(PolarisAuthorizableOperation.class), + any(PolarisResolvedPathWrapper.class), + (PolarisResolvedPathWrapper) isNull()); + + service = new MetricsReportsService(authorizer, principal, factory); + } + + @Test + void authorizedRequestReturnsNotImplemented() { + Response response = + service.listTableMetrics( + CATALOG, + NAMESPACE, + TABLE, + "scan", + null, + 10, + null, + null, + null, + null, + realmContext, + securityContext); + + assertThat(response.getStatus()).isEqualTo(Response.Status.NOT_IMPLEMENTED.getStatusCode()); + } + + @Test + void unauthorizedRequestThrowsForbiddenException() { + doThrow(new ForbiddenException("denied")) + .when(authorizer) + .authorizeOrThrow( + any(PolarisPrincipal.class), + any(Set.class), + eq(PolarisAuthorizableOperation.LIST_TABLE_METRICS), + any(PolarisResolvedPathWrapper.class), + (PolarisResolvedPathWrapper) isNull()); + + assertThatThrownBy( + () -> + service.listTableMetrics( + CATALOG, + NAMESPACE, + TABLE, + "scan", + null, + 10, + null, + null, + null, + null, + realmContext, + securityContext)) + .isInstanceOf(ForbiddenException.class); + } + + @Test + void tableNotFoundThrowsNotFoundException() { + when(manifest.getResolvedPath( + any(ResolvedPathKey.class), eq(PolarisEntitySubType.ANY_SUBTYPE), eq(true))) + .thenReturn(null); + + assertThatThrownBy( + () -> + service.listTableMetrics( + CATALOG, + NAMESPACE, + TABLE, + "scan", + null, + 10, + null, + null, + null, + null, + realmContext, + securityContext)) + .isInstanceOf(NotFoundException.class) + .hasMessageContaining(TABLE); + } + + @Test + void catalogNotFoundPropagatesNotFoundException() { + when(manifest.resolveAll()).thenReturn(new ResolverStatus(PolarisEntityType.CATALOG, CATALOG)); + + assertThatThrownBy( + () -> + service.listTableMetrics( + CATALOG, + NAMESPACE, + TABLE, + "scan", + null, + 10, + null, + null, + null, + null, + realmContext, + securityContext)) + .isInstanceOf(NotFoundException.class) + .hasMessageContaining(CATALOG); + } + + @Test + void pathNotFoundPropagatesNotFoundException() { + ResolverPath failedPath = new ResolverPath(List.of(NAMESPACE), PolarisEntityType.NAMESPACE); + when(manifest.resolveAll()).thenReturn(new ResolverStatus(failedPath, 0)); + + assertThatThrownBy( + () -> + service.listTableMetrics( + CATALOG, + NAMESPACE, + TABLE, + "scan", + null, + 10, + null, + null, + null, + null, + realmContext, + securityContext)) + .isInstanceOf(NotFoundException.class); + } + + @Test + void multiLevelNamespaceIsSplitCorrectly() { + // JAX-RS decodes %1F -> U+001F before injection; the service must split correctly. + // "dbschema" represents namespace ["db", "schema"]. + String encodedTwoLevel = "dbschema"; + when(factory.createResolutionManifest(eq(principal), eq(CATALOG))).thenReturn(manifest); + + Response response = + service.listTableMetrics( + CATALOG, + encodedTwoLevel, + TABLE, + "scan", + null, + 10, + null, + null, + null, + null, + realmContext, + securityContext); + + assertThat(response.getStatus()).isEqualTo(Response.Status.NOT_IMPLEMENTED.getStatusCode()); + } +} diff --git a/extensions/metrics-reports/spi/build.gradle.kts b/extensions/metrics-reports/spi/build.gradle.kts new file mode 100644 index 00000000000..b1b0a1898ff --- /dev/null +++ b/extensions/metrics-reports/spi/build.gradle.kts @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +plugins { + id("polaris-server") +} + +dependencies { + implementation(platform(libs.iceberg.bom)) + implementation("org.apache.iceberg:iceberg-api") + implementation(libs.guava) +} diff --git a/runtime/service/src/main/java/org/apache/polaris/service/reporting/PolarisMetricsReporter.java b/extensions/metrics-reports/spi/src/main/java/org/apache/polaris/extension/metrics/spi/IcebergMetricsReporter.java similarity index 60% rename from runtime/service/src/main/java/org/apache/polaris/service/reporting/PolarisMetricsReporter.java rename to extensions/metrics-reports/spi/src/main/java/org/apache/polaris/extension/metrics/spi/IcebergMetricsReporter.java index c621be2a5d8..585bcd07014 100644 --- a/runtime/service/src/main/java/org/apache/polaris/service/reporting/PolarisMetricsReporter.java +++ b/extensions/metrics-reports/spi/src/main/java/org/apache/polaris/extension/metrics/spi/IcebergMetricsReporter.java @@ -16,38 +16,35 @@ * specific language governing permissions and limitations * under the License. */ -package org.apache.polaris.service.reporting; +package org.apache.polaris.extension.metrics.spi; +import com.google.common.annotations.Beta; import java.time.Instant; import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.metrics.MetricsReport; /** - * SPI interface for reporting Iceberg metrics received by Polaris. + * SPI for reporting Iceberg metrics received by Polaris. * - *

Implementations can be used to send metrics to external systems for analysis and monitoring. - * Custom implementations can be annotated with appropriate {@code Quarkus} scope and {@link - * io.smallrye.common.annotation.Identifier @Identifier("my-reporter-type")} for CDI discovery. + *

Implementations receive a resolved table context (catalog name/id and table name/id) along + * with the raw Iceberg {@link MetricsReport}. Custom implementations should be annotated with the + * appropriate CDI scope and {@code @Identifier("my-type")} for selection via the {@code + * polaris.iceberg-metrics.reporting.type} configuration property. * - *

The implementation to use is selected via the {@code polaris.iceberg-metrics.reporting.type} - * configuration property, which defaults to {@code "default"}. - * - *

Implementations can inject other CDI beans for context. - * - * @see DefaultMetricsReporter - * @see MetricsReportingConfiguration + *

This interface is intentionally runtime/framework-agnostic. CDI and configuration concerns + * belong in the implementing class, not here. */ -public interface PolarisMetricsReporter { +@Beta +public interface IcebergMetricsReporter { /** - * Reports an Iceberg metrics report for a specific table. + * Reports an Iceberg metrics report for a resolved table. * * @param catalogName the name of the catalog containing the table * @param catalogId the internal Polaris ID of the catalog * @param table the identifier of the table the metrics are for * @param tableId the internal Polaris ID of the table entity - * @param metricsReport the Iceberg metrics report (e.g., {@link - * org.apache.iceberg.metrics.ScanReport} or {@link org.apache.iceberg.metrics.CommitReport}) + * @param metricsReport the Iceberg metrics report * @param receivedTimestamp the timestamp when the metrics were received by Polaris */ void reportMetric( diff --git a/gradle/projects.main.properties b/gradle/projects.main.properties index cae568b92e2..a907b82e8e1 100644 --- a/gradle/projects.main.properties +++ b/gradle/projects.main.properties @@ -23,6 +23,7 @@ polaris-core=polaris-core polaris-api-iceberg-service=api/iceberg-service polaris-api-management-model=api/management-model polaris-api-management-service=api/management-service +polaris-api-metrics-reports-service=api/metrics-reports-service polaris-api-catalog-service=api/polaris-catalog-service polaris-runtime-defaults=runtime/defaults polaris-runtime-service=runtime/service @@ -49,6 +50,8 @@ polaris-extensions-federation-hive=extensions/federation/hive polaris-extensions-federation-bigquery=extensions/federation/bigquery polaris-extensions-auth-opa=extensions/auth/opa polaris-extensions-auth-ranger=extensions/auth/ranger +polaris-extensions-metrics-reports-spi=extensions/metrics-reports/spi +polaris-extensions-metrics-reports=extensions/metrics-reports/base polaris-config-docs-annotations=tools/config-docs/annotations polaris-config-docs-generator=tools/config-docs/generator diff --git a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java index cc711cc646a..cdb408bc7a6 100644 --- a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java +++ b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java @@ -57,9 +57,6 @@ import org.apache.polaris.core.persistence.PolicyMappingAlreadyExistsException; import org.apache.polaris.core.persistence.PrincipalSecretsGenerator; import org.apache.polaris.core.persistence.RetryOnConcurrencyException; -import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; -import org.apache.polaris.core.persistence.metrics.MetricsPersistence; -import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; import org.apache.polaris.core.persistence.pagination.EntityIdToken; import org.apache.polaris.core.persistence.pagination.Page; import org.apache.polaris.core.persistence.pagination.PageToken; @@ -70,21 +67,18 @@ import org.apache.polaris.core.storage.PolarisStorageIntegration; import org.apache.polaris.core.storage.StorageLocation; import org.apache.polaris.persistence.relational.jdbc.models.EntityNameLookupRecordConverter; -import org.apache.polaris.persistence.relational.jdbc.models.ModelCommitMetricsReport; import org.apache.polaris.persistence.relational.jdbc.models.ModelEntity; import org.apache.polaris.persistence.relational.jdbc.models.ModelEvent; import org.apache.polaris.persistence.relational.jdbc.models.ModelGrantRecord; import org.apache.polaris.persistence.relational.jdbc.models.ModelPolicyMappingRecord; import org.apache.polaris.persistence.relational.jdbc.models.ModelPrincipalAuthenticationData; -import org.apache.polaris.persistence.relational.jdbc.models.ModelScanMetricsReport; import org.apache.polaris.persistence.relational.jdbc.models.SchemaVersion; import org.jspecify.annotations.NonNull; import org.jspecify.annotations.Nullable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -public class JdbcBasePersistenceImpl - implements BasePersistence, IntegrationPersistence, MetricsPersistence { +public class JdbcBasePersistenceImpl implements BasePersistence, IntegrationPersistence { private static final Logger LOGGER = LoggerFactory.getLogger(JdbcBasePersistenceImpl.class); @@ -1337,59 +1331,4 @@ public void persistStorageIntegrationIfNeeded( private interface QueryAction { Integer apply(Connection connection, QueryGenerator.PreparedQuery query) throws SQLException; } - - // ============================================================================ - // MetricsPersistence Implementation - // ============================================================================ - - /** Returns the datasource operations to use for metrics persistence. */ - private DatasourceOperations getMetricsDatasource() { - return datasourceOperations; - } - - @Override - public void writeScanReport(@NonNull ScanMetricsRecord record) { - ModelScanMetricsReport model = ModelScanMetricsReport.fromRecord(record, realmId); - writeScanMetricsReport(model); - } - - @Override - public void writeCommitReport(@NonNull CommitMetricsRecord record) { - ModelCommitMetricsReport model = ModelCommitMetricsReport.fromRecord(record, realmId); - writeCommitMetricsReport(model); - } - - // ========== Internal Metrics JDBC methods ========== - - private void writeScanMetricsReport(@NonNull ModelScanMetricsReport report) { - DatasourceOperations metricsOps = getMetricsDatasource(); - try { - PreparedQuery pq = - QueryGenerator.generateInsertQuery( - ModelScanMetricsReport.ALL_COLUMNS, - ModelScanMetricsReport.TABLE_NAME, - report.toMap(metricsOps.getDatabaseType()).values().stream().toList(), - realmId); - metricsOps.executeUpdate(pq); - } catch (SQLException e) { - throw new RuntimeException( - String.format("Failed to write scan metrics report due to %s", e.getMessage()), e); - } - } - - private void writeCommitMetricsReport(@NonNull ModelCommitMetricsReport report) { - DatasourceOperations metricsOps = getMetricsDatasource(); - try { - PreparedQuery pq = - QueryGenerator.generateInsertQuery( - ModelCommitMetricsReport.ALL_COLUMNS, - ModelCommitMetricsReport.TABLE_NAME, - report.toMap(metricsOps.getDatabaseType()).values().stream().toList(), - realmId); - metricsOps.executeUpdate(pq); - } catch (SQLException e) { - throw new RuntimeException( - String.format("Failed to write commit metrics report due to %s", e.getMessage()), e); - } - } } diff --git a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java index e88aac882af..57be471af31 100644 --- a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java +++ b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java @@ -252,7 +252,7 @@ public BasePersistence getOrCreateSession(RealmContext realmContext) { @Override public MetricsPersistence getOrCreateMetricsPersistence(RealmContext realmContext) { - return createJdbcPersistence(realmContext); + return new MetricsPersistence() {}; } @Override diff --git a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReport.java b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReport.java index 10dc193869e..47f777cdc86 100644 --- a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReport.java +++ b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReport.java @@ -20,9 +20,11 @@ import java.sql.ResultSet; import java.sql.SQLException; +import java.time.Instant; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Optional; import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; import org.apache.polaris.immutables.PolarisImmutable; import org.apache.polaris.persistence.relational.jdbc.DatabaseType; @@ -316,6 +318,64 @@ private static String toJsonString(Map map) { } } + private static Map parseMetadata(@Nullable String metadata) { + if (metadata == null || metadata.isBlank()) { + return Map.of(); + } + try { + Map parsed = + OBJECT_MAPPER.readValue( + metadata, new tools.jackson.core.type.TypeReference>() {}); + return parsed != null ? parsed : Map.of(); + } catch (JacksonException e) { + return Map.of(); + } + } + + /** + * Converts this JDBC model back to the backend-agnostic SPI record. + * + * @return a CommitMetricsRecord built from this model's fields + */ + default CommitMetricsRecord toRecord() { + return CommitMetricsRecord.builder() + .reportId(getReportId()) + .catalogId(getCatalogId()) + .tableId(getTableId()) + .timestamp(Instant.ofEpochMilli(getTimestampMs())) + .metadata(parseMetadata(getMetadata())) + .principalName(getPrincipalName()) + .requestId(getRequestId()) + .otelTraceId(getOtelTraceId()) + .otelSpanId(getOtelSpanId()) + .snapshotId(getSnapshotId()) + .sequenceNumber(Optional.ofNullable(getSequenceNumber())) + .operation(getOperation()) + .addedDataFiles(getAddedDataFiles()) + .removedDataFiles(getRemovedDataFiles()) + .totalDataFiles(getTotalDataFiles()) + .addedDeleteFiles(getAddedDeleteFiles()) + .removedDeleteFiles(getRemovedDeleteFiles()) + .totalDeleteFiles(getTotalDeleteFiles()) + .addedEqualityDeleteFiles(getAddedEqualityDeleteFiles()) + .removedEqualityDeleteFiles(getRemovedEqualityDeleteFiles()) + .addedPositionalDeleteFiles(getAddedPositionalDeleteFiles()) + .removedPositionalDeleteFiles(getRemovedPositionalDeleteFiles()) + .addedRecords(getAddedRecords()) + .removedRecords(getRemovedRecords()) + .totalRecords(getTotalRecords()) + .addedFileSizeBytes(getAddedFileSizeBytes()) + .removedFileSizeBytes(getRemovedFileSizeBytes()) + .totalFileSizeBytes(getTotalFileSizeBytes()) + // TODO(#4397): total_duration_ms is NOT NULL, so 0L is ambiguous between "unknown" and a + // genuine zero-duration commit. Make the column nullable as part of the metrics schema + // consolidation in #4397 so null vs 0 can be represented without this workaround. + .totalDurationMs( + getTotalDurationMs() == 0L ? Optional.empty() : Optional.of(getTotalDurationMs())) + .attempts(getAttempts()) + .build(); + } + /** Dummy instance to be used as a Converter when calling fromResultSet(). */ ModelCommitMetricsReport CONVERTER = ImmutableModelCommitMetricsReport.builder() diff --git a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReport.java b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReport.java index 49ad75c6a86..5c85ff7d43a 100644 --- a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReport.java +++ b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReport.java @@ -20,9 +20,13 @@ import java.sql.ResultSet; import java.sql.SQLException; +import java.time.Instant; +import java.util.Arrays; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.stream.Collectors; import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; import org.apache.polaris.immutables.PolarisImmutable; @@ -321,6 +325,85 @@ private static String toJsonString(Map map) { } } + private static Map parseMetadata(@Nullable String metadata) { + if (metadata == null || metadata.isBlank()) { + return Map.of(); + } + try { + Map parsed = + JsonMapper.shared() + .readValue( + metadata, new tools.jackson.core.type.TypeReference>() {}); + return parsed != null ? parsed : Map.of(); + } catch (JacksonException e) { + return Map.of(); + } + } + + /** + * Converts this JDBC model back to the backend-agnostic SPI record. + * + * @return a ScanMetricsRecord built from this model's fields + */ + default ScanMetricsRecord toRecord() { + String rawFieldIds = getProjectedFieldIds(); + String rawFieldNames = getProjectedFieldNames(); + List fieldIds = + rawFieldIds == null || rawFieldIds.isEmpty() + ? Collections.emptyList() + : Arrays.stream(rawFieldIds.split(",")) + .map(String::trim) + .filter(s -> !s.isEmpty()) + .flatMap( + s -> { + try { + return java.util.stream.Stream.of(Integer.parseInt(s)); + } catch (NumberFormatException e) { + return java.util.stream.Stream.empty(); + } + }) + .collect(Collectors.toList()); + List fieldNames = + rawFieldNames == null || rawFieldNames.isEmpty() + ? Collections.emptyList() + : Arrays.stream(rawFieldNames.split(",")) + .map(String::trim) + .filter(s -> !s.isEmpty()) + .collect(Collectors.toList()); + return ScanMetricsRecord.builder() + .reportId(getReportId()) + .catalogId(getCatalogId()) + .tableId(getTableId()) + .timestamp(Instant.ofEpochMilli(getTimestampMs())) + .metadata(parseMetadata(getMetadata())) + .principalName(getPrincipalName()) + .requestId(getRequestId()) + .otelTraceId(getOtelTraceId()) + .otelSpanId(getOtelSpanId()) + .snapshotId(Optional.ofNullable(getSnapshotId())) + .schemaId(Optional.ofNullable(getSchemaId())) + .filterExpression(Optional.ofNullable(getFilterExpression())) + .projectedFieldIds(fieldIds) + .projectedFieldNames(fieldNames) + .resultDataFiles(getResultDataFiles()) + .resultDeleteFiles(getResultDeleteFiles()) + .totalFileSizeBytes(getTotalFileSizeBytes()) + .totalDataManifests(getTotalDataManifests()) + .totalDeleteManifests(getTotalDeleteManifests()) + .scannedDataManifests(getScannedDataManifests()) + .scannedDeleteManifests(getScannedDeleteManifests()) + .skippedDataManifests(getSkippedDataManifests()) + .skippedDeleteManifests(getSkippedDeleteManifests()) + .skippedDataFiles(getSkippedDataFiles()) + .skippedDeleteFiles(getSkippedDeleteFiles()) + .totalPlanningDurationMs(getTotalPlanningDurationMs()) + .equalityDeleteFiles(getEqualityDeleteFiles()) + .positionalDeleteFiles(getPositionalDeleteFiles()) + .indexedDeleteFiles(getIndexedDeleteFiles()) + .totalDeleteFileSizeBytes(getTotalDeleteFileSizeBytes()) + .build(); + } + /** Dummy instance to be used as a Converter when calling fromResultSet(). */ ModelScanMetricsReport CONVERTER = ImmutableModelScanMetricsReport.builder() diff --git a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/MetricsReportPersistenceTest.java b/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/MetricsReportPersistenceTest.java deleted file mode 100644 index 37c7bb53916..00000000000 --- a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/MetricsReportPersistenceTest.java +++ /dev/null @@ -1,193 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.persistence.relational.jdbc; - -import java.io.InputStream; -import java.sql.SQLException; -import java.time.Instant; -import java.util.Optional; -import java.util.UUID; -import javax.sql.DataSource; -import org.apache.polaris.core.PolarisDefaultDiagServiceImpl; -import org.apache.polaris.core.PolarisDiagnostics; -import org.apache.polaris.core.persistence.PrincipalSecretsGenerator; -import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; -import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; -import org.h2.jdbcx.JdbcConnectionPool; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; - -/** - * Integration tests for metrics report persistence using JdbcBasePersistenceImpl. Tests the - * SPI-level write operations for scan and commit metrics reports. - */ -class MetricsReportPersistenceTest { - - private JdbcBasePersistenceImpl metricsPersistence; - private DataSource dataSource; - - @BeforeEach - void setUp() throws SQLException { - dataSource = - JdbcConnectionPool.create( - "jdbc:h2:mem:test_metrics_" + UUID.randomUUID() + ";DB_CLOSE_DELAY=-1", "sa", ""); - - DatasourceOperations datasourceOperations = - new DatasourceOperations(dataSource, new TestJdbcConfiguration()); - - // Execute main schema v4 (includes metrics tables) - ClassLoader classLoader = DatasourceOperations.class.getClassLoader(); - InputStream schemaStream = classLoader.getResourceAsStream("h2/schema-v4.sql"); - datasourceOperations.executeScript(schemaStream); - - PolarisDiagnostics diagnostics = new PolarisDefaultDiagServiceImpl(); - - metricsPersistence = - new JdbcBasePersistenceImpl( - diagnostics, - datasourceOperations, - PrincipalSecretsGenerator.RANDOM_SECRETS, - "TEST_REALM", - 4); - } - - @Test - void testWriteScanReport() { - ScanMetricsRecord record = - ScanMetricsRecord.builder() - .reportId(UUID.randomUUID().toString()) - .catalogId(12345L) - .tableId(67890L) - .timestamp(Instant.now()) - .principalName("test-user") - .requestId("req-123") - .otelTraceId("trace-abc") - .otelSpanId("span-xyz") - .resultDataFiles(10L) - .resultDeleteFiles(2L) - .totalFileSizeBytes(1024000L) - .totalDataManifests(5L) - .totalDeleteManifests(1L) - .scannedDataManifests(3L) - .scannedDeleteManifests(1L) - .skippedDataManifests(2L) - .skippedDeleteManifests(0L) - .skippedDataFiles(5L) - .skippedDeleteFiles(0L) - .totalPlanningDurationMs(150L) - .equalityDeleteFiles(1L) - .positionalDeleteFiles(1L) - .indexedDeleteFiles(0L) - .totalDeleteFileSizeBytes(10240L) - .build(); - - // Should not throw - uses SPI method - metricsPersistence.writeScanReport(record); - } - - @Test - void testWriteCommitReport() { - CommitMetricsRecord record = - CommitMetricsRecord.builder() - .reportId(UUID.randomUUID().toString()) - .catalogId(12345L) - .tableId(67890L) - .timestamp(Instant.now()) - .principalName("test-user") - .requestId("req-456") - .otelTraceId("trace-def") - .otelSpanId("span-uvw") - .snapshotId(12345L) - .operation("append") - .addedDataFiles(5L) - .removedDataFiles(0L) - .totalDataFiles(100L) - .addedDeleteFiles(0L) - .removedDeleteFiles(0L) - .totalDeleteFiles(2L) - .addedEqualityDeleteFiles(0L) - .removedEqualityDeleteFiles(0L) - .addedPositionalDeleteFiles(0L) - .removedPositionalDeleteFiles(0L) - .addedRecords(1000L) - .removedRecords(0L) - .totalRecords(50000L) - .addedFileSizeBytes(102400L) - .removedFileSizeBytes(0L) - .totalFileSizeBytes(5120000L) - .attempts(1) - .build(); - - // Should not throw - uses SPI method - metricsPersistence.writeCommitReport(record); - } - - @Test - void testWriteMultipleScanReports() { - for (int i = 0; i < 10; i++) { - ScanMetricsRecord record = - ScanMetricsRecord.builder() - .reportId(UUID.randomUUID().toString()) - .catalogId(12345L) - .tableId(100L + i) - .timestamp(Instant.now()) - .resultDataFiles((long) (i * 10)) - .resultDeleteFiles(0L) - .totalFileSizeBytes((long) (i * 1024)) - .totalDataManifests(1L) - .totalDeleteManifests(0L) - .scannedDataManifests(1L) - .scannedDeleteManifests(0L) - .skippedDataManifests(0L) - .skippedDeleteManifests(0L) - .skippedDataFiles(0L) - .skippedDeleteFiles(0L) - .totalPlanningDurationMs((long) (i * 10)) - .equalityDeleteFiles(0L) - .positionalDeleteFiles(0L) - .indexedDeleteFiles(0L) - .totalDeleteFileSizeBytes(0L) - .build(); - - metricsPersistence.writeScanReport(record); - } - } - - private static class TestJdbcConfiguration implements RelationalJdbcConfiguration { - @Override - public Optional maxRetries() { - return Optional.of(1); - } - - @Override - public Optional maxDurationInMs() { - return Optional.of(100L); - } - - @Override - public Optional initialDelayInMs() { - return Optional.of(10L); - } - - @Override - public Optional databaseType() { - return Optional.empty(); - } - } -} diff --git a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/SpiModelConverterTest.java b/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/SpiModelConverterTest.java deleted file mode 100644 index 4c889ddd291..00000000000 --- a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/SpiModelConverterTest.java +++ /dev/null @@ -1,209 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.persistence.relational.jdbc; - -import static org.assertj.core.api.Assertions.assertThat; - -import java.time.Instant; -import java.util.List; -import java.util.Map; -import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; -import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; -import org.apache.polaris.persistence.relational.jdbc.models.ModelCommitMetricsReport; -import org.apache.polaris.persistence.relational.jdbc.models.ModelScanMetricsReport; -import org.junit.jupiter.api.Test; - -/** Unit tests for metrics model conversion methods (SPI record -> JDBC model). */ -public class SpiModelConverterTest { - - private static final String TEST_REPORT_ID = "report-123"; - private static final String TEST_REALM_ID = "realm-1"; - private static final long TEST_CATALOG_ID = 12345L; - private static final long TEST_TABLE_ID = 67890L; - private static final Instant TEST_TIMESTAMP = Instant.ofEpochMilli(1704067200000L); - private static final long TEST_TIMESTAMP_MS = 1704067200000L; - - @Test - void testFromScanRecord() { - ScanMetricsRecord record = createTestScanRecord(); - - ModelScanMetricsReport model = ModelScanMetricsReport.fromRecord(record, TEST_REALM_ID); - - assertThat(model.getReportId()).isEqualTo(TEST_REPORT_ID); - assertThat(model.getRealmId()).isEqualTo(TEST_REALM_ID); - assertThat(model.getCatalogId()).isEqualTo(TEST_CATALOG_ID); - assertThat(model.getTableId()).isEqualTo(TEST_TABLE_ID); - assertThat(model.getTimestampMs()).isEqualTo(TEST_TIMESTAMP_MS); - assertThat(model.getSnapshotId()).isEqualTo(123456789L); - assertThat(model.getSchemaId()).isEqualTo(1); - assertThat(model.getFilterExpression()).isEqualTo("id > 100"); - assertThat(model.getProjectedFieldIds()).isEqualTo("1,2,3"); - assertThat(model.getProjectedFieldNames()).isEqualTo("id,name,value"); - assertThat(model.getResultDataFiles()).isEqualTo(10L); - assertThat(model.getResultDeleteFiles()).isEqualTo(2L); - assertThat(model.getTotalFileSizeBytes()).isEqualTo(1024000L); - assertThat(model.getMetadata()).isEqualTo("{\"custom\":\"value\"}"); - } - - @Test - void testFromCommitRecord() { - CommitMetricsRecord record = createTestCommitRecord(); - - ModelCommitMetricsReport model = ModelCommitMetricsReport.fromRecord(record, TEST_REALM_ID); - - assertThat(model.getReportId()).isEqualTo(TEST_REPORT_ID); - assertThat(model.getRealmId()).isEqualTo(TEST_REALM_ID); - assertThat(model.getCatalogId()).isEqualTo(TEST_CATALOG_ID); - assertThat(model.getTableId()).isEqualTo(TEST_TABLE_ID); - assertThat(model.getTimestampMs()).isEqualTo(TEST_TIMESTAMP_MS); - assertThat(model.getSnapshotId()).isEqualTo(987654321L); - assertThat(model.getSequenceNumber()).isEqualTo(5L); - assertThat(model.getOperation()).isEqualTo("append"); - assertThat(model.getAddedDataFiles()).isEqualTo(10L); - assertThat(model.getRemovedDataFiles()).isEqualTo(2L); - assertThat(model.getTotalDataFiles()).isEqualTo(100L); - assertThat(model.getAttempts()).isEqualTo(1); - } - - @Test - void testNullOptionalFields() { - ScanMetricsRecord record = - ScanMetricsRecord.builder() - .reportId(TEST_REPORT_ID) - .catalogId(TEST_CATALOG_ID) - .tableId(TEST_TABLE_ID) - .timestamp(TEST_TIMESTAMP) - .resultDataFiles(0L) - .resultDeleteFiles(0L) - .totalFileSizeBytes(0L) - .totalDataManifests(0L) - .totalDeleteManifests(0L) - .scannedDataManifests(0L) - .scannedDeleteManifests(0L) - .skippedDataManifests(0L) - .skippedDeleteManifests(0L) - .skippedDataFiles(0L) - .skippedDeleteFiles(0L) - .totalPlanningDurationMs(0L) - .equalityDeleteFiles(0L) - .positionalDeleteFiles(0L) - .indexedDeleteFiles(0L) - .totalDeleteFileSizeBytes(0L) - .build(); - - ModelScanMetricsReport model = ModelScanMetricsReport.fromRecord(record, TEST_REALM_ID); - assertThat(model.getSnapshotId()).isNull(); - assertThat(model.getSchemaId()).isNull(); - assertThat(model.getFilterExpression()).isNull(); - assertThat(model.getProjectedFieldIds()).isNull(); - assertThat(model.getProjectedFieldNames()).isNull(); - } - - @Test - void testEmptyMetadata() { - ScanMetricsRecord record = - ScanMetricsRecord.builder() - .reportId(TEST_REPORT_ID) - .catalogId(TEST_CATALOG_ID) - .tableId(TEST_TABLE_ID) - .timestamp(TEST_TIMESTAMP) - .resultDataFiles(0L) - .resultDeleteFiles(0L) - .totalFileSizeBytes(0L) - .totalDataManifests(0L) - .totalDeleteManifests(0L) - .scannedDataManifests(0L) - .scannedDeleteManifests(0L) - .skippedDataManifests(0L) - .skippedDeleteManifests(0L) - .skippedDataFiles(0L) - .skippedDeleteFiles(0L) - .totalPlanningDurationMs(0L) - .equalityDeleteFiles(0L) - .positionalDeleteFiles(0L) - .indexedDeleteFiles(0L) - .totalDeleteFileSizeBytes(0L) - .build(); - - ModelScanMetricsReport model = ModelScanMetricsReport.fromRecord(record, TEST_REALM_ID); - assertThat(model.getMetadata()).isEqualTo("{}"); - } - - private ScanMetricsRecord createTestScanRecord() { - return ScanMetricsRecord.builder() - .reportId(TEST_REPORT_ID) - .catalogId(TEST_CATALOG_ID) - .tableId(TEST_TABLE_ID) - .timestamp(TEST_TIMESTAMP) - .snapshotId(123456789L) - .schemaId(1) - .filterExpression("id > 100") - .projectedFieldIds(List.of(1, 2, 3)) - .projectedFieldNames(List.of("id", "name", "value")) - .resultDataFiles(10L) - .resultDeleteFiles(2L) - .totalFileSizeBytes(1024000L) - .totalDataManifests(5L) - .totalDeleteManifests(1L) - .scannedDataManifests(3L) - .scannedDeleteManifests(1L) - .skippedDataManifests(2L) - .skippedDeleteManifests(0L) - .skippedDataFiles(5L) - .skippedDeleteFiles(0L) - .totalPlanningDurationMs(150L) - .equalityDeleteFiles(1L) - .positionalDeleteFiles(1L) - .indexedDeleteFiles(0L) - .totalDeleteFileSizeBytes(2048L) - .metadata(Map.of("custom", "value")) - .build(); - } - - private CommitMetricsRecord createTestCommitRecord() { - return CommitMetricsRecord.builder() - .reportId(TEST_REPORT_ID) - .catalogId(TEST_CATALOG_ID) - .tableId(TEST_TABLE_ID) - .timestamp(TEST_TIMESTAMP) - .snapshotId(987654321L) - .sequenceNumber(5L) - .operation("append") - .addedDataFiles(10L) - .removedDataFiles(2L) - .totalDataFiles(100L) - .addedDeleteFiles(1L) - .removedDeleteFiles(0L) - .totalDeleteFiles(5L) - .addedEqualityDeleteFiles(0L) - .removedEqualityDeleteFiles(0L) - .addedPositionalDeleteFiles(1L) - .removedPositionalDeleteFiles(0L) - .addedRecords(1000L) - .removedRecords(50L) - .totalRecords(50000L) - .addedFileSizeBytes(102400L) - .removedFileSizeBytes(5120L) - .totalFileSizeBytes(5120000L) - .totalDurationMs(250L) - .attempts(1) - .metadata(Map.of("custom", "value")) - .build(); - } -} diff --git a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReportTest.java b/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReportTest.java deleted file mode 100644 index bdbb540853b..00000000000 --- a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/models/ModelCommitMetricsReportTest.java +++ /dev/null @@ -1,276 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.persistence.relational.jdbc.models; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertNull; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; - -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.Map; -import org.apache.polaris.persistence.relational.jdbc.DatabaseType; -import org.junit.jupiter.api.Test; -import org.postgresql.util.PGobject; - -public class ModelCommitMetricsReportTest { - - private static final String TEST_REPORT_ID = "commit-report-123"; - private static final String TEST_REALM_ID = "realm-1"; - private static final long TEST_CATALOG_ID = 12345L; - private static final long TEST_TABLE_ID = 67890L; - private static final long TEST_TIMESTAMP_MS = 1704067200000L; - private static final String TEST_PRINCIPAL = "user@example.com"; - private static final String TEST_REQUEST_ID = "req-456"; - private static final String TEST_OTEL_TRACE_ID = "trace-789"; - private static final String TEST_OTEL_SPAN_ID = "span-012"; - private static final String TEST_REPORT_TRACE_ID = "report-trace-345"; - private static final long TEST_SNAPSHOT_ID = 987654321L; - private static final Long TEST_SEQUENCE_NUMBER = 5L; - private static final String TEST_OPERATION = "append"; - private static final long TEST_ADDED_DATA_FILES = 10L; - private static final long TEST_REMOVED_DATA_FILES = 2L; - private static final long TEST_TOTAL_DATA_FILES = 50L; - private static final long TEST_ADDED_DELETE_FILES = 1L; - private static final long TEST_REMOVED_DELETE_FILES = 0L; - private static final long TEST_TOTAL_DELETE_FILES = 3L; - private static final long TEST_ADDED_EQUALITY_DELETE_FILES = 1L; - private static final long TEST_REMOVED_EQUALITY_DELETE_FILES = 0L; - private static final long TEST_ADDED_POSITIONAL_DELETE_FILES = 0L; - private static final long TEST_REMOVED_POSITIONAL_DELETE_FILES = 0L; - private static final long TEST_ADDED_RECORDS = 1000L; - private static final long TEST_REMOVED_RECORDS = 50L; - private static final long TEST_TOTAL_RECORDS = 10000L; - private static final long TEST_ADDED_FILE_SIZE = 1024000L; - private static final long TEST_REMOVED_FILE_SIZE = 51200L; - private static final long TEST_TOTAL_FILE_SIZE = 10240000L; - private static final long TEST_TOTAL_DURATION = 250L; - private static final int TEST_ATTEMPTS = 1; - private static final String TEST_METADATA = "{\"commit\":\"info\"}"; - - @Test - public void testFromResultSet() throws SQLException { - ResultSet mockResultSet = mock(ResultSet.class); - when(mockResultSet.getString(ModelCommitMetricsReport.REPORT_ID)).thenReturn(TEST_REPORT_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.REALM_ID)).thenReturn(TEST_REALM_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.CATALOG_ID)).thenReturn(TEST_CATALOG_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.TABLE_ID_COL)).thenReturn(TEST_TABLE_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.TIMESTAMP_MS)) - .thenReturn(TEST_TIMESTAMP_MS); - when(mockResultSet.getString(ModelCommitMetricsReport.PRINCIPAL_NAME)) - .thenReturn(TEST_PRINCIPAL); - when(mockResultSet.getString(ModelCommitMetricsReport.REQUEST_ID)).thenReturn(TEST_REQUEST_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.OTEL_TRACE_ID)) - .thenReturn(TEST_OTEL_TRACE_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.OTEL_SPAN_ID)) - .thenReturn(TEST_OTEL_SPAN_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.REPORT_TRACE_ID)) - .thenReturn(TEST_REPORT_TRACE_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.SNAPSHOT_ID)).thenReturn(TEST_SNAPSHOT_ID); - when(mockResultSet.getObject(ModelCommitMetricsReport.SEQUENCE_NUMBER, Long.class)) - .thenReturn(TEST_SEQUENCE_NUMBER); - when(mockResultSet.getString(ModelCommitMetricsReport.OPERATION)).thenReturn(TEST_OPERATION); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_DATA_FILES)) - .thenReturn(TEST_ADDED_DATA_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_DATA_FILES)) - .thenReturn(TEST_REMOVED_DATA_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_DATA_FILES)) - .thenReturn(TEST_TOTAL_DATA_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_DELETE_FILES)) - .thenReturn(TEST_ADDED_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_DELETE_FILES)) - .thenReturn(TEST_REMOVED_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_DELETE_FILES)) - .thenReturn(TEST_TOTAL_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_EQUALITY_DELETE_FILES)) - .thenReturn(TEST_ADDED_EQUALITY_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_EQUALITY_DELETE_FILES)) - .thenReturn(TEST_REMOVED_EQUALITY_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_POSITIONAL_DELETE_FILES)) - .thenReturn(TEST_ADDED_POSITIONAL_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_POSITIONAL_DELETE_FILES)) - .thenReturn(TEST_REMOVED_POSITIONAL_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_RECORDS)) - .thenReturn(TEST_ADDED_RECORDS); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_RECORDS)) - .thenReturn(TEST_REMOVED_RECORDS); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_RECORDS)) - .thenReturn(TEST_TOTAL_RECORDS); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_FILE_SIZE_BYTES)) - .thenReturn(TEST_ADDED_FILE_SIZE); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_FILE_SIZE_BYTES)) - .thenReturn(TEST_REMOVED_FILE_SIZE); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_FILE_SIZE_BYTES)) - .thenReturn(TEST_TOTAL_FILE_SIZE); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_DURATION_MS)) - .thenReturn(TEST_TOTAL_DURATION); - when(mockResultSet.getInt(ModelCommitMetricsReport.ATTEMPTS)).thenReturn(TEST_ATTEMPTS); - when(mockResultSet.getString(ModelCommitMetricsReport.METADATA)).thenReturn(TEST_METADATA); - - ModelCommitMetricsReport result = - ModelCommitMetricsReport.CONVERTER.fromResultSet(mockResultSet); - - assertEquals(TEST_REPORT_ID, result.getReportId()); - assertEquals(TEST_REALM_ID, result.getRealmId()); - assertEquals(TEST_CATALOG_ID, result.getCatalogId()); - assertEquals(TEST_TABLE_ID, result.getTableId()); - assertEquals(TEST_TIMESTAMP_MS, result.getTimestampMs()); - assertEquals(TEST_SNAPSHOT_ID, result.getSnapshotId()); - assertEquals(TEST_OPERATION, result.getOperation()); - assertEquals(TEST_ADDED_DATA_FILES, result.getAddedDataFiles()); - assertEquals(TEST_ADDED_RECORDS, result.getAddedRecords()); - assertEquals(TEST_TOTAL_DURATION, result.getTotalDurationMs()); - assertEquals(TEST_ATTEMPTS, result.getAttempts()); - assertEquals(TEST_METADATA, result.getMetadata()); - } - - @Test - public void testToMapWithH2DatabaseType() { - ModelCommitMetricsReport report = createTestReport(); - - Map resultMap = report.toMap(DatabaseType.H2); - - assertEquals(TEST_REPORT_ID, resultMap.get(ModelCommitMetricsReport.REPORT_ID)); - assertEquals(TEST_SNAPSHOT_ID, resultMap.get(ModelCommitMetricsReport.SNAPSHOT_ID)); - assertEquals(TEST_OPERATION, resultMap.get(ModelCommitMetricsReport.OPERATION)); - assertEquals(TEST_ADDED_DATA_FILES, resultMap.get(ModelCommitMetricsReport.ADDED_DATA_FILES)); - assertEquals(TEST_METADATA, resultMap.get(ModelCommitMetricsReport.METADATA)); - // realm_id is not included in toMap() - it's added by the persistence layer - assertNull(resultMap.get(ModelCommitMetricsReport.REALM_ID)); - } - - @Test - public void testToMapWithPostgresType() { - ModelCommitMetricsReport report = createTestReport(); - - Map resultMap = report.toMap(DatabaseType.POSTGRES); - - assertEquals(TEST_REPORT_ID, resultMap.get(ModelCommitMetricsReport.REPORT_ID)); - PGobject pgObject = (PGobject) resultMap.get(ModelCommitMetricsReport.METADATA); - assertEquals("jsonb", pgObject.getType()); - assertEquals(TEST_METADATA, pgObject.getValue()); - } - - @Test - public void testConverterFromResultSet() throws SQLException { - // Test the CONVERTER constant (used in query methods) - ResultSet mockResultSet = mock(ResultSet.class); - when(mockResultSet.getString(ModelCommitMetricsReport.REPORT_ID)).thenReturn(TEST_REPORT_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.REALM_ID)).thenReturn(TEST_REALM_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.CATALOG_ID)).thenReturn(TEST_CATALOG_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.TABLE_ID_COL)).thenReturn(TEST_TABLE_ID); - when(mockResultSet.getLong(ModelCommitMetricsReport.TIMESTAMP_MS)) - .thenReturn(TEST_TIMESTAMP_MS); - when(mockResultSet.getString(ModelCommitMetricsReport.PRINCIPAL_NAME)) - .thenReturn(TEST_PRINCIPAL); - when(mockResultSet.getString(ModelCommitMetricsReport.REQUEST_ID)).thenReturn(TEST_REQUEST_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.OTEL_TRACE_ID)) - .thenReturn(TEST_OTEL_TRACE_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.OTEL_SPAN_ID)) - .thenReturn(TEST_OTEL_SPAN_ID); - when(mockResultSet.getString(ModelCommitMetricsReport.REPORT_TRACE_ID)) - .thenReturn(TEST_REPORT_TRACE_ID); - when(mockResultSet.getObject(ModelCommitMetricsReport.SNAPSHOT_ID, Long.class)) - .thenReturn(TEST_SNAPSHOT_ID); - when(mockResultSet.getObject(ModelCommitMetricsReport.SEQUENCE_NUMBER, Long.class)) - .thenReturn(TEST_SEQUENCE_NUMBER); - when(mockResultSet.getString(ModelCommitMetricsReport.OPERATION)).thenReturn(TEST_OPERATION); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_DATA_FILES)) - .thenReturn(TEST_ADDED_DATA_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_DATA_FILES)) - .thenReturn(TEST_REMOVED_DATA_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_DATA_FILES)) - .thenReturn(TEST_TOTAL_DATA_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_DELETE_FILES)) - .thenReturn(TEST_ADDED_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_DELETE_FILES)) - .thenReturn(TEST_REMOVED_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_DELETE_FILES)) - .thenReturn(TEST_TOTAL_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_EQUALITY_DELETE_FILES)) - .thenReturn(TEST_ADDED_EQUALITY_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_EQUALITY_DELETE_FILES)) - .thenReturn(TEST_REMOVED_EQUALITY_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_POSITIONAL_DELETE_FILES)) - .thenReturn(TEST_ADDED_POSITIONAL_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_POSITIONAL_DELETE_FILES)) - .thenReturn(TEST_REMOVED_POSITIONAL_DELETE_FILES); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_RECORDS)) - .thenReturn(TEST_ADDED_RECORDS); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_RECORDS)) - .thenReturn(TEST_REMOVED_RECORDS); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_RECORDS)) - .thenReturn(TEST_TOTAL_RECORDS); - when(mockResultSet.getLong(ModelCommitMetricsReport.ADDED_FILE_SIZE_BYTES)) - .thenReturn(TEST_ADDED_FILE_SIZE); - when(mockResultSet.getLong(ModelCommitMetricsReport.REMOVED_FILE_SIZE_BYTES)) - .thenReturn(TEST_REMOVED_FILE_SIZE); - when(mockResultSet.getLong(ModelCommitMetricsReport.TOTAL_FILE_SIZE_BYTES)) - .thenReturn(TEST_TOTAL_FILE_SIZE); - when(mockResultSet.getObject(ModelCommitMetricsReport.TOTAL_DURATION_MS, Long.class)) - .thenReturn(TEST_TOTAL_DURATION); - when(mockResultSet.getObject(ModelCommitMetricsReport.ATTEMPTS, Integer.class)) - .thenReturn(TEST_ATTEMPTS); - when(mockResultSet.getString(ModelCommitMetricsReport.METADATA)).thenReturn(TEST_METADATA); - - ModelCommitMetricsReport result = - ModelCommitMetricsReport.CONVERTER.fromResultSet(mockResultSet); - - assertEquals(TEST_REPORT_ID, result.getReportId()); - assertEquals(TEST_REALM_ID, result.getRealmId()); - assertEquals(TEST_CATALOG_ID, result.getCatalogId()); - assertEquals(TEST_METADATA, result.getMetadata()); - } - - private ModelCommitMetricsReport createTestReport() { - return ImmutableModelCommitMetricsReport.builder() - .reportId(TEST_REPORT_ID) - .realmId(TEST_REALM_ID) - .catalogId(TEST_CATALOG_ID) - .tableId(TEST_TABLE_ID) - .timestampMs(TEST_TIMESTAMP_MS) - .principalName(TEST_PRINCIPAL) - .requestId(TEST_REQUEST_ID) - .otelTraceId(TEST_OTEL_TRACE_ID) - .snapshotId(TEST_SNAPSHOT_ID) - .sequenceNumber(TEST_SEQUENCE_NUMBER) - .operation(TEST_OPERATION) - .addedDataFiles(TEST_ADDED_DATA_FILES) - .removedDataFiles(TEST_REMOVED_DATA_FILES) - .totalDataFiles(TEST_TOTAL_DATA_FILES) - .addedDeleteFiles(TEST_ADDED_DELETE_FILES) - .removedDeleteFiles(TEST_REMOVED_DELETE_FILES) - .totalDeleteFiles(TEST_TOTAL_DELETE_FILES) - .addedEqualityDeleteFiles(TEST_ADDED_EQUALITY_DELETE_FILES) - .removedEqualityDeleteFiles(TEST_REMOVED_EQUALITY_DELETE_FILES) - .addedPositionalDeleteFiles(TEST_ADDED_POSITIONAL_DELETE_FILES) - .removedPositionalDeleteFiles(TEST_REMOVED_POSITIONAL_DELETE_FILES) - .addedRecords(TEST_ADDED_RECORDS) - .removedRecords(TEST_REMOVED_RECORDS) - .totalRecords(TEST_TOTAL_RECORDS) - .addedFileSizeBytes(TEST_ADDED_FILE_SIZE) - .removedFileSizeBytes(TEST_REMOVED_FILE_SIZE) - .totalFileSizeBytes(TEST_TOTAL_FILE_SIZE) - .totalDurationMs(TEST_TOTAL_DURATION) - .attempts(TEST_ATTEMPTS) - .metadata(TEST_METADATA) - .build(); - } -} diff --git a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReportTest.java b/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReportTest.java deleted file mode 100644 index f8068d9e041..00000000000 --- a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/models/ModelScanMetricsReportTest.java +++ /dev/null @@ -1,270 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.persistence.relational.jdbc.models; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertNull; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; - -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.Map; -import org.apache.polaris.persistence.relational.jdbc.DatabaseType; -import org.junit.jupiter.api.Test; -import org.postgresql.util.PGobject; - -public class ModelScanMetricsReportTest { - - private static final String TEST_REPORT_ID = "report-123"; - private static final String TEST_REALM_ID = "realm-1"; - private static final long TEST_CATALOG_ID = 12345L; - private static final long TEST_TABLE_ID = 67890L; - private static final long TEST_TIMESTAMP_MS = 1704067200000L; - private static final String TEST_PRINCIPAL = "user@example.com"; - private static final String TEST_REQUEST_ID = "req-456"; - private static final String TEST_OTEL_TRACE_ID = "trace-789"; - private static final String TEST_OTEL_SPAN_ID = "span-012"; - private static final String TEST_REPORT_TRACE_ID = "report-trace-345"; - private static final Long TEST_SNAPSHOT_ID = 123456789L; - private static final Integer TEST_SCHEMA_ID = 1; - private static final String TEST_FILTER = "id > 100"; - private static final String TEST_PROJECTED_IDS = "1,2,3"; - private static final String TEST_PROJECTED_NAMES = "id,name,value"; - private static final long TEST_RESULT_DATA_FILES = 10L; - private static final long TEST_RESULT_DELETE_FILES = 2L; - private static final long TEST_TOTAL_FILE_SIZE = 1024000L; - private static final long TEST_TOTAL_DATA_MANIFESTS = 5L; - private static final long TEST_TOTAL_DELETE_MANIFESTS = 1L; - private static final long TEST_SCANNED_DATA_MANIFESTS = 3L; - private static final long TEST_SCANNED_DELETE_MANIFESTS = 1L; - private static final long TEST_SKIPPED_DATA_MANIFESTS = 2L; - private static final long TEST_SKIPPED_DELETE_MANIFESTS = 0L; - private static final long TEST_SKIPPED_DATA_FILES = 5L; - private static final long TEST_SKIPPED_DELETE_FILES = 0L; - private static final long TEST_PLANNING_DURATION = 150L; - private static final long TEST_EQUALITY_DELETE_FILES = 1L; - private static final long TEST_POSITIONAL_DELETE_FILES = 1L; - private static final long TEST_INDEXED_DELETE_FILES = 0L; - private static final long TEST_DELETE_FILE_SIZE = 2048L; - private static final String TEST_METADATA = "{\"custom\":\"value\"}"; - - @Test - public void testFromResultSet() throws SQLException { - ResultSet mockResultSet = mock(ResultSet.class); - when(mockResultSet.getString(ModelScanMetricsReport.REPORT_ID)).thenReturn(TEST_REPORT_ID); - when(mockResultSet.getString(ModelScanMetricsReport.REALM_ID)).thenReturn(TEST_REALM_ID); - when(mockResultSet.getLong(ModelScanMetricsReport.CATALOG_ID)).thenReturn(TEST_CATALOG_ID); - when(mockResultSet.getLong(ModelScanMetricsReport.TABLE_ID_COL)).thenReturn(TEST_TABLE_ID); - when(mockResultSet.getLong(ModelScanMetricsReport.TIMESTAMP_MS)).thenReturn(TEST_TIMESTAMP_MS); - when(mockResultSet.getString(ModelScanMetricsReport.PRINCIPAL_NAME)).thenReturn(TEST_PRINCIPAL); - when(mockResultSet.getString(ModelScanMetricsReport.REQUEST_ID)).thenReturn(TEST_REQUEST_ID); - when(mockResultSet.getString(ModelScanMetricsReport.OTEL_TRACE_ID)) - .thenReturn(TEST_OTEL_TRACE_ID); - when(mockResultSet.getString(ModelScanMetricsReport.OTEL_SPAN_ID)) - .thenReturn(TEST_OTEL_SPAN_ID); - when(mockResultSet.getString(ModelScanMetricsReport.REPORT_TRACE_ID)) - .thenReturn(TEST_REPORT_TRACE_ID); - when(mockResultSet.getObject(ModelScanMetricsReport.SNAPSHOT_ID, Long.class)) - .thenReturn(TEST_SNAPSHOT_ID); - when(mockResultSet.getObject(ModelScanMetricsReport.SCHEMA_ID, Integer.class)) - .thenReturn(TEST_SCHEMA_ID); - when(mockResultSet.getString(ModelScanMetricsReport.FILTER_EXPRESSION)).thenReturn(TEST_FILTER); - when(mockResultSet.getString(ModelScanMetricsReport.PROJECTED_FIELD_IDS)) - .thenReturn(TEST_PROJECTED_IDS); - when(mockResultSet.getString(ModelScanMetricsReport.PROJECTED_FIELD_NAMES)) - .thenReturn(TEST_PROJECTED_NAMES); - when(mockResultSet.getLong(ModelScanMetricsReport.RESULT_DATA_FILES)) - .thenReturn(TEST_RESULT_DATA_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.RESULT_DELETE_FILES)) - .thenReturn(TEST_RESULT_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_FILE_SIZE_BYTES)) - .thenReturn(TEST_TOTAL_FILE_SIZE); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_DATA_MANIFESTS)) - .thenReturn(TEST_TOTAL_DATA_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_DELETE_MANIFESTS)) - .thenReturn(TEST_TOTAL_DELETE_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SCANNED_DATA_MANIFESTS)) - .thenReturn(TEST_SCANNED_DATA_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SCANNED_DELETE_MANIFESTS)) - .thenReturn(TEST_SCANNED_DELETE_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DATA_MANIFESTS)) - .thenReturn(TEST_SKIPPED_DATA_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DELETE_MANIFESTS)) - .thenReturn(TEST_SKIPPED_DELETE_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DATA_FILES)) - .thenReturn(TEST_SKIPPED_DATA_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DELETE_FILES)) - .thenReturn(TEST_SKIPPED_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_PLANNING_DURATION_MS)) - .thenReturn(TEST_PLANNING_DURATION); - when(mockResultSet.getLong(ModelScanMetricsReport.EQUALITY_DELETE_FILES)) - .thenReturn(TEST_EQUALITY_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.POSITIONAL_DELETE_FILES)) - .thenReturn(TEST_POSITIONAL_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.INDEXED_DELETE_FILES)) - .thenReturn(TEST_INDEXED_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_DELETE_FILE_SIZE_BYTES)) - .thenReturn(TEST_DELETE_FILE_SIZE); - when(mockResultSet.getString(ModelScanMetricsReport.METADATA)).thenReturn(TEST_METADATA); - - ModelScanMetricsReport result = ModelScanMetricsReport.CONVERTER.fromResultSet(mockResultSet); - - assertEquals(TEST_REPORT_ID, result.getReportId()); - assertEquals(TEST_REALM_ID, result.getRealmId()); - assertEquals(TEST_CATALOG_ID, result.getCatalogId()); - assertEquals(TEST_TABLE_ID, result.getTableId()); - assertEquals(TEST_TIMESTAMP_MS, result.getTimestampMs()); - assertEquals(TEST_PRINCIPAL, result.getPrincipalName()); - assertEquals(TEST_REQUEST_ID, result.getRequestId()); - assertEquals(TEST_OTEL_TRACE_ID, result.getOtelTraceId()); - assertEquals(TEST_SNAPSHOT_ID, result.getSnapshotId()); - assertEquals(TEST_RESULT_DATA_FILES, result.getResultDataFiles()); - assertEquals(TEST_TOTAL_FILE_SIZE, result.getTotalFileSizeBytes()); - assertEquals(TEST_PLANNING_DURATION, result.getTotalPlanningDurationMs()); - assertEquals(TEST_METADATA, result.getMetadata()); - } - - @Test - public void testToMapWithH2DatabaseType() { - ModelScanMetricsReport report = createTestReport(); - - Map resultMap = report.toMap(DatabaseType.H2); - - assertEquals(TEST_REPORT_ID, resultMap.get(ModelScanMetricsReport.REPORT_ID)); - assertEquals(TEST_CATALOG_ID, resultMap.get(ModelScanMetricsReport.CATALOG_ID)); - assertEquals(TEST_TABLE_ID, resultMap.get(ModelScanMetricsReport.TABLE_ID_COL)); - assertEquals(TEST_TIMESTAMP_MS, resultMap.get(ModelScanMetricsReport.TIMESTAMP_MS)); - assertEquals(TEST_RESULT_DATA_FILES, resultMap.get(ModelScanMetricsReport.RESULT_DATA_FILES)); - assertEquals(TEST_METADATA, resultMap.get(ModelScanMetricsReport.METADATA)); - // realm_id is not included in toMap() - it's added by the persistence layer - assertNull(resultMap.get(ModelScanMetricsReport.REALM_ID)); - } - - @Test - public void testToMapWithPostgresType() { - ModelScanMetricsReport report = createTestReport(); - - Map resultMap = report.toMap(DatabaseType.POSTGRES); - - assertEquals(TEST_REPORT_ID, resultMap.get(ModelScanMetricsReport.REPORT_ID)); - PGobject pgObject = (PGobject) resultMap.get(ModelScanMetricsReport.METADATA); - assertEquals("jsonb", pgObject.getType()); - assertEquals(TEST_METADATA, pgObject.getValue()); - } - - @Test - public void testConverterFromResultSet() throws SQLException { - // Test the CONVERTER constant (used in query methods) - ResultSet mockResultSet = mock(ResultSet.class); - when(mockResultSet.getString(ModelScanMetricsReport.REPORT_ID)).thenReturn(TEST_REPORT_ID); - when(mockResultSet.getString(ModelScanMetricsReport.REALM_ID)).thenReturn(TEST_REALM_ID); - when(mockResultSet.getLong(ModelScanMetricsReport.CATALOG_ID)).thenReturn(TEST_CATALOG_ID); - when(mockResultSet.getLong(ModelScanMetricsReport.TABLE_ID_COL)).thenReturn(TEST_TABLE_ID); - when(mockResultSet.getLong(ModelScanMetricsReport.TIMESTAMP_MS)).thenReturn(TEST_TIMESTAMP_MS); - when(mockResultSet.getString(ModelScanMetricsReport.PRINCIPAL_NAME)).thenReturn(TEST_PRINCIPAL); - when(mockResultSet.getString(ModelScanMetricsReport.REQUEST_ID)).thenReturn(TEST_REQUEST_ID); - when(mockResultSet.getString(ModelScanMetricsReport.OTEL_TRACE_ID)) - .thenReturn(TEST_OTEL_TRACE_ID); - when(mockResultSet.getString(ModelScanMetricsReport.OTEL_SPAN_ID)) - .thenReturn(TEST_OTEL_SPAN_ID); - when(mockResultSet.getString(ModelScanMetricsReport.REPORT_TRACE_ID)) - .thenReturn(TEST_REPORT_TRACE_ID); - when(mockResultSet.getObject(ModelScanMetricsReport.SNAPSHOT_ID, Long.class)) - .thenReturn(TEST_SNAPSHOT_ID); - when(mockResultSet.getObject(ModelScanMetricsReport.SCHEMA_ID, Integer.class)) - .thenReturn(TEST_SCHEMA_ID); - when(mockResultSet.getString(ModelScanMetricsReport.FILTER_EXPRESSION)).thenReturn(TEST_FILTER); - when(mockResultSet.getString(ModelScanMetricsReport.PROJECTED_FIELD_IDS)) - .thenReturn(TEST_PROJECTED_IDS); - when(mockResultSet.getString(ModelScanMetricsReport.PROJECTED_FIELD_NAMES)) - .thenReturn(TEST_PROJECTED_NAMES); - when(mockResultSet.getLong(ModelScanMetricsReport.RESULT_DATA_FILES)) - .thenReturn(TEST_RESULT_DATA_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.RESULT_DELETE_FILES)) - .thenReturn(TEST_RESULT_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_FILE_SIZE_BYTES)) - .thenReturn(TEST_TOTAL_FILE_SIZE); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_DATA_MANIFESTS)) - .thenReturn(TEST_TOTAL_DATA_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_DELETE_MANIFESTS)) - .thenReturn(TEST_TOTAL_DELETE_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SCANNED_DATA_MANIFESTS)) - .thenReturn(TEST_SCANNED_DATA_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SCANNED_DELETE_MANIFESTS)) - .thenReturn(TEST_SCANNED_DELETE_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DATA_MANIFESTS)) - .thenReturn(TEST_SKIPPED_DATA_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DELETE_MANIFESTS)) - .thenReturn(TEST_SKIPPED_DELETE_MANIFESTS); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DATA_FILES)) - .thenReturn(TEST_SKIPPED_DATA_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.SKIPPED_DELETE_FILES)) - .thenReturn(TEST_SKIPPED_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_PLANNING_DURATION_MS)) - .thenReturn(TEST_PLANNING_DURATION); - when(mockResultSet.getLong(ModelScanMetricsReport.EQUALITY_DELETE_FILES)) - .thenReturn(TEST_EQUALITY_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.POSITIONAL_DELETE_FILES)) - .thenReturn(TEST_POSITIONAL_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.INDEXED_DELETE_FILES)) - .thenReturn(TEST_INDEXED_DELETE_FILES); - when(mockResultSet.getLong(ModelScanMetricsReport.TOTAL_DELETE_FILE_SIZE_BYTES)) - .thenReturn(TEST_DELETE_FILE_SIZE); - when(mockResultSet.getString(ModelScanMetricsReport.METADATA)).thenReturn(TEST_METADATA); - - ModelScanMetricsReport result = ModelScanMetricsReport.CONVERTER.fromResultSet(mockResultSet); - - assertEquals(TEST_REPORT_ID, result.getReportId()); - assertEquals(TEST_REALM_ID, result.getRealmId()); - assertEquals(TEST_CATALOG_ID, result.getCatalogId()); - assertEquals(TEST_METADATA, result.getMetadata()); - } - - private ModelScanMetricsReport createTestReport() { - return ImmutableModelScanMetricsReport.builder() - .reportId(TEST_REPORT_ID) - .realmId(TEST_REALM_ID) - .catalogId(TEST_CATALOG_ID) - .tableId(TEST_TABLE_ID) - .timestampMs(TEST_TIMESTAMP_MS) - .principalName(TEST_PRINCIPAL) - .requestId(TEST_REQUEST_ID) - .otelTraceId(TEST_OTEL_TRACE_ID) - .snapshotId(TEST_SNAPSHOT_ID) - .resultDataFiles(TEST_RESULT_DATA_FILES) - .resultDeleteFiles(TEST_RESULT_DELETE_FILES) - .totalFileSizeBytes(TEST_TOTAL_FILE_SIZE) - .totalDataManifests(TEST_TOTAL_DATA_MANIFESTS) - .totalDeleteManifests(TEST_TOTAL_DELETE_MANIFESTS) - .scannedDataManifests(TEST_SCANNED_DATA_MANIFESTS) - .scannedDeleteManifests(TEST_SCANNED_DELETE_MANIFESTS) - .skippedDataManifests(TEST_SKIPPED_DATA_MANIFESTS) - .skippedDeleteManifests(TEST_SKIPPED_DELETE_MANIFESTS) - .skippedDataFiles(TEST_SKIPPED_DATA_FILES) - .skippedDeleteFiles(TEST_SKIPPED_DELETE_FILES) - .totalPlanningDurationMs(TEST_PLANNING_DURATION) - .equalityDeleteFiles(TEST_EQUALITY_DELETE_FILES) - .positionalDeleteFiles(TEST_POSITIONAL_DELETE_FILES) - .indexedDeleteFiles(TEST_INDEXED_DELETE_FILES) - .totalDeleteFileSizeBytes(TEST_DELETE_FILE_SIZE) - .metadata(TEST_METADATA) - .build(); - } -} diff --git a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java index c016749459b..39c92153c5c 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizableOperation.java @@ -56,6 +56,7 @@ public enum PolarisAuthorizableOperation { VIEW_EXISTS, RENAME_VIEW, REPORT_READ_METRICS, + LIST_TABLE_METRICS, REPORT_WRITE_METRICS, SEND_NOTIFICATIONS, LIST_CATALOGS, diff --git a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java index 25fa6ae774d..0e4fe62801a 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/auth/PolarisAuthorizerImpl.java @@ -98,6 +98,7 @@ import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_MANAGE_GRANTS_ON_SECURABLE; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_MANAGE_STRUCTURE; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_DATA; +import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_METRICS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_PROPERTIES; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PARTITION_SPECS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PROPERTIES; @@ -456,6 +457,9 @@ public class PolarisAuthorizerImpl implements PolarisAuthorizer { SUPER_PRIVILEGES.putAll( TABLE_READ_DATA, List.of(CATALOG_MANAGE_CONTENT, TABLE_READ_DATA, TABLE_WRITE_DATA)); SUPER_PRIVILEGES.putAll(TABLE_WRITE_DATA, List.of(CATALOG_MANAGE_CONTENT, TABLE_WRITE_DATA)); + SUPER_PRIVILEGES.putAll( + TABLE_READ_METRICS, + List.of(CATALOG_MANAGE_CONTENT, TABLE_FULL_METADATA, TABLE_READ_DATA, TABLE_READ_METRICS)); SUPER_PRIVILEGES.putAll( NAMESPACE_FULL_METADATA, List.of(CATALOG_MANAGE_CONTENT, CATALOG_MANAGE_METADATA, NAMESPACE_FULL_METADATA)); diff --git a/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java b/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java index f63999959dd..4880f8dddad 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/auth/RbacOperationSemantics.java @@ -77,6 +77,7 @@ import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_PRINCIPAL_ROLES; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_PRINCIPAL_ROLES_ASSIGNED; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_TABLES; +import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_TABLE_METRICS; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LIST_VIEWS; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LOAD_NAMESPACE_METADATA; import static org.apache.polaris.core.auth.PolarisAuthorizableOperation.LOAD_POLICY; @@ -193,6 +194,7 @@ import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_LIST; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_MANAGE_GRANTS_ON_SECURABLE; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_DATA; +import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_METRICS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_READ_PROPERTIES; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PARTITION_SPECS; import static org.apache.polaris.core.entity.PolarisPrivilege.TABLE_REMOVE_PROPERTIES; @@ -325,6 +327,7 @@ private static void register( // Metrics and notifications register(REPORT_READ_METRICS, TABLE_READ_DATA); + register(LIST_TABLE_METRICS, TABLE_READ_METRICS); register(REPORT_WRITE_METRICS, TABLE_WRITE_DATA); register( SEND_NOTIFICATIONS, diff --git a/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java b/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java index ff7e029ba54..483d18b2e45 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/entity/PolarisPrivilege.java @@ -256,6 +256,15 @@ public enum PolarisPrivilege { PolarisEntityType.TABLE_LIKE, List.of(PolarisEntitySubType.ICEBERG_TABLE, PolarisEntitySubType.GENERIC_TABLE), PolarisEntityType.CATALOG_ROLE), + /** + * Read-only access to table scan and commit metrics reports. Does not grant access to table data. + * Implied by TABLE_READ_DATA and TABLE_FULL_METADATA. + */ + TABLE_READ_METRICS( + 103, + PolarisEntityType.TABLE_LIKE, + List.of(PolarisEntitySubType.ICEBERG_TABLE, PolarisEntitySubType.GENERIC_TABLE), + PolarisEntityType.CATALOG_ROLE), ; /** diff --git a/polaris-core/src/main/java/org/apache/polaris/core/metrics/iceberg/MetricsRecordConverter.java b/polaris-core/src/main/java/org/apache/polaris/core/metrics/iceberg/MetricsRecordConverter.java deleted file mode 100644 index 111f77a286e..00000000000 --- a/polaris-core/src/main/java/org/apache/polaris/core/metrics/iceberg/MetricsRecordConverter.java +++ /dev/null @@ -1,339 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.core.metrics.iceberg; - -import java.time.Instant; -import java.util.Collections; -import java.util.Map; -import java.util.Optional; -import java.util.UUID; -import org.apache.iceberg.metrics.CommitMetricsResult; -import org.apache.iceberg.metrics.CommitReport; -import org.apache.iceberg.metrics.CounterResult; -import org.apache.iceberg.metrics.ScanMetricsResult; -import org.apache.iceberg.metrics.ScanReport; -import org.apache.iceberg.metrics.TimerResult; -import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; -import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; - -/** - * Converts Iceberg metrics reports to SPI record types using a fluent builder API. - * - *

This converter extracts all relevant metrics from Iceberg's {@link ScanReport} and {@link - * CommitReport} and combines them with context information to create persistence-ready records. - * - *

Example usage: - * - * 

{@code
- * ScanMetricsRecord record = MetricsRecordConverter.forScanReport(scanReport)
- *     .catalogId(catalog.getId())
- *     .tableId(tableEntity.getId())
- *     .build();
- * }
- */ -public final class MetricsRecordConverter { - - private MetricsRecordConverter() { - // Utility class - } - - /** - * Creates a builder for converting a ScanReport to a ScanMetricsRecord. - * - * @param scanReport the Iceberg scan report - * @return builder for configuring the conversion - */ - public static ScanReportBuilder forScanReport(ScanReport scanReport) { - return new ScanReportBuilder(scanReport); - } - - /** - * Creates a builder for converting a CommitReport to a CommitMetricsRecord. - * - * @param commitReport the Iceberg commit report - * @return builder for configuring the conversion - */ - public static CommitReportBuilder forCommitReport(CommitReport commitReport) { - return new CommitReportBuilder(commitReport); - } - - /** Well-known metadata key for client-provided report trace ID. */ - public static final String REPORT_TRACE_ID_KEY = "report-trace-id"; - - /** Builder for converting ScanReport to ScanMetricsRecord. */ - public static final class ScanReportBuilder { - private final ScanReport scanReport; - private long catalogId; - private long tableId; - private Instant timestamp; - private String principalName; - private String requestId; - private String otelTraceId; - private String otelSpanId; - - private ScanReportBuilder(ScanReport scanReport) { - this.scanReport = scanReport; - } - - public ScanReportBuilder catalogId(long catalogId) { - this.catalogId = catalogId; - return this; - } - - /** - * Sets the table entity ID. - * - *

This is the internal Polaris entity ID for the table. - * - * @param tableId the table entity ID - * @return this builder - */ - public ScanReportBuilder tableId(long tableId) { - this.tableId = tableId; - return this; - } - - /** - * Sets the timestamp for the metrics record. - * - *

This should be the time the metrics report was received by the server, which may differ - * from the time it was recorded by the client. - * - * @param timestamp the timestamp - * @return this builder - */ - public ScanReportBuilder timestamp(Instant timestamp) { - this.timestamp = timestamp; - return this; - } - - /** Sets the principal name for request context. */ - public ScanReportBuilder principalName(String principalName) { - this.principalName = principalName; - return this; - } - - /** Sets the server-generated request ID. */ - public ScanReportBuilder requestId(String requestId) { - this.requestId = requestId; - return this; - } - - /** Sets the OpenTelemetry trace ID. */ - public ScanReportBuilder otelTraceId(String otelTraceId) { - this.otelTraceId = otelTraceId; - return this; - } - - /** Sets the OpenTelemetry span ID. */ - public ScanReportBuilder otelSpanId(String otelSpanId) { - this.otelSpanId = otelSpanId; - return this; - } - - public ScanMetricsRecord build() { - ScanMetricsResult metrics = scanReport.scanMetrics(); - Map reportMetadata = - scanReport.metadata() != null ? scanReport.metadata() : Collections.emptyMap(); - - return ScanMetricsRecord.builder() - .reportId(UUID.randomUUID().toString()) - .catalogId(catalogId) - .tableId(tableId) - .timestamp(timestamp != null ? timestamp : Instant.now()) - .principalName(principalName) - .requestId(requestId) - .otelTraceId(otelTraceId) - .otelSpanId(otelSpanId) - .snapshotId(Optional.of(scanReport.snapshotId())) - .schemaId(Optional.of(scanReport.schemaId())) - .filterExpression( - scanReport.filter() != null - ? Optional.of(scanReport.filter().toString()) - : Optional.empty()) - .projectedFieldIds( - scanReport.projectedFieldIds() != null - ? scanReport.projectedFieldIds() - : Collections.emptyList()) - .projectedFieldNames( - scanReport.projectedFieldNames() != null - ? scanReport.projectedFieldNames() - : Collections.emptyList()) - .resultDataFiles(getCounterValue(metrics.resultDataFiles())) - .resultDeleteFiles(getCounterValue(metrics.resultDeleteFiles())) - .totalFileSizeBytes(getCounterValue(metrics.totalFileSizeInBytes())) - .totalDataManifests(getCounterValue(metrics.totalDataManifests())) - .totalDeleteManifests(getCounterValue(metrics.totalDeleteManifests())) - .scannedDataManifests(getCounterValue(metrics.scannedDataManifests())) - .scannedDeleteManifests(getCounterValue(metrics.scannedDeleteManifests())) - .skippedDataManifests(getCounterValue(metrics.skippedDataManifests())) - .skippedDeleteManifests(getCounterValue(metrics.skippedDeleteManifests())) - .skippedDataFiles(getCounterValue(metrics.skippedDataFiles())) - .skippedDeleteFiles(getCounterValue(metrics.skippedDeleteFiles())) - .totalPlanningDurationMs(getTimerValueMs(metrics.totalPlanningDuration())) - .equalityDeleteFiles(getCounterValue(metrics.equalityDeleteFiles())) - .positionalDeleteFiles(getCounterValue(metrics.positionalDeleteFiles())) - .indexedDeleteFiles(getCounterValue(metrics.indexedDeleteFiles())) - .totalDeleteFileSizeBytes(getCounterValue(metrics.totalDeleteFileSizeInBytes())) - .metadata(reportMetadata) - .build(); - } - } - - /** Builder for converting CommitReport to CommitMetricsRecord. */ - public static final class CommitReportBuilder { - private final CommitReport commitReport; - private long catalogId; - private long tableId; - private Instant timestamp; - private String principalName; - private String requestId; - private String otelTraceId; - private String otelSpanId; - - private CommitReportBuilder(CommitReport commitReport) { - this.commitReport = commitReport; - } - - public CommitReportBuilder catalogId(long catalogId) { - this.catalogId = catalogId; - return this; - } - - /** - * Sets the table entity ID. - * - *

This is the internal Polaris entity ID for the table. - * - * @param tableId the table entity ID - * @return this builder - */ - public CommitReportBuilder tableId(long tableId) { - this.tableId = tableId; - return this; - } - - /** - * Sets the timestamp for the metrics record. - * - *

This should be the time the metrics report was received by the server, which may differ - * from the time it was recorded by the client. - * - * @param timestamp the timestamp - * @return this builder - */ - public CommitReportBuilder timestamp(Instant timestamp) { - this.timestamp = timestamp; - return this; - } - - /** Sets the principal name for request context. */ - public CommitReportBuilder principalName(String principalName) { - this.principalName = principalName; - return this; - } - - /** Sets the server-generated request ID. */ - public CommitReportBuilder requestId(String requestId) { - this.requestId = requestId; - return this; - } - - /** Sets the OpenTelemetry trace ID. */ - public CommitReportBuilder otelTraceId(String otelTraceId) { - this.otelTraceId = otelTraceId; - return this; - } - - /** Sets the OpenTelemetry span ID. */ - public CommitReportBuilder otelSpanId(String otelSpanId) { - this.otelSpanId = otelSpanId; - return this; - } - - public CommitMetricsRecord build() { - CommitMetricsResult metrics = commitReport.commitMetrics(); - Map reportMetadata = - commitReport.metadata() != null ? commitReport.metadata() : Collections.emptyMap(); - - return CommitMetricsRecord.builder() - .reportId(UUID.randomUUID().toString()) - .catalogId(catalogId) - .tableId(tableId) - .timestamp(timestamp != null ? timestamp : Instant.now()) - .principalName(principalName) - .requestId(requestId) - .otelTraceId(otelTraceId) - .otelSpanId(otelSpanId) - .snapshotId(commitReport.snapshotId()) - .sequenceNumber(Optional.of(commitReport.sequenceNumber())) - .operation(commitReport.operation()) - .addedDataFiles(getCounterValue(metrics.addedDataFiles())) - .removedDataFiles(getCounterValue(metrics.removedDataFiles())) - .totalDataFiles(getCounterValue(metrics.totalDataFiles())) - .addedDeleteFiles(getCounterValue(metrics.addedDeleteFiles())) - .removedDeleteFiles(getCounterValue(metrics.removedDeleteFiles())) - .totalDeleteFiles(getCounterValue(metrics.totalDeleteFiles())) - .addedEqualityDeleteFiles(getCounterValue(metrics.addedEqualityDeleteFiles())) - .removedEqualityDeleteFiles(getCounterValue(metrics.removedEqualityDeleteFiles())) - .addedPositionalDeleteFiles(getCounterValue(metrics.addedPositionalDeleteFiles())) - .removedPositionalDeleteFiles(getCounterValue(metrics.removedPositionalDeleteFiles())) - .addedRecords(getCounterValue(metrics.addedRecords())) - .removedRecords(getCounterValue(metrics.removedRecords())) - .totalRecords(getCounterValue(metrics.totalRecords())) - .addedFileSizeBytes(getCounterValue(metrics.addedFilesSizeInBytes())) - .removedFileSizeBytes(getCounterValue(metrics.removedFilesSizeInBytes())) - .totalFileSizeBytes(getCounterValue(metrics.totalFilesSizeInBytes())) - .totalDurationMs(getTimerValueMsOpt(metrics.totalDuration())) - .attempts(getCounterValueInt(metrics.attempts())) - .metadata(reportMetadata) - .build(); - } - } - - // === Helper Methods === - - private static long getCounterValue(CounterResult counter) { - if (counter == null) { - return 0L; - } - return counter.value(); - } - - private static int getCounterValueInt(CounterResult counter) { - if (counter == null) { - return 0; - } - return (int) counter.value(); - } - - private static long getTimerValueMs(TimerResult timer) { - if (timer == null || timer.totalDuration() == null) { - return 0L; - } - return timer.totalDuration().toMillis(); - } - - private static Optional getTimerValueMsOpt(TimerResult timer) { - if (timer == null || timer.totalDuration() == null) { - return Optional.empty(); - } - return Optional.of(timer.totalDuration().toMillis()); - } -} diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java index 59081179f98..6aaf0dd5bd0 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java @@ -52,11 +52,6 @@ *

Note that APIs to the actual persistence store are very basic, often point read or write to * the underlying data store. The goal is to make it really easy to back this using databases like * Postgres or simpler KV store. - * - *

Metrics-related persistence is intentionally decoupled and lives in {@code - * MetricsPersistence}. A concrete backend may implement both SPIs on the same class, but callers - * that only need metrics persistence should depend on {@code MetricsPersistence} directly rather - * than on {@link BasePersistence}. */ public interface BasePersistence extends PolicyMappingPersistence { /** diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/metrics/MetricsRecordIdentity.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/metrics/MetricsRecordIdentity.java index 7d4e427677b..fa98a537bfa 100644 --- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/metrics/MetricsRecordIdentity.java +++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/metrics/MetricsRecordIdentity.java @@ -26,105 +26,39 @@ /** * Base interface containing common identification fields shared by all metrics records. * - *

This interface defines the common fields that identify the source of a metrics report, - * including the report ID, catalog ID, table ID, timestamp, and metadata. - * *

Both {@link ScanMetricsRecord} and {@link CommitMetricsRecord} extend this interface to * inherit these common fields while adding their own specific metrics. * - *

Design Decisions

- * - *

Entity IDs only (no names): We store only catalog ID and table ID, not their names or - * namespace paths. Names can change over time (via rename operations), which would make querying - * historical metrics by name challenging and lead to correctness issues. Queries should resolve - * names to IDs using the current catalog state. The table ID uniquely identifies the table, and the - * namespace can be derived from the table entity if needed. - * - *

Realm ID: Realm ID is intentionally not included in this interface. Multi-tenancy realm - * context should be obtained from the CDI-injected {@code RealmContext} at persistence time. This - * keeps catalog-specific code from needing to manage realm concerns. - * *

Note: This type is part of the experimental Metrics Persistence SPI and may change in * future releases. */ @Beta public interface MetricsRecordIdentity { - /** - * Unique identifier for this report (UUID). - * - *

This ID is generated when the record is created and serves as the primary key for the - * metrics record in persistence storage. - */ + /** Unique identifier for this report (UUID). */ String reportId(); - /** - * Internal catalog ID. - * - *

This matches the catalog entity ID in Polaris persistence, as defined by {@code - * PolarisEntityCore#getId()}. The catalog name is not stored since it can change over time; - * queries should resolve names to IDs using the current catalog state. - */ + /** Internal catalog ID. */ long catalogId(); - /** - * Internal table entity ID. - * - *

This matches the table entity ID in Polaris persistence, as defined by {@code - * PolarisEntityCore#getId()}. The table name is not stored since it can change over time; queries - * should resolve names to IDs using the current catalog state. The namespace can be derived from - * the table entity if needed. - */ + /** Internal table entity ID. */ long tableId(); - /** - * Timestamp when the report was received. - * - *

This is the server-side timestamp when the metrics report was processed, not the client-side - * timestamp when the operation occurred. - */ + /** Timestamp when the report was received. */ Instant timestamp(); - /** - * Additional metadata as key-value pairs. - * - *

This map can contain additional contextual information from the original Iceberg report, - * including client-provided trace IDs or other correlation data. Persistence implementations can - * store and index specific metadata fields as needed. - */ + /** Additional metadata as key-value pairs. */ Map metadata(); - // === Request Context === - - /** - * Name of the principal who made the request. - * - *

This is resolved from the authenticated principal at the time the metrics report was - * received. May be null if authentication is not enabled or the principal cannot be determined. - */ + /** Name of the principal who made the request. */ @Nullable String principalName(); - /** - * Server-generated request ID for correlation. - * - *

This ID is generated by the server for each incoming request and can be used to correlate - * metrics with server logs and traces. - */ + /** Server-generated request ID for correlation. */ @Nullable String requestId(); - /** - * OpenTelemetry trace ID for distributed tracing correlation. - * - *

When OpenTelemetry is configured, this provides the trace ID from the current span, enabling - * correlation of metrics with distributed traces. - */ + /** OpenTelemetry trace ID for distributed tracing correlation. */ @Nullable String otelTraceId(); - /** - * OpenTelemetry span ID for distributed tracing correlation. - * - *

When OpenTelemetry is configured, this provides the span ID from the current span, enabling - * correlation of metrics with specific spans in a trace. - */ + /** OpenTelemetry span ID for distributed tracing correlation. */ @Nullable String otelSpanId(); } diff --git a/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java b/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java index 14596911fd7..048f9ce8893 100644 --- a/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java +++ b/polaris-core/src/test/java/org/apache/polaris/core/entity/PolarisPrivilegeTest.java @@ -131,7 +131,8 @@ static Stream polarisPrivileges() { Arguments.of(100, PolarisPrivilege.TABLE_REMOVE_STATISTICS), Arguments.of(101, PolarisPrivilege.TABLE_REMOVE_PARTITION_SPECS), Arguments.of(102, PolarisPrivilege.TABLE_MANAGE_STRUCTURE), - Arguments.of(103, null)); + Arguments.of(103, PolarisPrivilege.TABLE_READ_METRICS), + Arguments.of(104, null)); } @ParameterizedTest diff --git a/runtime/defaults/src/main/resources/application-test.properties b/runtime/defaults/src/main/resources/application-test.properties index c8d2d1672db..262b625c87b 100644 --- a/runtime/defaults/src/main/resources/application-test.properties +++ b/runtime/defaults/src/main/resources/application-test.properties @@ -25,6 +25,8 @@ quarkus.http.test-port=0 quarkus.management.test-port=0 +polaris.iceberg-metrics.reporting.type=no-op + quarkus.datasource.devservices.enabled=false quarkus.keycloak.devservices.enabled=false quarkus.mongodb.devservices.enabled=false diff --git a/runtime/defaults/src/main/resources/application.properties b/runtime/defaults/src/main/resources/application.properties index e5b48cda46e..4f322240d02 100644 --- a/runtime/defaults/src/main/resources/application.properties +++ b/runtime/defaults/src/main/resources/application.properties @@ -321,9 +321,9 @@ polaris.oidc.principal-roles-mapper.type=default polaris.credential-manager.type=default # Configuration for the behaviour of the metrics endpoint -polaris.iceberg-metrics.reporting.type=default +polaris.iceberg-metrics.reporting.type=log # Set to INFO if you want to see iceberg metric reports logged -quarkus.log.category."org.apache.polaris.service.reporting".level=OFF +quarkus.log.category."org.apache.polaris.extension.metrics.reports".level=OFF quarkus.arc.ignored-split-packages=\ org.apache.polaris.service.catalog.api,\ diff --git a/runtime/server/build.gradle.kts b/runtime/server/build.gradle.kts index 14f3524d9f2..6ef88525d3f 100644 --- a/runtime/server/build.gradle.kts +++ b/runtime/server/build.gradle.kts @@ -41,6 +41,7 @@ dependencies { runtimeOnly(project(":polaris-extensions-federation-hadoop")) runtimeOnly(project(":polaris-extensions-auth-opa")) runtimeOnly(project(":polaris-extensions-auth-ranger")) + runtimeOnly(project(":polaris-extensions-metrics-reports")) val nonRestCatalogs = providers.gradleProperty("NonRESTCatalogs").orNull if (nonRestCatalogs?.contains("HIVE") == true) { diff --git a/runtime/service/build.gradle.kts b/runtime/service/build.gradle.kts index d19b3d8bd3a..92984d22398 100644 --- a/runtime/service/build.gradle.kts +++ b/runtime/service/build.gradle.kts @@ -30,9 +30,9 @@ dependencies { implementation(project(":polaris-api-management-service")) implementation(project(":polaris-api-iceberg-service")) implementation(project(":polaris-api-catalog-service")) + implementation(project(":polaris-extensions-metrics-reports-spi")) runtimeOnly(project(":polaris-relational-jdbc")) - implementation(project(":polaris-runtime-defaults")) implementation(project(":polaris-runtime-common")) @@ -144,6 +144,8 @@ dependencies { testImplementation("io.quarkus:quarkus-rest-client") testImplementation("io.quarkus:quarkus-rest-client-jackson") testImplementation("io.quarkus:quarkus-jdbc-h2") + // Provides jakarta.ws.rs.ext.RuntimeDelegate needed to build Response objects in plain unit tests + testRuntimeOnly("io.quarkus.resteasy.reactive:resteasy-reactive") testImplementation("io.opentelemetry:opentelemetry-sdk-testing") @@ -171,6 +173,7 @@ dependencies { testImplementation(project(":polaris-persistence-nosql-impl")) testFixturesImplementation(project(":polaris-core")) + testFixturesImplementation(project(":polaris-extensions-metrics-reports-spi")) testFixturesImplementation(project(":polaris-api-management-model")) testFixturesImplementation(project(":polaris-api-management-service")) testFixturesImplementation(project(":polaris-api-iceberg-service")) diff --git a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java index 930ce966723..912b4fe3919 100644 --- a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java +++ b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java @@ -117,6 +117,7 @@ import org.apache.polaris.core.storage.PolarisStorageActions; import org.apache.polaris.core.storage.StorageAccessConfig; import org.apache.polaris.core.storage.StorageUtil; +import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter; import org.apache.polaris.immutables.PolarisImmutable; import org.apache.polaris.service.catalog.AccessDelegationMode; import org.apache.polaris.service.catalog.AccessDelegationModeResolver; @@ -130,7 +131,6 @@ import org.apache.polaris.service.events.EventAttributes; import org.apache.polaris.service.http.IcebergHttpUtil; import org.apache.polaris.service.http.IfNoneMatch; -import org.apache.polaris.service.reporting.PolarisMetricsReporter; import org.apache.polaris.service.types.NotificationRequest; import org.jspecify.annotations.NonNull; import org.jspecify.annotations.Nullable; @@ -209,7 +209,7 @@ public abstract class IcebergCatalogHandler extends CatalogHandler implements Au protected abstract EventAttributeMap eventAttributeMap(); - protected abstract PolarisMetricsReporter metricsReporter(); + protected abstract IcebergMetricsReporter metricsReporter(); protected abstract Clock clock(); diff --git a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerFactory.java b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerFactory.java index 31ba19550a1..39225234d53 100644 --- a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerFactory.java +++ b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerFactory.java @@ -33,12 +33,12 @@ import org.apache.polaris.core.persistence.PolarisMetaStoreManager; import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory; import org.apache.polaris.core.persistence.resolver.ResolverFactory; +import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter; import org.apache.polaris.service.catalog.AccessDelegationModeResolver; import org.apache.polaris.service.catalog.CatalogPrefixParser; import org.apache.polaris.service.catalog.io.StorageAccessConfigProvider; import org.apache.polaris.service.config.ReservedProperties; import org.apache.polaris.service.events.EventAttributeMap; -import org.apache.polaris.service.reporting.PolarisMetricsReporter; @RequestScoped public class IcebergCatalogHandlerFactory { @@ -57,7 +57,7 @@ public class IcebergCatalogHandlerFactory { @Inject @Any Instance federatedCatalogFactories; @Inject StorageAccessConfigProvider storageAccessConfigProvider; @Inject EventAttributeMap eventAttributeMap; - @Inject PolarisMetricsReporter metricsReporter; + @Inject IcebergMetricsReporter metricsReporter; @Inject Clock clock; @Inject AccessDelegationModeResolver accessDelegationModeResolver; diff --git a/runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java b/runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java index be424489437..ac2b5584cb6 100644 --- a/runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java +++ b/runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java @@ -62,6 +62,7 @@ import org.apache.polaris.core.secrets.UserSecretsManagerFactory; import org.apache.polaris.core.storage.cache.StorageCredentialCache; import org.apache.polaris.core.storage.cache.StorageCredentialCacheConfig; +import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter; import org.apache.polaris.service.auth.AuthenticationConfiguration; import org.apache.polaris.service.auth.AuthenticationRealmConfiguration; import org.apache.polaris.service.auth.AuthenticationType; @@ -83,7 +84,6 @@ import org.apache.polaris.service.ratelimiter.TokenBucketConfiguration; import org.apache.polaris.service.ratelimiter.TokenBucketFactory; import org.apache.polaris.service.reporting.MetricsReportingConfiguration; -import org.apache.polaris.service.reporting.PolarisMetricsReporter; import org.apache.polaris.service.secrets.SecretsManagerConfiguration; import org.apache.polaris.service.storage.StorageConfiguration; import org.apache.polaris.service.storage.aws.S3AccessConfig; @@ -458,9 +458,19 @@ public RequestIdSupplier requestIdSupplier() { @Produces @ApplicationScoped - public PolarisMetricsReporter metricsReporter( - MetricsReportingConfiguration config, @Any Instance reporters) { - return reporters.select(Identifier.Literal.of(config.type())).get(); + public IcebergMetricsReporter metricsReporter( + MetricsReportingConfiguration config, @Any Instance reporters) { + var selected = reporters.select(Identifier.Literal.of(config.type())); + if (selected.isUnsatisfied()) { + // NoOpMetricsReporter and LoggingMetricsReporter live in polaris-extensions-metrics-reports + // (not SPI). If that module is absent from the classpath, fall back to a silent no-op so + // core service startup is not blocked by a missing metrics extension. + LOGGER.warn( + "No IcebergMetricsReporter found for type '{}'; Iceberg metrics will be dropped", + config.type()); + return (catalogName, catalogId, table, tableId, metricsReport, receivedTimestamp) -> {}; + } + return selected.get(); } @Produces diff --git a/runtime/service/src/main/java/org/apache/polaris/service/reporting/MetricsReportingConfiguration.java b/runtime/service/src/main/java/org/apache/polaris/service/reporting/MetricsReportingConfiguration.java index 3d60302ab3f..86add73fa4b 100644 --- a/runtime/service/src/main/java/org/apache/polaris/service/reporting/MetricsReportingConfiguration.java +++ b/runtime/service/src/main/java/org/apache/polaris/service/reporting/MetricsReportingConfiguration.java @@ -23,6 +23,6 @@ @ConfigMapping(prefix = "polaris.iceberg-metrics.reporting") public interface MetricsReportingConfiguration { - @WithDefault("default") + @WithDefault("log") String type(); } diff --git a/runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java b/runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java deleted file mode 100644 index 8925213819e..00000000000 --- a/runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java +++ /dev/null @@ -1,152 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.service.reporting; - -import io.opentelemetry.api.trace.Span; -import io.opentelemetry.api.trace.SpanContext; -import io.smallrye.common.annotation.Identifier; -import jakarta.enterprise.context.RequestScoped; -import jakarta.enterprise.inject.Instance; -import jakarta.inject.Inject; -import java.time.Instant; -import org.apache.iceberg.catalog.TableIdentifier; -import org.apache.iceberg.metrics.CommitReport; -import org.apache.iceberg.metrics.MetricsReport; -import org.apache.iceberg.metrics.ScanReport; -import org.apache.polaris.core.auth.PolarisPrincipal; -import org.apache.polaris.core.context.RequestIdSupplier; -import org.apache.polaris.core.metrics.iceberg.MetricsRecordConverter; -import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; -import org.apache.polaris.core.persistence.metrics.MetricsPersistence; -import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Implementation of {@link PolarisMetricsReporter} that persists metrics via the request-scoped - * {@link MetricsPersistence} backend. - * - *

This reporter is selected when {@code polaris.iceberg-metrics.reporting.type} is set to {@code - * "persisting"}. - * - *

If the underlying persistence does not support metrics, they are silently discarded. - * - *

The reporter receives catalog and table IDs from the caller (already resolved during - * authorization), avoiding redundant entity lookups. It uses {@link MetricsRecordConverter} to - * convert Iceberg metrics reports to SPI records before persisting them. - * - * @see PolarisMetricsReporter - * @see MetricsPersistence - * @see MetricsRecordConverter - */ -@RequestScoped -@Identifier("persisting") -public class PersistingMetricsReporter implements PolarisMetricsReporter { - private static final Logger LOGGER = LoggerFactory.getLogger(PersistingMetricsReporter.class); - - private final MetricsPersistence metricsPersistence; - private final Instance polarisPrincipal; - private final Instance requestIdSupplier; - - @Inject - public PersistingMetricsReporter( - MetricsPersistence metricsPersistence, - Instance polarisPrincipal, - Instance requestIdSupplier) { - this.metricsPersistence = metricsPersistence; - this.polarisPrincipal = polarisPrincipal; - this.requestIdSupplier = requestIdSupplier; - } - - @Override - public void reportMetric( - String catalogName, - long catalogId, - TableIdentifier table, - long tableId, - MetricsReport metricsReport, - Instant receivedTimestamp) { - - // Resolve request context - String principalName = resolvePrincipalName(); - String requestId = resolveRequestId(); - String otelTraceId = null; - String otelSpanId = null; - - // Get OpenTelemetry context if available - SpanContext spanContext = Span.current().getSpanContext(); - if (spanContext.isValid()) { - otelTraceId = spanContext.getTraceId(); - otelSpanId = spanContext.getSpanId(); - } - - if (metricsReport instanceof ScanReport scanReport) { - ScanMetricsRecord record = - MetricsRecordConverter.forScanReport(scanReport) - .catalogId(catalogId) - .tableId(tableId) - .timestamp(receivedTimestamp) - .principalName(principalName) - .requestId(requestId) - .otelTraceId(otelTraceId) - .otelSpanId(otelSpanId) - .build(); - metricsPersistence.writeScanReport(record); - LOGGER.debug( - "Persisted scan metrics for {}.{} (reportId={})", catalogName, table, record.reportId()); - } else if (metricsReport instanceof CommitReport commitReport) { - CommitMetricsRecord record = - MetricsRecordConverter.forCommitReport(commitReport) - .catalogId(catalogId) - .tableId(tableId) - .timestamp(receivedTimestamp) - .principalName(principalName) - .requestId(requestId) - .otelTraceId(otelTraceId) - .otelSpanId(otelSpanId) - .build(); - metricsPersistence.writeCommitReport(record); - LOGGER.debug( - "Persisted commit metrics for {}.{} (reportId={})", - catalogName, - table, - record.reportId()); - } else { - LOGGER.warn( - "Unknown metrics report type: {}. Metrics will not be stored.", - metricsReport.getClass().getName()); - } - } - - private String resolvePrincipalName() { - if (polarisPrincipal.isResolvable()) { - PolarisPrincipal principal = polarisPrincipal.get(); - return principal != null ? principal.getName() : null; - } - return null; - } - - private String resolveRequestId() { - if (requestIdSupplier.isResolvable()) { - RequestIdSupplier supplier = requestIdSupplier.get(); - return supplier != null ? supplier.getRequestId() : null; - } - return null; - } -} diff --git a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java index ebf1feac31b..7f02a415a8d 100644 --- a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java +++ b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandlerTest.java @@ -60,12 +60,12 @@ import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory; import org.apache.polaris.core.persistence.resolver.ResolverFactory; import org.apache.polaris.core.storage.StorageAccessConfig; +import org.apache.polaris.extension.metrics.spi.IcebergMetricsReporter; import org.apache.polaris.service.catalog.AccessDelegationModeResolver; import org.apache.polaris.service.catalog.CatalogPrefixParser; import org.apache.polaris.service.catalog.io.StorageAccessConfigProvider; import org.apache.polaris.service.config.ReservedProperties; import org.apache.polaris.service.events.EventAttributeMap; -import org.apache.polaris.service.reporting.PolarisMetricsReporter; import org.junit.jupiter.api.Test; class IcebergCatalogHandlerTest { @@ -126,7 +126,7 @@ private IcebergCatalogHandler newHandler() { .catalogHandlerUtils(mock(CatalogHandlerUtils.class)) .storageAccessConfigProvider(storageAccessConfigProvider) .eventAttributeMap(mock(EventAttributeMap.class)) - .metricsReporter(mock(PolarisMetricsReporter.class)) + .metricsReporter(mock(IcebergMetricsReporter.class)) .clock(mock(Clock.class)) .accessDelegationModeResolver(accessDelegationModeResolver) .build(); diff --git a/runtime/service/src/test/java/org/apache/polaris/service/reporting/PersistingMetricsReporterTest.java b/runtime/service/src/test/java/org/apache/polaris/service/reporting/PersistingMetricsReporterTest.java deleted file mode 100644 index ac76f8dcc1d..00000000000 --- a/runtime/service/src/test/java/org/apache/polaris/service/reporting/PersistingMetricsReporterTest.java +++ /dev/null @@ -1,162 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.polaris.service.reporting; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.ArgumentMatchers.any; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.never; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; - -import jakarta.enterprise.inject.Instance; -import java.time.Instant; -import java.util.Map; -import org.apache.iceberg.catalog.Namespace; -import org.apache.iceberg.catalog.TableIdentifier; -import org.apache.iceberg.expressions.Expressions; -import org.apache.iceberg.metrics.CommitMetrics; -import org.apache.iceberg.metrics.CommitMetricsResult; -import org.apache.iceberg.metrics.CommitReport; -import org.apache.iceberg.metrics.ImmutableCommitReport; -import org.apache.iceberg.metrics.ImmutableScanReport; -import org.apache.iceberg.metrics.MetricsReport; -import org.apache.iceberg.metrics.ScanMetrics; -import org.apache.iceberg.metrics.ScanMetricsResult; -import org.apache.iceberg.metrics.ScanReport; -import org.apache.polaris.core.auth.PolarisPrincipal; -import org.apache.polaris.core.context.RequestIdSupplier; -import org.apache.polaris.core.persistence.metrics.CommitMetricsRecord; -import org.apache.polaris.core.persistence.metrics.MetricsPersistence; -import org.apache.polaris.core.persistence.metrics.ScanMetricsRecord; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.mockito.ArgumentCaptor; - -/** - * Tests for {@link PersistingMetricsReporter}. - * - *

Note: The reporter now receives catalogId and tableId directly from the caller (already - * resolved during authorization in IcebergCatalogHandler), so there's no need to mock entity - * lookups. The reporter uses {@link MetricsPersistence} to persist metrics. - */ -public class PersistingMetricsReporterTest { - - private static final String CATALOG_NAME = "test-catalog"; - private static final long CATALOG_ID = 12345L; - private static final long TABLE_ID = 67890L; - private static final String TABLE_NAME = "test_table"; - private static final TableIdentifier TABLE_IDENTIFIER = - TableIdentifier.of(Namespace.of("db", "schema"), TABLE_NAME); - - private MetricsPersistence metricsPersistence; - private PersistingMetricsReporter reporter; - - @SuppressWarnings("unchecked") - @BeforeEach - void setUp() { - metricsPersistence = mock(MetricsPersistence.class); - - // Mock Instance beans (not resolvable in test context) - Instance principalInstance = mock(Instance.class); - when(principalInstance.isResolvable()).thenReturn(false); - - Instance requestIdInstance = mock(Instance.class); - when(requestIdInstance.isResolvable()).thenReturn(false); - - reporter = - new PersistingMetricsReporter(metricsPersistence, principalInstance, requestIdInstance); - } - - @Test - void testReportScanMetrics() { - // Create a scan report - ScanReport scanReport = createScanReport(); - - // Call the reporter with pre-resolved IDs - reporter.reportMetric( - CATALOG_NAME, CATALOG_ID, TABLE_IDENTIFIER, TABLE_ID, scanReport, Instant.now()); - - // Verify metricsPersistence was called with correct record - ArgumentCaptor captor = ArgumentCaptor.forClass(ScanMetricsRecord.class); - verify(metricsPersistence).writeScanReport(captor.capture()); - - ScanMetricsRecord record = captor.getValue(); - assertThat(record.catalogId()).isEqualTo(CATALOG_ID); - assertThat(record.tableId()).isEqualTo(TABLE_ID); - assertThat(record.reportId()).isNotNull(); - } - - @Test - void testReportCommitMetrics() { - // Create a commit report - CommitReport commitReport = createCommitReport(); - - // Call the reporter with pre-resolved IDs - reporter.reportMetric( - CATALOG_NAME, CATALOG_ID, TABLE_IDENTIFIER, TABLE_ID, commitReport, Instant.now()); - - // Verify metricsPersistence was called with correct record - ArgumentCaptor captor = ArgumentCaptor.forClass(CommitMetricsRecord.class); - verify(metricsPersistence).writeCommitReport(captor.capture()); - - CommitMetricsRecord record = captor.getValue(); - assertThat(record.catalogId()).isEqualTo(CATALOG_ID); - assertThat(record.tableId()).isEqualTo(TABLE_ID); - assertThat(record.reportId()).isNotNull(); - } - - @Test - void testUnknownReportType() { - // Create an unknown report type (using a mock) - MetricsReport unknownReport = mock(MetricsReport.class); - - // Call the reporter - should not throw - reporter.reportMetric( - CATALOG_NAME, CATALOG_ID, TABLE_IDENTIFIER, TABLE_ID, unknownReport, Instant.now()); - - // Verify metricsPersistence was NOT called since report type is unknown - verify(metricsPersistence, never()).writeScanReport(any()); - verify(metricsPersistence, never()).writeCommitReport(any()); - } - - private ScanReport createScanReport() { - return ImmutableScanReport.builder() - .tableName("db.schema.test_table") - .snapshotId(123456789L) - .schemaId(1) - .filter(Expressions.alwaysTrue()) - .scanMetrics(ScanMetricsResult.fromScanMetrics(ScanMetrics.noop())) - .build(); - } - - private CommitReport createCommitReport() { - CommitMetrics commitMetrics = - CommitMetrics.of(new org.apache.iceberg.metrics.DefaultMetricsContext()); - CommitMetricsResult metricsResult = CommitMetricsResult.from(commitMetrics, Map.of()); - - return ImmutableCommitReport.builder() - .tableName("db.schema.test_table") - .snapshotId(987654321L) - .sequenceNumber(5L) - .operation("append") - .commitMetrics(metricsResult) - .build(); - } -} diff --git a/runtime/service/src/testFixtures/java/org/apache/polaris/service/TestServices.java b/runtime/service/src/testFixtures/java/org/apache/polaris/service/TestServices.java index bb4779eedf7..fa730ec0990 100644 --- a/runtime/service/src/testFixtures/java/org/apache/polaris/service/TestServices.java +++ b/runtime/service/src/testFixtures/java/org/apache/polaris/service/TestServices.java @@ -99,7 +99,6 @@ import org.apache.polaris.service.events.listeners.InMemoryEventCollector; import org.apache.polaris.service.identity.provider.DefaultServiceIdentityProvider; import org.apache.polaris.service.persistence.InMemoryPolarisMetaStoreManagerFactory; -import org.apache.polaris.service.reporting.DefaultMetricsReporter; import org.apache.polaris.service.secrets.UnsafeInMemorySecretsManagerFactory; import org.apache.polaris.service.storage.PolarisStorageIntegrationProviderImpl; import org.apache.polaris.service.task.TaskExecutor; @@ -358,7 +357,7 @@ public IcebergCatalogHandler createHandler( .federatedCatalogFactories(federatedCatalogFactory) .storageAccessConfigProvider(storageAccessConfigProvider) .eventAttributeMap(eventAttributeMap) - .metricsReporter(new DefaultMetricsReporter()) + .metricsReporter((_catName, _catId, _table, _tableId, _report, _ts) -> {}) .clock(clock) .accessDelegationModeResolver( new DefaultAccessDelegationModeResolver(realmConfig)) diff --git a/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_iceberg_metrics_reporting.md b/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_iceberg_metrics_reporting.md index eb3a21a0dc8..a118b4720bf 100644 --- a/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_iceberg_metrics_reporting.md +++ b/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_iceberg_metrics_reporting.md @@ -25,4 +25,4 @@ build: | Property | Default Value | Type | Description | |----------|---------------|------|-------------| -| `polaris.iceberg-metrics.reporting.type` | `default` | `string` | | +| `polaris.iceberg-metrics.reporting.type` | `log` | `string` | | diff --git a/site/content/in-dev/unreleased/managing-security/access-control.md b/site/content/in-dev/unreleased/managing-security/access-control.md index 40752651b30..6bb45888649 100644 --- a/site/content/in-dev/unreleased/managing-security/access-control.md +++ b/site/content/in-dev/unreleased/managing-security/access-control.md @@ -115,6 +115,7 @@ To grant the full set of privileges (drop, list, read, write, etc.) on an object | TABLE_WRITE_PROPERTIES | Enables configuring properties for the table. | | TABLE_READ_DATA | Enables reading data from the table by receiving short-lived read-only storage credentials from the catalog. | | TABLE_WRITE_DATA | Enables writing data to the table by receiving short-lived read+write storage credentials from the catalog. | +| TABLE_READ_METRICS | Enables reading persisted Iceberg scan and commit metrics reports for the table via the Metrics Reports API. | | TABLE_FULL_METADATA | Grants all table privileges, except TABLE_READ_DATA and TABLE_WRITE_DATA, which need to be granted individually. | | TABLE_ATTACH_POLICY | Enables attaching policy to a table. | | TABLE_DETACH_POLICY | Enables detaching policy from a table. | diff --git a/site/content/in-dev/unreleased/polaris-api-specs/polaris-metrics-reports-api.md b/site/content/in-dev/unreleased/polaris-api-specs/polaris-metrics-reports-api.md new file mode 100644 index 00000000000..fd6a6a8cb31 --- /dev/null +++ b/site/content/in-dev/unreleased/polaris-api-specs/polaris-metrics-reports-api.md @@ -0,0 +1,27 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: 'Apache Polaris Metrics Reports Service OpenAPI Specification (Experimental)' +linkTitle: 'Metrics Reports API (Experimental) ↗' +weight: 300 +params: + show_page_toc: false +--- + +{{< redoc-polaris "metrics-reports-service.yml" >}} diff --git a/site/content/in-dev/unreleased/telemetry.md b/site/content/in-dev/unreleased/telemetry.md index 59a9b763ee6..b65a6fbce16 100644 --- a/site/content/in-dev/unreleased/telemetry.md +++ b/site/content/in-dev/unreleased/telemetry.md @@ -191,6 +191,43 @@ polaris.log.mdc.region=us-west-2 MDC context is propagated across threads, including in `TaskExecutor` threads. +## Iceberg Metrics Reports API + +> **Experimental**: The Metrics Reports REST API and its related Java types (annotated `@Beta`) are +> experimental. They are subject to change, including **breaking changes**, in any future release +> without prior notice. The `@Beta` label indicates early-access / proof-of-concept status, not +> production stability. Do not depend on this API in production environments until it is declared +> stable. + +Polaris collects Iceberg scan and commit metrics reports submitted by clients and routes them to a +configured reporter. Two built-in reporters are provided: + +| Type | Description | +|------|-------------| +| `log` (default) | Logs each report at INFO level via SLF4J. | +| `no-op` | Silently discards all reports. | + +The active reporter is selected with: + +```properties +polaris.iceberg-metrics.reporting.type=log +``` + +### Durable persistence (extension) + +Durable storage of metrics reports and querying via the REST API at +`/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` requires the +`polaris-extensions-metrics-reports-jdbc` extension. Without it, the query endpoint returns HTTP +501. See the extension documentation for installation and configuration details. + +### Prerequisites for the query API + +The caller must hold the `TABLE_READ_METRICS` privilege on the target table (or a privilege that +implies it, such as `TABLE_READ_DATA`, `TABLE_FULL_METADATA`, or `CATALOG_MANAGE_CONTENT`). + +See the [Metrics Reports API specification]({{% relref "polaris-api-specs/polaris-metrics-reports-api" %}}) +for the full schema reference. + ## Links Visit [Using Polaris with telemetry tools]({{% relref "getting-started/using-polaris/telemetry-tools" %}}) to see sample Polaris config with Prometheus and Jaeger. diff --git a/spec/metrics-reports-service.yml b/spec/metrics-reports-service.yml new file mode 100644 index 00000000000..ff6eb353661 --- /dev/null +++ b/spec/metrics-reports-service.yml @@ -0,0 +1,498 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +openapi: 3.0.3 +info: + title: Apache Polaris Metrics Reports API + description: > + **Experimental / Beta**: Read-only API for querying Iceberg table metrics (scan and commit + reports) from Apache Polaris. + + + **This API is experimental and subject to change, including breaking changes, in any future + release without prior notice. It should not be used in production environments. The "beta" + label indicates early-access status, not stability.** + + + Requires TABLE_READ_METRICS privilege on the target table. Durable persistence backing for this + API is provided by the polaris-extensions-metrics-reports-jdbc extension; without it the + endpoint returns HTTP 501. + version: 0.1.0 + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0.html + +servers: + - url: "{scheme}://{host}/api/metrics-reports/v1" + variables: + scheme: + default: https + host: + default: localhost + +paths: + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}: + parameters: + - $ref: '#/components/parameters/catalogName' + - $ref: '#/components/parameters/namespace' + - $ref: '#/components/parameters/table' + get: + operationId: listTableMetrics + summary: List metrics reports for a table + description: > + Returns persisted metrics reports for the specified table. The required `metricType` + parameter selects between scan reports (produced during table reads) and commit reports + (produced during table writes). Results are ordered by timestamp descending. + Requires TABLE_READ_METRICS privilege on the target table. + tags: + - Metrics + parameters: + - name: metricType + in: query + required: true + description: Type of metrics to retrieve + schema: + type: string + enum: [scan, commit] + - name: pageToken + in: query + required: false + schema: + type: string + description: Opaque cursor from a previous response's nextPageToken field + - name: pageSize + in: query + required: false + schema: + type: integer + minimum: 1 + default: 100 + description: Maximum number of results to return per page + - name: snapshotId + in: query + required: false + schema: + type: integer + format: int64 + description: Filter results to a specific snapshot ID + - name: principalName + in: query + required: false + schema: + type: string + description: Filter results to a specific principal (e.g. service account name) + - name: timestampFrom + in: query + required: false + schema: + type: integer + format: int64 + description: Inclusive lower bound on report timestamp (Unix epoch milliseconds) + - name: timestampTo + in: query + required: false + schema: + type: integer + format: int64 + description: Exclusive upper bound on report timestamp (Unix epoch milliseconds) + responses: + '200': + description: Paginated list of metrics reports + content: + application/json: + schema: + $ref: '#/components/schemas/ListMetricsResponse' + '400': + description: Bad request (missing or invalid parameters) + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + '403': + description: Insufficient privileges (TABLE_READ_METRICS required) + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + '404': + description: Catalog, namespace, or table not found + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + '501': + description: Durable metrics query backing is not available in this deployment + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + +components: + parameters: + catalogName: + name: catalogName + in: path + required: true + schema: + type: string + namespace: + name: namespace + in: path + required: true + description: > + Namespace path encoded using the same convention as the Polaris Iceberg REST API: + multi-level namespaces use the unit separator (0x1F) between levels, URL-encoded as %1F. + schema: + type: string + table: + name: table + in: path + required: true + schema: + type: string + + schemas: + ListMetricsResponse: + description: > + Polymorphic response for metrics queries. The concrete type is determined by the + metricType discriminator field, which echoes the requested metricType query parameter. + oneOf: + - $ref: '#/components/schemas/ListScanMetricsResponse' + - $ref: '#/components/schemas/ListCommitMetricsResponse' + discriminator: + propertyName: metricType + mapping: + scan: '#/components/schemas/ListScanMetricsResponse' + commit: '#/components/schemas/ListCommitMetricsResponse' + + ListScanMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + nullable: true + description: > + Opaque cursor for fetching the next page. Null or absent when no further pages exist. + metricType: + type: string + enum: [scan] + description: Discriminator — always "scan" for this response type + reports: + type: array + items: + $ref: '#/components/schemas/ScanMetricsReport' + + ListCommitMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + nullable: true + description: > + Opaque cursor for fetching the next page. Null or absent when no further pages exist. + metricType: + type: string + enum: [commit] + description: Discriminator — always "commit" for this response type + reports: + type: array + items: + $ref: '#/components/schemas/CommitMetricsReport' + + MetricsActor: + type: object + description: Identity of the principal who triggered the operation + properties: + principalName: + type: string + nullable: true + + MetricsRequest: + type: object + description: Request context for correlation with logs and traces + properties: + requestId: + type: string + nullable: true + otelTraceId: + type: string + nullable: true + description: OpenTelemetry trace ID + otelSpanId: + type: string + nullable: true + description: OpenTelemetry span ID + + ScanMetricsObject: + type: object + description: Resource context for the scanned table operation + properties: + snapshotId: + type: integer + format: int64 + nullable: true + + CommitMetricsObject: + type: object + description: Resource context for the committed table operation + required: + - snapshotId + properties: + snapshotId: + type: integer + format: int64 + + ScanPayloadData: + type: object + description: Iceberg scan metrics data + properties: + schemaId: + type: integer + nullable: true + filterExpression: + type: string + nullable: true + projectedFieldIds: + type: array + nullable: true + items: + type: integer + description: Projected field IDs + projectedFieldNames: + type: array + nullable: true + items: + type: string + description: Projected field names + resultDataFiles: + type: integer + format: int64 + resultDeleteFiles: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDataManifests: + type: integer + format: int64 + totalDeleteManifests: + type: integer + format: int64 + scannedDataManifests: + type: integer + format: int64 + scannedDeleteManifests: + type: integer + format: int64 + skippedDataManifests: + type: integer + format: int64 + skippedDeleteManifests: + type: integer + format: int64 + skippedDataFiles: + type: integer + format: int64 + skippedDeleteFiles: + type: integer + format: int64 + totalPlanningDurationMs: + type: integer + format: int64 + equalityDeleteFiles: + type: integer + format: int64 + positionalDeleteFiles: + type: integer + format: int64 + indexedDeleteFiles: + type: integer + format: int64 + totalDeleteFileSizeBytes: + type: integer + format: int64 + + ScanPayload: + type: object + required: + - type + - version + - data + properties: + type: + type: string + enum: [iceberg.metrics.scan] + version: + type: integer + enum: [1] + data: + $ref: '#/components/schemas/ScanPayloadData' + + CommitPayloadData: + type: object + description: Iceberg commit metrics data + properties: + sequenceNumber: + type: integer + format: int64 + nullable: true + operation: + type: string + description: Commit operation (append, overwrite, delete, replace) + addedDataFiles: + type: integer + format: int64 + removedDataFiles: + type: integer + format: int64 + totalDataFiles: + type: integer + format: int64 + addedDeleteFiles: + type: integer + format: int64 + removedDeleteFiles: + type: integer + format: int64 + totalDeleteFiles: + type: integer + format: int64 + addedEqualityDeleteFiles: + type: integer + format: int64 + removedEqualityDeleteFiles: + type: integer + format: int64 + addedPositionalDeleteFiles: + type: integer + format: int64 + removedPositionalDeleteFiles: + type: integer + format: int64 + addedRecords: + type: integer + format: int64 + removedRecords: + type: integer + format: int64 + totalRecords: + type: integer + format: int64 + addedFileSizeBytes: + type: integer + format: int64 + removedFileSizeBytes: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDurationMs: + type: integer + format: int64 + nullable: true + attempts: + type: integer + + CommitPayload: + type: object + required: + - type + - version + - data + properties: + type: + type: string + enum: [iceberg.metrics.commit] + version: + type: integer + enum: [1] + data: + $ref: '#/components/schemas/CommitPayloadData' + + ScanMetricsReport: + type: object + description: Stable envelope for a persisted Iceberg scan metrics report + required: + - id + - timestampMs + - object + - payload + properties: + id: + type: string + description: Unique identifier for this report + timestampMs: + type: integer + format: int64 + description: Server-side timestamp when the report was received (Unix epoch milliseconds) + actor: + $ref: '#/components/schemas/MetricsActor' + nullable: true + request: + $ref: '#/components/schemas/MetricsRequest' + nullable: true + object: + $ref: '#/components/schemas/ScanMetricsObject' + payload: + $ref: '#/components/schemas/ScanPayload' + + CommitMetricsReport: + type: object + description: Stable envelope for a persisted Iceberg commit metrics report + required: + - id + - timestampMs + - object + - payload + properties: + id: + type: string + description: Unique identifier for this report + timestampMs: + type: integer + format: int64 + description: Server-side timestamp when the report was received (Unix epoch milliseconds) + actor: + $ref: '#/components/schemas/MetricsActor' + nullable: true + request: + $ref: '#/components/schemas/MetricsRequest' + nullable: true + object: + $ref: '#/components/schemas/CommitMetricsObject' + payload: + $ref: '#/components/schemas/CommitPayload' + + ErrorResponse: + type: object + properties: + message: + type: string + type: + type: string + code: + type: integer