feat(crossplane-observability): ship the optional core metrics Service

The chart offers `coreServiceMonitor` but never shipped anything for it to
select. Crossplane core serves /metrics with `metrics.enabled=true`, but
upstream ships no Service in front of that port, so on any cluster without a
hand-applied one the monitor matches nothing and every core rule and panel
evaluates against no data — silently.

Found live on eu-2: the addon deployed clean (14 objects, 21 rules health=ok)
and the dashboard was still empty, because eu-2 has no crossplane-metrics
Service, no ServiceMonitor/PodMonitor and no inventory exporter. us-2 only
works because three unowned `kubectl apply`ed platform objects happen to exist
there. Probing the eu-2 core pod IP directly returned HTTP 200 and 1.84 MB of
metrics, so the data was there all along with nothing to scrape it.

Adds `prometheus.monitors.coreMetricsService` (default off), a headless Service
reproducing the object us-2 proved works. Two couplings are derived rather than
restated, so the pair cannot drift apart:

  * the Service name defaults to `crossplane.core.job`, because
    prometheus-operator takes a ServiceMonitor's job label from the Service —
    the same label the alert and recording expressions filter on;
  * its labels are `crossplane.core.serviceMonitorSelector.matchLabels`, so the
    monitor this chart ships always matches the Service it ships.

`selector` stays install-specific and must be set per cluster: the upstream
chart labels core pods `release: <helm release name>`, which is `crossplane` on
eu-2 but `crossplane-operator` on us-2.

Verified: rendered Service is label-, port- and headless-identical to us-2's
working hand-applied one; `kubectl apply --dry-run=server` on eu-2 accepts it
and the default selector matches the live core pod. `./tests/validate.sh` green
(25 objects strict-validated, up from 24; 29 rules; 39 panel queries).

Also documents in docs/values-stakater-cloud-example.yaml that "the cluster
already scrapes Crossplane" is a per-cluster assumption, not a Stakater Cloud
guarantee, plus the PodMonitor job-label trap (`<ns>/<podmonitor-name>`, which
does not match the example's `crossplane.providers.job`).

Refs SA-8539, #283

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: 2 people

crossplane-observability/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: crossplane-observability
 description: A Helm chart for Crossplane observability (ServiceMonitor + PrometheusRule + Grafana dashboard), integrated with OpenShift user-workload monitoring.
 type: application
-version: 0.1.6
+version: 0.1.7
 appVersion: "0.0.1"

crossplane-observability/README.md

@@ -52,6 +52,11 @@ UWM evaluates these `PrometheusRule`s in `thanos-ruler-user-workload`.
 ### 2. Crossplane metrics exposed
 
 - **Core:** start core with `--set metrics.enabled=true` (Helm) so `/metrics` is served.
+  Upstream ships **no Service** in front of that port, so a `ServiceMonitor` has nothing to
+  select. Either point `crossplane.core.serviceMonitorSelector` at a metrics Service your
+  cluster already has, or let this chart create one with
+  `prometheus.monitors.coreMetricsService.enabled=true` (set its `selector` to your core pod
+  labels — the upstream chart labels them `app: crossplane`, `release: <release name>`).
 - **Providers:** expose a metrics port on each provider via its `DeploymentRuntimeConfig`.
   Every crossplane-runtime provider emits `crossplane_managed_resource_*` natively
   (ready/synced/TTR/drift) — no external exporter is needed for leaf-MR health.
@@ -61,6 +66,7 @@ Then turn on the scrapers (both default to `false`):
 ```yaml
 prometheus:
   monitors:
+    coreMetricsService: { enabled: true }   # only if the cluster has no metrics Service
     coreServiceMonitor: { enabled: true }
     providerPodMonitor: { enabled: true }
 ```
@@ -223,6 +229,8 @@ These need cluster/Grafana context an agent can't safely guess:
 | `grafana.compositeAlerts.enabled` | `false` | Story 4.1 composite alerts as **Grafana-managed** rules (use on UWM instead of the PrometheusRule composite alerts). |
 | `grafana.compositeAlerts.datasourceUid` | `""` | **Required when enabled** — UID of the cross-namespace Prometheus/Thanos datasource in Grafana. |
 | `grafana.namespace` | release ns | Namespace to create the GrafanaDashboard in. |
+| `prometheus.monitors.coreMetricsService.enabled` | `false` | Create the headless metrics Service for Crossplane core (upstream ships none, so `coreServiceMonitor` has nothing to select without it). Its name defaults to `crossplane.core.job` — which is what makes the `job` label match — and its labels to `crossplane.core.serviceMonitorSelector.matchLabels`. |
+| `prometheus.monitors.coreMetricsService.selector` | `{app: crossplane, release: crossplane}` | Pod labels of the Crossplane core Deployment. **Install-specific** — the upstream chart sets `release: <helm release name>`. |
 | `prometheus.monitors.coreServiceMonitor.enabled` | `false` | Scrape Crossplane core. |
 | `prometheus.monitors.providerPodMonitor.enabled` | `false` | Scrape provider pods. |
 | `prometheus.recordingRules.enabled` | `true` | Emit the SLO recording rules. |

crossplane-observability/docs/values-stakater-cloud-example.yaml

@@ -6,12 +6,33 @@
 
 # The cluster already scrapes core + providers via its own ServiceMonitor/PodMonitor, so the
 # chart's monitors stay OFF; the rules just point at the existing Prometheus jobs.
+#
+# ⚠️ CONFIRM this per cluster — it is NOT true of every Stakater Cloud cluster. Those
+# monitors and the `crossplane-metrics` Service they select are hand-applied platform
+# objects, not part of any chart, so a cluster can be missing them entirely and every rule
+# and panel then evaluates against nothing. Check with:
+#   kubectl get servicemonitor,podmonitor -A | grep -i crossplane
+#   kubectl get svc -n crossplane-system | grep metrics
+# If they are absent, turn the chart's own scrape on instead (see the commented block
+# below) rather than hand-applying more unowned objects.
 prometheus:
   monitors:
     coreServiceMonitor:
       enabled: false
     providerPodMonitor:
       enabled: false
+    # If the cluster has NO crossplane-metrics Service, uncomment these three blocks. The
+    # Service is named after `crossplane.core.job` below, so the core rules keep matching;
+    # set `selector` to your core pods' labels (the upstream chart uses
+    # `release: <helm release name>` — `crossplane-operator` on us-2, `crossplane` on eu-2)
+    # and override `crossplane.providers.job` to
+    # `crossplane-system/<release>-crossplane-observability-providers`, because a PodMonitor's
+    # job label is `<namespace>/<podmonitor-name>` and will not match the value set below.
+    # coreMetricsService:
+    #   enabled: true
+    #   selector:
+    #     app: crossplane
+    #     release: crossplane
   rules:
     fleet:
       # On UWM, keep the PrometheusRule composite alerts OFF — Grafana handles them

crossplane-observability/templates/prometheus/monitors/core-metrics-service.yaml

@@ -0,0 +1,37 @@
+{{- if and ( .Values.prometheus.monitors.coreMetricsService ) ( .Values.prometheus.monitors.coreMetricsService.enabled ) }}
+{{- $svc := .Values.prometheus.monitors.coreMetricsService }}
+{{- $sm := .Values.prometheus.monitors.coreServiceMonitor | default dict }}
+{{- $sel := .Values.crossplane.core.serviceMonitorSelector | default dict }}
+# Headless Service in front of the Crossplane core pods' /metrics port.
+#
+# Crossplane core serves /metrics (with `metrics.enabled=true`) but upstream ships NO Service
+# for it, so `coreServiceMonitor` selects nothing unless the cluster happens to have one
+# hand-applied. Enable this and the chart owns both halves of the scrape.
+#
+# Two couplings are derived rather than restated, so the pair cannot drift apart:
+#   * the Service NAME becomes the `job` label on every core metric (prometheus-operator
+#     derives a ServiceMonitor's job from the Service), so it defaults to
+#     `crossplane.core.job` — the value the alert and recording expressions filter on;
+#   * its LABELS are the ServiceMonitor's own selector `matchLabels`, so the monitor this
+#     chart ships always matches the Service this chart ships. (Only `matchLabels` is
+#     reflected — a selector written with `matchExpressions` must be matched by hand.)
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ $svc.name | default .Values.crossplane.core.job }}
+  namespace: {{ include "crossplane-observability.namespace" . }}
+  labels:
+    {{- /* the monitor's selector labels win over the chart defaults — see the note above */}}
+    {{- merge (deepCopy ($sel.matchLabels | default dict)) (fromYaml (include "crossplane-observability.labels" .)) | toYaml | nindent 4 }}
+spec:
+  # Headless: this Service exists only to produce Endpoints for the scrape. Nothing dials
+  # it by name, so it needs no ClusterIP.
+  clusterIP: None
+  selector:
+    {{- toYaml $svc.selector | nindent 4 }}
+  ports:
+    - name: {{ $sm.port | default "metrics" }}
+      port: {{ $svc.port }}
+      targetPort: {{ $svc.targetPort }}
+      protocol: TCP
+{{- end }}

crossplane-observability/tests/validate.sh

@@ -24,6 +24,7 @@ green() { printf "\033[32m%s\033[0m\n" "$*"; }
 step()  { printf "\n\033[1m== %s ==\033[0m\n" "$*"; }
 
 ALL_FLAGS=(
+  --set prometheus.monitors.coreMetricsService.enabled=true
   --set prometheus.monitors.coreServiceMonitor.enabled=true
   --set prometheus.monitors.providerPodMonitor.enabled=true
   --set upjet.enabled=true

crossplane-observability/values.yaml

@@ -33,6 +33,8 @@ crossplane:
     # Label selector for the core metrics Service (used only when coreServiceMonitor is
     # enabled). Default targets the metrics Service by component; the upstream metrics
     # Service has no per-instance label, so don't require one.
+    # `prometheus.monitors.coreMetricsService` labels the Service it creates with exactly
+    # these matchLabels, so leaving both at their defaults always pairs.
     serviceMonitorSelector:
       matchLabels:
         app.kubernetes.io/name: crossplane
@@ -148,11 +150,35 @@ grafana:
 # Prometheus configuration
 # ---------------------------------------------------------------------------
 prometheus:
-  # Scrape configuration. Both default to disabled — turn them on once the matching
+  # Scrape configuration. All default to disabled — turn them on once the matching
   # metrics ports are confirmed exposed (see docs/roadmap.md, Decision 2).
   monitors:
+    # Headless Service in front of the core pods' /metrics port. Upstream Crossplane
+    # serves /metrics but ships NO Service for it, so `coreServiceMonitor` below selects
+    # nothing unless the cluster has one hand-applied. Enable this to have the chart own
+    # both halves of the core scrape.
+    coreMetricsService:
+      enabled: false
+      # Service name. Prometheus-operator derives a ServiceMonitor's `job` label from the
+      # Service, so this name IS the job label — it defaults to `crossplane.core.job` so
+      # the alert/recording expressions match without a second setting to keep in sync.
+      # Override only if your Prometheus relabels the job.
+      name: ""
+      # Pod labels of the Crossplane core Deployment. Varies by install: the upstream
+      # chart sets `release: <helm release name>`, so this default fits a release named
+      # `crossplane`. Check with
+      #   kubectl get pod -n <ns> -l app=crossplane --show-labels
+      selector:
+        app: crossplane
+        release: crossplane
+      # Port the core pods serve /metrics on. Crossplane's Deployment does not declare it
+      # as a containerPort, which is fine — a Service may target a raw port number. The
+      # port NAME is taken from `coreServiceMonitor.port` so the two always agree.
+      port: 8080
+      targetPort: 8080
     # Scrapes Crossplane core /metrics (controller-runtime, workqueue, circuit
-    # breaker, function-pipeline metrics).
+    # breaker, function-pipeline metrics). Needs a metrics Service to select — either
+    # `coreMetricsService` above, or one already present on the cluster.
     coreServiceMonitor:
       enabled: false
       port: metrics