fix(crossplane-observability): derive the provider job label from the PodMonitor the chart ships

[TehreemNisa]

What

crossplane.providers.job now defaults to empty, meaning "the PodMonitor this chart ships", and is derived from the rendered object. An explicit value still wins.

Follows #289 (merged, published 0.0.8) — together these are the two chart-side prerequisites for turning the scrape on in the saap-catalog wrapper baseline, so the addon works from a ksp upgrade with no per-cluster values.

Why

The old default was crossplane-providers — a plausible-looking name no PodMonitor can ever produce. Prometheus-operator derives a PodMonitor's job label as <namespace>/<name>, so the chart's own provider scrape lands under crossplane-system/<release>-crossplane-observability-providers, while 11 rules filtered on job="crossplane-providers".

Enabling providerPodMonitor therefore gave you a working scrape and still-empty rules, and nothing reported a problem: the objects are valid, the PromQL parses, the rules evaluate — against no series, forever. This is the same silent-empty failure class as #288 (no Service to select) and #289 (a selector matching no pods), and it is why eu-2's dashboard is empty today.

How

The core half already avoids this by construction: crossplane.core.job names the Service, and prometheus-operator takes a ServiceMonitor's job from the Service — one value, no agreement to maintain. The provider half now works the same way via a crossplane-observability.providersJob helper:

crossplane.providers.job	resolves to
unset (default)	<namespace>/<fullname>-providers — the chart's own PodMonitor
set	verbatim (use when a platform scrape already covers the provider pods — then keep providerPodMonitor off so both don't scrape)

Regression guard

New validate.sh step 1c walks the all-enabled render and asserts every shipped monitor's job label is one the rules actually query — a ServiceMonitor's being the Service name, a PodMonitor's being <namespace>/<name>:

== 1c. scrape/job coherence — the monitors this chart ships match the job labels its rules query ==
job labels coherent: Service ['crossplane'] + PodMonitor ['default/cp-crossplane-observability-providers'] all queried by rules

Verified it fails on the old default before the fix — reinstating job: crossplane-providers gives:

scrape/job MISMATCH:
  - PodMonitor job="default/cp-crossplane-observability-providers" is queried by no rule (crossplane.providers.job disagrees)
  jobs queried by rules: crossplane, crossplane-providers

The scope is deliberately narrow — only the couplings this chart owns — so crossplane.inventory.job (ksm-crossplane, an external exporter) is not flagged.

The promtool unit tests now render with an explicit job, keeping their 16 fixture series independent of the release name: they test rule logic, step 1c tests the wiring.

Verification

./tests/validate.sh fully green — lint, both renders, step 1c, promtool check/test rules (11 groups / 29 rules), metric gate 18/28 captured, dashboard JSON + 39 panel queries, kubeconform strict 25/25.

Not verified live: both cluster tokens are expired this session. The claim being made is about how prometheus-operator names jobs and what this chart renders, both of which are checked in the render — and the live evidence for the trap is already recorded (eu-2's deployed rules query crossplane-system/crossplane-providers-and-functions, us-2's hand-applied PodMonitor name, which the chart never produces).

Follow-up (not in this PR)

The saap-catalog wrapper pins crossplane.providers.job: crossplane-system/crossplane-providers-and-functions — a description of one cluster's hand-applied PodMonitor, wrong on every other cluster. With this merged it can drop the override and enable the chart's own monitors instead.

Refs SA-8539, #283. Follows #288, #289.

🤖 Generated with Claude Code