diff --git a/AGENTS.md b/AGENTS.md
index f0c52c4053..915c83d269 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -77,7 +77,6 @@ When working on OpenShift:
| **Webhooks** | `./ai-docs/platform/operator-patterns/webhooks.md` | Validation/mutation/conversion |
| **Finalizers** | `./ai-docs/platform/operator-patterns/finalizers.md` | Cleanup external resources on deletion |
| **RBAC** | `./ai-docs/platform/operator-patterns/rbac.md` | Service account permissions |
-| **must-gather** | `./ai-docs/platform/operator-patterns/must-gather.md` | Debugging and diagnostics |
| **Upgrade safety** | `./ai-docs/platform/openshift-specifics/upgrade-strategies.md` | N→N+1 version skew, CVO coordination |
---
@@ -86,7 +85,7 @@ When working on OpenShift:
| Area | Index | Description |
|------|-------|-------------|
-| **Testing** | `./ai-docs/practices/testing/` | Pyramid (60/30/10), e2e framework |
+| **Testing** | `./ai-docs/practices/testing/` | Testing pyramid, e2e framework |
| **Security** | `./ai-docs/practices/security/` | STRIDE, RBAC patterns, secret handling |
| **Reliability** | `./ai-docs/practices/reliability/` | SLI/SLO/SLA, degraded-mode patterns |
| **Development** | `./ai-docs/practices/development/` | API evolution, compatibility |
@@ -121,9 +120,9 @@ When working on OpenShift:
**Index**: See `./ai-docs/decisions/index.md`
**Common ADRs**:
-- Why etcd as backend
- Why CVO orchestration model
- Why immutable nodes (RHCOS + rpm-ostree)
+- Why standardized status conditions pattern
---
@@ -166,8 +165,8 @@ When working on OpenShift:
│ ├── operator-patterns/ # Controller runtime, status, webhooks
│ └── openshift-specifics/ # Upgrade safety, CVO coordination
├── domain/ # Core API concepts
-│ ├── kubernetes/ # Pod, Service, CRD, Node
-│ └── openshift/ # ClusterOperator, ClusterVersion, Machine
+│ ├── kubernetes/ # Pod, Service, CRD
+│ └── openshift/ # ClusterOperator, ClusterVersion
├── practices/ # Cross-cutting concerns
│ ├── testing/ # Pyramid, e2e framework
│ ├── security/ # STRIDE, RBAC, secrets
diff --git a/ai-docs/DESIGN_PHILOSOPHY.md b/ai-docs/DESIGN_PHILOSOPHY.md
index 739716c8ab..cd47ec8ca4 100644
--- a/ai-docs/DESIGN_PHILOSOPHY.md
+++ b/ai-docs/DESIGN_PHILOSOPHY.md
@@ -2,7 +2,7 @@
**Purpose**: Core principles guiding OpenShift architecture and development
-**Last Updated**: YYYY-MM-DD
+**Last Updated**: 2026-06-24
---
diff --git a/ai-docs/KNOWLEDGE_GRAPH.md b/ai-docs/KNOWLEDGE_GRAPH.md
index 2f2b0c7b03..1ab90aa476 100644
--- a/ai-docs/KNOWLEDGE_GRAPH.md
+++ b/ai-docs/KNOWLEDGE_GRAPH.md
@@ -19,9 +19,10 @@
| Task | Start Here | Then Read | Finally |
|------|-----------|----------|---------|
-| **Build an operator** | DESIGN_PHILOSOPHY.md | platform/operator-patterns/controller-runtime.md
platform/operator-patterns/status-conditions.md | domain/openshift/clusteroperator.md |
+| **Build a core operator** | DESIGN_PHILOSOPHY.md | platform/operator-patterns/controller-runtime.md
platform/operator-patterns/status-conditions.md | domain/openshift/clusteroperator.md |
+| **Build an OLM operator** | DESIGN_PHILOSOPHY.md | platform/operator-patterns/controller-runtime.md
platform/operator-patterns/status-conditions.md | [OLM packaging](https://olm.operatorframework.io/docs/tasks/creating-operator-manifests/) |
| **Add a feature** | workflows/index.md (links to enhancement process) | practices/development/index.md (links to API conventions) | practices/testing/index.md |
-| **Debug an issue** | practices/reliability/index.md | platform/operator-patterns/must-gather.md | references/repo-index.md |
+| **Debug an issue** | practices/reliability/index.md | references/repo-index.md | `oc adm must-gather --help` |
| **Understand a concept** | DESIGN_PHILOSOPHY.md | domain/kubernetes/ or domain/openshift/ | platform/operator-patterns/ |
| **Find a component** | references/repo-index.md | Component's AGENTS.md | Component's agentic/ docs |
@@ -69,8 +70,8 @@
## Concept Dependencies
-### Operator Development Path (Core / Platform Operators)
-
+### Operator Development Paths
+
```
DESIGN_PHILOSOPHY.md
↓
@@ -78,11 +79,31 @@ controller-runtime.md (how reconciliation works)
↓
status-conditions.md (how to report health)
↓
-clusteroperator.md (how CVO sees your operator)
- ↓
-webhooks.md, rbac.md, finalizers.md (advanced patterns)
+ ├─── Core / Platform Operators ───────── OLM-Managed Operators ──┐
+ │ │
+ │ clusteroperator.md Your CRD's .status │
+ │ (report to CVO via ClusterOperator) (conditions on your CR) │
+ │ │
+ │ CVO orchestrates upgrades OLM manages lifecycle │
+ │ Available/Progressing/Degraded/ metav1.Condition types │
+ │ Upgradeable CSV defines install/RBAC │
+ │ configv1 condition types │
+ │ │
+ └─────────────────────┬───────────────────────────┘
+ ↓
+ webhooks.md, rbac.md, finalizers.md (advanced patterns)
```
+**Key difference**: Core operators create a `ClusterOperator` resource so the
+CVO can track their health and gate upgrades (the CVO advances when
+`Available=True` and `Degraded=False`; setting `Upgradeable=False` blocks
+minor-version updates). OLM-managed operators report status on their own
+Custom Resource using standard `metav1.Condition` and rely on OLM's
+`ClusterServiceVersion` for lifecycle management. OLM does **not** proxy
+operator health to the CVO — a degraded OLM-managed operator does not block
+cluster upgrades (though a CSV's `olm.maxOpenShiftVersion` annotation can
+block minor-version upgrades as a static compatibility constraint).
+
### API Development Path
```
DESIGN_PHILOSOPHY.md
diff --git a/ai-docs/decisions/adr-0001-cvo-orchestration.md b/ai-docs/decisions/adr-0001-cvo-orchestration.md
new file mode 100644
index 0000000000..129644e4c8
--- /dev/null
+++ b/ai-docs/decisions/adr-0001-cvo-orchestration.md
@@ -0,0 +1,115 @@
+# ADR-0001: CVO Orchestrates Cluster Upgrades
+
+**Status**: Accepted
+**Date**: 2026-06-24
+**Scope**: Cross-repository
+
+## Context
+
+OpenShift clusters contain dozens of operators and platform components that must be upgraded together when moving between versions. These components have ordering dependencies: for example, kube-apiserver must be updated before kube-controller-manager, and core API servers must be available before higher-level operators can reconcile.
+
+The upgrade process must:
+- Handle component interdependencies safely
+- Complete control plane upgrades within a reasonable time period (30m–1h)
+- Avoid disrupting running application workloads
+- Be resilient to failures, including node reboots and pod evictions mid-upgrade
+- Enforce N-1 minor version compatibility between components
+
+## Decision
+
+The Cluster Version Operator (CVO) is the single, centralized orchestrator for all cluster upgrades. It installs and updates all "second-level" operators by applying their manifests from the release image in a deterministic order.
+
+### Ordering model
+
+The CVO loads manifests from the `/release-manifests` directory within the release image. During upgrades, manifests are applied in lexicographic order using a runlevel convention:
+
+```
+0000___
+```
+
+Assigned runlevels (from [CVO dev guide: operators.md](../../dev-guide/cluster-version-operator/dev/operators.md) and [upgrades.md](../../dev-guide/cluster-version-operator/dev/upgrades.md)):
+
+| Runlevel | Components |
+|----------|-----------|
+| 00-04 | CVO itself |
+| 05 | cluster-config-operator |
+| 07 | Network operator |
+| 08 | DNS operator |
+| 09 | Service certificate authority, machine approver |
+| 10-29 | Kubernetes operators (kube-apiserver, kube-controller-manager, kube-scheduler, etcd) |
+| 30-39 | Machine API |
+| 50-59 | Operator Lifecycle Manager (OLM) |
+| 60-69 | OpenShift core operators (openshift-apiserver, etc.) |
+| 70 | Disruptive node-level components, monitoring, samples |
+| 80 | Machine operators |
+
+Components sharing the same runlevel run in parallel (e.g., `0000_70_cluster-monitoring-operator_*` and `0000_70_cluster-samples-operator_*` execute concurrently). Within a component, manifests apply in lexicographic order.
+
+Install flattens inter-runlevel barriers (components across different runlevels can start in parallel), but intra-component ordering is preserved (e.g., a CRD manifest is applied before the Deployment within the same component). Upgrades apply strict runlevel ordering — the next runlevel does not start until the previous one completes.
+
+### Upgrade completion criteria
+
+The CVO uses ClusterOperator status conditions to determine when each component has finished upgrading:
+
+| Operation | Version | Available | Degraded | Progressing | Upgradeable |
+|-----------|---------|-----------|----------|-------------|-------------|
+| Install completion | any | true | any | any | any |
+| Begin patch upgrade | any | any | any | any | any |
+| Begin minor upgrade | any | any | any | any | not false |
+| Upgrade completion | target version (declared in operator status) | true | false | any | any |
+
+### Self-managing design
+
+The CVO is itself a regular pod in the cluster. During upgrades, when the MCO updates the operating system or when the release image updates the CVO image, the CVO gets drained and restarted like any other pod. The new CVO pod observes current cluster state and resumes reconciliation — there is no special handoff mechanism.
+
+This is intentional: by not special-casing its own upgrade, the CVO restart works the same way as it would after a kernel panic, hardware failure, or network partition. The "normal" code path and the "exceptional" path are identical, keeping the upgrade process robust and continuously tested.
+
+## Rationale
+
+- **Deterministic ordering** prevents subtle failures from components upgrading before their prerequisites are ready. Operators that lack sophistication about detecting their own prerequisites get safety from runlevel ordering.
+- **Centralized enforcement** means upgrade safety invariants (version gating, Upgradeable condition checks, override blocking) are enforced in one place rather than distributed across every operator.
+- **Self-managing resilience** ensures the upgrade process is robust against any disruption — the same reconciliation logic handles both normal restarts and failure recovery.
+- **Parallel execution within runlevels** keeps upgrade times reasonable while preserving safety between runlevels.
+
+## Consequences
+
+### Positive
+
+- All component upgrades follow a single, predictable ordering model
+- Upgrade failures are observable through ClusterVersion and ClusterOperator conditions
+- The system self-heals after any disruption during upgrade
+- N-1 version compatibility is enforced across all components
+
+### Negative
+
+- Adding a new component to the payload requires choosing a runlevel and understanding the ordering implications
+- The conservative ordering is sometimes stricter than necessary — most components tolerate version skew during upgrades, but ordering is kept conservative for edge cases
+- Node upgrades currently proceed in the same sequence as control plane upgrades; future improvements may allow independent node upgrade scheduling
+
+## Alternatives Considered
+
+### Alternative 1: OLM-managed upgrades
+
+**Description**: Use the Operator Lifecycle Manager to coordinate all operator upgrades through its dependency resolution system.
+
+**Rejected because**: OLM is itself a second-level operator managed by CVO (runlevel 50–59). It manages optional/add-on operators, not the platform itself. Platform components need to be available before OLM can function, creating a bootstrap dependency that cannot be resolved from within OLM.
+
+### Alternative 2: Distributed coordination
+
+**Description**: Each operator independently detects when its prerequisites are ready and upgrades itself, using leader election and CRD-based signaling.
+
+**Rejected because**: Distributing upgrade coordination across dozens of operators would require every operator to implement prerequisite detection correctly. The CVO's bias toward predictable ordering exists precisely because "operators that lack sophistication about detecting their prerequisites" benefit from centralized ordering. A distributed model trades simplicity for flexibility that is rarely needed.
+
+### Alternative 3: Manual operator ordering
+
+**Description**: Administrators manually trigger component upgrades in the correct order.
+
+**Rejected because**: OpenShift 4's design goal is that clusters "self-manage" by default. Manual ordering is error-prone, scales poorly, and conflicts with the operator-driven automation model.
+
+## References
+
+- Dev guide: [Upgrades and order](../../dev-guide/cluster-version-operator/dev/upgrades.md)
+- Dev guide: [Operator integration with CVO](../../dev-guide/cluster-version-operator/dev/operators.md)
+- Dev guide: [ClusterOperator conditions](../../dev-guide/cluster-version-operator/dev/clusteroperator.md)
+- AI docs: [ClusterVersion resource](../domain/openshift/clusterversion.md)
+- AI docs: [Upgrade strategies](../platform/openshift-specifics/upgrade-strategies.md)
diff --git a/ai-docs/decisions/adr-0002-immutable-nodes.md b/ai-docs/decisions/adr-0002-immutable-nodes.md
new file mode 100644
index 0000000000..ba837b4ff0
--- /dev/null
+++ b/ai-docs/decisions/adr-0002-immutable-nodes.md
@@ -0,0 +1,109 @@
+# ADR-0002: Immutable Node Infrastructure
+
+**Status**: Accepted
+**Date**: 2026-06-24
+**Scope**: Cross-repository
+
+## Context
+
+OpenShift nodes run the kubelet, container runtime (CRI-O), and other system-level services that must be consistent, predictable, and updatable across the cluster. Traditional Linux node management using package managers (yum/dnf) and configuration management tools (Ansible, Puppet) creates challenges:
+
+- **Configuration drift**: Nodes diverge over time due to ad-hoc changes, failed partial updates, or inconsistent package versions
+- **Non-atomic updates**: Package-based updates can fail partway through, leaving the system in an inconsistent state
+- **Rollback difficulty**: Reverting a failed update requires understanding exactly which packages and configs changed
+- **Testing combinatorial explosion**: Every combination of packages, versions, and configuration must be tested independently
+
+## Decision
+
+OpenShift nodes use an immutable infrastructure model built on four components:
+
+1. **RHCOS (Red Hat CoreOS)**: A purpose-built, minimal operating system designed for running containerized workloads. Ships as a single versioned image tested with each OpenShift release.
+
+2. **rpm-ostree**: Provides transactional, atomic operating system updates. The entire OS is deployed as an image-like tree — updates either fully succeed or fully roll back. Extensions (like `kernel-rt` or `usbguard`) can be layered atomically on top of the base image.
+
+3. **Ignition**: Performs one-shot OS configuration at provision time based on the Ignition specification associated with each node's MachineConfigPool. This establishes the initial node state declaratively.
+
+4. **Machine Config Operator (MCO)**: Manages day-2 node configuration changes through MachineConfig resources. The MCO renders desired configuration, and the MachineConfigDaemon (MCD) on each node applies changes by draining the node, applying updates (OS image, files, systemd units), rebooting, and uncordoning.
+
+### Update flow
+
+1. A new MachineConfig is created (by MCO or administrator)
+2. MCO renders the desired configuration for each MachineConfigPool
+3. MCD on each node detects the change
+4. Nodes update one at a time per pool: drain → apply files and OS → reboot → uncordon
+5. If the update fails, rpm-ostree rolls back to the previous OS tree on next boot
+
+### Reboot-by-default principle
+
+All MachineConfig changes require a node drain and reboot by default. This ensures the node reaches a known-good state rather than attempting live patching of a running system. Administrators can opt into specific no-reboot changes for selected files via Node Disruption Policy, but the immutable model remains the foundation.
+
+## Rationale
+
+- **Predictable state**: Every node in a pool runs the same OS image and configuration. There is no accumulated drift from manual edits or imperative commands.
+- **Atomic rollback**: rpm-ostree's transactional model means a failed OS update automatically rolls back to the previous known-good tree. This is not possible with traditional package-by-package updates.
+- **Reduced testing surface**: The entire OS stack (base + extensions) ships as a single version tested with each OpenShift release. All content is versioned with and tested with the OS, eliminating per-package combinatorial testing.
+- **Security**: The read-only root filesystem and image-based deployment reduce the attack surface. Unauthorized modifications to the OS are detectable.
+
+## Consequences
+
+### Positive
+
+- Nodes are reproducible — a new node joining a pool is identical to existing nodes
+- OS updates are atomic and rollback-safe
+- Configuration is declarative and auditable through MachineConfig resources
+- Consistent across the fleet eliminates "works on my node" debugging
+
+### Negative
+
+- All configuration changes require a drain and reboot by default, which causes temporary workload disruption on that node
+- Installing custom packages or agents requires building custom OS layers (on-cluster layering) rather than running `yum install`
+- Ignition's one-shot model means some configuration fields are irreconcilable after initial provisioning — they cannot be changed without reprovisioning the node
+
+### Neutral
+
+- Administrators accustomed to SSH-based node management must adapt to the declarative MachineConfig model
+- Debugging node-level issues uses `oc debug node/` rather than direct SSH access
+
+## Alternatives Considered
+
+### Alternative 1: Traditional package management (yum/dnf on RHEL)
+
+**Description**: Run standard RHEL on nodes, manage packages via yum/dnf, configure via SSH or Ansible.
+
+**Pros**:
+- Familiar to Linux administrators
+- Flexible, can install any package at any time
+- No reboot required for many changes
+
+**Cons**:
+- Configuration drift over time
+- Non-atomic updates can leave nodes in inconsistent state
+- Rollback requires manual intervention
+- Combinatorial testing burden across package versions
+
+**Rejected because**: The operational cost of managing drift and non-atomic updates across a fleet of nodes outweighs the flexibility benefits. OpenShift's operator model depends on predictable, consistent node state.
+
+### Alternative 2: Mutable configuration management (Ansible/Puppet/Chef)
+
+**Description**: Use configuration management tools to continuously converge node state toward a desired specification.
+
+**Pros**:
+- Handles drift through continuous reconciliation
+- Mature tooling ecosystem
+- Does not require reboots
+
+**Cons**:
+- Convergence is not atomic — intermediate states during a run may be inconsistent
+- Agent-based systems add operational complexity
+- Configuration management DSLs add another abstraction layer alongside Kubernetes APIs
+
+**Rejected because**: The Kubernetes operator model already provides declarative reconciliation. Adding a separate configuration management layer for nodes creates a split-brain between Kubernetes-managed and externally-managed state. MCO provides the same continuous reconciliation but integrated into the Kubernetes API model.
+
+## References
+
+- AI docs: [Design Philosophy — Immutable Infrastructure](../DESIGN_PHILOSOPHY.md)
+- AI docs: [Upgrade strategies](../platform/openshift-specifics/upgrade-strategies.md)
+- Enhancement: [Machine Config Node](../../enhancements/machine-config/machine-config-node.md)
+- Enhancement: [RHCOS extensions](../../enhancements/rhcos/extensions.md)
+- Enhancement: [On-cluster layering](../../enhancements/machine-config/on-cluster-layering.md)
+- Enhancement: [Node Disruption Policy](../../enhancements/machine-config/admin-defined-node-disruption-policy.md)
diff --git a/ai-docs/decisions/adr-0003-status-conditions-pattern.md b/ai-docs/decisions/adr-0003-status-conditions-pattern.md
new file mode 100644
index 0000000000..9f22d1f2bc
--- /dev/null
+++ b/ai-docs/decisions/adr-0003-status-conditions-pattern.md
@@ -0,0 +1,157 @@
+# ADR-0003: Standardized Status Conditions Pattern
+
+**Status**: Accepted
+**Date**: 2026-06-24
+**Scope**: Cross-repository
+
+## Context
+
+OpenShift platform operators report health and progress through ClusterOperator resources. The CVO reads these status reports to make upgrade decisions — determining when a component has finished upgrading, whether the cluster is healthy enough to proceed, and whether a minor upgrade should be allowed.
+
+Without a standardized condition scheme:
+- Each operator would report status differently, making cluster-wide health assessment impossible to automate
+- The CVO would need per-operator logic to interpret health signals
+- Administrators would have no consistent way to diagnose operator problems across the platform
+- Monitoring and alerting could not use uniform queries across all operators
+
+## Decision
+
+All ClusterOperators must report three required status conditions: **Available**, **Progressing**, and **Degraded**. Two additional optional conditions are defined: **Upgradeable** and **EvaluationConditionsDetected**.
+
+### Required conditions
+
+**Available**
+- `True`: The component (operator and all configured operands) is functional and available in the cluster
+- `False`: At least part of the component is non-functional and requires immediate administrator intervention
+- A component must **not** report `Available=False` during the course of a normal upgrade
+
+**Progressing**
+- `True`: The component is actively rolling out new code, propagating config changes, or moving from one steady state to another
+- `False`: The component has reached its desired state; no changes are in progress
+- Should not report `Progressing=True` for normal reconciliation without action, or for resource adjustments from node scaling
+- A component in a cluster with fewer than 250 nodes must complete a version change within 20 minutes (90 minutes for MCO, which must restart control plane nodes)
+
+**Degraded**
+- `True`: The component does not match its desired state over a period of time, resulting in lower quality of service
+- `False`: The component is fully healthy
+- Represents a **persistent** observation — should not oscillate in and out of Degraded state
+- A component must **not** report `Degraded=True` during the course of a normal upgrade
+- A component may be both Available and Degraded (e.g., 3 replicas desired, 1 crash-looping — serving requests but impaired)
+
+### Optional conditions
+
+**Upgradeable**
+- `False`: The cluster-version operator will prevent minor OpenShift updates (patch updates are not blocked)
+- The message field should explain what the administrator must do to unblock the upgrade
+- Missing, `True`, or `Unknown` all allow upgrades to proceed
+
+**EvaluationConditionsDetected**
+- Indicates the result of detection logic evaluating the introduction of an invasive change that could cause alerts, breakages, or upgrade failures
+
+### CVO upgrade decision table
+
+| Operation | Version | Available | Degraded | Progressing | Upgradeable |
+|-----------|---------|-----------|----------|-------------|-------------|
+| Install completion | any | true | any | any | any |
+| Begin patch upgrade | any | any | any | any | any |
+| Begin minor upgrade | any | any | any | any | not false |
+| Upgrade completion | target version (declared in operator status) | true | false | any | any |
+
+Install flattens inter-runlevel barriers (components across different runlevels can start in parallel, though intra-component ordering is preserved). Upgrade will not proceed to the next runlevel until the previous runlevel completes (Available=true AND Degraded=false AND target version declared in operator status).
+
+### Message conventions
+
+- **Reason**: Machine-readable CamelCase (e.g., `AsExpected`, `PodsNotReady`, `RollingOut`)
+- **Available message**: Single sentence without punctuation describing what is available
+- **Progressing message**: Terse, 5–10 words describing current state (shown as default column in CLI)
+- **Degraded message**: Detailed (a few sentences at most) explanation sufficient for triage
+- All messages start with a capital letter
+- Operators should set reason and message for both happy and sad conditions (e.g., `AsExpected` / `All is well` for healthy state)
+
+### Example
+
+```yaml
+apiVersion: config.openshift.io/v1
+kind: ClusterOperator
+metadata:
+ name: authentication
+status:
+ conditions:
+ - type: Available
+ status: "True"
+ reason: AsExpected
+ message: "Cluster has deployed 4.17.0"
+ - type: Progressing
+ status: "False"
+ reason: AsExpected
+ message: "Cluster version is 4.17.0"
+ - type: Degraded
+ status: "False"
+ reason: AsExpected
+```
+
+## Rationale
+
+- **Three orthogonal concerns**: Available answers "can you use it now?", Progressing answers "is something changing?", Degraded answers "is there a persistent problem?" This captures all meaningful operator states without an explosion of condition types.
+- **CVO automation**: The CVO can make upgrade decisions using a simple, uniform table across all operators without per-operator interpretation logic.
+- **Consistent observability**: Administrators and monitoring systems use the same condition types across all operators. Prometheus queries like `cluster_operator_conditions{condition="Degraded"}` work uniformly.
+- **Explicit happy reasons** (`AsExpected`) enable aggregation queries that distinguish healthy from unhealthy without mixing in time-series values.
+
+## Consequences
+
+### Positive
+
+- Cluster-wide health is assessable through a single `oc get clusteroperators` command
+- Upgrade decisions are automated and consistent
+- Monitoring and alerting use uniform queries across all platform operators
+- New operators get a clear contract for status reporting
+
+### Negative
+
+- The three-condition model may not capture every nuance of an operator's state — operators sometimes add custom conditions for component-specific signals
+- Time bounds on Progressing (20 minutes / 90 minutes) can trigger false alerts if an operator's reconciliation is legitimately slow
+- The prohibition against reporting Degraded during normal upgrades requires operators to distinguish upgrade-related transient errors from genuine degradation
+
+### Neutral
+
+- The pattern is specific to ClusterOperator resources; individual operator CRDs may use different condition types for their own operands
+
+## Alternatives Considered
+
+### Alternative 1: Free-form status fields
+
+**Description**: Each operator reports status using whatever fields and values make sense for its domain (e.g., `.status.phase`, `.status.health`, free-text `.status.message`).
+
+**Pros**:
+- Maximum flexibility for each operator
+- No need to agree on shared semantics
+
+**Cons**:
+- CVO cannot automate upgrade decisions without per-operator parsing logic
+- Administrators must learn each operator's status conventions
+- Monitoring cannot use uniform queries
+
+**Rejected because**: The value of standardized conditions grows with the number of operators. OpenShift's platform has dozens of ClusterOperators — free-form status would make automated upgrade orchestration and cluster-wide health assessment impractical.
+
+### Alternative 2: Kubernetes-native conditions without standardization
+
+**Description**: Use the Kubernetes `conditions` array but without requiring specific condition types. Each operator defines its own condition types.
+
+**Pros**:
+- Uses standard Kubernetes API conventions
+- Operators can define domain-specific conditions
+
+**Cons**:
+- No guaranteed condition types means CVO cannot rely on any specific condition existing
+- Cluster-wide queries require knowing each operator's condition vocabulary
+- No uniform contract for upgrade readiness
+
+**Rejected because**: The Kubernetes conditions API provides the mechanism but not the semantics. Standardizing on Available/Progressing/Degraded gives the CVO and administrators a guaranteed, uniform contract while still allowing operators to add custom conditions alongside the required three.
+
+## References
+
+- Dev guide: [ClusterOperator conditions](../../dev-guide/cluster-version-operator/dev/clusteroperator.md)
+- AI docs: [Status conditions pattern](../platform/operator-patterns/status-conditions.md)
+- AI docs: [ClusterOperator resource](../domain/openshift/clusteroperator.md)
+- AI docs: [Design Philosophy — Observability by Default](../DESIGN_PHILOSOPHY.md)
+- API: [ClusterStatusConditionType](https://pkg.go.dev/github.com/openshift/api/config/v1#ClusterStatusConditionType)
diff --git a/ai-docs/decisions/adr-template.md b/ai-docs/decisions/adr-template.md
new file mode 100644
index 0000000000..9dde7e4cd7
--- /dev/null
+++ b/ai-docs/decisions/adr-template.md
@@ -0,0 +1,100 @@
+# ADR-NNNN: [Title]
+
+**Status**: Proposed | Accepted | Deprecated | Superseded by [ADR-XXXX]
+**Date**: YYYY-MM-DD
+**Authors**: @username1, @username2
+**Scope**: Cross-repository
+
+## Context
+
+What is the issue or situation that motivates this decision?
+
+- Background information
+- Current state
+- Problem to solve
+- Constraints (technical, organizational, timeline)
+
+## Decision
+
+What is the decision we're making?
+
+State the decision clearly and concisely. Be specific about:
+- What we will do
+- What we won't do
+- How it will work
+
+### Example
+
+```yaml
+# If the decision involves API or code changes, include examples
+apiVersion: config.openshift.io/v1
+kind: ClusterOperator
+status:
+ conditions:
+ - type: Available
+ status: "True"
+```
+
+## Rationale
+
+Why did we make this decision?
+
+- Benefits of this approach
+- How it solves the problem
+- Why this is better than alternatives
+
+## Consequences
+
+What are the implications of this decision?
+
+### Positive
+
+- Benefit 1
+- Benefit 2
+
+### Negative
+
+- Trade-off 1
+- Trade-off 2
+
+### Neutral
+
+- Change 1 (neither good nor bad, just different)
+
+## Alternatives Considered
+
+What other options did we evaluate?
+
+### Alternative 1: [Name]
+
+**Description**: What this alternative would look like
+
+**Pros**:
+- Pro 1
+- Pro 2
+
+**Cons**:
+- Con 1
+- Con 2
+
+**Rejected because**: Reason for rejection
+
+### Alternative 2: [Name]
+
+...
+
+## Implementation
+
+How will this decision be implemented?
+
+- Changes required (API, operators, components)
+- Migration path (if applicable)
+- Timeline (if applicable)
+- Affected components
+
+## References
+
+- Enhancement: [Link]
+- Related ADRs: [Link]
+- External docs: [Link]
+- Discussions: [Link to GitHub issue/PR]
diff --git a/ai-docs/decisions/index.md b/ai-docs/decisions/index.md
new file mode 100644
index 0000000000..1839ee7119
--- /dev/null
+++ b/ai-docs/decisions/index.md
@@ -0,0 +1,69 @@
+# Architectural Decision Records (ADRs)
+
+Cross-repository architectural decisions that shape OpenShift platform.
+
+## Purpose
+
+ADRs document significant decisions affecting multiple components. They explain:
+- What decision was made
+- Why it was made
+- What alternatives were considered
+- Consequences and trade-offs
+
+## Template
+
+Use [adr-template.md](adr-template.md) for new ADRs.
+
+## Scope
+
+**Include here** (cross-repo decisions):
+- Platform-wide patterns (why etcd, why CVO orchestration)
+- API design principles (why status conditions pattern)
+- Architectural constraints (why immutable nodes)
+
+**Exclude** (component-specific):
+- Component implementation details (belongs in component repo)
+- Technology choices for single component
+- Temporary decisions
+
+## ADR List
+
+- [ADR-0001: CVO Orchestrates Cluster Upgrades](adr-0001-cvo-orchestration.md) — Why a single centralized operator orchestrates all upgrades via runlevel ordering
+- [ADR-0002: Immutable Node Infrastructure](adr-0002-immutable-nodes.md) — Why RHCOS + rpm-ostree + Ignition + MachineConfig instead of traditional package management
+- [ADR-0003: Standardized Status Conditions Pattern](adr-0003-status-conditions-pattern.md) — Why Available/Progressing/Degraded as the uniform operator health contract
+
+## Creating ADRs
+
+```bash
+# Copy template
+cp ai-docs/decisions/adr-template.md ai-docs/decisions/adr-0001-my-decision.md
+
+# Fill in:
+# - Title, Date, Status
+# - Context, Decision, Consequences
+# - Alternatives considered
+
+# Add to this index
+
+# Create PR
+```
+
+## ADR Numbering
+
+- Use 4-digit numbers with leading zeros: `adr-0001-`
+- Sequential numbering (next available number)
+- Include slug in filename: `adr-0001-topic-name.md`
+
+## ADR Status
+
+| Status | Meaning |
+|--------|---------|
+| **Proposed** | Under discussion |
+| **Accepted** | Decision made, being implemented |
+| **Deprecated** | Superseded by another ADR |
+| **Superseded** | Replaced (link to new ADR) |
+
+## Related
+
+- **Pattern**: [Architectural Decision Records](https://adr.github.io/)
+- **Enhancement Process**: [../workflows/enhancement-process.md](../workflows/enhancement-process.md)
diff --git a/ai-docs/domain/kubernetes/crds.md b/ai-docs/domain/kubernetes/crds.md
index 21f590655b..f1a6d9510b 100644
--- a/ai-docs/domain/kubernetes/crds.md
+++ b/ai-docs/domain/kubernetes/crds.md
@@ -105,7 +105,7 @@ versions:
### Conversion Webhook
-**⚠️ OpenShift Practice**: OpenShift operators typically **do not serve multiple API versions simultaneously**, so conversion webhooks are rarely needed. Version transitions happen during OpenShift upgrades (one version at a time), not via runtime conversion.
+**⚠️ OpenShift Practice**: Core platform operators (CVO-managed) typically **do not serve multiple API versions simultaneously**, so conversion webhooks are rarely needed for platform APIs. Version transitions happen during OpenShift upgrades (one version at a time), not via runtime conversion. Note: many OLM-managed/optional operators do use conversion webhooks in practice.
**Contrast with upstream Kubernetes**: The [Kubernetes documentation](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) presents serving multiple versions simultaneously as the standard API evolution pattern, stating "it is perfectly safe for some clients to use the old version while others use the new version." Conversion webhooks enable this multi-version serving model.
@@ -218,7 +218,7 @@ foo 3 3 5m
shortNames: [co] # kubectl get co
```
- **OpenShift API team practice**: Short names should only be used for APIs accessed frequently by the majority of cluster users (e.g., `co` for ClusterOperator, `mc` for MachineConfig). Most operator-specific APIs should **not** define short names to avoid namespace pollution.
+ **OpenShift best practice**: Short names should only be used for APIs accessed frequently by the majority of cluster users (e.g., `co` for ClusterOperator, `mc` for MachineConfig). Most operator-specific APIs should **not** define short names to avoid namespace pollution and potential collisions.
## Common Patterns
diff --git a/ai-docs/domain/openshift/clusteroperator.md b/ai-docs/domain/openshift/clusteroperator.md
index 149786f5ed..32e2b1842c 100644
--- a/ai-docs/domain/openshift/clusteroperator.md
+++ b/ai-docs/domain/openshift/clusteroperator.md
@@ -76,7 +76,15 @@ status:
| **Available** | Component functional | Pods ready, API serving | Pods not ready, API down |
| **Progressing** | Reconciliation in progress | Upgrade/rollout active | Desired state reached |
| **Degraded** | Impaired functionality | Partial failure | Fully healthy |
+
+## Optional Conditions
+
+| Condition | Meaning | True When | False When |
+|-----------|---------|-----------|------------|
| **Upgradeable** | Safe to upgrade | No blockers | Manual intervention needed |
+| **EvaluationConditionsDetected** | Detects impact of invasive changes | Deprecated/risky config detected | No issues detected |
+
+**Note**: `Upgradeable` defaults to allowing upgrades when missing, True, or Unknown — only `False` blocks minor upgrades. `EvaluationConditionsDetected` is set by individual operators on their own ClusterOperator resources to report the results of detection logic for invasive changes; the CVO monitors it generically for metrics but does not exclusively manage it.
See [status-conditions.md](../../platform/operator-patterns/status-conditions.md) for detailed semantics.
@@ -119,47 +127,14 @@ oc get co -o json | jq '.items[] | select(.status.conditions[] | select(.type=="
## Creating ClusterOperator
-```go
-import (
- configv1 "github.com/openshift/api/config/v1"
- metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-)
-
-func ensureClusterOperator(ctx context.Context, client client.Client, name string) error {
- co := &configv1.ClusterOperator{
- ObjectMeta: metav1.ObjectMeta{
- Name: name,
- },
- }
-
- _, err := controllerutil.CreateOrUpdate(ctx, client, co, func() error {
- // Set conditions
- setCondition(&co.Status.Conditions,
- configv1.OperatorAvailable,
- configv1.ConditionTrue,
- "AsExpected",
- "Component is available")
-
- // Set versions
- co.Status.Versions = []configv1.OperandVersion{
- {Name: "operator", Version: os.Getenv("RELEASE_VERSION")},
- }
-
- // Set related objects
- co.Status.RelatedObjects = []configv1.ObjectReference{
- {
- Group: "",
- Resource: "namespaces",
- Name: "my-operator-namespace",
- },
- }
-
- return nil
- })
-
- return err
-}
-```
+**Two-step process** — ClusterOperator has a status subresource (`+kubebuilder:subresource:status`), so the resource and its status must be updated via separate API calls:
+
+1. Create/update the ClusterOperator resource via the main endpoint (`controllerutil.CreateOrUpdate` or `client.Create`)
+2. Update status (conditions, versions, relatedObjects) via `client.Status().Update()` or `client.Status().Patch()`
+
+**Key types**: `configv1.ClusterOperator`, `configv1.OperatorAvailable`, `configv1.ConditionTrue`, `configv1.OperandVersion`, `configv1.ObjectReference` (all in `github.com/openshift/api/config/v1`)
+
+**Caution**: Status mutations inside a `CreateOrUpdate` mutate function are silently discarded by the API server. Always use `Status().Update()` for status changes.
## Versions
diff --git a/ai-docs/domain/openshift/clusterversion.md b/ai-docs/domain/openshift/clusterversion.md
index c56f85204f..a7df48ffb7 100644
--- a/ai-docs/domain/openshift/clusterversion.md
+++ b/ai-docs/domain/openshift/clusterversion.md
@@ -80,14 +80,15 @@ status:
↓
4. CVO applies manifests in runlevel order:
- Runlevel 00-09: Core platform (network, DNS, certs)
- - Runlevel 10-29: Kubernetes operators (API server, controllers, scheduler)
+ - Runlevel 10-29: Kubernetes operators (etcd, API server, controllers, scheduler)
- Runlevel 30+: Other operators (Machine API, OLM, OpenShift core)
- Components at same runlevel apply in parallel
↓
5. CVO waits for each operator to reach expected state:
- - Operator version matches desired version
- - Operator reports Progressing=False
- Operator reports Available=True
+ - Operator reports Degraded=False
+ - Operator version matches desired version
+ - Note: Progressing is NOT checked by CVO for runlevel advancement
↓
6. Once all operators reach expected state:
- CVO sets status.conditions.Progressing=False
@@ -149,7 +150,7 @@ oc logs -n openshift-cluster-version deployment/cluster-version-operator
**What EUS provides:**
- **Extended support lifecycle**: Designated releases receive ~14 months additional support
- **OCP 4.x**: Even-numbered releases (4.8, 4.10, 4.12, 4.14, 4.16, etc.)
- - **OCP 5.x**: Every third release (5.2, 5.5, 5.8, etc.)
+ - **OCP 5.x**: Every third release (5.2, 5.5, 5.8, etc.) — planned, subject to change
- **EUS-to-EUS upgrade path**: Streamlined upgrade with reduced worker node reboots
**What EUS does NOT provide:**
@@ -176,16 +177,22 @@ oc logs -n openshift-cluster-version deployment/cluster-version-operator
4.16.0 → 4.16.1 → 4.16.2
```
-**CVO validates**: Upgrade path exists in Cincinnati graph
+**CVO validates**: Upgrade path exists in the OpenShift Update Service (OSUS, also known as Cincinnati) graph
## Upgrade Conditions
| Condition | Meaning | True When |
|-----------|---------|-----------|
-| **Available** | CVO is functional | CVO pod running |
-| **Progressing** | Upgrade in progress | Operators being updated |
-| **Failing** | Upgrade failed | Operator reconciliation failed |
-| **RetrievedUpdates** | Update graph fetched | Cincinnati API reachable |
+| **Available** | CVO is functional | Desired cluster version reached |
+| **Progressing** | Upgrade in progress | CVO actively applying new version |
+| **Failing** | CVO cannot reconcile cluster to desired release image | Manifest reconciliation or ClusterOperator health check failed |
+| **RetrievedUpdates** | Update graph fetched | OSUS API reachable |
+| **ReleaseAccepted** | Release payload loaded | Payload verified without errors |
+| **Upgradeable** | Safe to upgrade | No in-progress update, no blockers |
+| **ImplicitlyEnabledCapabilities** | Enabled capabilities diverge from spec | Capabilities auto-enabled beyond explicit request |
+| **Invalid** | Error in cluster version | Prevents CVO from taking action |
+
+**Note**: ClusterVersion uses `Failing` (not `Degraded`) to indicate the cluster cannot reach its desired state. `Degraded` is a [ClusterOperator condition](clusteroperator.md), not a ClusterVersion condition.
## History
@@ -211,16 +218,18 @@ status:
CVO applies manifests in lexicographic order by filename during upgrades. Manifests use the naming convention:
`0000___.yaml`
-**Assigned runlevels** (from [CVO dev docs](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/operators.md)):
+**Assigned runlevels** (from [CVO dev docs: operators.md](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/operators.md) and [upgrades.md](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/upgrades.md)):
- **00-04**: CVO itself
- **05**: cluster-config-operator
- **07**: Network operator
- **08**: DNS operator
- **09**: Service certificate authority, machine approver
-- **10-29**: Kubernetes operators (e.g., kube-apiserver, kube-controller-manager, kube-scheduler)
+- **10-29**: Kubernetes operators (e.g., etcd, kube-apiserver, kube-controller-manager, kube-scheduler)
- **30-39**: Machine API
- **50-59**: Operator Lifecycle Manager (OLM)
- **60-69**: OpenShift core operators
+- **70**: Disruptive node-level components, monitoring, samples
+- **80**: Machine operators
**Key behaviors**:
- Components at same runlevel execute in **parallel**
diff --git a/ai-docs/platform/openshift-specifics/index.md b/ai-docs/platform/openshift-specifics/index.md
new file mode 100644
index 0000000000..bd28ce3e15
--- /dev/null
+++ b/ai-docs/platform/openshift-specifics/index.md
@@ -0,0 +1,13 @@
+# OpenShift-Specific Patterns
+
+Patterns unique to OpenShift platform.
+
+## Patterns
+
+- [upgrade-strategies.md](upgrade-strategies.md) - CVO orchestration, N→N+1 version skew, zero-downtime upgrades
+
+## Related
+
+- [Operator Patterns](../operator-patterns/) - Generic operator patterns
+- [ClusterOperator](../../domain/openshift/clusteroperator.md) - Status reporting for platform components
+- [ClusterVersion](../../domain/openshift/clusterversion.md) - Upgrade orchestration
diff --git a/ai-docs/platform/openshift-specifics/upgrade-strategies.md b/ai-docs/platform/openshift-specifics/upgrade-strategies.md
index b8ae36b934..7746a11cf7 100644
--- a/ai-docs/platform/openshift-specifics/upgrade-strategies.md
+++ b/ai-docs/platform/openshift-specifics/upgrade-strategies.md
@@ -25,22 +25,24 @@ OpenShift upgrades are orchestrated by the Cluster Version Operator (CVO). Every
| Phase | CVO Action | Operator Responsibility |
|-------|-----------|------------------------|
-| **Pre-upgrade** | Check Upgradeable=True | Set Upgradeable=False if unsafe |
+| **Pre-upgrade** | Check Upgradeable not False (minor only) | Set Upgradeable=False if unsafe |
| **Upgrade** | Update operator image | Reconcile new version |
-| **Rollout** | Wait for Progressing=False | Update workloads, report progress |
-| **Post-upgrade** | Verify Available=True | Resume normal operation |
+| **Rollout** | Wait for Available=True, Degraded=False, version match | Update workloads, report progress |
+| **Post-upgrade** | All operators at target version | Resume normal operation |
## CVO Orchestration
CVO applies manifests in lexicographic order by filename prefix during upgrades.
-**Runlevel ordering** (from [CVO dev docs](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/operators.md)):
+**Runlevel ordering** (from [CVO dev docs: operators.md](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/operators.md) and [upgrades.md](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/upgrades.md)):
```
Runlevel 00-09: Core platform (CVO, network, DNS, certs)
-Runlevel 10-29: Kubernetes operators (API server, controllers, scheduler)
+Runlevel 10-29: Kubernetes operators (etcd, API server, controllers, scheduler)
Runlevel 30-39: Machine API
Runlevel 50-59: Operator Lifecycle Manager
Runlevel 60-69: OpenShift core operators
+Runlevel 70: Disruptive node-level components, monitoring, samples
+Runlevel 80: Machine operators
```
**Key behaviors**:
@@ -294,14 +296,15 @@ oc get co -o json | jq '.items[] | select(.status.conditions[] | select(.type=="
**Static Pod + PDB Guard Pattern**: All control plane components use this pattern for upgrades:
1. **Static pod** runs the actual component (etcd, kube-apiserver, kube-controller-manager, kube-scheduler)
-2. **Guard pod** (Deployment) mirrors the static pod state
-3. **PodDisruptionBudget** on guard pod prevents node drain if it would violate availability
+2. **Guard pod** (bare Pod pinned to the node via `pod.Spec.NodeName`) mirrors the static pod's health via a readiness probe pointed at the static pod's health endpoint (`/readyz` for kube-apiserver and etcd; `/healthz` for kube-controller-manager and kube-scheduler)
+3. **PodDisruptionBudget** on guard pods (selected by label, `minAvailable: nodes-1`) prevents node drain if it would violate availability
4. Ensures at least N-1 replicas available during node upgrades
**Why this pattern?**
- Control plane runs as static pods (not managed by scheduler)
-- PDB only works on pods managed by controllers (Deployment, StatefulSet)
-- Guard pods enable PDB protection for static pods
+- PDB requires pod objects that can be tracked by the eviction API
+- Guard pods are bare Pods (not Deployments) to avoid depending on the scheduler during control plane degradation
+- The `GuardController` in `library-go` manages guard pod lifecycle across all four operators
## Examples in Components
diff --git a/ai-docs/platform/operator-patterns/controller-runtime.md b/ai-docs/platform/operator-patterns/controller-runtime.md
new file mode 100644
index 0000000000..417fecf553
--- /dev/null
+++ b/ai-docs/platform/operator-patterns/controller-runtime.md
@@ -0,0 +1,223 @@
+# Controller Runtime Pattern
+
+**Category**: Platform Pattern
+**Applies To**: All Operators
+**Last Updated**: 2026-05-26
+**Scope**: All form factors (see HCP note below)
+
+## Overview
+
+The controller-runtime pattern implements the Kubernetes reconciliation loop: continuously compare desired state (spec) with current state (status) and take actions to converge.
+
+**Core Principle**: Watch → Reconcile → Update Status → Repeat
+
+**⚠️ Form Factor Note**: In **Hypershift/HCP**, operators may run in:
+- **Guest cluster**: Same as standalone (operator manages guest workloads)
+- **Management cluster**: Operator manages hosted control plane components
+- **Split across both**: Some controllers in management, some in guest (requires careful design)
+
+See [Hypershift Operator Placement](#hypershift-operator-placement) section below for guidance.
+
+## Key Concepts
+
+- **Reconcile Loop**: Core function that runs when resources change
+- **Watch**: Monitor specific resources for changes (create/update/delete events)
+- **Requeue**: Return from reconcile with delay to retry later
+- **Idempotence**: Reconcile must handle being called multiple times safely
+- **Level-Triggered**: React to current state, not edge events
+
+## Implementation
+
+```go
+import (
+ ctrl "sigs.k8s.io/controller-runtime"
+ "sigs.k8s.io/controller-runtime/pkg/reconcile"
+)
+
+func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+ // 1. Fetch current resource
+ obj := &MyResource{}
+ if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
+ return ctrl.Result{}, client.IgnoreNotFound(err)
+ }
+
+ // 2. Compare desired (obj.Spec) vs current state
+ current := r.getCurrentState()
+
+ // 3. Take action to converge
+ if !stateMatches(obj.Spec, current) {
+ if err := r.reconcileState(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+ }
+
+ // 4. Update status
+ obj.Status.ObservedGeneration = obj.Generation
+ if err := r.Status().Update(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+
+ return ctrl.Result{}, nil
+}
+```
+
+## Best Practices
+
+1. **Idempotent Reconciliation**: Calling reconcile multiple times = same result
+ - Check current state before creating resources
+ - Use `CreateOrUpdate()` instead of `Create()`
+
+2. **Status Updates Separate from Spec**:
+ - Use `Status().Update()` not `Update()`
+ - ObservedGeneration pattern to detect spec changes
+
+3. **Requeue Strategy**:
+ - `Result{Requeue: true}` - retry immediately
+ - `Result{RequeueAfter: 5*time.Minute}` - retry after delay
+ - Return error - exponential backoff
+
+4. **Garbage Collection**:
+ - Use `OwnerReferences` for automatic cleanup
+ - Set `controller: true` for primary owner
+
+## Common Patterns
+
+### Basic Reconcile Structure
+```go
+func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+ // Pattern: Fetch → Check → Act → Status → Requeue
+
+ // 1. Fetch
+ obj := &v1.MyResource{}
+ if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
+ return ctrl.Result{}, client.IgnoreNotFound(err)
+ }
+
+ // 2. Check deletion
+ if !obj.DeletionTimestamp.IsZero() {
+ return r.handleDeletion(ctx, obj)
+ }
+
+ // 3. Reconcile owned resources
+ if err := r.ensureDeployment(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+
+ if err := r.ensureService(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+
+ // 4. Update status
+ return r.updateStatus(ctx, obj)
+}
+```
+
+### Watch Setup
+```go
+func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error {
+ return ctrl.NewControllerManagedBy(mgr).
+ For(&v1.MyResource{}). // Primary resource
+ Owns(&appsv1.Deployment{}). // Owned resources (auto-watch)
+ Watches(
+ &corev1.ConfigMap{}, // External resource
+ handler.EnqueueRequestsFromMapFunc(r.mapConfigMapToOwner),
+ ).
+ Complete(r)
+}
+```
+
+### Error Handling
+```go
+// Transient error - retry with backoff
+if err := r.createPod(ctx, obj); err != nil {
+ return ctrl.Result{}, fmt.Errorf("failed to create pod: %w", err)
+}
+
+// Expected condition - requeue later
+if !r.isReady(obj) {
+ return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+}
+
+// Success - no requeue
+return ctrl.Result{}, nil
+```
+
+## Watches and Events
+
+| Watch Type | When to Use | Example |
+|------------|-------------|---------|
+| `For()` | Primary resource | `.For(&MyResource{})` |
+| `Owns()` | Resources with OwnerReference | `.Owns(&Deployment{})` |
+| `Watches()` | External resources | Cluster-scoped resources, ConfigMaps |
+
+## Examples in Components
+
+| Component | Pattern | Notes |
+|-----------|---------|-------|
+| cluster-version-operator | CVO reconciles ClusterVersion | Orchestrates operator upgrades |
+| machine-api-operator | Machine controller | Creates cloud instances |
+| cluster-network-operator | Network config reconciliation | Multi-resource coordination |
+
+## Antipatterns
+
+❌ **Imperative logic**: Storing state in controller (use cluster state as source of truth)
+❌ **Long-running operations**: Blocking reconcile loop (use requeue instead)
+❌ **Event-driven**: Relying on event order (level-triggered, not edge-triggered)
+❌ **Side effects without idempotency**: Creating resources without checking existence
+
+## Hypershift Operator Placement
+
+In **Hypershift/HCP**, operators run in different locations depending on what they manage.
+
+### Decision Matrix
+
+| Operator Manages | Runs In | Example | Why |
+|------------------|---------|---------|-----|
+| **Control plane components** | Management cluster | kube-apiserver-operator, etcd-operator | Control plane pods run in management cluster |
+| **Guest workloads** | Guest cluster | openshift-monitoring (workload metrics) | Monitors workloads running in guest cluster |
+| **Platform infrastructure** | Management cluster | HyperShift Operator | Orchestrates HostedCluster lifecycle |
+| **Both control plane and guest** | Split deployment | cluster-network-operator | Control plane networking in mgmt, pod networking in guest |
+
+### Design Considerations
+
+1. **Where does the operator watch resources?**
+ - Management cluster: `HostedCluster`, `HostedControlPlane`, control plane pods
+ - Guest cluster: `ClusterOperator`, workload resources (Pods, Deployments)
+
+2. **Version skew tolerance**
+ - Management cluster operator may be newer than guest cluster
+ - Must tolerate N→N+1 version differences
+
+3. **RBAC and permissions**
+ - Management cluster: Needs RBAC for `HostedCluster`, infrastructure resources
+ - Guest cluster: Same RBAC as standalone operator
+
+4. **Cross-cluster communication**
+ - If operator in management needs to watch guest resources → requires KAS access
+ - If operator in guest needs to watch management resources → rarely needed, avoid
+
+### Testing HCP Operators
+
+```bash
+# Check where operator runs
+oc --context=management get pods -A | grep my-operator
+oc --context=guest get pods -A | grep my-operator
+
+# Check what operator watches
+oc --context=management logs -n my-namespace my-operator-pod | grep "Watching"
+
+# Verify RBAC in correct cluster
+oc --context=management get clusterrole my-operator-role
+```
+
+See [topology-considerations-guide.md](../../workflows/topology-considerations-guide.md) for comprehensive HCP guidance.
+
+## References
+
+- **Upstream**: [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime)
+- **Book**: [Kubebuilder Book - Controllers](https://book.kubebuilder.io/cronjob-tutorial/controller-implementation.html)
+- **OpenShift**: [status-conditions.md](./status-conditions.md)
+- **Pattern**: Implements "Desired State" from [DESIGN_PHILOSOPHY.md](../../DESIGN_PHILOSOPHY.md)
+- **HCP Enhancements** (authoritative sources for HCP operator placement):
+ - [hypershift-control-plane-version-status.md](../../../enhancements/hypershift/hypershift-control-plane-version-status.md) - CPO in management, CVO in guest
+ - [node-tuning.md](../../../enhancements/hypershift/node-tuning.md) - Example of split operator pattern
diff --git a/ai-docs/platform/operator-patterns/finalizers.md b/ai-docs/platform/operator-patterns/finalizers.md
new file mode 100644
index 0000000000..52706ae00d
--- /dev/null
+++ b/ai-docs/platform/operator-patterns/finalizers.md
@@ -0,0 +1,277 @@
+# Finalizers Pattern
+
+**Category**: Platform Pattern
+**Applies To**: Operators managing external resources
+**Last Updated**: 2026-04-29
+
+## Overview
+
+Finalizers enable cleanup of external resources before a Kubernetes object is deleted. They block deletion until the finalizer is removed, allowing controllers to perform cleanup logic.
+
+**Use Case**: Delete external resources (cloud VMs, load balancers) when Kubernetes object is deleted.
+
+## Key Concepts
+
+- **Finalizer**: String in `metadata.finalizers` that blocks deletion
+- **DeletionTimestamp**: Set when object is marked for deletion
+- **Cleanup Logic**: Controller removes external resources, then removes finalizer
+- **Blocking Deletion**: Object stays in "deleting" state until finalizers removed
+
+## Lifecycle
+
+```
+1. User creates object
+ → Controller adds finalizer
+
+2. User deletes object
+ → DeletionTimestamp set (object marked for deletion)
+ → Object still exists (blocked by finalizer)
+
+3. Controller sees DeletionTimestamp
+ → Cleanup external resources
+ → Remove finalizer
+
+4. No finalizers remain
+ → Kubernetes deletes object from etcd
+```
+
+## Implementation
+
+### Adding Finalizer
+
+```go
+import (
+ "sigs.k8s.io/controller-runtime/pkg/controllerutil"
+)
+
+const finalizerName = "myoperator.example.com/finalizer"
+
+func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+ obj := &v1.MyResource{}
+ if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
+ return ctrl.Result{}, client.IgnoreNotFound(err)
+ }
+
+ // Add finalizer if not present
+ if obj.DeletionTimestamp.IsZero() {
+ if !controllerutil.ContainsFinalizer(obj, finalizerName) {
+ controllerutil.AddFinalizer(obj, finalizerName)
+ if err := r.Update(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+ }
+ } else {
+ // Handle deletion
+ if controllerutil.ContainsFinalizer(obj, finalizerName) {
+ if err := r.cleanup(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+
+ controllerutil.RemoveFinalizer(obj, finalizerName)
+ if err := r.Update(ctx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+ }
+ }
+
+ return ctrl.Result{}, nil
+}
+```
+
+### Cleanup Logic
+
+```go
+func (r *Reconciler) cleanup(ctx context.Context, obj *v1.MyResource) error {
+ // Delete external resources
+ if err := r.deleteCloudInstance(ctx, obj); err != nil {
+ return fmt.Errorf("failed to delete cloud instance: %w", err)
+ }
+
+ if err := r.deleteLoadBalancer(ctx, obj); err != nil {
+ return fmt.Errorf("failed to delete load balancer: %w", err)
+ }
+
+ log.Info("cleaned up external resources", "name", obj.Name)
+ return nil
+}
+```
+
+## Best Practices
+
+1. **Use Domain-Specific Finalizer Names**: Include domain to avoid conflicts
+ ```go
+ const finalizerName = "machine.openshift.io/finalizer" // ✅ Good
+ const finalizerName = "finalizer" // ❌ Bad (generic)
+ ```
+
+2. **Idempotent Cleanup**: Handle cleanup being called multiple times
+ ```go
+ func (r *Reconciler) cleanup(ctx context.Context, obj *v1.MyResource) error {
+ // Check if resource exists before deleting
+ instance, err := r.cloudProvider.GetInstance(obj.Spec.InstanceID)
+ if err != nil {
+ if isNotFoundError(err) {
+ return nil // Already deleted
+ }
+ return err
+ }
+
+ return r.cloudProvider.DeleteInstance(instance.ID)
+ }
+ ```
+
+3. **Handle Errors Gracefully**: Don't remove finalizer if cleanup fails
+ ```go
+ if err := r.cleanup(ctx, obj); err != nil {
+ // Log error, requeue for retry
+ return ctrl.Result{}, err // Don't remove finalizer
+ }
+ ```
+
+4. **Set Status During Cleanup**: Update status to show cleanup in progress
+ ```go
+ if !obj.DeletionTimestamp.IsZero() {
+ obj.Status.Phase = "Terminating"
+ r.Status().Update(ctx, obj)
+ }
+ ```
+
+5. **Timeout Cleanup**: Don't block forever on stuck cleanup
+ ```go
+ cleanupCtx, cancel := context.WithTimeout(ctx, 5*time.Minute)
+ defer cancel()
+
+ if err := r.cleanup(cleanupCtx, obj); err != nil {
+ return ctrl.Result{}, err
+ }
+ ```
+
+## Common Patterns
+
+### Machine Finalizer (Machine API)
+
+```yaml
+apiVersion: machine.openshift.io/v1beta1
+kind: Machine
+metadata:
+ name: my-machine
+ finalizers:
+ - machine.machine.openshift.io # Blocks deletion until cloud VM deleted
+spec:
+ providerSpec:
+ value:
+ instanceType: m5.large
+```
+
+### Multiple Finalizers
+
+```go
+const (
+ finalizerVM = "myoperator.example.com/vm"
+ finalizerStorage = "myoperator.example.com/storage"
+)
+
+func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+ obj := &v1.MyResource{}
+ r.Get(ctx, req.NamespacedName, obj)
+
+ if obj.DeletionTimestamp.IsZero() {
+ // Add both finalizers
+ controllerutil.AddFinalizer(obj, finalizerVM)
+ controllerutil.AddFinalizer(obj, finalizerStorage)
+ r.Update(ctx, obj)
+ } else {
+ // Remove in order: storage first, then VM
+ if controllerutil.ContainsFinalizer(obj, finalizerStorage) {
+ r.cleanupStorage(ctx, obj)
+ controllerutil.RemoveFinalizer(obj, finalizerStorage)
+ r.Update(ctx, obj)
+ }
+
+ if controllerutil.ContainsFinalizer(obj, finalizerVM) {
+ r.cleanupVM(ctx, obj)
+ controllerutil.RemoveFinalizer(obj, finalizerVM)
+ r.Update(ctx, obj)
+ }
+ }
+
+ return ctrl.Result{}, nil
+}
+```
+
+## Debugging
+
+### View Finalizers
+
+```bash
+# Check finalizers on object
+oc get machine my-machine -o jsonpath='{.metadata.finalizers}'
+
+# View object stuck in deletion
+oc get machines | grep Terminating
+```
+
+### Remove Stuck Finalizer (Emergency)
+
+```bash
+# Only use if cleanup is stuck and manual intervention needed
+oc patch machine my-machine -p '{"metadata":{"finalizers":[]}}' --type=merge
+```
+
+**Warning**: This bypasses cleanup logic and may leak external resources.
+
+## Examples in Components
+
+| Component | Finalizer Use | Cleanup Action |
+|-----------|--------------|----------------|
+| machine-api-operator | `machine.machine.openshift.io` | Delete cloud VM |
+| cluster-network-operator | `network.operator.openshift.io/finalizer` | Remove network config |
+| cluster-autoscaler | `autoscaler.openshift.io/finalizer` | Delete autoscaler config |
+
+## Troubleshooting
+
+### Object Stuck in Terminating
+
+**Symptoms**: `oc get` shows object in "Terminating" state for >5 minutes
+
+**Causes**:
+1. Cleanup logic failing (check operator logs)
+2. External resource doesn't exist (idempotent cleanup not implemented)
+3. Controller not running (no reconciliation)
+
+**Debug**:
+```bash
+# Check controller logs
+oc logs -n my-operator deployment/my-operator
+
+# Check finalizers
+oc get myresource my-obj -o yaml | grep -A5 finalizers
+
+# View events
+oc describe myresource my-obj
+```
+
+### Finalizer Not Added
+
+**Symptoms**: Object deleted immediately without cleanup
+
+**Causes**:
+1. Finalizer not added during creation
+2. Race condition (object deleted before finalizer added)
+
+**Fix**: Add finalizer in admission webhook or immediately on creation
+
+## Antipatterns
+
+❌ **Removing finalizer before cleanup**: Leaks external resources
+❌ **Generic finalizer names**: Conflicts with other operators
+❌ **Not handling cleanup errors**: Object stuck forever
+❌ **Synchronous long-running cleanup**: Blocks reconcile loop
+❌ **No idempotent cleanup**: Fails if cleanup called twice
+
+## References
+
+- **Kubernetes**: [Finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/)
+- **controller-runtime**: [controllerutil](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil)
+- **OpenShift**: Machine API uses finalizers extensively
+- **Related**: [controller-runtime.md](./controller-runtime.md)
diff --git a/ai-docs/platform/operator-patterns/index.md b/ai-docs/platform/operator-patterns/index.md
new file mode 100644
index 0000000000..6745e75329
--- /dev/null
+++ b/ai-docs/platform/operator-patterns/index.md
@@ -0,0 +1,18 @@
+# Operator Patterns
+
+Standard patterns used across all OpenShift operators.
+
+## Core Patterns
+
+- [controller-runtime.md](controller-runtime.md) - Reconciliation loop pattern (watch → reconcile → update status)
+- [status-conditions.md](status-conditions.md) - Available/Progressing/Degraded health reporting
+- [webhooks.md](webhooks.md) - Validation, mutation, and conversion webhooks
+
+## Resource Management
+
+- [finalizers.md](finalizers.md) - Cleanup external resources before deletion
+- [rbac.md](rbac.md) - Service account and RBAC permissions
+
+## Related
+
+- [OpenShift-Specific Patterns](../openshift-specifics/) - Upgrade strategies, CVO coordination
diff --git a/ai-docs/platform/operator-patterns/rbac.md b/ai-docs/platform/operator-patterns/rbac.md
new file mode 100644
index 0000000000..db161a77f2
--- /dev/null
+++ b/ai-docs/platform/operator-patterns/rbac.md
@@ -0,0 +1,309 @@
+# RBAC Patterns
+
+**Category**: Platform Pattern
+**Applies To**: All Operators
+**Last Updated**: 2026-04-29
+
+## Overview
+
+Role-Based Access Control (RBAC) restricts access to Kubernetes resources. Operators need appropriate permissions to manage resources, and should follow least-privilege principles.
+
+**Pattern**: ServiceAccount + Role/ClusterRole + RoleBinding/ClusterRoleBinding
+
+## Key Concepts
+
+- **ServiceAccount**: Identity for pods
+- **Role**: Namespaced permissions
+- **ClusterRole**: Cluster-wide permissions
+- **RoleBinding**: Grants Role to ServiceAccount (namespaced)
+- **ClusterRoleBinding**: Grants ClusterRole to ServiceAccount (cluster-wide)
+
+## RBAC Resources
+
+| Resource | Scope | Use Case |
+|----------|-------|----------|
+| **Role** | Namespace | Permissions for namespaced resources in one namespace |
+| **ClusterRole** | Cluster | Permissions for cluster-scoped or all-namespace resources |
+| **RoleBinding** | Namespace | Grant Role to users/groups/ServiceAccounts |
+| **ClusterRoleBinding** | Cluster | Grant ClusterRole to users/groups/ServiceAccounts |
+
+## Operator RBAC Pattern
+
+```yaml
+# ServiceAccount
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: my-operator
+ namespace: my-operator-namespace
+
+---
+# ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: my-operator
+rules:
+# Watch and manage custom resources
+- apiGroups: ["example.com"]
+ resources: ["myresources"]
+ verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+# Update status subresource
+- apiGroups: ["example.com"]
+ resources: ["myresources/status"]
+ verbs: ["get", "update", "patch"]
+
+# Manage deployments (operands)
+- apiGroups: ["apps"]
+ resources: ["deployments"]
+ verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+# Read configmaps
+- apiGroups: [""]
+ resources: ["configmaps"]
+ verbs: ["get", "list", "watch"]
+
+---
+# ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: my-operator
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: my-operator
+subjects:
+- kind: ServiceAccount
+ name: my-operator
+ namespace: my-operator-namespace
+```
+
+## Best Practices
+
+1. **Least Privilege**: Grant only required permissions
+ ```yaml
+ # ✅ Good: Specific resources and verbs
+ - apiGroups: ["apps"]
+ resources: ["deployments"]
+ verbs: ["get", "list", "watch", "update"]
+
+ # ❌ Bad: Wildcard permissions
+ - apiGroups: ["*"]
+ resources: ["*"]
+ verbs: ["*"]
+ ```
+
+2. **Separate Status Permissions**: Use status subresource
+ ```yaml
+ # Spec updates (user-facing)
+ - apiGroups: ["example.com"]
+ resources: ["myresources"]
+ verbs: ["update", "patch"]
+
+ # Status updates (controller-only)
+ - apiGroups: ["example.com"]
+ resources: ["myresources/status"]
+ verbs: ["update", "patch"]
+ ```
+
+3. **Use ClusterRole for CRDs**: Even if namespaced, CRD management needs ClusterRole
+ ```yaml
+ - apiGroups: ["apiextensions.k8s.io"]
+ resources: ["customresourcedefinitions"]
+ verbs: ["get", "list", "watch"] # Usually read-only
+ ```
+
+4. **Scope Appropriately**: Use Role for namespace-specific operators
+ ```yaml
+ # Namespace-scoped operator (single namespace)
+ kind: Role # Not ClusterRole
+
+ # Multi-namespace operator
+ kind: ClusterRole
+ ```
+
+## Common Permission Sets
+
+### CRD Management
+
+```yaml
+rules:
+# Watch CRD
+- apiGroups: ["apiextensions.k8s.io"]
+ resources: ["customresourcedefinitions"]
+ verbs: ["get", "list", "watch"]
+
+# Manage custom resources
+- apiGroups: ["example.com"]
+ resources: ["myresources"]
+ verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+# Update status
+- apiGroups: ["example.com"]
+ resources: ["myresources/status"]
+ verbs: ["get", "update", "patch"]
+```
+
+### Core Resources
+
+```yaml
+rules:
+# Pods
+- apiGroups: [""]
+ resources: ["pods"]
+ verbs: ["get", "list", "watch"]
+
+# Services
+- apiGroups: [""]
+ resources: ["services"]
+ verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+
+# ConfigMaps (read-only)
+- apiGroups: [""]
+ resources: ["configmaps"]
+ verbs: ["get", "list", "watch"]
+
+# Secrets (read-only, specific names)
+- apiGroups: [""]
+ resources: ["secrets"]
+ resourceNames: ["my-operator-tls"]
+ verbs: ["get"]
+```
+
+### Events
+
+```yaml
+rules:
+# Create events for debugging
+- apiGroups: [""]
+ resources: ["events"]
+ verbs: ["create", "patch"]
+```
+
+### Leader Election
+
+```yaml
+rules:
+# ConfigMap-based leader election
+- apiGroups: [""]
+ resources: ["configmaps"]
+ resourceNames: ["my-operator-leader"]
+ verbs: ["get", "update", "patch"]
+
+- apiGroups: [""]
+ resources: ["configmaps"]
+ verbs: ["create"]
+
+# Lease-based leader election (preferred)
+- apiGroups: ["coordination.k8s.io"]
+ resources: ["leases"]
+ verbs: ["get", "create", "update", "patch"]
+```
+
+## OpenShift-Specific RBAC
+
+### Security Context Constraints (SCCs)
+
+```yaml
+# ClusterRole to use specific SCC
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: my-operator-scc
+rules:
+- apiGroups: ["security.openshift.io"]
+ resources: ["securitycontextconstraints"]
+ resourceNames: ["privileged"] # Or custom SCC
+ verbs: ["use"]
+```
+
+### OpenShift Config Resources
+
+```yaml
+rules:
+# Read cluster config
+- apiGroups: ["config.openshift.io"]
+ resources: ["clusterversions", "infrastructures", "networks"]
+ verbs: ["get", "list", "watch"]
+
+# Update ClusterOperator status
+- apiGroups: ["config.openshift.io"]
+ resources: ["clusteroperators"]
+ verbs: ["get", "list", "watch"]
+
+- apiGroups: ["config.openshift.io"]
+ resources: ["clusteroperators/status"]
+ verbs: ["update", "patch"]
+```
+
+## Kubebuilder Markers
+
+```go
+// Generate RBAC manifests using kubebuilder markers
+
+//+kubebuilder:rbac:groups=example.com,resources=myresources,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=example.com,resources=myresources/status,verbs=get;update;patch
+//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups="",resources=configmaps,verbs=get;list;watch
+//+kubebuilder:rbac:groups="",resources=events,verbs=create;patch
+
+func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+ // Controller logic
+}
+```
+
+**Generate manifests**: `make manifests` creates RBAC YAML from markers.
+
+## Debugging RBAC
+
+### Check Permissions
+
+```bash
+# Can I create deployments?
+oc auth can-i create deployments --as=system:serviceaccount:my-ns:my-sa
+
+# What can this SA do?
+oc auth can-i --list --as=system:serviceaccount:my-ns:my-sa
+
+# View ClusterRole
+oc describe clusterrole my-operator
+
+# View bindings
+oc get clusterrolebinding | grep my-operator
+```
+
+### Common Errors
+
+**Error**: `forbidden: User "system:serviceaccount:my-ns:my-sa" cannot create resource`
+
+**Cause**: Missing RBAC permissions
+
+**Fix**:
+1. Check ClusterRole has required verbs
+2. Verify ClusterRoleBinding references correct SA
+3. Ensure SA exists
+
+## Examples in Components
+
+| Component | RBAC Pattern | Notes |
+|-----------|-------------|-------|
+| machine-api-operator | ClusterRole for Machines | Cluster-scoped resources |
+| kube-apiserver | ClusterRole + privileged SCC | Needs elevated permissions |
+| cluster-network-operator | ClusterRole for network config | Cluster-wide networking |
+
+## Antipatterns
+
+❌ **Wildcard permissions**: `apiGroups: ["*"]`, `resources: ["*"]`
+❌ **Overly broad verbs**: `verbs: ["*"]` instead of specific verbs
+❌ **No status subresource**: Mixing spec and status permissions
+❌ **Hardcoded namespace**: ClusterRoleBinding with hardcoded namespace
+❌ **Secrets without resourceNames**: Allowing read of all secrets
+
+## References
+
+- **Kubernetes**: [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
+- **OpenShift**: [RBAC Guide](https://docs.openshift.com/container-platform/latest/authentication/using-rbac.html)
+- **SCCs**: [Security Context Constraints](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html)
+- **Kubebuilder**: [RBAC Markers](https://book.kubebuilder.io/reference/markers/rbac.html)
diff --git a/ai-docs/platform/operator-patterns/status-conditions.md b/ai-docs/platform/operator-patterns/status-conditions.md
index 3a040c63dc..4887c63606 100644
--- a/ai-docs/platform/operator-patterns/status-conditions.md
+++ b/ai-docs/platform/operator-patterns/status-conditions.md
@@ -6,7 +6,7 @@
## Overview
-Status conditions provide standardized health reporting for OpenShift components. Every ClusterOperator must report Available, Progressing, and Degraded conditions.
+Status conditions provide standardized health reporting for OpenShift components. Core (platform) operators report via `configv1.ClusterOperatorStatusCondition` on a `ClusterOperator` resource; OLM-managed operators report via standard `metav1.Condition` on their own Custom Resources. The pattern is the same — the Go types and reporting targets differ.
**Purpose**: Enable cluster-wide health monitoring and upgrade orchestration.
@@ -73,6 +73,37 @@ func setCondition(conditions *[]configv1.ClusterOperatorStatusCondition,
}
```
+## OLM-Managed Operators
+
+OLM-managed operators use `metav1.Condition` (from `k8s.io/apimachinery/pkg/apis/meta/v1`) instead of `configv1.ClusterOperatorStatusCondition`. Conditions are set on the operator's own Custom Resource, not on a `ClusterOperator`.
+
+```go
+import (
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/apimachinery/pkg/api/meta"
+)
+
+func setStatusCondition(conditions *[]metav1.Condition,
+ condType string,
+ status metav1.ConditionStatus,
+ reason, message string,
+ generation int64) {
+ meta.SetStatusCondition(conditions, metav1.Condition{
+ Type: condType,
+ Status: status,
+ Reason: reason,
+ Message: message,
+ ObservedGeneration: generation,
+ })
+}
+```
+
+Key differences from core operators:
+- **Type field**: Open vocabulary string (e.g., `"Ready"`, `"Available"`, `"Degraded"`) rather than the fixed constants in `configv1.ClusterStatusConditionType`
+- **ObservedGeneration**: Supported in `metav1.Condition` (absent in `configv1.ClusterOperatorStatusCondition`)
+- **No CVO visibility**: Conditions on a CR do not propagate to the CVO and do not gate cluster upgrades
+- **OLM also creates an `OperatorCondition` CR** (OLMv0 only, deprecated in OLMv1) per managed operator, where the operator can set `Upgradeable=False` in `spec.conditions` to block its own CSV upgrade (not cluster upgrades)
+
## Best Practices
1. **Always Set All Three**: Available, Progressing, Degraded must always be set
diff --git a/ai-docs/platform/operator-patterns/webhooks.md b/ai-docs/platform/operator-patterns/webhooks.md
new file mode 100644
index 0000000000..2b956c9994
--- /dev/null
+++ b/ai-docs/platform/operator-patterns/webhooks.md
@@ -0,0 +1,269 @@
+# Webhooks Pattern
+
+**Category**: Platform Pattern
+**Applies To**: Operators with API validation/mutation needs
+**Last Updated**: 2026-05-26
+**Scope**: All form factors with networking considerations (see below)
+
+## Overview
+
+Webhooks extend Kubernetes API server with custom validation, mutation, and conversion logic. They intercept API requests before objects are persisted to etcd.
+
+**Types**: ValidatingWebhook, MutatingWebhook, ConversionWebhook
+
+**⚠️ Form Factor Note**: In **Hypershift/HCP**, the documented pattern is deploying webhooks as sidecars to kube-apiserver (in the management cluster). See [azure-workload-identity-webhook.md](../../../enhancements/hypershift/azure-workload-identity-webhook.md) for the authoritative HCP webhook pattern.
+
+## Key Concepts
+
+- **Admission Webhook**: Intercepts CREATE/UPDATE/DELETE operations
+- **Validating**: Approve or reject requests (cannot modify)
+- **Mutating**: Modify requests before persistence (set defaults, inject sidecars)
+- **Conversion**: Convert between API versions (v1alpha1 ↔ v1beta1)
+- **Fail Policy**: FailClosed (reject on error) vs FailOpen (allow on error)
+
+## Webhook Types
+
+| Type | Purpose | Can Modify | Use Cases |
+|------|---------|------------|-----------|
+| **Validating** | Enforce invariants | No | Reject invalid configs |
+| **Mutating** | Set defaults, inject | Yes | Default values, sidecar injection |
+| **Conversion** | API version migration | Yes | v1alpha1 ↔ v1 |
+
+## Implementation
+
+### Validating Webhook
+
+```go
+import (
+ "k8s.io/api/admission/v1"
+ "sigs.k8s.io/controller-runtime/pkg/webhook/admission"
+)
+
+type MyValidator struct {
+ decoder *admission.Decoder
+}
+
+func (v *MyValidator) Handle(ctx context.Context, req admission.Request) admission.Response {
+ obj := &MyResource{}
+ if err := v.decoder.Decode(req, obj); err != nil {
+ return admission.Errored(http.StatusBadRequest, err)
+ }
+
+ // Validate
+ if obj.Spec.Replicas > 10 {
+ return admission.Denied("replicas cannot exceed 10")
+ }
+
+ if obj.Spec.Image == "" {
+ return admission.Denied("image is required")
+ }
+
+ return admission.Allowed("")
+}
+```
+
+### Mutating Webhook
+
+```go
+func (m *MyMutator) Handle(ctx context.Context, req admission.Request) admission.Response {
+ obj := &MyResource{}
+ if err := m.decoder.Decode(req, obj); err != nil {
+ return admission.Errored(http.StatusBadRequest, err)
+ }
+
+ // Set defaults
+ if obj.Spec.Replicas == 0 {
+ obj.Spec.Replicas = 3
+ }
+
+ if obj.Spec.Strategy == "" {
+ obj.Spec.Strategy = "RollingUpdate"
+ }
+
+ // Return patched object (compare raw original with modified bytes)
+ marshaledObj, err := json.Marshal(obj)
+ if err != nil {
+ return admission.Errored(http.StatusInternalServerError, err)
+ }
+ return admission.PatchResponseFromRaw(req.Object.Raw, marshaledObj)
+}
+```
+
+### Conversion Webhook
+
+```go
+func (c *MyConverter) ConvertTo(dst conversion.Hub) error {
+ // Convert from this version to hub version
+ dstObj := dst.(*v1.MyResource)
+ dstObj.Spec.NewField = c.Spec.OldField
+ return nil
+}
+
+func (c *MyConverter) ConvertFrom(src conversion.Hub) error {
+ // Convert from hub version to this version
+ srcObj := src.(*v1.MyResource)
+ c.Spec.OldField = srcObj.Spec.NewField
+ return nil
+}
+```
+
+## Best Practices
+
+1. **Fail Policies**:
+ - **FailClosed** (default): Reject requests if webhook unavailable (safer)
+ - **FailOpen**: Allow requests if webhook unavailable (for non-critical validation)
+
+2. **Idempotent Mutations**: Applying mutation multiple times = same result
+ ```go
+ // ✅ Idempotent
+ if obj.Labels == nil {
+ obj.Labels = make(map[string]string)
+ }
+ obj.Labels["injected"] = "true"
+
+ // ❌ Not idempotent
+ obj.Spec.Env = append(obj.Spec.Env, newVar)
+ ```
+
+3. **Validation Order**: Mutating webhooks run before validating webhooks
+
+4. **Namespace Selectors**: Limit webhook scope to avoid performance issues
+ ```yaml
+ namespaceSelector:
+ matchExpressions:
+ - key: admission.openshift.io/ignore
+ operator: DoesNotExist
+ ```
+
+5. **Timeout**: Default 10s (can configure 1-30s)
+
+## Common Patterns
+
+### WebhookConfiguration
+
+```yaml
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+ name: my-validator
+webhooks:
+- name: validate.myresource.example.com
+ clientConfig:
+ service:
+ name: my-webhook-service
+ namespace: my-operator
+ path: /validate
+ caBundle:
+ rules:
+ - operations: ["CREATE", "UPDATE"]
+ apiGroups: ["example.com"]
+ apiVersions: ["v1"]
+ resources: ["myresources"]
+ failurePolicy: Fail
+ sideEffects: None
+ admissionReviewVersions: ["v1"]
+ timeoutSeconds: 10
+```
+
+### Certificate Management
+
+```go
+// Use cert-manager or controller-runtime's cert provisioning
+import (
+ "sigs.k8s.io/controller-runtime/pkg/webhook"
+)
+
+func main() {
+ mgr, _ := ctrl.NewManager(cfg, ctrl.Options{
+ CertDir: "/tmp/k8s-webhook-server/serving-certs",
+ })
+
+ // Certs automatically provisioned
+ mgr.GetWebhookServer().Register("/validate", &webhook.Admission{
+ Handler: &MyValidator{},
+ })
+}
+```
+
+### Validation Example: Cross-Field
+
+```go
+func validateCrossField(obj *MyResource) error {
+ // Ensure replicas matches tier
+ if obj.Spec.Tier == "production" && obj.Spec.Replicas < 3 {
+ return fmt.Errorf("production tier requires at least 3 replicas")
+ }
+
+ // Ensure resources set for production
+ if obj.Spec.Tier == "production" && obj.Spec.Resources == nil {
+ return fmt.Errorf("production tier requires resource limits")
+ }
+
+ return nil
+}
+```
+
+## Testing Webhooks
+
+```go
+func TestWebhook(t *testing.T) {
+ decoder, _ := admission.NewDecoder(scheme)
+ validator := &MyValidator{decoder: decoder}
+
+ obj := &MyResource{
+ Spec: MyResourceSpec{
+ Replicas: 15, // Invalid
+ },
+ }
+
+ req := admission.Request{
+ AdmissionRequest: v1.AdmissionRequest{
+ Object: runtime.RawExtension{Object: obj},
+ },
+ }
+
+ resp := validator.Handle(context.TODO(), req)
+ assert.False(t, resp.Allowed)
+ assert.Contains(t, resp.Result.Message, "cannot exceed 10")
+}
+```
+
+## Examples in Components
+
+| Component | Webhook Type | Purpose |
+|-----------|--------------|---------|
+| machine-api-operator | Validating | Validate Machine/MachineSet specs |
+| kube-apiserver | Mutating | Inject service account tokens |
+| cluster-network-operator | Validating | Prevent invalid network config changes |
+
+## Performance Considerations
+
+- **Timeout**: Keep webhook logic fast (<1s ideal, 10s max)
+- **Namespace Filtering**: Use selectors to reduce webhook invocations
+- **Caching**: Cache expensive lookups (don't query API in every request)
+- **Monitoring**: Track webhook latency and failure rates
+
+```promql
+# Webhook latency
+histogram_quantile(0.99, rate(apiserver_admission_webhook_request_duration_seconds_bucket[5m]))
+
+# Webhook rejections
+rate(apiserver_admission_webhook_rejection_count[5m])
+```
+
+## Antipatterns
+
+❌ **Slow webhooks**: Blocking API server for >1s
+❌ **External dependencies**: Webhook depends on external service (network partition = cluster unavailable)
+❌ **FailOpen for critical validation**: Allows invalid configs during webhook downtime
+❌ **No namespace filtering**: Webhook called for every pod/deployment in cluster
+❌ **Non-idempotent mutations**: Applying mutation twice gives different results
+
+## References
+
+- **Upstream**: [Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)
+- **controller-runtime**: [Webhook Guide](https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation.html)
+- **OpenShift**: [Admission Plugins](https://docs.openshift.com/container-platform/latest/architecture/admission-plug-ins.html)
+- **Pattern**: Implements "API-First Design" from [DESIGN_PHILOSOPHY.md](../../DESIGN_PHILOSOPHY.md)
+- **HCP Enhancements** (authoritative webhook pattern for HCP):
+ - [azure-workload-identity-webhook.md](../../../enhancements/hypershift/azure-workload-identity-webhook.md) - KAS sidecar webhook pattern in HyperShift
diff --git a/ai-docs/practices/development/api-evolution.md b/ai-docs/practices/development/api-evolution.md
new file mode 100644
index 0000000000..bae8485567
--- /dev/null
+++ b/ai-docs/practices/development/api-evolution.md
@@ -0,0 +1,317 @@
+# API Evolution
+
+**Category**: Engineering Practice
+**Last Updated**: 2026-04-29
+
+## Overview
+
+Evolving APIs requires backward compatibility. OpenShift follows Kubernetes API conventions for stability and versioning.
+
+**Key Principle**: Never break existing clients.
+
+## API Stability Levels
+
+| Level | Meaning | Breaking Changes Allowed | Removal Allowed |
+|-------|---------|-------------------------|-----------------|
+| **v1** | Stable | No | No (deprecated only) |
+| **v1beta1** | Pre-release | Minimal | After deprecation period |
+| **v1alpha1** | Experimental | Yes | Yes |
+
+## Compatibility Rules
+
+### Safe Changes ✅
+
+| Change | Example | Why Safe |
+|--------|---------|----------|
+| Add optional field | `newField *string` | Old clients ignore unknown fields |
+| Add new API version | `v1beta1 → v1` | Conversion preserves old version |
+| Add values to enum | `enum: [A, B, C]` | Old clients handle unknown values |
+| Deprecate field | `// Deprecated: use newField` | Field still works |
+
+### Breaking Changes ❌
+
+| Change | Example | Why Breaking |
+|--------|---------|--------------|
+| Remove field | Delete `oldField` | Old clients fail validation |
+| Rename field | `replicas → count` | Old clients send wrong field name |
+| Change field type | `string → int` | Old clients send wrong type |
+| Make optional field required | `*string → string` | Old resources lack required field |
+| Remove enum value | `enum: [A, B]` (was `[A, B, C]`) | Old resources have invalid value |
+
+## Graduation Path Policy
+
+OpenShift generally follows the **alpha → beta → GA** progression, but this is **not always required**.
+
+### When You Can Skip Beta/TechPreview
+
+You may go **alpha → GA directly** when:
+- ✅ API is stable and well-understood
+- ✅ Implementation is production-quality from the start
+- ✅ No expectation of API changes before GA
+- ✅ Sufficient confidence to commit to long-term support
+
+**BUT** you must still meet GA quality requirements (see below).
+
+### When Beta/TechPreview Is Required
+
+Use **TechPreview** stage when:
+- ⚠️ API design needs real-world validation
+- ⚠️ Implementation quality unknown (scalability, stability)
+- ⚠️ Want flexibility to change API without migration path
+- ⚠️ Feature requires special enablement (feature gate)
+
+### Testing Requirements for GA (Cannot Be Skipped)
+
+**Whether you go alpha → GA directly or alpha → beta → GA, GA requires:**
+
+| Requirement | Why |
+|-------------|-----|
+| **Sufficient test coverage** | Unit, integration, e2e tests |
+| **Upgrade/downgrade testing** | N→N+1 and rollback scenarios |
+| **Scale testing** | Load testing, resource usage validation |
+| **End-to-end tests** | Required for non-optional features |
+| **SLI/SLO definition** | Service level indicators and objectives |
+| **User documentation** | openshift-docs PRs |
+| **Sufficient feedback time** | Real-world usage validation |
+
+**Skipping TechPreview does NOT skip these requirements** - you still need production-quality testing.
+
+**Key Rule**: If unsure about API or implementation, use TechPreview. Going directly to GA means **no breaking changes ever** AND **full production quality from day one**.
+
+See [supportability.md](../../../guidelines/supportability.md) for official Tech Preview policy and [enhancement_template.md](../../../guidelines/enhancement_template.md) for graduation criteria details.
+
+## Versioning Strategy
+
+### Adding New API Version
+
+```go
+// v1alpha1 (initial)
+type MyResourceSpec struct {
+ Replicas int `json:"replicas"`
+ Image string `json:"image"`
+}
+
+// v1beta1 (add features) - OPTIONAL stage
+type MyResourceSpec struct {
+ Replicas int `json:"replicas"`
+ Image string `json:"image"`
+ Strategy string `json:"strategy,omitempty"` // New optional field
+}
+
+// v1 (stable) - can skip v1beta1 if confident
+type MyResourceSpec struct {
+ Replicas int `json:"replicas"`
+ Image string `json:"image"`
+ Strategy string `json:"strategy,omitempty"`
+}
+```
+
+### Deprecating Fields
+
+```go
+type MyResourceSpec struct {
+ // Deprecated: Use Strategy instead
+ // +optional
+ OldStrategy string `json:"oldStrategy,omitempty"`
+
+ Strategy string `json:"strategy,omitempty"`
+}
+
+func (r *Reconciler) reconcile(obj *MyResource) {
+ // Handle both old and new fields
+ strategy := obj.Spec.Strategy
+ if strategy == "" && obj.Spec.OldStrategy != "" {
+ strategy = obj.Spec.OldStrategy
+ }
+}
+```
+
+### Deprecation Timeline
+
+```
+Version N: Field announced as deprecated (still works)
+Version N+1: Field still works, warnings in logs
+Version N+2: Field removed (only if v1alpha1/v1beta1)
+```
+
+**Stable APIs (v1)**: Cannot remove deprecated fields (mark deprecated forever)
+
+## Conversion Webhooks
+
+```go
+// Hub version (v1)
+type MyResource struct {
+ Spec MyResourceSpec `json:"spec"`
+}
+
+// Spoke version (v1beta1)
+func (src *MyResource) ConvertTo(dstRaw conversion.Hub) error {
+ dst := dstRaw.(*v1.MyResource)
+
+ // Convert fields
+ dst.Spec.Replicas = src.Spec.Replicas
+ dst.Spec.NewField = convertOldToNew(src.Spec.OldField)
+
+ return nil
+}
+
+func (dst *MyResource) ConvertFrom(srcRaw conversion.Hub) error {
+ src := srcRaw.(*v1.MyResource)
+
+ // Reverse conversion
+ dst.Spec.Replicas = src.Spec.Replicas
+ dst.Spec.OldField = convertNewToOld(src.Spec.NewField)
+
+ return nil
+}
+```
+
+## Default Values
+
+```yaml
+# CRD schema with defaults
+schema:
+ openAPIV3Schema:
+ properties:
+ spec:
+ properties:
+ replicas:
+ type: integer
+ default: 3 # Applied if not specified
+ strategy:
+ type: string
+ default: RollingUpdate
+```
+
+**Alternative**: Mutating webhook sets defaults
+
+## Validation
+
+```yaml
+# CRD validation
+schema:
+ openAPIV3Schema:
+ properties:
+ spec:
+ required: ["image"] # Required field
+ properties:
+ replicas:
+ type: integer
+ minimum: 1
+ maximum: 100
+ tier:
+ type: string
+ enum: ["dev", "staging", "prod"]
+```
+
+## Migration Strategies
+
+### Adding Required Field (Without Breaking)
+
+```go
+// Step 1: Add optional field (version N)
+type MyResourceSpec struct {
+ Image string `json:"image"`
+ Tier *string `json:"tier,omitempty"` // Optional
+}
+
+// Step 2: Set default via webhook (version N)
+func (m *Mutator) Default(obj *MyResource) {
+ if obj.Spec.Tier == nil {
+ tier := "dev"
+ obj.Spec.Tier = &tier
+ }
+}
+
+// Step 3: Make required after all resources migrated (version N+2)
+type MyResourceSpec struct {
+ Image string `json:"image"`
+ Tier string `json:"tier"` // Now required
+}
+```
+
+### Renaming Field
+
+```go
+// Step 1: Add new field, deprecate old (version N)
+type MyResourceSpec struct {
+ // Deprecated: Use Count instead
+ Replicas *int `json:"replicas,omitempty"`
+ Count *int `json:"count,omitempty"`
+}
+
+// Step 2: Handle both in controller
+func (r *Reconciler) getCount(obj *MyResource) int {
+ if obj.Spec.Count != nil {
+ return *obj.Spec.Count
+ }
+ if obj.Spec.Replicas != nil {
+ return *obj.Spec.Replicas
+ }
+ return 3 // default
+}
+
+// Step 3: Remove old field (version N+2, only if not v1)
+```
+
+## Storage Version
+
+```yaml
+versions:
+- name: v1
+ served: true
+ storage: true # Stored in etcd as v1
+
+- name: v1beta1
+ served: true
+ storage: false # Converted to v1 before storing
+```
+
+**Migration**: `oc adm migrate storage` to rewrite old versions
+
+## Testing API Changes
+
+```go
+func TestBackwardCompatibility(t *testing.T) {
+ // Old object (v1beta1)
+ old := &v1beta1.MyResource{
+ Spec: v1beta1.MyResourceSpec{
+ OldField: "value",
+ },
+ }
+
+ // Convert to new version (v1)
+ new := &v1.MyResource{}
+ old.ConvertTo(new)
+
+ // Verify conversion preserves data
+ assert.Equal(t, "value", new.Spec.NewField)
+
+ // Convert back
+ roundtrip := &v1beta1.MyResource{}
+ roundtrip.ConvertFrom(new)
+
+ // Verify roundtrip preserves data
+ assert.Equal(t, old.Spec.OldField, roundtrip.Spec.OldField)
+}
+```
+
+## Decision Table: Can I Make This Change?
+
+| Current API | Change | Allowed | Alternative |
+|-------------|--------|---------|-------------|
+| v1 | Remove field | ❌ | Deprecate only |
+| v1 | Rename field | ❌ | Add new, keep old deprecated |
+| v1 | Add optional field | ✅ | Safe |
+| v1 | Add required field | ❌ | Add optional + default |
+| v1beta1 | Remove field | ✅ (after deprecation) | Deprecate in N, remove in N+1 |
+| v1beta1 | Change type | ✅ (with conversion) | Conversion webhook |
+| v1alpha1 | Any change | ✅ | No guarantees |
+
+## References
+
+- **Kubernetes**: [API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md)
+- **Dev Guide**: [api-conventions.md](../../../dev-guide/api-conventions.md)
+- **OpenShift**: [API Review Process](https://github.com/openshift/enhancements/blob/master/dev-guide/api-conventions.md)
+- **Supportability**: [supportability.md](../../../guidelines/supportability.md) - Tech Preview vs GA policy
+- **Graduation Criteria**: [enhancement_template.md](../../../guidelines/enhancement_template.md) - Testing requirements for GA
diff --git a/ai-docs/practices/development/index.md b/ai-docs/practices/development/index.md
new file mode 100644
index 0000000000..bd1f48ac13
--- /dev/null
+++ b/ai-docs/practices/development/index.md
@@ -0,0 +1,24 @@
+# Development Practices
+
+Software development practices and conventions.
+
+## Guides
+
+- [api-evolution.md](api-evolution.md) - API versioning, compatibility, and migration strategies
+
+## Dev Guide References
+
+For authoritative development conventions, see:
+- [../../dev-guide/api-conventions.md](../../../dev-guide/api-conventions.md) - API design conventions
+- [../../dev-guide/breaking-changes.md](../../../dev-guide/breaking-changes.md) - Breaking change policy
+- [../../dev-guide/operators.md](../../../dev-guide/operators.md) - Operator development guide
+- [../../dev-guide/feature-zero-to-hero.md](../../../dev-guide/feature-zero-to-hero.md) - Feature development workflow
+- [../../guidelines/commit_and_pr_text.md](../../../guidelines/commit_and_pr_text.md) - PR title and commit message conventions
+- [../../dev-guide/new-components.md](../../../dev-guide/new-components.md) - Policy for adding components to payload vs OLM
+- [../../dev-guide/host-port-registry.md](../../../dev-guide/host-port-registry.md) - Authoritative host port allocation registry (9000-9999)
+- [../../dev-guide/release-blocker-definition.md](../../../dev-guide/release-blocker-definition.md) - When bugs block releases
+
+## Related
+
+- [Platform Patterns](../../platform/operator-patterns/) - Operator implementation patterns
+- [Domain Concepts](../../domain/) - Core API documentation
diff --git a/ai-docs/practices/index.md b/ai-docs/practices/index.md
new file mode 100644
index 0000000000..945a59538f
--- /dev/null
+++ b/ai-docs/practices/index.md
@@ -0,0 +1,23 @@
+# Engineering Practices
+
+Cross-cutting engineering practices for OpenShift development.
+
+## Practice Areas
+
+- [testing/](testing/) - Testing pyramid, test conventions
+- [development/](development/) - API evolution, compatibility
+- [security/](security/) - RBAC, threat modeling, secret handling
+- [reliability/](reliability/) - SLI/SLO/SLA, degraded mode, high availability
+
+## Dev Guide References
+
+Most authoritative guidance lives in:
+- [../../dev-guide/](../../dev-guide/) - Development conventions
+- [../../guidelines/](../../guidelines/) - Enhancement process
+
+These practice docs provide AI-optimized structure (tables, checklists) and fill gaps.
+
+## Related
+
+- **Platform Patterns**: [../platform/](../platform/) - Operator implementation patterns
+- **Workflows**: [../workflows/](../workflows/) - Process guides
diff --git a/ai-docs/practices/reliability/index.md b/ai-docs/practices/reliability/index.md
new file mode 100644
index 0000000000..84845b1287
--- /dev/null
+++ b/ai-docs/practices/reliability/index.md
@@ -0,0 +1,39 @@
+# Reliability Practices
+
+Reliability patterns and SLI/SLO/SLA definitions for OpenShift.
+
+## Guidance
+
+Reliability practices are primarily documented in:
+- [../../dev-guide/](../../../dev-guide/) - Component readiness, release blockers
+- [Status Conditions Pattern](../../platform/operator-patterns/status-conditions.md)
+
+## SLI/SLO/SLA
+
+| Term | Definition | Example |
+|------|------------|---------|
+| **SLI** | Service Level Indicator (metric) | API server 99th percentile latency |
+| **SLO** | Service Level Objective (target) | p99 latency < 1s |
+| **SLA** | Service Level Agreement (contract) | 99.95% uptime guarantee |
+
+## Degraded Mode Patterns
+
+| Pattern | Use Case | Example |
+|---------|----------|---------|
+| **Partial Availability** | Some replicas down | 2/3 pods ready → Available=True, Degraded=True |
+| **Reduced Capacity** | Performance degraded | Serving requests but slower |
+| **Read-Only Mode** | Write path broken | API reads work, writes fail |
+
+## High Availability
+
+| Pattern | When to Use | Reference |
+|---------|-------------|-----------|
+| **PodDisruptionBudget** | Prevent all replicas down | [PDB Docs](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) |
+| **Multiple Replicas** | Redundancy | 3+ replicas for HA |
+| **Leader Election** | Single writer | [controller-runtime leader election](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/leaderelection) |
+
+## Related
+
+- **Dev Guide**: [../../dev-guide/component-readiness.md](../../../dev-guide/component-readiness.md)
+- **Status Conditions**: [../../platform/operator-patterns/status-conditions.md](../../platform/operator-patterns/status-conditions.md)
+- **Upgrade Strategies**: [../../platform/openshift-specifics/upgrade-strategies.md](../../platform/openshift-specifics/upgrade-strategies.md)
diff --git a/ai-docs/practices/security/index.md b/ai-docs/practices/security/index.md
new file mode 100644
index 0000000000..88d3d47660
--- /dev/null
+++ b/ai-docs/practices/security/index.md
@@ -0,0 +1,34 @@
+# Security Practices
+
+Security patterns and threat modeling for OpenShift components.
+
+## Guidance
+
+Security practices are primarily documented in:
+- [../../dev-guide/](../../../dev-guide/) - Security conventions
+- [OpenShift Security Guide](https://docs.openshift.com/container-platform/latest/security/index.html)
+
+## Key Security Patterns
+
+| Pattern | When to Use | Reference |
+|---------|-------------|-----------|
+| **RBAC** | API access control | [RBAC Docs](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) |
+| **SCCs** | Pod security | [SCC Guide](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html) |
+| **Secret Handling** | Credentials, tokens | Never log/expose secrets |
+| **Network Policies** | Network segmentation | [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) |
+
+## Threat Modeling (STRIDE)
+
+| Threat | Mitigation |
+|--------|------------|
+| **Spoofing** | Use service accounts, RBAC |
+| **Tampering** | Immutable infrastructure, admission controllers |
+| **Repudiation** | Audit logs, structured logging |
+| **Information Disclosure** | Secret encryption, RBAC |
+| **Denial of Service** | Resource limits, rate limiting |
+| **Elevation of Privilege** | RBAC, SCCs, least privilege |
+
+## Related
+
+- **Dev Guide**: [../../dev-guide/](../../../dev-guide/)
+- **Platform Patterns**: [../../platform/](../../platform/)
diff --git a/ai-docs/practices/testing/index.md b/ai-docs/practices/testing/index.md
new file mode 100644
index 0000000000..225389dcaa
--- /dev/null
+++ b/ai-docs/practices/testing/index.md
@@ -0,0 +1,18 @@
+# Testing Practices
+
+Testing strategies and conventions for OpenShift components.
+
+## Guides
+
+- [pyramid.md](pyramid.md) - Testing pyramid: unit, integration, e2e guidance
+
+## Dev Guide References
+
+For authoritative testing conventions, see:
+- [../../dev-guide/test-conventions.md](../../../dev-guide/test-conventions.md) - Official test conventions
+- [../../dev-guide/ci/](../../../dev-guide/ci/) - CI configuration and integration
+
+## Related
+
+- [E2E Framework](https://github.com/openshift/origin/tree/master/test/extended)
+- [CI Documentation](https://docs.ci.openshift.org/)
diff --git a/ai-docs/practices/testing/pyramid.md b/ai-docs/practices/testing/pyramid.md
new file mode 100644
index 0000000000..e777efd147
--- /dev/null
+++ b/ai-docs/practices/testing/pyramid.md
@@ -0,0 +1,207 @@
+# Testing Pyramid
+
+**Category**: Engineering Practice
+**Last Updated**: 2026-04-29
+
+## Overview
+
+OpenShift follows the testing pyramid: many unit tests, fewer integration tests, minimal e2e tests.
+
+## Test Levels
+
+| Level | Speed | Scope | When to Use |
+|-------|-------|-------|-------------|
+| **Unit** | <100ms | Single function/method | Logic, edge cases, error handling |
+| **Integration** | <5s | Multiple components | API contracts, DB interactions |
+| **E2E** | 1-30min | Full system | Critical user flows, upgrade paths |
+
+## Unit Tests
+
+**Characteristics**:
+- Fast (<100ms per test)
+- No external dependencies (mock DB, API calls)
+- Test single functions/methods
+- High coverage (>80% for new code)
+
+```go
+// Example: Unit test with mocking
+func TestReconcileDeployment(t *testing.T) {
+ client := fake.NewClientBuilder().WithObjects(
+ &appsv1.Deployment{...},
+ ).Build()
+
+ r := &Reconciler{client: client}
+
+ result, err := r.reconcileDeployment(context.TODO(), req)
+
+ assert.NoError(t, err)
+ assert.False(t, result.Requeue)
+}
+```
+
+**Best practices**:
+- ✅ Mock external dependencies
+- ✅ Test error paths
+- ✅ Test edge cases (nil, empty, max values)
+- ❌ Don't test framework code (controller-runtime handles watches)
+
+## Integration Tests
+
+**Characteristics**:
+- Moderate speed (<5s per test)
+- Real components (etcd via envtest)
+- Test API contracts and multi-component interactions
+- Run in CI on every PR
+
+```go
+// Example: Integration test with envtest
+func TestControllerIntegration(t *testing.T) {
+ testEnv := &envtest.Environment{
+ CRDDirectoryPaths: []string{"config/crd/bases"},
+ }
+
+ cfg, _ := testEnv.Start()
+ defer testEnv.Stop()
+
+ k8sClient, _ := client.New(cfg, client.Options{})
+
+ // Create resource
+ obj := &v1.MyResource{...}
+ k8sClient.Create(ctx, obj)
+
+ // Wait for reconciliation
+ Eventually(func() bool {
+ k8sClient.Get(ctx, key, obj)
+ return obj.Status.Ready
+ }, timeout, interval).Should(BeTrue())
+}
+```
+
+**Best practices**:
+- ✅ Use envtest for realistic API server
+- ✅ Test CRD validation, webhooks
+- ✅ Test controller watches and reconciliation
+- ❌ Don't test full cluster setup (use e2e instead)
+
+## E2E Tests
+
+**Characteristics**:
+- Slow (1-30 minutes)
+- Real cluster (OpenShift CI)
+- Test critical user flows
+- Run on merge and nightly
+
+```go
+// Example: E2E test
+func TestUpgrade(t *testing.T) {
+ // Install operator
+ installOperator(t)
+
+ // Deploy workload
+ deployApp(t)
+
+ // Trigger upgrade
+ upgradeCluster(t, "4.16.0", "4.16.1")
+
+ // Verify workload still running
+ assertAppHealthy(t)
+}
+```
+
+**Best practices**:
+- ✅ Test critical paths only (upgrade, install, day-2 ops)
+- ✅ Clean up resources (don't leak)
+- ✅ Use retries and timeouts
+- ❌ Don't test every feature (use integration tests)
+
+## When to Use Each Level
+
+| Scenario | Test Level | Rationale |
+|----------|-----------|-----------|
+| Validate input parsing | Unit | Pure logic, no dependencies |
+| Check CRD schema validation | Integration | Needs API server CRD handling |
+| Verify operator reconciles resource | Integration | Needs controller-runtime + envtest |
+| Confirm cluster upgrade works | E2E | Needs full cluster + CVO |
+| Test edge case (nil pointer) | Unit | Fast, isolated |
+| Verify webhook rejects invalid config | Integration | Needs admission controller |
+| Test N→N+1 version skew | E2E | Needs multi-version cluster |
+
+## Test Organization
+
+```
+my-operator/
+├── pkg/
+│ └── controller/
+│ ├── reconcile.go
+│ └── reconcile_test.go # Unit tests
+├── test/
+│ ├── integration/
+│ │ └── controller_test.go # Integration tests (envtest)
+│ └── e2e/
+│ ├── install_test.go # E2E tests
+│ └── upgrade_test.go
+└── Makefile
+ ├── test-unit
+ ├── test-integration
+ └── test-e2e
+```
+
+## CI Integration
+
+```yaml
+# .ci-operator.yaml
+tests:
+- as: unit
+ commands: make test-unit
+ container:
+ from: src
+
+- as: integration
+ commands: make test-integration
+ container:
+ from: src
+
+- as: e2e
+ steps:
+ cluster_profile: aws
+ test:
+ - as: test
+ commands: make test-e2e
+ from: src
+```
+
+## Coverage Targets
+
+| Test Level | Coverage Target | Measurement |
+|-----------|----------------|-------------|
+| Unit | >80% | `go test -cover` |
+| Integration | API contracts | All CRDs, webhooks tested |
+| E2E | Critical paths | Install, upgrade, day-2 ops |
+
+## Common Antipatterns
+
+❌ **Inverted pyramid**: More e2e than unit tests (slow CI)
+❌ **Testing framework**: Unit testing controller-runtime watch logic
+❌ **No mocks**: Unit tests calling real API server
+❌ **Flaky e2e**: No retries, tight timeouts
+❌ **Missing cleanup**: e2e tests leak resources
+
+## Examples in Components
+
+| Component | Test Strategy | Notes |
+|-----------|--------------|-------|
+| cluster-version-operator | Heavy integration, critical e2e | Tests upgrade orchestration |
+| machine-api-operator | Integration for API, e2e for cloud | Cloud provider interactions |
+| kube-apiserver | Unit for logic, e2e for availability | HA and upgrade critical |
+
+## Tools
+
+- **Unit**: `go test`, `testify/assert`, `gomock`
+- **Integration**: `controller-runtime/envtest`, `ginkgo`
+- **E2E**: OpenShift CI, `e2e-framework`
+
+## References
+
+- **Dev Guide**: [test-conventions.md](../../../dev-guide/test-conventions.md)
+- **E2E Framework**: [origin/test/extended](https://github.com/openshift/origin/tree/master/test/extended)
+- **CI**: [ci-operator](https://docs.ci.openshift.org/)
diff --git a/ai-docs/workflows/enhancement-process.md b/ai-docs/workflows/enhancement-process.md
new file mode 100644
index 0000000000..3494fe3e7e
--- /dev/null
+++ b/ai-docs/workflows/enhancement-process.md
@@ -0,0 +1,164 @@
+# Enhancement Process
+
+**Purpose**: AI-optimized guide for writing enhancement proposals
+**Authoritative Source**: [../guidelines/enhancement_template.md](../../guidelines/enhancement_template.md)
+**Last Updated**: 2026-04-29
+
+## Quick Start
+
+| Step | Action | Output |
+|------|--------|--------|
+| 1 | Copy template | `cp guidelines/enhancement_template.md enhancements/my-feature.md` |
+| 2 | Fill required sections | Summary, Motivation, Proposal |
+| 3 | Create PR | Tag with `kind/enhancement` |
+| 4 | Address review | Update based on feedback |
+| 5 | Merge | Enhancement approved |
+
+## Required Sections
+
+| Section | Purpose | Length | Required |
+|---------|---------|--------|----------|
+| **Summary** | 1-sentence description | 1 sentence | ✅ |
+| **Motivation** | Why this feature | 2-5 paragraphs | ✅ |
+| **Goals** | What we will do | Bulleted list | ✅ |
+| **Non-Goals** | What we won't do | Bulleted list | ✅ |
+| **Proposal** | How it works | 1-3 pages | ✅ |
+| **Risks and Mitigations** | What could go wrong | Table | ✅ |
+| **Drawbacks** | Why not to do this | 1-2 paragraphs | Optional |
+| **Alternatives** | Other approaches considered | List with rationale | Optional |
+| **Implementation Details** | Technical design | API examples, diagrams | ✅ |
+| **Test Plan** | How to verify | Test scenarios | ✅ |
+| **Graduation Criteria** | When to promote | DevPreview → TechPreview → GA (or alpha → GA if confident) | ✅ |
+| **Upgrade/Downgrade** | Migration strategy | Version compatibility | If applicable |
+
+## Enhancement States
+
+```
+Idea → Proposal → Implementable → Implemented → Stable
+ ↓ ↓ ↓ ↓ ↓
+Draft Review Approved Merged GA
+```
+
+## Template Structure
+
+```yaml
+---
+title: my-feature
+authors:
+ - "@myusername"
+reviewers:
+ - "@reviewer1"
+approvers:
+ - "@approver1"
+creation-date: 2026-04-29
+status: implementable
+---
+
+# My Feature
+
+## Summary
+
+One sentence describing the feature.
+
+## Motivation
+
+### Goals
+- Goal 1
+- Goal 2
+
+### Non-Goals
+- Non-goal 1
+
+## Proposal
+
+### User Stories
+- As a cluster admin, I want...
+
+### API Changes
+```yaml
+apiVersion: config.openshift.io/v1
+kind: MyResource
+spec:
+ newField: value
+```
+
+### Risks and Mitigations
+
+| Risk | Mitigation |
+|------|------------|
+| Risk 1 | Mitigation 1 |
+
+### Test Plan
+- Unit tests for...
+- E2E tests for...
+
+### Graduation Criteria
+- DevPreview: API defined, basic implementation
+- TechPreview: Feature complete, tested (optional if confident in API)
+- GA: Production-ready, stable API, no breaking changes allowed
+
+See [../guidelines/supportability.md](../../guidelines/supportability.md) for Tech Preview policy and when you can skip TechPreview stage.
+
+## Implementation History
+- 2026-04-29: Initial proposal
+```
+
+## API Design Checklist
+
+- [ ] API group and version chosen (config.openshift.io/v1, etc.)
+- [ ] CRD schema defined with validation
+- [ ] Status conditions follow Available/Progressing/Degraded pattern
+- [ ] Backward compatibility considered
+- [ ] Upgrade/downgrade path documented
+- [ ] Default values specified
+
+## Review Process
+
+| Stage | Criteria | Reviewers |
+|-------|----------|-----------|
+| **Initial Review** | Complete template, clear motivation | Enhancement team |
+| **API Review** | API follows conventions | API review team |
+| **Implementation Review** | Technical feasibility | Component owners |
+| **Approval** | All concerns addressed | Approvers |
+
+## Common Mistakes
+
+❌ **Missing motivation**: "We need feature X" (why?)
+❌ **Unclear proposal**: High-level description without implementation details
+❌ **No test plan**: How will we verify this works?
+❌ **Ignoring upgrades**: What happens to existing clusters?
+❌ **Breaking changes**: Not considering backward compatibility
+❌ **Wrong delivery mechanism**: Should this be in payload or OLM? See [new-components.md](../../dev-guide/new-components.md)
+
+## Examples
+
+| Enhancement | Type | Notes |
+|-------------|------|-------|
+| [ClusterOperator Conditions](https://github.com/openshift/enhancements/blob/master/dev-guide/cluster-version-operator/dev/clusteroperator.md) | Pattern | Status reporting standard |
+| [Machine API](https://github.com/openshift/enhancements/blob/master/enhancements/machine-api/) | Feature | Node lifecycle management |
+| [Feature Gates](https://github.com/openshift/enhancements/blob/master/dev-guide/featuresets.md) | Process | Graduation criteria |
+
+## Workflow Commands
+
+```bash
+# Create enhancement
+cp guidelines/enhancement_template.md enhancements/my-feature.md
+
+# Validate format (run markdown linter)
+make lint
+
+# Create PR
+gh pr create --title "Enhancement: My Feature" --body "..." --label kind/enhancement
+
+# View enhancements
+ls enhancements/
+```
+
+## Related
+
+- **Authoritative Template**: [../guidelines/enhancement_template.md](../../guidelines/enhancement_template.md)
+- **Supportability Policy**: [../guidelines/supportability.md](../../guidelines/supportability.md) - Tech Preview vs GA, graduation paths
+- **New Component Policy**: [../dev-guide/new-components.md](../../dev-guide/new-components.md) - Payload vs OLM decision criteria
+- **Topology Considerations**: [topology-considerations-guide.md](topology-considerations-guide.md) - Guide for SNO, MicroShift, Hypershift, OKE sections
+- **API Conventions**: [../practices/development/api-evolution.md](../practices/development/api-evolution.md)
+- **Feature Development**: [implementing-features.md](implementing-features.md)
diff --git a/ai-docs/workflows/exec-plans/README.md b/ai-docs/workflows/exec-plans/README.md
new file mode 100644
index 0000000000..3b99282ade
--- /dev/null
+++ b/ai-docs/workflows/exec-plans/README.md
@@ -0,0 +1,165 @@
+# Exec-Plans: Feature Implementation Tracking
+
+**Purpose**: Track multi-week feature implementation progress (Platform guidance)
+**Last Updated**: 2026-04-29
+
+## Overview
+
+Exec-plans are ephemeral documents that track feature implementation. They bridge the gap between enhancement (design) and implementation (code).
+
+**Key Points**:
+- Guidance lives here (Platform docs)
+- Actual exec-plans live in component repos (`agentic/exec-plans/active/`)
+- Exec-plans are temporary → extract to permanent docs, then delete
+
+## When to Use Exec-Plans
+
+| Scenario | Use Exec-Plan? | Why |
+|----------|---------------|-----|
+| Multi-week feature (3+ PRs) | ✅ Yes | Track progress, coordinate PRs |
+| Single PR feature | ❌ No | PR description sufficient |
+| Bug fix | ❌ No | Issue tracker sufficient |
+| Multi-component coordination | ✅ Yes | Track cross-repo dependencies |
+| Experimental spike | ❌ No | Use spike branch, not exec-plan |
+
+**Rule of thumb**: If you'll have 3+ PRs over 2+ weeks, use an exec-plan.
+
+## Exec-Plan vs Enhancement
+
+| Aspect | Enhancement | Exec-Plan |
+|--------|-------------|-----------|
+| **Purpose** | Design approval | Implementation tracking |
+| **Location** | `enhancements/` (permanent) | `agentic/exec-plans/active/` (ephemeral) |
+| **Scope** | What and why | How and when |
+| **Lifetime** | Permanent (historical record) | Temporary (delete after completion) |
+| **Audience** | Reviewers, future developers | Current implementation team |
+
+**Relationship**: Enhancement = design spec, Exec-plan = implementation plan
+
+## Structure
+
+```
+component-repo/
+└── agentic/
+ └── exec-plans/
+ ├── active/
+ │ ├── YYYY-MM-my-feature.md # Active plans
+ │ └── YYYY-MM-another-feature.md
+ ├── archive/ # Completed (or use git history)
+ └── template.md # Copy this for new plans
+```
+
+## Template
+
+See [template.md](template.md) for the full template.
+
+**Core sections**:
+1. **Overview**: What, why, timeline
+2. **Implementation Tasks**: Checklist of PRs/work items
+3. **Dependencies**: Blockers, cross-component coordination
+4. **Progress Log**: Weekly updates
+5. **Decisions**: Key technical decisions made during implementation
+6. **Completion**: Outcome, follow-ups, knowledge extraction
+
+## Workflow
+
+### 1. Start: Create Exec-Plan
+
+```bash
+# Copy template
+cp agentic/exec-plans/template.md agentic/exec-plans/active/2026-04-my-feature.md
+
+# Fill in:
+# - Overview (what, why, timeline)
+# - Implementation tasks (checklist)
+# - Dependencies
+```
+
+### 2. During: Update Progress
+
+```bash
+# Weekly updates:
+# - Check off completed tasks
+# - Add progress log entries
+# - Document decisions
+# - Update timeline if needed
+```
+
+### 3. Complete: Extract & Delete
+
+```bash
+# Extract knowledge:
+# - Key decisions → ADR in decisions/
+# - Architecture changes → Update architecture.md
+# - Patterns → Update relevant pattern docs
+
+# Then delete or archive exec-plan:
+git rm agentic/exec-plans/active/2026-04-my-feature.md
+# Or: mv active/2026-04-my-feature.md archive/
+```
+
+## Best Practices
+
+1. **Keep Updated**: Update at least weekly (more often for active development)
+
+2. **Check Off Tasks**: Visible progress motivates and informs stakeholders
+
+3. **Document Decisions**: Capture "why" not just "what"
+ - Why we chose approach X over Y
+ - Why we changed direction mid-implementation
+
+4. **Extract Knowledge**: Don't let decisions disappear
+ - Significant decisions → ADR
+ - Architecture changes → Permanent docs
+ - Patterns → Platform or component guides
+
+5. **Delete When Done**: Exec-plans are ephemeral
+ - Extract value to permanent docs
+ - Delete to avoid doc rot
+
+## Example Timeline
+
+```
+Week 1:
+ [x] Create exec-plan
+ [x] PR #123: API changes
+ [ ] PR #124: Controller logic
+
+Week 2:
+ [x] PR #124: Controller logic
+ [ ] PR #125: E2E tests
+ Decision: Changed approach from X to Y because...
+
+Week 3:
+ [x] PR #125: E2E tests
+ [x] All PRs merged
+
+Week 4:
+ [x] Extract decisions to ADR
+ [x] Update architecture docs
+ [x] Delete exec-plan
+```
+
+## Integration with Other Docs
+
+| Doc Type | Relationship | Example |
+|----------|--------------|---------|
+| **Enhancement** | Exec-plan implements enhancement | Enhancement: design, Exec-plan: tracking |
+| **ADR** | Extract key decisions from exec-plan | "Why we chose etcd" → ADR |
+| **Architecture** | Update with changes from exec-plan | New component → architecture.md |
+| **Pattern Docs** | Extract patterns discovered | New pattern → pattern doc |
+
+## Antipatterns
+
+❌ **Permanent exec-plans**: Exec-plans should be deleted after completion
+❌ **No updates**: Exec-plan created but never updated (use PR description instead)
+❌ **Too detailed**: Line-by-line code plan (let PRs evolve)
+❌ **No extraction**: Delete without extracting decisions to permanent docs
+❌ **Single PR features**: Don't need exec-plan (PR description sufficient)
+
+## Related
+
+- **Template**: [template.md](template.md) - Copy this for new exec-plans
+- **Enhancement Process**: [../enhancement-process.md](../enhancement-process.md)
+- **Feature Implementation**: [../implementing-features.md](../implementing-features.md)
+- **ADRs**: [../../decisions/](../../decisions/) - Where to extract key decisions
diff --git a/ai-docs/workflows/exec-plans/template.md b/ai-docs/workflows/exec-plans/template.md
new file mode 100644
index 0000000000..3a126fb940
--- /dev/null
+++ b/ai-docs/workflows/exec-plans/template.md
@@ -0,0 +1,169 @@
+# Exec-Plan: [Feature Name]
+
+**Status**: Active | Blocked | Completed
+**Start Date**: YYYY-MM-DD
+**Target Completion**: YYYY-MM-DD
+**Owner**: @username
+**Related Enhancement**: [Link to enhancement](../../../enhancements/my-feature.md)
+
+---
+
+## Overview
+
+### What
+Brief description of what this feature does (1-2 sentences).
+
+### Why
+Why we're building this (link to enhancement for details).
+
+### Timeline
+- **Start**: YYYY-MM-DD
+- **Target**: YYYY-MM-DD
+- **Status**: On track | At risk | Delayed
+
+---
+
+## Implementation Tasks
+
+### Phase 1: API Changes
+- [ ] Define CRD schema (PR #XXX)
+- [ ] Add validation webhook (PR #XXX)
+- [ ] Update API types (PR #XXX)
+- [ ] API review approved
+
+**Target**: Week 1
+
+### Phase 2: Controller Implementation
+- [ ] Implement reconciliation logic (PR #XXX)
+- [ ] Add unit tests (>80% coverage)
+- [ ] Add integration tests
+- [ ] Handle error cases
+
+**Target**: Week 2-3
+
+### Phase 3: Testing & Documentation
+- [ ] E2E tests for critical paths (PR #XXX)
+- [ ] Upgrade/downgrade tests
+- [ ] Documentation (user-facing)
+- [ ] Metrics and alerts
+
+**Target**: Week 4
+
+---
+
+## Dependencies
+
+### Blockers
+- [ ] Dependency 1: Description (blocked by #XXX)
+- [ ] Dependency 2: Description (waiting on team Y)
+
+### Cross-Component Coordination
+- [ ] Component A: API contract defined
+- [ ] Component B: Integration tested
+
+---
+
+## Progress Log
+
+### YYYY-MM-DD (Week 4)
+**Progress**:
+- Completed PR #125 (E2E tests)
+- All PRs merged
+
+**Next**:
+- Extract decisions to ADR
+- Delete exec-plan
+
+### YYYY-MM-DD (Week 3)
+**Progress**:
+- Completed PR #124 (controller logic)
+- Started PR #125 (E2E tests)
+
+**Blockers**: None
+
+**Next**:
+- Finish E2E tests
+- Final review
+
+### YYYY-MM-DD (Week 2)
+**Progress**:
+- PR #123 merged (API changes)
+- Started controller implementation
+
+**Decisions**:
+- Changed from approach X to Y because... (extract to ADR)
+
+**Next**:
+- Complete controller logic
+- Add integration tests
+
+### YYYY-MM-DD (Week 1)
+**Progress**:
+- Created exec-plan
+- Opened PR #123 (API changes)
+
+**Next**:
+- Complete API review
+- Start controller implementation
+
+---
+
+## Decisions
+
+### Decision 1: [Title]
+**Date**: YYYY-MM-DD
+**Decision**: What we decided
+**Rationale**: Why we decided this
+**Alternatives**: What we considered
+**Status**: Extract to ADR: [ ] Yes [x] No
+
+### Decision 2: [Title]
+**Date**: YYYY-MM-DD
+**Decision**: ...
+**Rationale**: ...
+**Status**: Extract to ADR: [x] Yes [ ] No
+
+---
+
+## Completion Checklist
+
+### Code
+- [ ] All PRs merged
+- [ ] Tests passing in CI
+- [ ] Documentation updated
+- [ ] Feature gate configured (if TechPreview)
+
+### Knowledge Extraction
+- [ ] Key decisions extracted to ADRs
+- [ ] Architecture docs updated
+- [ ] Patterns documented (if new patterns emerged)
+- [ ] Lessons learned captured
+
+### Cleanup
+- [ ] Exec-plan archived or deleted
+- [ ] Related issues closed
+- [ ] Team notified of completion
+
+---
+
+## Outcome
+
+**Status**: [Completed | Deferred | Cancelled]
+**Completion Date**: YYYY-MM-DD
+
+### What Shipped
+- Feature X in version Y.Z
+- PRs: #123, #124, #125
+
+### Follow-Ups
+- [ ] Follow-up 1: Description (issue #XXX)
+- [ ] Follow-up 2: Description (issue #XXX)
+
+### Knowledge Extracted
+- ADR: [Link to decision doc]
+- Architecture update: [Link]
+- Pattern doc: [Link]
+
+---
+
+**Note**: This exec-plan is ephemeral. After completion, extract key decisions and delete this file.
diff --git a/ai-docs/workflows/implementing-features.md b/ai-docs/workflows/implementing-features.md
new file mode 100644
index 0000000000..fccceaa01a
--- /dev/null
+++ b/ai-docs/workflows/implementing-features.md
@@ -0,0 +1,225 @@
+# Implementing Features
+
+**Purpose**: Structured workflow from enhancement to production
+**Last Updated**: 2026-04-29
+
+## Overview
+
+Feature implementation follows: Spec → Plan → Build → Test → Review → Ship
+
+## Workflow Stages
+
+| Stage | Input | Output | Duration |
+|-------|-------|--------|----------|
+| **1. Spec** | Enhancement proposal | Approved enhancement | 1-2 weeks |
+| **2. Plan** | Enhancement | Implementation plan | 1-3 days |
+| **3. Build** | Plan | Code + tests | 1-8 weeks |
+| **4. Test** | Code | Passing CI | Continuous |
+| **5. Review** | PR | Approved PR | 1-3 days |
+| **6. Ship** | Merged PR | Production release | 1-4 weeks |
+
+## Stage 1: Spec (Enhancement)
+
+**Goal**: Get approval for feature design
+
+**Checklist**:
+- [ ] Enhancement proposal written (see [enhancement-process.md](enhancement-process.md))
+- [ ] API design reviewed
+- [ ] Risks and mitigations identified
+- [ ] Test plan defined
+- [ ] Graduation criteria set (DevPreview → TechPreview → GA, or DevPreview → GA if API stable)
+- [ ] Enhancement PR merged
+
+**Artifacts**: Merged enhancement in `enhancements/`
+
+## Stage 2: Plan (Implementation Plan)
+
+**Goal**: Break enhancement into implementable tasks
+
+**Checklist**:
+- [ ] Identify affected components
+- [ ] List required API changes
+- [ ] Define PR sequence (API → implementation → tests)
+- [ ] Estimate timeline
+- [ ] Identify dependencies
+- [ ] Create exec-plan if multi-week (see [exec-plans/](exec-plans/))
+
+**Artifacts**: Implementation plan (PR description or exec-plan doc)
+
+**Example Plan**:
+```markdown
+## Implementation Plan
+
+### PR 1: API Changes
+- Add new CRD field `spec.newFeature`
+- Update validation webhook
+- Files: api/, webhook/
+
+### PR 2: Controller Logic
+- Implement reconciliation for new field
+- Add unit tests
+- Files: controllers/
+
+### PR 3: E2E Tests
+- Add upgrade test
+- Add feature test
+- Files: test/e2e/
+
+Timeline: 3 weeks
+Dependencies: None
+```
+
+## Stage 3: Build (Code + Tests)
+
+**Goal**: Implement feature with tests
+
+**Checklist**:
+- [ ] API changes (CRD, types, validation)
+- [ ] Controller logic (reconciliation)
+- [ ] Unit tests (>80% coverage)
+- [ ] Integration tests (API contracts)
+- [ ] E2E tests (critical paths)
+- [ ] Documentation (godoc, user-facing)
+- [ ] Feature gate if TechPreview
+
+**Test Pyramid**:
+- Unit tests (fast, isolated) — most tests
+- Integration tests (envtest) — API contracts
+- E2E tests (full cluster) — critical paths only
+
+See [../practices/testing/pyramid.md](../practices/testing/pyramid.md)
+
+## Stage 4: Test (CI Validation)
+
+**Goal**: Ensure tests pass in CI
+
+**Checklist**:
+- [ ] Unit tests pass (`make test-unit`)
+- [ ] Integration tests pass (`make test-integration`)
+- [ ] E2E tests pass (CI job)
+- [ ] Linting passes (`make verify`)
+- [ ] No flaky tests
+- [ ] Coverage meets target
+
+**Common CI Jobs**:
+- `pull-ci----unit`
+- `pull-ci----e2e-aws`
+- `pull-ci----verify`
+
+## Stage 5: Review (PR Review)
+
+**Goal**: Get code review and approval
+
+**Checklist**:
+- [ ] PR description explains changes
+- [ ] Tests included
+- [ ] Documentation updated
+- [ ] CHANGELOG entry added (if applicable)
+- [ ] API review approved (if API changes)
+- [ ] Code review approved (/lgtm)
+- [ ] Required approvers approved (/approve)
+
+**Review Areas**:
+- Code quality (readability, maintainability)
+- Test coverage (edge cases, error paths)
+- API design (backward compatibility)
+- Performance (no obvious regressions)
+- Security (no vulnerabilities)
+
+## Stage 6: Ship (Production Release)
+
+**Goal**: Feature available in release
+
+**Checklist**:
+- [ ] PR merged to main branch
+- [ ] Feature included in next release image
+- [ ] Release notes written
+- [ ] Documentation published
+- [ ] Metrics dashboard created (if applicable)
+- [ ] Alerts configured (if applicable)
+
+**Graduation Path**:
+```
+DevPreview (4.16) → TechPreview (4.17) → GA (4.18)
+OR
+DevPreview (4.16) → GA (4.17) # If API stable and confident
+```
+
+See [../guidelines/supportability.md](../../guidelines/supportability.md) for when you can skip TechPreview.
+
+## Feature States
+
+| State | Meaning | Support Level | API Stability |
+|-------|---------|--------------|---------------|
+| **DevPreview** | Experimental | Best-effort | May change |
+| **TechPreview** | Feature-complete | Supported (no SLA) | Mostly stable |
+| **GA** | Production-ready | Fully supported | Stable |
+
+## Multi-PR Strategy
+
+**Pattern**: API → Implementation → Tests
+
+```
+PR 1 (API):
+ - Add CRD fields
+ - Update validation
+ - Merge: Week 1
+
+PR 2 (Implementation):
+ - Add controller logic
+ - Add unit tests
+ - Depends on: PR 1
+ - Merge: Week 2
+
+PR 3 (E2E):
+ - Add e2e tests
+ - Depends on: PR 2
+ - Merge: Week 3
+```
+
+**Benefits**:
+- Smaller, easier-to-review PRs
+- Incremental progress
+- Easier to revert if needed
+
+## Rollout Strategy
+
+| Strategy | Use Case | Example |
+|----------|----------|---------|
+| **Feature Gate** | TechPreview features | `FeatureGates: [MyFeature]` |
+| **Operator Flag** | Opt-in features | `--enable-my-feature=true` |
+| **Progressive Rollout** | Gradual activation | Enable 10% → 50% → 100% |
+| **API Version** | Breaking changes | v1alpha1 → v1beta1 → v1 |
+
+## Monitoring Post-Ship
+
+**Checklist**:
+- [ ] Monitor error rates (Prometheus)
+- [ ] Check for support cases (Jira)
+- [ ] Watch for regression bugs
+- [ ] Verify upgrade scenarios work
+- [ ] Gather user feedback
+
+## Common Pitfalls
+
+❌ **No test plan**: Feature works locally but breaks in CI
+❌ **Missing upgrade path**: Old clusters break on upgrade
+❌ **Insufficient testing**: Only happy path tested
+❌ **No feature gate**: Can't disable if bugs found
+❌ **Unclear PR sequence**: Dependencies not obvious
+
+## Examples
+
+| Feature | PRs | Timeline | Notes |
+|---------|-----|----------|-------|
+| Machine API | 5 PRs | 8 weeks | API, controller, provider, tests |
+| ClusterVersion | 3 PRs | 4 weeks | API, CVO logic, e2e |
+| Feature Gates | 2 PRs | 2 weeks | Implementation, docs |
+
+## Related
+
+- **Enhancement Process**: [enhancement-process.md](enhancement-process.md)
+- **Supportability Policy**: [../guidelines/supportability.md](../../guidelines/supportability.md) - Tech Preview vs GA, graduation paths
+- **Exec Plans**: [exec-plans/](exec-plans/) - Track multi-week features
+- **Testing**: [../practices/testing/pyramid.md](../practices/testing/pyramid.md)
+- **API Evolution**: [../practices/development/api-evolution.md](../practices/development/api-evolution.md)
diff --git a/ai-docs/workflows/index.md b/ai-docs/workflows/index.md
new file mode 100644
index 0000000000..62c9c1dc2f
--- /dev/null
+++ b/ai-docs/workflows/index.md
@@ -0,0 +1,23 @@
+# Workflows
+
+AI-optimized guides for OpenShift development processes.
+
+## Workflow Guides
+
+- [enhancement-process.md](enhancement-process.md) - Writing enhancement proposals (structured version of `../guidelines/enhancement_template.md`)
+- [topology-considerations-guide.md](topology-considerations-guide.md) - Guide for addressing SNO, MicroShift, Hypershift, and OKE in enhancement proposals
+- [implementing-features.md](implementing-features.md) - Feature implementation workflow (spec → plan → build → test → review → ship)
+- [exec-plans/](exec-plans/) - Feature tracking templates and guidance (Platform)
+
+## Authoritative Sources
+
+These workflows are AI-optimized versions of content from:
+- [../guidelines/](../../guidelines/) - Enhancement process, PR conventions
+- [../dev-guide/](../../dev-guide/) - Development conventions, feature development
+
+**Difference**: Same information, but formatted for AI agents (tables, checklists, YAML examples instead of prose).
+
+## Related
+
+- **Practices**: [../practices/](../practices/) - Testing, API evolution, security
+- **Platform Patterns**: [../platform/](../platform/) - Operator patterns
diff --git a/ai-docs/workflows/topology-considerations-guide.md b/ai-docs/workflows/topology-considerations-guide.md
index 053b0f785d..0c9b39b893 100644
--- a/ai-docs/workflows/topology-considerations-guide.md
+++ b/ai-docs/workflows/topology-considerations-guide.md
@@ -122,13 +122,14 @@ graph TD
| **Distributed quorum** | No quorum with 1 member | Use single-member mode or disable |
| **Network partitions** | Can't happen (1 node) | Simplifies some failure modes |
-**Topology detection**:
+**Topology detection** (API-level, used by operators at runtime):
```bash
-oc get node -l node.openshift.io/single-node-cluster
+oc get infrastructure cluster -o jsonpath='{.status.controlPlaneTopology}'
+# Returns "SingleReplica" for SNO, "HighlyAvailable" for multi-node
```
**Good example**:
-> **SNO impact**: This adds a DaemonSet consuming 100MB RAM and 50m CPU per node. In SNO, this is 100MB total. The feature detects single-node topology (via `node.openshift.io/single-node-cluster` label) and reduces replica count from 3 to 1 for the StatefulSet.
+> **SNO impact**: This adds a DaemonSet consuming 100MB RAM and 50m CPU per node. In SNO, this is 100MB total. The feature detects single-node topology (via `Infrastructure.Status.ControlPlaneTopology == SingleReplica`) and reduces replica count from 3 to 1 for the StatefulSet.
**Bad example**:
> No special considerations for single-node deployments.
@@ -266,7 +267,7 @@ When reviewing enhancement PRs, verify:
- [ ] Quantifies per-node memory/CPU overhead
- [ ] Addresses replica count assumptions (if any)
- [ ] Documents behavior if HA is assumed
-- [ ] Tests with `node.openshift.io/single-node-cluster` label
+- [ ] Tests with SNO topology (`Infrastructure.Status.ControlPlaneTopology == SingleReplica`)
**MicroShift**:
- [ ] Lists operator dependencies (are they in MicroShift?)