Agentic docs part 2: operator patterns, practices, workflows, and ADRs

AGENTS.md

@@ -77,7 +77,6 @@ When working on OpenShift:
 | **Webhooks** | `./ai-docs/platform/operator-patterns/webhooks.md` | Validation/mutation/conversion |
 | **Finalizers** | `./ai-docs/platform/operator-patterns/finalizers.md` | Cleanup external resources on deletion |
 | **RBAC** | `./ai-docs/platform/operator-patterns/rbac.md` | Service account permissions |
-| **must-gather** | `./ai-docs/platform/operator-patterns/must-gather.md` | Debugging and diagnostics |
 | **Upgrade safety** | `./ai-docs/platform/openshift-specifics/upgrade-strategies.md` | N→N+1 version skew, CVO coordination |
 
 ---
@@ -86,7 +85,7 @@ When working on OpenShift:
 
 | Area | Index | Description |
 |------|-------|-------------|
-| **Testing** | `./ai-docs/practices/testing/` | Pyramid (60/30/10), e2e framework |
+| **Testing** | `./ai-docs/practices/testing/` | Testing pyramid, e2e framework |
 | **Security** | `./ai-docs/practices/security/` | STRIDE, RBAC patterns, secret handling |
 | **Reliability** | `./ai-docs/practices/reliability/` | SLI/SLO/SLA, degraded-mode patterns |
 | **Development** | `./ai-docs/practices/development/` | API evolution, compatibility |
@@ -121,9 +120,9 @@ When working on OpenShift:
 **Index**: See `./ai-docs/decisions/index.md`
 
 **Common ADRs**:
-- Why etcd as backend
 - Why CVO orchestration model
 - Why immutable nodes (RHCOS + rpm-ostree)
+- Why standardized status conditions pattern
 
 ---
 
@@ -166,8 +165,8 @@ When working on OpenShift:
 │   ├── operator-patterns/        # Controller runtime, status, webhooks
 │   └── openshift-specifics/      # Upgrade safety, CVO coordination
 ├── domain/                       # Core API concepts
-│   ├── kubernetes/               # Pod, Service, CRD, Node
-│   └── openshift/                # ClusterOperator, ClusterVersion, Machine
+│   ├── kubernetes/               # Pod, Service, CRD
+│   └── openshift/                # ClusterOperator, ClusterVersion
 ├── practices/                    # Cross-cutting concerns
 │   ├── testing/                  # Pyramid, e2e framework
 │   ├── security/                 # STRIDE, RBAC, secrets

ai-docs/DESIGN_PHILOSOPHY.md

@@ -2,7 +2,7 @@
 
 **Purpose**: Core principles guiding OpenShift architecture and development
 
-**Last Updated**: YYYY-MM-DD
+**Last Updated**: 2026-06-24
 
 ---
 

ai-docs/KNOWLEDGE_GRAPH.md

@@ -19,9 +19,10 @@
 
 | Task | Start Here | Then Read | Finally |
 |------|-----------|----------|---------|
-| **Build an operator** | DESIGN_PHILOSOPHY.md | platform/operator-patterns/controller-runtime.md<br>platform/operator-patterns/status-conditions.md | domain/openshift/clusteroperator.md |
+| **Build a core operator** | DESIGN_PHILOSOPHY.md | platform/operator-patterns/controller-runtime.md<br>platform/operator-patterns/status-conditions.md | domain/openshift/clusteroperator.md |
+| **Build an OLM operator** | DESIGN_PHILOSOPHY.md | platform/operator-patterns/controller-runtime.md<br>platform/operator-patterns/status-conditions.md | [OLM packaging](https://olm.operatorframework.io/docs/tasks/creating-operator-manifests/) |
 | **Add a feature** | workflows/index.md (links to enhancement process) | practices/development/index.md (links to API conventions) | practices/testing/index.md |
-| **Debug an issue** | practices/reliability/index.md | platform/operator-patterns/must-gather.md | references/repo-index.md |
+| **Debug an issue** | practices/reliability/index.md | references/repo-index.md | `oc adm must-gather --help` |
 | **Understand a concept** | DESIGN_PHILOSOPHY.md | domain/kubernetes/ or domain/openshift/ | platform/operator-patterns/ |
 | **Find a component** | references/repo-index.md | Component's AGENTS.md | Component's agentic/ docs |
 
@@ -69,20 +70,40 @@
 
 ## Concept Dependencies
 
-### Operator Development Path (Core / Platform Operators)
-<!-- TODO: add a parallel path for non-core (OLM-managed) operators that use status conditions but don't report via ClusterOperator -->
+### Operator Development Paths
+
 ```
 DESIGN_PHILOSOPHY.md
   ↓
 controller-runtime.md (how reconciliation works)
   ↓
 status-conditions.md (how to report health)
   ↓
-clusteroperator.md (how CVO sees your operator)
-  ↓
-webhooks.md, rbac.md, finalizers.md (advanced patterns)
+  ├─── Core / Platform Operators ───────── OLM-Managed Operators ──┐
+  │                                                                 │
+  │  clusteroperator.md                    Your CRD's .status       │
+  │  (report to CVO via ClusterOperator)   (conditions on your CR)  │
+  │                                                                 │
+  │  CVO orchestrates upgrades             OLM manages lifecycle    │
+  │  Available/Progressing/Degraded/       metav1.Condition types    │
+  │    Upgradeable                         CSV defines install/RBAC │
+  │  configv1 condition types                                       │
+  │                                                                 │
+  └─────────────────────┬───────────────────────────┘
+                        ↓
+        webhooks.md, rbac.md, finalizers.md (advanced patterns)
 ```
 
+**Key difference**: Core operators create a `ClusterOperator` resource so the
+CVO can track their health and gate upgrades (the CVO advances when
+`Available=True` and `Degraded=False`; setting `Upgradeable=False` blocks
+minor-version updates). OLM-managed operators report status on their own
+Custom Resource using standard `metav1.Condition` and rely on OLM's
+`ClusterServiceVersion` for lifecycle management. OLM does **not** proxy
+operator health to the CVO — a degraded OLM-managed operator does not block
+cluster upgrades (though a CSV's `olm.maxOpenShiftVersion` annotation can
+block minor-version upgrades as a static compatibility constraint).
+
 ### API Development Path
 ```
 DESIGN_PHILOSOPHY.md

ai-docs/decisions/adr-0001-cvo-orchestration.md

@@ -0,0 +1,115 @@
+# ADR-0001: CVO Orchestrates Cluster Upgrades
+
+**Status**: Accepted  
+**Date**: 2026-06-24  
+**Scope**: Cross-repository  
+
+## Context
+
+OpenShift clusters contain dozens of operators and platform components that must be upgraded together when moving between versions. These components have ordering dependencies: for example, kube-apiserver must be updated before kube-controller-manager, and core API servers must be available before higher-level operators can reconcile.
+
+The upgrade process must:
+- Handle component interdependencies safely
+- Complete control plane upgrades within a reasonable time period (30m–1h)
+- Avoid disrupting running application workloads
+- Be resilient to failures, including node reboots and pod evictions mid-upgrade
+- Enforce N-1 minor version compatibility between components
+
+## Decision
+
+The Cluster Version Operator (CVO) is the single, centralized orchestrator for all cluster upgrades. It installs and updates all "second-level" operators by applying their manifests from the release image in a deterministic order.
+
+### Ordering model
+
+The CVO loads manifests from the `/release-manifests` directory within the release image. During upgrades, manifests are applied in lexicographic order using a runlevel convention:
+
+```
+0000_<runlevel>_<component-name>_<manifest-filename>
+```
+
+Assigned runlevels (from [CVO dev guide: operators.md](../../dev-guide/cluster-version-operator/dev/operators.md) and [upgrades.md](../../dev-guide/cluster-version-operator/dev/upgrades.md)):
+
+| Runlevel | Components |
+|----------|-----------|
+| 00-04 | CVO itself |
+| 05 | cluster-config-operator |
+| 07 | Network operator |
+| 08 | DNS operator |
+| 09 | Service certificate authority, machine approver |
+| 10-29 | Kubernetes operators (kube-apiserver, kube-controller-manager, kube-scheduler, etcd) |
+| 30-39 | Machine API |
+| 50-59 | Operator Lifecycle Manager (OLM) |
+| 60-69 | OpenShift core operators (openshift-apiserver, etc.) |
+| 70 | Disruptive node-level components, monitoring, samples |
+| 80 | Machine operators |
+
+Components sharing the same runlevel run in parallel (e.g., `0000_70_cluster-monitoring-operator_*` and `0000_70_cluster-samples-operator_*` execute concurrently). Within a component, manifests apply in lexicographic order.
+
+Install flattens inter-runlevel barriers (components across different runlevels can start in parallel), but intra-component ordering is preserved (e.g., a CRD manifest is applied before the Deployment within the same component). Upgrades apply strict runlevel ordering — the next runlevel does not start until the previous one completes.
+
+### Upgrade completion criteria
+
+The CVO uses ClusterOperator status conditions to determine when each component has finished upgrading:
+
+| Operation | Version | Available | Degraded | Progressing | Upgradeable |
+|-----------|---------|-----------|----------|-------------|-------------|
+| Install completion | any | true | any | any | any |
+| Begin patch upgrade | any | any | any | any | any |
+| Begin minor upgrade | any | any | any | any | not false |
+| Upgrade completion | target version (declared in operator status) | true | false | any | any |
+
+### Self-managing design
+
+The CVO is itself a regular pod in the cluster. During upgrades, when the MCO updates the operating system or when the release image updates the CVO image, the CVO gets drained and restarted like any other pod. The new CVO pod observes current cluster state and resumes reconciliation — there is no special handoff mechanism.
+
+This is intentional: by not special-casing its own upgrade, the CVO restart works the same way as it would after a kernel panic, hardware failure, or network partition. The "normal" code path and the "exceptional" path are identical, keeping the upgrade process robust and continuously tested.
+
+## Rationale
+
+- **Deterministic ordering** prevents subtle failures from components upgrading before their prerequisites are ready. Operators that lack sophistication about detecting their own prerequisites get safety from runlevel ordering.
+- **Centralized enforcement** means upgrade safety invariants (version gating, Upgradeable condition checks, override blocking) are enforced in one place rather than distributed across every operator.
+- **Self-managing resilience** ensures the upgrade process is robust against any disruption — the same reconciliation logic handles both normal restarts and failure recovery.
+- **Parallel execution within runlevels** keeps upgrade times reasonable while preserving safety between runlevels.
+
+## Consequences
+
+### Positive
+
+- All component upgrades follow a single, predictable ordering model
+- Upgrade failures are observable through ClusterVersion and ClusterOperator conditions
+- The system self-heals after any disruption during upgrade
+- N-1 version compatibility is enforced across all components
+
+### Negative
+
+- Adding a new component to the payload requires choosing a runlevel and understanding the ordering implications
+- The conservative ordering is sometimes stricter than necessary — most components tolerate version skew during upgrades, but ordering is kept conservative for edge cases
+- Node upgrades currently proceed in the same sequence as control plane upgrades; future improvements may allow independent node upgrade scheduling
+
+## Alternatives Considered
+
+### Alternative 1: OLM-managed upgrades
+
+**Description**: Use the Operator Lifecycle Manager to coordinate all operator upgrades through its dependency resolution system.
+
+**Rejected because**: OLM is itself a second-level operator managed by CVO (runlevel 50–59). It manages optional/add-on operators, not the platform itself. Platform components need to be available before OLM can function, creating a bootstrap dependency that cannot be resolved from within OLM.
+
+### Alternative 2: Distributed coordination
+
+**Description**: Each operator independently detects when its prerequisites are ready and upgrades itself, using leader election and CRD-based signaling.
+
+**Rejected because**: Distributing upgrade coordination across dozens of operators would require every operator to implement prerequisite detection correctly. The CVO's bias toward predictable ordering exists precisely because "operators that lack sophistication about detecting their prerequisites" benefit from centralized ordering. A distributed model trades simplicity for flexibility that is rarely needed.
+
+### Alternative 3: Manual operator ordering
+
+**Description**: Administrators manually trigger component upgrades in the correct order.
+
+**Rejected because**: OpenShift 4's design goal is that clusters "self-manage" by default. Manual ordering is error-prone, scales poorly, and conflicts with the operator-driven automation model.
+
+## References
+
+- Dev guide: [Upgrades and order](../../dev-guide/cluster-version-operator/dev/upgrades.md)
+- Dev guide: [Operator integration with CVO](../../dev-guide/cluster-version-operator/dev/operators.md)
+- Dev guide: [ClusterOperator conditions](../../dev-guide/cluster-version-operator/dev/clusteroperator.md)
+- AI docs: [ClusterVersion resource](../domain/openshift/clusterversion.md)
+- AI docs: [Upgrade strategies](../platform/openshift-specifics/upgrade-strategies.md)

ai-docs/decisions/adr-0002-immutable-nodes.md

@@ -0,0 +1,109 @@
+# ADR-0002: Immutable Node Infrastructure
+
+**Status**: Accepted  
+**Date**: 2026-06-24  
+**Scope**: Cross-repository  
+
+## Context
+
+OpenShift nodes run the kubelet, container runtime (CRI-O), and other system-level services that must be consistent, predictable, and updatable across the cluster. Traditional Linux node management using package managers (yum/dnf) and configuration management tools (Ansible, Puppet) creates challenges:
+
+- **Configuration drift**: Nodes diverge over time due to ad-hoc changes, failed partial updates, or inconsistent package versions
+- **Non-atomic updates**: Package-based updates can fail partway through, leaving the system in an inconsistent state
+- **Rollback difficulty**: Reverting a failed update requires understanding exactly which packages and configs changed
+- **Testing combinatorial explosion**: Every combination of packages, versions, and configuration must be tested independently
+
+## Decision
+
+OpenShift nodes use an immutable infrastructure model built on four components:
+
+1. **RHCOS (Red Hat CoreOS)**: A purpose-built, minimal operating system designed for running containerized workloads. Ships as a single versioned image tested with each OpenShift release.
+
+2. **rpm-ostree**: Provides transactional, atomic operating system updates. The entire OS is deployed as an image-like tree — updates either fully succeed or fully roll back. Extensions (like `kernel-rt` or `usbguard`) can be layered atomically on top of the base image.
+
+3. **Ignition**: Performs one-shot OS configuration at provision time based on the Ignition specification associated with each node's MachineConfigPool. This establishes the initial node state declaratively.
+
+4. **Machine Config Operator (MCO)**: Manages day-2 node configuration changes through MachineConfig resources. The MCO renders desired configuration, and the MachineConfigDaemon (MCD) on each node applies changes by draining the node, applying updates (OS image, files, systemd units), rebooting, and uncordoning.
+
+### Update flow
+
+1. A new MachineConfig is created (by MCO or administrator)
+2. MCO renders the desired configuration for each MachineConfigPool
+3. MCD on each node detects the change
+4. Nodes update one at a time per pool: drain → apply files and OS → reboot → uncordon
+5. If the update fails, rpm-ostree rolls back to the previous OS tree on next boot
+
+### Reboot-by-default principle
+
+All MachineConfig changes require a node drain and reboot by default. This ensures the node reaches a known-good state rather than attempting live patching of a running system. Administrators can opt into specific no-reboot changes for selected files via Node Disruption Policy, but the immutable model remains the foundation.
+
+## Rationale
+
+- **Predictable state**: Every node in a pool runs the same OS image and configuration. There is no accumulated drift from manual edits or imperative commands.
+- **Atomic rollback**: rpm-ostree's transactional model means a failed OS update automatically rolls back to the previous known-good tree. This is not possible with traditional package-by-package updates.
+- **Reduced testing surface**: The entire OS stack (base + extensions) ships as a single version tested with each OpenShift release. All content is versioned with and tested with the OS, eliminating per-package combinatorial testing.
+- **Security**: The read-only root filesystem and image-based deployment reduce the attack surface. Unauthorized modifications to the OS are detectable.
+
+## Consequences
+
+### Positive
+
+- Nodes are reproducible — a new node joining a pool is identical to existing nodes
+- OS updates are atomic and rollback-safe
+- Configuration is declarative and auditable through MachineConfig resources
+- Consistent across the fleet eliminates "works on my node" debugging
+
+### Negative
+
+- All configuration changes require a drain and reboot by default, which causes temporary workload disruption on that node
+- Installing custom packages or agents requires building custom OS layers (on-cluster layering) rather than running `yum install`
+- Ignition's one-shot model means some configuration fields are irreconcilable after initial provisioning — they cannot be changed without reprovisioning the node
+
+### Neutral
+
+- Administrators accustomed to SSH-based node management must adapt to the declarative MachineConfig model
+- Debugging node-level issues uses `oc debug node/` rather than direct SSH access
+
+## Alternatives Considered
+
+### Alternative 1: Traditional package management (yum/dnf on RHEL)
+
+**Description**: Run standard RHEL on nodes, manage packages via yum/dnf, configure via SSH or Ansible.
+
+**Pros**:
+- Familiar to Linux administrators
+- Flexible, can install any package at any time
+- No reboot required for many changes
+
+**Cons**:
+- Configuration drift over time
+- Non-atomic updates can leave nodes in inconsistent state
+- Rollback requires manual intervention
+- Combinatorial testing burden across package versions
+
+**Rejected because**: The operational cost of managing drift and non-atomic updates across a fleet of nodes outweighs the flexibility benefits. OpenShift's operator model depends on predictable, consistent node state.
+
+### Alternative 2: Mutable configuration management (Ansible/Puppet/Chef)
+
+**Description**: Use configuration management tools to continuously converge node state toward a desired specification.
+
+**Pros**:
+- Handles drift through continuous reconciliation
+- Mature tooling ecosystem
+- Does not require reboots
+
+**Cons**:
+- Convergence is not atomic — intermediate states during a run may be inconsistent
+- Agent-based systems add operational complexity
+- Configuration management DSLs add another abstraction layer alongside Kubernetes APIs
+
+**Rejected because**: The Kubernetes operator model already provides declarative reconciliation. Adding a separate configuration management layer for nodes creates a split-brain between Kubernetes-managed and externally-managed state. MCO provides the same continuous reconciliation but integrated into the Kubernetes API model.
+
+## References
+
+- AI docs: [Design Philosophy — Immutable Infrastructure](../DESIGN_PHILOSOPHY.md)
+- AI docs: [Upgrade strategies](../platform/openshift-specifics/upgrade-strategies.md)
+- Enhancement: [Machine Config Node](../../enhancements/machine-config/machine-config-node.md)
+- Enhancement: [RHCOS extensions](../../enhancements/rhcos/extensions.md)
+- Enhancement: [On-cluster layering](../../enhancements/machine-config/on-cluster-layering.md)
+- Enhancement: [Node Disruption Policy](../../enhancements/machine-config/admin-defined-node-disruption-policy.md)