White Paper: Gating the Gates — A Release Pipeline Strategy for a Mid-Sized Live-Service Game Studio
Author: Edison Kardirsson
Course: SYS25D – DevOps
Assignment: Assignment 2
Date: 2026-04-24
- Executive Summary
- Company Description
- Background and History
- Proposed Solution – Overview
- Proposed Solution – Technical
- Roles and Responsibilities
- Career and Skills Reflection
- References
Nordwind Interactive AB is a mid-sized Swedish game studio whose live-service title Planet Reavers serves roughly 180 000 daily players. Since launch, about 20 % of backend deployments have caused a user-visible incident within 48 hours, and an estimated 30–35 % of backend engineering capacity is spent on firefighting rather than on new features. Secrets live as plain-text environment variables, hotfixes are deployed manually through the Azure Portal, and there is no tested path for the required .NET 8 → .NET 10 migration before .NET 8 leaves support in November 2026.
This white paper proposes a release pipeline that acts as an automated gatekeeper between developers and production. The pipeline is built on GitHub Actions, Azure Container Apps, Bicep, Azure Key Vault and Managed Identities, with GitHub Advanced Security, Azure App Configuration for feature flags, and Azure Monitor plus Application Insights for observability.
The strategy rests on three pillars. First, blocking quality gates at every stage — tests, static analysis, code review, contract verification and canary health — enforced automatically rather than by convention. Second, layered release strategies: Canary as the default for player-facing services, Blue/Green for the .NET 10 migration itself, and feature flags for behavior-level risk. Third, Continuous Delivery with a single human approval for player-facing services, evolving toward Continuous Deployment as the pipeline earns trust through evidence.
A six-month rollout prioritizes risk reduction — secrets, branch protection and blocking CI — before advanced deployment strategies. The goal is to cut the incident rate per deployment from ~20 % to under 5 %, free up 30 % of backend capacity that is currently lost to firefighting, and turn the looming .NET 10 deadline into a rehearsed, low-risk event rather than a cliff-edge.
Nordwind Interactive AB is a mid-sized Swedish independent game studio based in Stockholm, founded in 2018. The studio has 78 employees and sits in the indie/AA segment rather than AAA.
| Discipline | Headcount | Notes |
|---|---|---|
| Client engineering (Unreal Engine 5, C++) | 14 | Gameplay, engine, tools, netcode |
| Backend engineering (.NET, Azure) | 11 | Scope of this white paper |
| Art (2D, 3D, VFX, animation) | 24 | |
| Audio (music, SFX) | 5 | |
| Design (game, level, narrative, economy) | 8 | |
| Quality Assurance | 5 | Primarily client/playtest-focused |
| Production, community, marketing, business, leadership | 11 | |
| Total | 78 |
The studio's flagship title, Planet Reavers, is a sci-fi extraction shooter (PvPvE) launched 18 months ago on PC (Steam, Epic), PlayStation 5 and Xbox Series X/S. It has around 180 000 DAU and 650 000 MAU. Revenue is driven by an initial game purchase plus a seasonal battle pass and in-game store.
This white paper focuses on Nordwind's backend platform — the services the client talks to for authentication, matchmaking, inventory, the store, progression, seasonal events and telemetry. The Unreal-based client is out of scope. Its releases follow platform certification (Sony, Microsoft, Valve) that works very differently from backend releases.
Backend technology base: .NET 8 (LTS) / ASP.NET Core microservices on Microsoft Azure. A migration to .NET 10 (LTS) before November 2026 is an explicit requirement. Persistence: Azure SQL, Cosmos DB, Redis. Messaging: Azure Service Bus, Event Hubs. Dedicated multiplayer game servers run as containerized C++ workloads on AKS with Agones; they are stateful and only mentioned where relevant.
Nordwind was founded in 2018 by six former AAA developers. The original backend was a small monolithic ASP.NET service written by two engineers just before beta, under the pragmatic philosophy of "ship the game, then we'll figure out the rest." When Planet Reavers launched successfully 18 months ago, the studio grew from under 30 people to ~78 within a year; the backend team went from 2 to 11 engineers, and the monolith was gradually broken into microservices. The processes, tooling and culture around releases did not keep pace.
The studio also moved from a "ship and patch" model to live-service — seasonal content, weekly hotfixes, economy tuning — reactively rather than by design. Deadlines today are set by season launches and marketing moments (collaborations, esports-events, platform sales) rather than by engineering readiness. Every 10–12 weeks a major content drop is pushed under time pressure, followed by 1–2 weeks of firefighting that compresses the next season and repeats the cycle.
Organizationally, the client team (Unreal, C++) and the backend team (.NET, Azure) operate as separate organizations with separate leadership, sprint cadences and definitions of done. API contracts are often changed late in a cycle to fit gameplay needs. QA is split: the 5-person QA team is mostly client-focused, and there is no dedicated backend QA role or formal integration test suite covering the client–backend contract.
Symptoms and consequences:
- About 20 % of backend deployments cause a user-visible incident within 48 hours; each major incident during a season launch costs an estimated 0.5–2 MSEK.
- Hotfixes are deployed manually via the Azure Portal, bypassing CI/CD.
- Secrets live as plain environment variables in App Service config, readable broadly across the backend team.
- There is no staging environment mirroring production scale; outages are often found on Discord and Reddit before internal monitoring reports them.
- Two senior backend engineers have left in the last six months citing hotfix burnout, and an estimated 30–35 % of backend capacity is spent on reactive firefighting.
- The Steam review trend has drifted from "Very Positive" at launch to "Mixed", with stability a common complaint.
- There is no tested path to .NET 10 before .NET 8 leaves support in November 2026, which makes the upgrade a latent strategic risk on top of the operational ones.
The root causes are a mix of cultural ("shipping is heroism"), structural (no dedicated platform/SRE function — the backend team does features and operations), technical (tests exist but are not enforced as blocking) and process-related. (Release-cadence driven by the marketing calendar rather than engineering readiness.)
The guiding principle is that the release pipeline itself becomes the quality gate between engineers and production. Human judgment is used at well-defined points, but the default path from a merged pull request to production is automated, observable and reversible. No engineer should be able to push to production via the Azure Portal — and none should need to, because the pipeline is faster and safer than the manual route.
The existing stack (.NET 8, Azure, Azure SQL, Cosmos DB, Redis, Service Bus, Event Hubs, AKS + Agones) is kept as the foundation. The pipeline is built with GitHub Actions (CI/CD orchestration), Docker and Azure Container Registry (immutable artefacts), Azure Container Apps for stateless services (native traffic splitting for Blue/Green and Canary), Bicep for Infrastructure as Code, Azure Key Vault with Managed Identities for secrets, Azure App Configuration for feature flags, Azure Monitor + Application Insights + Grafana for observability, GitHub Advanced Security for scanning, and Azure Load Testing for performance gates.
- A Platform / SRE function of 2–3 engineers is carved out of the backend team. It owns the pipeline, IaC, observability and on-call. Other backend engineers become consumers of the platform.
- Backend-aware QA. One QA role is expanded or upskilled. Its job is not to press buttons but to design the gates the pipeline evaluates automatically.
- Version control discipline. Trunk-based development with short-lived branches (< 3 days); Conventional Commits;
mainprotected with mandatory PRs, required reviewers, required status checks and linear history;CODEOWNERSroutes PRs; a short "Version control & release playbook" in the repo is paired with mentoring rather than read in isolation. - Cultural and cadence changes. Blameless post-mortem for every production incident, with action items fed back into the pipeline. Release cadence decoupled from the marketing calendar — engineering readiness (gates green, error budget is available) is non-negotiable. A formal on-call rotation replaces ad-hoc "whoever is still awake" firefighting.
The rollout is staged over roughly six months and deliberately front-loaded with risk reduction rather than technology adoption. Nordwind's most urgent problem is not the absence of Blue/Green deployments — it is plain-text secrets, unblocked merges and manual Portal deployments. Concretely: first, move secrets to Key Vault, revoke Portal access and introduce blocking CI gates; then provision Bicep-managed infrastructure and a production-like staging environment; then add integration and contract tests, canary deployments and feature flags; finally, run a parallel .NET 8 / .NET 10 build matrix and migrate the first non-critical service to .NET 10 via Blue/Green as the full-migration dry-run.
The pipeline is implemented as GitHub Actions workflows living alongside the services they build. The same pipeline is used for every backend service; service-specific configuration is injected via Bicep parameter files rather than by duplicating workflows. This is what makes the pipeline the quality gate rather than a quality gate — there is one path to production, and it is the same for everyone.
The stages are:
- Source / commit. A PR is opened against
main. Branch protection requires aCODEOWNERSapproval (author cannot self-approve) and all status checks below passing. - Build & unit tests. The service is built inside a pinned .NET SDK container image. The build runs as a matrix across runtime versions — initially only .NET 8, expanded to .NET 8 + .NET 10 in the final rollout phase. Unit tests run in parallel and must complete in under three minutes per service.
- Static analysis & security scanning. CodeQL, secret scanning, Dependabot and
dotnet formatrun concurrently. High or critical findings block merge; lower severities are annotated on the PR. - Artifact packaging. The service is packaged as a Docker image, tagged with the commit SHA, signed with cosign, and pushed to Azure Container Registry via OIDC.
- Deploy to Staging. After merge to
main, the image is deployed to a staging environment that mirrors production topology. Database migrations use the expand/contract pattern so staging remains compatible with the previous revision for rollback purposes. - Integration, contract and smoke tests. Run against the service in staging. Contract tests validate the public API against versioned contracts consumed by the client and other services. Full load tests (Azure Load Testing) run for season release candidates.
- Promotion gate to Production. The only routinely manual gate. A Platform engineer or release manager approves through GitHub Environments; the approver cannot be a commit author in the release.
- Deploy to Production with Canary, then progressive rollout and post-deploy monitoring. Traffic shifts 5 % → 25 % → 50 % → 100 % with automated health evaluation between steps, followed by a 30-minute monitoring window on the "four golden signals" (latency, traffic, errors, saturation). A signal breach at any point triggers automated rollback and an incident.
The target median time from merged PR to 100 % production traffic is 35–60 minutes. The point is not speed for its own sake, but to ensure the pipeline is never the slow option compared to a manual Portal deployment — because a pipeline slower than its workaround is a pipeline that will be worked around.
The two terms differ in one specific way: whether the final step to production requires a human approval.
- Continuous Delivery: every change that passes the gates is deployable, but a human explicitly authorizes each production release.
- Continuous Deployment: the final approval is removed. Every change that passes the gates is deployed without further intervention.
Both models require the same engineering rigour upstream; the difference is whether the final trust sits with a human or with the gates.
Recommendation for Nordwind: Continuous Delivery with selective Continuous Deployment for low-risk services. Player-facing services (store, matchmaking, progression) keep a human approval step until the pipeline has a track record of low-incident releases. Internal and non-player-facing services (telemetry ingestion, admin tools) can run full Continuous Deployment from the start. Three reasons not to go straight to full CD everywhere: trust is earned by evidence, not declared; season launches and esports events create hours when even a "safe" deploy is unacceptable and a human approval is the simplest freeze mechanism; and some backend changes indirectly affect certified client behavior on PlayStation and Xbox, where coordinated communication is sometimes required. Over time, services migrate toward Continuous Deployment as trust grows.
The following (abbreviated) GitHub Actions workflow shows how triggers and early stages are expressed as configuration.
# .github/workflows/ci-cd.yml
name: ci-cd
on:
pull_request:
branches: [ main ] # Run CI gates on every proposed change.
push:
branches: [ main ] # Run CI and promote to Staging/Production.
workflow_dispatch: # Audited manual trigger (break-glass hotfixes).
inputs:
reason:
description: "Incident ID / justification"
required: true
permissions:
contents: read # Minimal default; jobs elevate only what they need (PoLP).
jobs:
build-test:
runs-on: ubuntu-latest
strategy:
matrix:
dotnet: [ "8.0.x" ] # Expanded to [ "8.0.x", "10.0.x" ] during migration.
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: ${{ matrix.dotnet }}
- run: |
dotnet restore
dotnet build --configuration Release --no-restore
dotnet test --configuration Release --no-build
- uses: github/codeql-action/analyze@v3
deploy-production:
needs: build-test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: production # GitHub Environments enforces the required-reviewer gate.
permissions:
id-token: write # OIDC federation to Azure; no long-lived secrets.
steps:
- run: ./scripts/deploy.sh production "${{ github.sha }}" --strategy canaryThe test strategy follows the classical test pyramid but bends it to reflect the incident history: the worst production issues have come from contract drift between the client and the backend, not from logic errors within a single service. Contract testing is therefore a first-class layer rather than a subset of integration or E2E tests.
| Layer | Share of executions | Primary purpose |
|---|---|---|
| Unit tests | ~70 % | Individual classes and domain rules behave as specified. |
| Integration tests | ~15 % | Services work against real dependencies (Azure SQL, Cosmos DB, Redis, Service Bus) using Testcontainers or ephemeral Azure resources. |
| Contract tests | ~10 % | Public APIs match the versioned contract consumed by the client and other services, implemented with Pact or equivalent. |
| E2E tests | ~5 % | A small set of high-value player journeys works across the full backend in staging. |
Key principles: unit tests are owned by the feature author and enforced by code review rather than by a coverage- percentage gate (coverage is a trend, not a truth); integration tests run against real infrastructure rather than hand-rolled mocks; contract tests are the main defense against inter-team breakage — a failing backend blocks the client change, inverting today's dynamic; E2E tests are deliberately few (5–10 journeys that would constitute a de-facto outage if broken); and flaky tests are quarantined, not retried into green — a test that flakes more than twice in a rolling week is moved to a non-blocking lane until fixed.
A Quality Gate is a point where a signal is compared to a threshold, and a failure stops the pipeline. Gates are binary, but the policy around them is not.
| # | Gate | Stage | Owner | Action on fail |
|---|---|---|---|---|
| G1 | Unit tests | CI | Feature author | Block merge |
| G2 | Static analysis & security scanning (CodeQL, secret scanning, Dependabot) | CI | Author + Platform | Block merge |
| G3 | Code review (≥ 1 CODEOWNERS approval, author ≠ sole reviewer) | CI | CODEOWNERS | Block merge |
| G4 | Integration & contract tests | CI | Backend QA lead | Block merge |
| G5 | Staging smoke tests | Staging | Backend QA lead | Block promotion |
| G6 | Canary health (30-min post-deploy window) | Production | Automated + on-call | Auto-rollback + incident |
Every gate has a clear owner — feature authors own their own code (G1–G3), the Backend QA lead owns inter-service checks
(G4–G5), and the Platform function owns production (G6). Flaky gates are quarantined rather than retried into green,
because a flaky gate loses trust faster than a missing gate: an honest "we don't test this yet" is safer than an
unreliable test nobody takes seriously. Every post-mortem is required to answer "which gate should have caught this?"
— if the answer is "none", a new gate is proposed. For genuine emergencies, a hotfix/* branch skips the load-test
gate, requires two on-call approvers, and auto-creates a post-mortem ticket that must close within 5 working days.
A pipeline with strong gates can still cause an incident at the last mile: when real traffic meets new code. Release strategies address that moment, and they are not interchangeable — each mitigates a different failure class.
Blue/Green Deployment. Two environments (Blue and Green) run in parallel; the new release is deployed to the idle one, validated, and promoted by switching the router so all traffic moves to it. Mitigates catastrophic deploy-time failures — the active environment is never touched during deployment. Does not mitigate bugs that only appear under real traffic, since the switch is instant and all-at-once. Cost: roughly 2× compute during cut-over. Best fit: the .NET 8 → .NET 10 migration itself, where a binary cut-over is simpler and safer than a gradual mix of runtimes.
Canary Releases. A new release receives a small and growing share of production traffic (5 % → 25 % → 50 % → 100 %), with golden signals compared to the previous revision between each step. Mitigates bugs that only surface under real traffic or at scale by bounding the blast radius. Does not mitigate failures invisible at low-traffic fractions. Cost: marginal on Azure Container Apps. Best fit: most player-facing services — a broken store visible to 5 % of players for 10 minutes is a survivable incident; to 100 % for the same duration it is a Reddit front-page event.
Feature Toggles. Code is deployed to production disabled and enabled gradually via Azure App Configuration, first for the dev team, then a small cohort, then globally. Mitigates behavior-level risk — a misbehaving feature is flipped off in seconds without a redeployment, the fastest rollback mechanism available. Does not mitigate bugs at load time or in shared code paths the flag does not fence. Cost: zero infrastructure, but non-trivial code cost if flags accumulate into permanent "flag debt". Best fit: seasonal events, economy tuning, store layouts, matchmaking variants — anywhere the risk is in the behavior, not the deployment.
None of the three is sufficient alone. For Nordwind specifically, the biggest single risk reduction comes from Canary releases with automated rollback — because it directly attacks the dominant failure pattern (runtime misbehavior under real traffic), which is exactly what today's 20 % incident rate is made of. Blue/Green is reserved for runtime migrations, and feature flags cover behavior-level risk on top of the canary.
A deployment that is technically successful (image pulled, pods ready, health endpoint green) can still misbehave at the application level: error-rate regressions, latency spikes, store transactions failing silently, seasonal events returning the wrong rewards. Rollback must therefore be triggered by application-level signals, not just by infrastructure health.
The pipeline uses a two-layer approach:
- Automated rollback during the canary window. While traffic is stepped up through 5 % → 25 % → 50 % → 100 %, a rollback decision function continuously compares the new revision's golden signals (error rate, p95/p99 latency, saturation) and a small set of business metrics (store checkout success rate, match-start success rate) against a baseline from the previous revision. If any signal breaches its tolerance for more than a short dampening window (to avoid reacting to single spikes), Azure Container Apps' traffic split is reverted to the previous revision automatically and an incident is opened. Database migrations follow the expand/contract pattern so the previous revision stays compatible for at least one release cycle, which is what makes automatic reversion safe.
- Feature-flag kill-switch for behavioral regressions. When a bug manifests only when a specific new behavior is enabled — a new store layout, a new matchmaking variant, a new seasonal event — the correct response is not a rollback of the deployment but a flag flip. This is faster (seconds rather than minutes), less disruptive, and narrower in scope.
Ownership is shared between the Platform function (who owns the rollback decision function and the canary tooling) and the on-call engineer (who owns the incident response). The rollback itself is automatic; the post-mortem is mandatory and must answer the question of which gate or signal could have caught the regression before traffic was exposed to it.
Pipeline security at Nordwind is built on three ideas: no long-lived secrets, least privilege by default, and a verifiable supply chain.
- Secrets management. All secrets live in Azure Key Vault and are retrieved at runtime via Managed Identity.
No secret ever appears in source code, config files, CI logs or environment variables in plain form. The pipeline
itself authenticates to Azure via OIDC federated credentials, scoped to a specific repository, branch and
workflow — a stolen workflow log is worthless, unlike a leaked
AZURE_CREDENTIALSblob. - Principle of Least Privilege. GitHub Actions workflows start with
permissions: contents: readand elevate per job only where needed (e.g.id-token: writefor Azure login,packages: writefor image publishing). Azure role assignments follow the same pattern — services get the minimum RBAC role on the minimum scope (resource group or individual resource), not subscription-wide Contributor. - Supply chain. Dependencies are scanned by Dependabot on every PR. Container images are built from pinned base images, signed with cosign at packaging time, and verified against the signature before deployment. A Software Bill of Materials (SBOM) is produced per build and stored alongside the artifact. This turns "what exactly is in production right now?" from an investigation into a query.
- Audit and compliance. Every production deployment is traceable through GitHub Environments approvals to a named approver, a commit and an artifact digest. Break-glass deployments are tagged automatically and paired with a mandatory post-mortem, making the exceptions as auditable as the normal path.
Observability is treated as a feature of every service, not as an afterthought. A change is not considered done until it emits the signals needed to detect its failure modes in production. Three data types are collected consistently: structured logs from every service, metrics including the four golden signals (latency, traffic, errors, saturation) plus a small set of business-level metrics (store checkout success, match-start success, seasonal-event completion), and distributed traces that follow a request across services to locate latency and failure hotspots.
The tooling is Azure Monitor + Application Insights for collection and alerting, with Grafana on top for dashboards that are visible studio-wide — not just to the backend team. SLOs (e.g. "99.5 % of store checkouts succeed in under 500 ms") are published openly, and burn-rate alerts fire when a service is consuming its error budget faster than sustainable. This directly addresses one of the most striking symptoms from the current state: that outages are today first noticed on Discord and Reddit, not internally. The rule going forward is that the team should know before the players do — and when it doesn't, that specific failure is itself treated as an incident and a gap in the signals.
| Activity | Responsible (R) | Accountable (A) | Consulted (C) | Informed (I) |
|---|---|---|---|---|
| Feature development & unit tests | Feature author | Backend team lead | Backend QA lead | Platform |
| CI pipeline & gates | Platform | Backend team lead | Backend QA lead, Feature authors | All engineering |
| Integration / contract / E2E tests | Backend QA lead | Backend team lead | Feature authors, Client team | Platform |
| Production deployment approval | Release manager (rotating, Platform) | Backend team lead | On-call, Platform | Backend team, Production |
| Incident response & rollback | On-call engineer | Platform lead | Backend QA lead, Feature author | Backend team, Production |
| Post-mortems & pipeline improvements | Platform | Backend team lead | Everyone involved in the incident | All engineering |
| Release strategy & .NET 10 migration | Platform | Backend team lead + CTO | Backend engineers, Client team | Production, Leadership |
Working through this scenario has clarified a few things for me that I want to carry into LIA and my first roles after graduation.
- New insights. DevOps is less about tools and more about where trust is placed. The same CI/CD tool can produce very different outcomes depending on whether the organization has the discipline to treat gates as non-negotiable. "Automation" without that discipline just moves the chaos downstream.
- Strengths I want to highlight. My background is .NET-focused programming combined with earlier experience in game development. That combination is less common than either alone, and it lets me reason about both the backend services and the constraints of a live-service game client consuming them.
- Areas to develop. Hands-on time with real Azure environments, Bicep or Terraform at scale, and running incidents on a real on-call rotation. A white paper is reasoning about these — a LIA placement is the place to actually do them.
- Potential niche. .NET-based backend platforms for live-service games: the intersection of DevOps maturity and a domain (games) where the stakes of bad releases are immediate and visible. It is narrow enough to be memorable and broad enough to be employable.
- Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press. Readable Online
- Humble, J., & Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley. Readable Online
- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. Readable Online
- Microsoft. .NET and .NET Core release lifecycle. https://learn.microsoft.com/dotnet/core/releases-and-support/
- Microsoft. Azure Container Apps — revisions and traffic splitting. https://learn.microsoft.com/azure/container-apps/revisions
- Microsoft. Bicep documentation. https://learn.microsoft.com/azure/azure-resource-manager/bicep/
- GitHub. Using OpenID Connect with reusable workflows. https://docs.github.com/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect
- GitHub. Configuring environments for deployments. https://docs.github.com/actions/deployment/targeting-different-environments/using-environments-for-deployment
- Pact Foundation. Consumer-driven contract testing. https://docs.pact.io/
- Azure Architecture Center. Blue-green deployment and Canary release patterns. https://learn.microsoft.com/azure/architecture/guide/