added real playbooks

Author: dylanwongtencent

pvm-firecracker-ansible/README.md

@@ -0,0 +1,86 @@
+# PVM Firecracker — Ansible playbook
+
+Run **Firecracker microVMs without KVM / hardware virtualization** on a regular
+cloud VM, using **PVM** (Pagetable-based Virtual Machine). This playbook codifies
+the whole manual build: a PVM-patched Firecracker, a PVM host kernel, a PVM guest
+kernel, an Ubuntu rootfs, GRUB config, reboot, and an end-to-end microVM smoke test.
+
+## The one hard requirement
+
+PVM needs the **`fsgsbase`** CPU feature exposed to the guest vCPU. Many cloud
+hosts hide it (for live-migration compatibility), and PVM **cannot** run there.
+Check first:
+
+```bash
+grep -o fsgsbase /proc/cpuinfo   # must print "fsgsbase"
+```
+
+The playbook asserts this in preflight and stops immediately if it's missing.
+
+Other requirements: x86_64, RHEL family (Rocky/Alma/RHEL **10** tested), root via SSH,
+~10 GB free disk, internet access. First run compiles two kernels + Firecracker
+(~20–40 min on 32 vCPUs).
+
+## Usage
+
+```bash
+# 1. point inventory.ini at your host
+# 2. run it
+ansible-playbook playbook.yml
+
+# stage everything but don't reboot / test (e.g. you want to control the reboot)
+ansible-playbook playbook.yml -e pvm_reboot=false -e pvm_run_smoke_test=false
+
+# re-run only part of it (every expensive step is idempotent via `creates:`)
+ansible-playbook playbook.yml --tags firecracker
+ansible-playbook playbook.yml --tags host_kernel,guest_kernel
+```
+
+## What it does (stages / tags)
+
+| Tag | Stage |
+|-----|-------|
+| `preflight`     | assert x86_64 + RHEL + **fsgsbase**, make work dirs |
+| `packages`      | install build + runtime dependencies |
+| `firecracker`   | install Rust, clone the fork, **re-apply the PVM source patches**, build, install `firecracker`/`jailer` |
+| `host_kernel`   | build + install the `pvm-612` host kernel (`CONFIG_KVM_PVM=m`, `pti=off` fixes) |
+| `guest_kernel`  | build the PVM guest `vmlinux` (stripped) |
+| `rootfs`        | build an Ubuntu ext4 rootfs with an injected SSH key |
+| `boot`          | `pti=off`, auto-load `kvm-pvm`, set default kernel, **reboot**, verify `/dev/kvm` |
+| `smoke_test`    | boot a microVM, ping + SSH into the guest, assert it's up |
+
+## Key variables (`roles/pvm_firecracker/defaults/main.yml`)
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `pvm_reboot` | `true` | reboot into the PVM kernel and verify |
+| `pvm_set_default_kernel` | `true` | make the PVM kernel the persistent default |
+| `pvm_run_smoke_test` | `true` | boot a microVM and prove it |
+| `firecracker_repo` / `firecracker_version` | rusternetes-labs / `main` | patches re-applied regardless |
+| `pvm_linux_branch` | `pvm-612` | Linux 6.12.33 base |
+| `rootfs_squashfs_url` | FC CI Ubuntu 24.04 | userland is PVM-independent |
+| `make_jobs` | all vCPUs | parallel build |
+
+## Result
+
+After a successful run the target has, in `/root/pvm/`: `firecracker`, `jailer`,
+`vmlinux-pvm`, `rootfs.ext4`, `id_rsa`, `vmconfig.json`, `run-test.sh`. It boots the
+PVM kernel by default with `/dev/kvm` provided by `kvm-pvm`, and:
+
+```bash
+/root/pvm/run-test.sh   # boots a microVM (full Ubuntu guest) over PVM, no KVM hardware
+```
+
+## Notes & caveats
+
+- **Custom kernel + reboot.** Have console access to your cloud VM in case the
+  kernel fails to boot. `pvm_reboot=false` lets you stage everything and reboot
+  yourself. (`saved_entry` keeps your stock kernel as a fallback.)
+- **gnu build target.** Firecracker is built for `x86_64-unknown-linux-gnu`, which
+  uses an empty seccomp filter (no `libseccomp` at runtime). For a hardened
+  musl/static build, build via the upstream `tools/devtool` in a container.
+- **Idempotent.** Safe to re-run; finished steps are skipped. To force a kernel
+  rebuild, remove its `creates:` marker (e.g. the `bzImage`) or wipe `build_root`.
+- Live-migration support from the fork is intentionally out of scope (not needed
+  to run microVMs without KVM).
+```

pvm-firecracker-ansible/ansible.cfg

@@ -0,0 +1,13 @@
+[defaults]
+inventory = inventory.ini
+roles_path = roles
+host_key_checking = False
+retry_files_enabled = False
+stdout_callback = yaml
+# Kernel builds are long-running; keep the SSH connection alive.
+timeout = 60
+
+[ssh_connection]
+pipelining = True
+# Keep long-running tasks (kernel compiles) from dropping the control socket.
+ssh_args = -o ControlMaster=auto -o ControlPersist=120m -o ServerAliveInterval=30 -o ServerAliveCountMax=120

pvm-firecracker-ansible/inventory.ini

@@ -0,0 +1,10 @@
+# Target host(s) that will run the no-KVM PVM Firecracker stack.
+# Requirements: x86_64, a cloud VM whose vCPU EXPOSES the `fsgsbase` flag
+# (check with: grep -o fsgsbase /proc/cpuinfo). RHEL/Rocky/Alma 10 family.
+#
+# Example:
+[pvm]
+deployment ansible_host=170.106.40.67 ansible_user=root
+
+[pvm:vars]
+ansible_python_interpreter=/usr/bin/python3

pvm-firecracker-ansible/playbook.yml

@@ -0,0 +1,21 @@
+---
+# =============================================================================
+#  Run Firecracker microVMs WITHOUT KVM / hardware virtualization, using PVM
+#  (Pagetable-based Virtual Machine) on a regular cloud VM.
+#
+#  Usage:
+#     ansible-playbook playbook.yml
+#     ansible-playbook playbook.yml -e pvm_reboot=false        # stage, don't reboot
+#     ansible-playbook playbook.yml -e pvm_run_smoke_test=false
+#     ansible-playbook playbook.yml --tags firecracker,host_kernel
+#
+#  NOTE: This compiles two Linux kernels and Firecracker from source. On a
+#  32-vCPU box budget ~20-40 min for a first run. It is safe to re-run: every
+#  expensive step is guarded with `creates:` and will be skipped if already done.
+# =============================================================================
+- name: Set up the no-KVM PVM Firecracker stack
+  hosts: pvm
+  become: true
+  gather_facts: true
+  roles:
+    - role: pvm_firecracker

pvm-firecracker-ansible/roles/pvm_firecracker/defaults/main.yml

@@ -0,0 +1,41 @@
+---
+# ---------- Source repositories ----------
+# The Firecracker fork carrying the PVM forward-port (this role re-applies the
+# two essential source changes itself, so any reasonably-recent commit works).
+firecracker_repo: "https://github.com/rusternetes-labs/firecracker"
+firecracker_version: "main"
+
+# PVM host + guest kernel source (Linux 6.12.33 base).
+pvm_linux_repo: "https://github.com/virt-pvm/linux"
+pvm_linux_branch: "pvm-612"
+pvm_guest_config_url: "https://raw.githubusercontent.com/virt-pvm/misc/main/pvm-guest-6.12.33.config"
+
+# Ubuntu rootfs published by the Firecracker CI (userland is PVM-independent).
+rootfs_squashfs_url: "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.12/x86_64/ubuntu-24.04.squashfs"
+rootfs_size: "1G"
+
+# ---------- Working directories ----------
+build_root: "/root/pvm-build"            # scratch space for clones + compiles
+host_kernel_src: "{{ build_root }}/host-linux"
+guest_kernel_src: "{{ build_root }}/guest-linux"
+firecracker_src: "{{ build_root }}/firecracker"
+pvm_dir: "/root/pvm"                      # runtime artifacts live here
+
+# ---------- Build knobs ----------
+make_jobs: "{{ ansible_processor_vcpus | default(ansible_processor_cores) | default(4) }}"
+host_localversion: "-pvm"                 # host kernel becomes 6.12.33-pvm+
+guest_localversion: "-pvmguest"
+fc_target: "x86_64-unknown-linux-gnu"     # gnu build => empty seccomp filter, no libseccomp at runtime
+
+# ---------- Behaviour toggles ----------
+pvm_reboot: true                          # reboot into the PVM kernel at the end
+pvm_set_default_kernel: true              # make the PVM kernel the persistent default
+pvm_run_smoke_test: true                  # boot a microVM and prove it after reboot
+
+# ---------- microVM smoke-test settings ----------
+vm_vcpus: 2
+vm_mem_mib: 1024
+vm_tap_dev: "tap0"
+vm_host_ip: "172.16.0.1"
+vm_guest_ip: "172.16.0.2"
+vm_guest_mac: "06:00:AC:10:00:02"         # derives 172.16.0.2 in the CI rootfs

pvm-firecracker-ansible/roles/pvm_firecracker/handlers/main.yml

@@ -0,0 +1,3 @@
+---
+# Reboot is handled inline in tasks/60-boot.yml (it must be followed by
+# verification tasks in the same play), so no handlers are required here.

pvm-firecracker-ansible/roles/pvm_firecracker/meta/main.yml

@@ -0,0 +1,17 @@
+---
+galaxy_info:
+  role_name: pvm_firecracker
+  author: deployment
+  description: Run Firecracker microVMs without KVM using PVM on regular cloud VMs.
+  license: Apache-2.0
+  min_ansible_version: "2.14"
+  platforms:
+    - name: EL
+      versions:
+        - "10"
+  galaxy_tags:
+    - firecracker
+    - pvm
+    - virtualization
+    - microvm
+dependencies: []

pvm-firecracker-ansible/roles/pvm_firecracker/tasks/00-preflight.yml

@@ -0,0 +1,47 @@
+---
+# PVM has hard requirements. Fail loudly NOW rather than after a 30-minute build.
+
+- name: Require x86_64
+  ansible.builtin.assert:
+    that: ansible_architecture == "x86_64"
+    fail_msg: "PVM is x86_64-only (found {{ ansible_architecture }})."
+
+- name: Require a RHEL-family distro (Rocky/Alma/RHEL 10 tested)
+  ansible.builtin.assert:
+    that: ansible_os_family == "RedHat"
+    fail_msg: "This role targets the RHEL family (found {{ ansible_distribution }})."
+
+- name: Read CPU flags
+  ansible.builtin.command: grep -m1 '^flags' /proc/cpuinfo
+  register: cpuflags
+  changed_when: false
+
+- name: >-
+    Require FSGSBASE (the show-stopper). PVM's kvm-pvm module refuses to load
+    without it, and a cloud host that hides it from guest CPUID cannot run PVM.
+  ansible.builtin.assert:
+    that: "'fsgsbase' in cpuflags.stdout"
+    fail_msg: >-
+      This VM's vCPU does NOT expose `fsgsbase`, so PVM cannot run here.
+      Move to an instance type / CPU model that exposes it
+      (verify with: grep -o fsgsbase /proc/cpuinfo).
+    success_msg: "FSGSBASE present — PVM is viable on this host."
+
+- name: Detect pre-existing /dev/kvm (real KVM present — PVM not strictly needed)
+  ansible.builtin.stat:
+    path: /dev/kvm
+  register: _preexisting_kvm
+
+- name: Note if hardware KVM is already available
+  ansible.builtin.debug:
+    msg: "Note: /dev/kvm already exists; this host may have real KVM and not need PVM."
+  when: _preexisting_kvm.stat.exists
+
+- name: Ensure working directories exist
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    mode: "0755"
+  loop:
+    - "{{ build_root }}"
+    - "{{ pvm_dir }}"

pvm-firecracker-ansible/roles/pvm_firecracker/tasks/10-packages.yml

@@ -0,0 +1,35 @@
+---
+- name: Install toolchain, kernel-build, Firecracker-build and rootfs dependencies
+  ansible.builtin.dnf:
+    name:
+      # base toolchain
+      - gcc
+      - gcc-c++
+      - make
+      - git
+      - perl
+      - python3
+      - curl
+      - wget
+      - xz
+      - rsync
+      - hostname
+      # kernel build
+      - bison
+      - flex
+      - openssl-devel
+      - elfutils-libelf-devel
+      - bc
+      - dwarves
+      - ncurses-devel
+      - dracut
+      # Firecracker build (aws-lc-sys/bindgen/seccompiler)
+      - cmake
+      - clang
+      - clang-devel
+      - libseccomp-devel
+      # rootfs + microVM networking
+      - squashfs-tools
+      - e2fsprogs
+      - iptables
+    state: present

pvm-firecracker-ansible/roles/pvm_firecracker/tasks/20-firecracker.yml

@@ -0,0 +1,69 @@
+---
+# Build the PVM-aware Firecracker. The fork's `main` tracks upstream and the
+# PVM source changes are tiny + surgical, so we re-apply them here idempotently
+# instead of depending on them being committed.
+
+- name: Install rustup + toolchain (user-space, no root toolchain needed)
+  ansible.builtin.shell: |
+    set -e
+    curl -sSf --proto '=https' --tlsv1.2 https://sh.rustup.rs -o /tmp/rustup-init.sh
+    sh /tmp/rustup-init.sh -y --default-toolchain stable --profile minimal
+  args:
+    creates: "{{ ansible_env.HOME }}/.cargo/bin/cargo"
+
+- name: Clone the Firecracker fork
+  ansible.builtin.git:
+    repo: "{{ firecracker_repo }}"
+    dest: "{{ firecracker_src }}"
+    version: "{{ firecracker_version }}"
+    depth: 1
+    single_branch: true
+    force: false
+
+# ---- PVM forward-port: change #1 ----------------------------------------
+# Disable per-vCPU TSC scaling. PVM does not support KVM_CAP_TSC_CONTROL, so
+# the upstream call would fail and Firecracker would refuse to boot a guest.
+- name: "PVM patch: make set_tsc_khz() a no-op"
+  ansible.builtin.replace:
+    path: "{{ firecracker_src }}/src/vmm/src/arch/x86_64/vcpu.rs"
+    regexp: 'self\.fd\.set_tsc_khz\(tsc_freq\)\.map_err\(SetTscError\)'
+    replace: '{ let _ = tsc_freq; Ok(()) } // PVM: TSC scaling unsupported'
+
+# ---- PVM forward-port: change #2 ----------------------------------------
+# Relax the static-CPU-template model check so templates work across CPUs.
+# `return Err(InvalidCpuModel);` (note: no enum prefix) is unique to the live
+# code path; the test-only references use GetCpuTemplateError::InvalidCpuModel.
+- name: "PVM patch: relax static CPU-template model check"
+  ansible.builtin.replace:
+    path: "{{ firecracker_src }}/src/vmm/src/cpu_config/x86_64/custom_cpu_template.rs"
+    regexp: '^\s*return Err\(InvalidCpuModel\);'
+    replace: '                        { /* PVM: skip CPU-model compatibility check */ }'
+
+# A gnu-target build has no matching seccomp JSON, so Firecracker's build.rs
+# falls back to an empty filter automatically (no libseccomp at runtime).
+- name: Build Firecracker + jailer ({{ fc_target }}, release)
+  ansible.builtin.command:
+    cmd: cargo build --release --target {{ fc_target }} --package firecracker --package jailer
+    chdir: "{{ firecracker_src }}"
+    creates: "{{ firecracker_src }}/build/cargo_target/{{ fc_target }}/release/firecracker"
+  environment:
+    PATH: "{{ ansible_env.HOME }}/.cargo/bin:{{ ansible_env.PATH }}"
+    CARGO_TERM_PROGRESS_WHEN: "never"
+  async: 1800
+  poll: 20
+
+- name: Install firecracker + jailer binaries
+  ansible.builtin.copy:
+    src: "{{ firecracker_src }}/build/cargo_target/{{ fc_target }}/release/{{ item }}"
+    dest: "{{ pvm_dir }}/{{ item }}"
+    mode: "0755"
+    remote_src: true
+  loop:
+    - firecracker
+    - jailer
+
+- name: Symlink firecracker into PATH
+  ansible.builtin.file:
+    src: "{{ pvm_dir }}/firecracker"
+    dest: /usr/local/bin/firecracker
+    state: link