Skip to content

dev-loop.md

Latest commit

 

History

History
353 lines (257 loc) · 12.6 KB

dev-loop.md

File metadata and controls

353 lines (257 loc) · 12.6 KB
Raw

Dev Loop — Cuttlefish-Based Development

The PersonaPhone dev loop runs the entire multi-persona stack inside Cuttlefish on a Linux box. You edit, build, boot in Cuttlefish, test, repeat — all without touching a Pixel until you're ready for hardware-specific validation.

Why Cuttlefish

Cuttlefish is Google's reference virtual Android device. It:

  • Uses crosvm as its VMM (same VMM we're patching for snapshot/restore)
  • Supports nested KVM — Cuttlefish itself runs as a VM; PersonaPhone's base layer can spawn pVMs inside it
  • Is the official AOSP CI target — it tracks upstream AOSP exactly
  • Runs on standard x86_64 Linux hosts — no Pixel needed for dev
  • Boots in seconds to a fully-functional Android with display (WebRTC streaming) + input + audio + sensors + fake-NFC + fake-RIL + fake-biometric

What this means for us: every line of code — pKVM patches, VirtualizationService extensions, crosvm snapshot/restore, virt HALs, arbitrator daemons, PersonaLauncher UI — is exercised end-to-end on Cuttlefish before any hardware contact. The same source produces the same behavior in Cuttlefish and on a real Pixel, with the documented exceptions in PLAN.md § 4.

The loop

       ┌───────────────────────┐
       │  Edit code            │
       └──────────┬────────────┘
                  │
                  ▼
       ┌───────────────────────┐
       │  ./tools/build/inc.sh │     incremental build (~30s–5min)
       └──────────┬────────────┘
                  │
                  ▼
       ┌───────────────────────┐
       │  cvd start            │     boot PersonaPhone in Cuttlefish
       │  --image=<artifact>   │     (~15–30s to base layer ready)
       └──────────┬────────────┘
                  │
                  ▼
       ┌───────────────────────┐
       │  ./tools/test/run.sh  │     run the test(s) you care about
       │  --suite cuttlefish   │     (full suite: ~30 min; single test: ~30s–2min)
       │  --test <name>        │
       └──────────┬────────────┘
                  │
            ┌─────┴─────┐
            │           │
            ▼           ▼
         green         red
       (commit       (read logs, edit, loop)
        or            
        deploy
        to Pixel)

Total cycle for "small edit → test result": ~2-5 minutes for most changes. Kernel/pKVM changes are slower (~15 minutes) because of the full kernel build.

Setting up the dev host

One-time setup. Skip if ./tools/build/setup-host.sh has been run.

Hardware requirements

  • Linux x86_64 (Ubuntu 24.04 LTS or Debian 13 recommended)
  • 16-core CPU minimum, 32+ recommended
  • 64GB RAM minimum (Cuttlefish + base layer + 2 pVMs simultaneously)
  • 1TB NVMe — 250GB for AOSP source, 200GB for build artifacts, rest for snapshots
  • Hardware virtualization enabled in BIOS (VT-x + VT-d on Intel, AMD-V + IOMMU on AMD)
  • Nested virtualization enabled in kernel: 
    echo "options kvm-intel nested=1" | sudo tee /etc/modprobe.d/kvm.conf
echo "options kvm-amd nested=1" | sudo tee -a /etc/modprobe.d/kvm.conf
sudo modprobe -r kvm_intel; sudo modprobe kvm_intel   # or kvm_amd
cat /sys/module/kvm_intel/parameters/nested  # should print "Y"

Software installation

Done by tools/build/setup-host.sh:

# AOSP build deps
sudo apt update
sudo apt install -y git-core gnupg flex bison build-essential zip curl \
  zlib1g-dev libc6-dev-i386 libncurses5 lib32ncurses5-dev x11proto-core-dev \
  libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc unzip fontconfig \
  python-is-python3 openjdk-21-jdk

# repo tool
mkdir -p ~/bin
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod +x ~/bin/repo

# Cuttlefish host packages
sudo apt install -y cuttlefish-base cuttlefish-user

# Rust (for crosvm work)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup install nightly
rustup component add rust-src --toolchain nightly

# Additional: avbroot, qemu, docker for builds
sudo apt install -y qemu-system-arm qemu-user-static docker.io
cargo install avbroot

Validation

After setup:

./tools/build/check-host.sh

Should print all green checks. If anything fails, see docs/runbook.md § "Dev host setup issues."

Syncing sources

./tools/build/sync-sources.sh

This pulls:

Source Path Size What
AOSP main aosp/ ~150GB Android Open Source Project
Android KVM kernel kernel/android-kvm/ ~3GB pKVM-bearing kernel tree
crosvm upstream crosvm/upstream/ ~500MB google/crosvm
crosvm Android fork crosvm/android-fork/ ~500MB AOSP external/crosvm
GrapheneOS sources aosp-graphene/ ~150GB Hardened AOSP for the GrapheneOS persona
Cuttlefish images cf/ ~5GB Latest cuttlefish reference images
avbroot third_party/avbroot/ ~50MB Boot signing tooling
References third_party/refs/ ~20GB Pinned copies of nfcgate, wg-netns, etc. (read-only, for code-lifting)

Total: ~330GB. Cached aggressively via git's own object reuse; second sync of GrapheneOS is fast because it shares objects with AOSP.

Building

Full build

./tools/build/build-all.sh

Builds (in order, parallel where possible):

  1. GKI kernel with our pKVM patches → out/kernel/
  2. pvmfw with our AVB key in trust root → out/pvmfw.img
  3. crosvm with snapshot patches → out/bin/crosvm
  4. avf (VirtualizationService) with our AIDL extensions → out/avf/
  5. Host AOSP (personaphone product) → out/target/product/personaphone/
  6. Guest images: one per persona makefile in guest/personas/*.mk → out/guest/<persona>.img
  7. Cuttlefish images of the same build → out/cf/
  8. Sign everything with our AVB key → out/signed/

Full clean build: ~3-4 hours on a 32-core machine. Incremental: typically 30 seconds to 5 minutes depending on what changed.

Incremental build

./tools/build/inc.sh

Detects what changed since last build (by mtime + git status) and rebuilds the minimum set. Common scenarios:

You changed Rebuilds Time
PersonaLauncher Kotlin Just the APK ~30s
A virt HAL .cpp Just the HAL + reassemble guest image ~2min
personad C++ Daemon binary + base layer system image ~3min
Arbitrator daemon Daemon binary + base layer system image ~3min
crosvm Rust crosvm binary + redeploy to image ~1min (Rust incremental)
AVF AIDL extension AVF service + base layer ~5min
pKVM kernel patch Full kernel rebuild + reflash ~15min
AVB config Full re-sign ~5min

Running Cuttlefish

./tools/build/cuttlefish-up.sh

Internally:

cvd start \
  --num_instances=1 \
  --kernel_path=out/kernel/Image.lz4 \
  --initramfs_path=out/cf/initrd.img \
  --system_image_dir=out/cf/ \
  --gpu_mode=gfxstream \
  --enable_host_nfc_proxy=true \      # talks to our host NFC arbitrator
  --enable_host_ril_proxy=true \      # talks to our host RIL arbitrator
  --num_persona_pvms=3 \              # spawn 3 nested pVMs for testing
  --pvm_image_dir=out/guest/ \
  --webrtc_public_ip=stun

This boots Cuttlefish, which boots the PersonaPhone base layer, which then spawns 3 nested pVM personas.

Viewing the display

Cuttlefish streams the display + microphone + camera via WebRTC. Open in your browser:

https://localhost:1443

You see the PersonaPhone launcher. Click "switch persona" — the surface swaps to the next pVM. Just like on hardware.

ADB access

adb devices
# 0.0.0.0:6520  device

adb shell
# you're in the base layer
adb -s 0.0.0.0:6521 shell
# you're in persona 0's pVM
adb -s 0.0.0.0:6522 shell
# persona 1, etc.

Stopping

cvd stop

Running tests

./tools/test/run.sh --suite cuttlefish

This:

  1. Spins up a fresh Cuttlefish instance
  2. Runs each test in tests/cuttlefish/*.sh in sequence
  3. Captures: stdout/stderr per test, ADB logcat from base layer + each persona, screenshots at test boundaries, timing data
  4. Outputs a JUnit-format XML to out/test-results/cuttlefish-<timestamp>.xml
  5. Tears down Cuttlefish, leaves logs in out/test-logs/

For a single test:

./tools/test/run.sh --suite cuttlefish --test snapshot-stress

For a flame graph of slow tests:

./tools/test/run.sh --suite cuttlefish --profile

The fake peripherals in Cuttlefish

Cuttlefish doesn't have real NFC / Shannon modem / Titan M2. We add fake backends so the arbitrator daemons can be tested end-to-end:

Fake NFC

tools/test/fake-nfc/ is a Python service that mocks the PN553 NCI protocol over the same /dev/nq-nci device path. The NFC arbitrator daemon talks to it indistinguishably from real hardware. We can scriptedly inject "card detected" events to test foreground routing.

Fake RIL

Cuttlefish already provides a fake RIL via cuttlefish-ril. We extend it via tools/test/fake-ril/ to support test-driven scenarios: inject incoming calls, SMS, signal strength changes. RIL arbitrator daemon doesn't know it's fake.

Fake biometric

tools/test/fake-biometric/ exposes a vsock interface that the biometric arbitrator treats as a fingerprint sensor. Scriptable: "press finger now," "match success," "match fail," "sensor error."

Fake TEE

Trusty TEE doesn't run in Cuttlefish by default. We deploy a software-only Trusty replica via tools/test/fake-trusty/ that implements the same Keymint AIDL but stores keys in a host file. For confidentiality testing we deploy this with a host-protected directory; for security testing we explicitly mark these tests as "Cuttlefish stub, full validation requires real Trusty on Pixel."

What only works on real Pixel (and gets tested in tests/e2e/)

Test Why Cuttlefish can't
nfc-tap-real-card.sh No physical NFC controller
ril-real-call.sh No Shannon modem
biometric-real-fingerprint.sh No fingerprint sensor + no Trusty hardware-bound seal
tee-real-attestation.sh No Titan M2
gpu-mali-performance.sh Cuttlefish uses host GPU or swiftshader, not Mali
pmu-power-consumption.sh No real PMIC
boot-from-fastboot.sh Cuttlefish boot is different

These are validated on a dedicated test Pixel attached to the CI runner.

Debugging in Cuttlefish

Tracing pKVM hyp calls

adb shell setprop persist.dev.pkvm.trace 1
adb shell cat /sys/kernel/debug/pkvm/trace

Debugging crosvm snapshot

./tools/build/cuttlefish-up.sh --crosvm-log=trace
adb logcat | grep crosvm

Inspecting a pVM mid-snapshot

adb shell vm tool inspect --vm-id=persona-2 --include=memory,devices,vcpu

Attaching gdb to a guest kernel

tools/test/gdb-pvm.sh persona-2 opens gdb on the guest kernel via a vsock-bridged gdbserver running inside the pVM.

Reproducibility

Same git SHA → bit-identical Cuttlefish images. CI verifies by building each commit twice and diffing:

./tools/build/verify-reproducibility.sh <git-sha>

Performance budgets (Cuttlefish vs Pixel)

Cuttlefish is somewhat slower than real Pixel (no Tensor accelerators, virtualized I/O), but the ordering is preserved — if a test passes its timing budget on Cuttlefish, it'll pass on Pixel comfortably:

Operation Cuttlefish budget Pixel target
Base layer cold boot <30s <12s
First persona pVM start <15s <8s
Hot persona switch <5s <2s
Cold persona switch (snapshot from disk) <12s <5s
NFC arbiter switch handoff <500ms <200ms
RIL call route after switch <2s <1s

These are encoded as test assertions in tests/cuttlefish/*.sh.

Real-Pixel deploy

When all tests/cuttlefish/ are green:

./tools/flash/flash-pixel.sh --device <adb-serial>

Flashes the same artifacts (signed factory image) you've been testing. Then:

./tools/test/run.sh --suite e2e --device <adb-serial>

The e2e suite mirrors the Cuttlefish suite plus the Pixel-only tests. Expected outcome: everything green that was green on Cuttlefish, plus the Pixel-only tests pass on first run.

If a test passes on Cuttlefish but fails on Pixel, that's a divergence bug — file in tests/divergence/ and treat as a blocker.