Dynamic Jenkins agents on GCP Spot VMs

At a glance — the problem, the approach, and what it delivered.

Problem. An always-on self-hosted GitHub Actions runner faced constant 24/7 billing regardless of actual usage and single-machine concurrency limits. The goal was to achieve elastic CI capacity priced at spot rates that only incurs charges during active builds.

Approach. A Jenkins controller paired with the Google Compute Engine plugin provisions a fresh Spot VM per build from a pre-baked golden image, executes the job, and removes the VM upon completion. Spot preemptions are treated as expected events with automatic retry mechanisms onto fresh capacity.

Scale. Approximately 1,400 single-use Spot agents provisioned monthly (~47 daily); CI builds run 10–23 minutes each; agents boot in seconds from the golden image rather than the minutes required for at-boot toolchain installation.

Outcome. Replaced one always-on 8-vCPU on-demand runner with per-build Spot agents billed only during builds — an estimated 5–10× reduction in CI compute costs (list-price basis), with unlimited concurrency. Zero Spot preemptions recorded across approximately 1,400 agent starts over 30 days.

Provenance — Based on a production CI system designed and operated by the author. Figures are real and measured but de-identified: no hostnames, project IDs, credentials, or proprietary code.

Why Move Off an Always-On Runner

The starting point involved a single self-hosted GitHub Actions runner on an always-on 8-vCPU VM. The fundamental problem: 24/7 billing regardless of build activity, with peak concurrency capped at one machine. CI workloads are inherently bursty, so this model wasted resources during idle periods. The desired outcome was capacity appearing when builds queue and disappearing upon completion, priced near spot market rates.

How It Works

The Jenkins controller serves as the only persistent component. GitHub multibranch pipelines (one per branch/PR) trigger jobs. When queued, the Google Compute Engine plugin provisions a fresh Spot VM from a pre-baked golden image, the agent connects, runs the build, and the VM is deleted afterward. No pooled agents, no idle machines.

Two instance templates support the fleet — a smaller default (2 vCPU) and a larger option (4 vCPU) for heavier workloads — both Spot. In a typical month, the plugin manages ~1,400 single-use agents (approximately 47 daily).

The Decision That Actually Mattered: Bake a Golden Image

Initial attempts installed the toolchain via boot-time startup scripts. Testing succeeded but production failed with an unhelpful error:

Agent failed to connect, even though the launcher didn't report it.

The actual cause: the startup script performed full toolchain installation at boot — slow, and it used set -euo pipefail, so any transient apt failure aborted the entire process. This raced against the agent-launch timeout, leaving the VM unable to function as an agent. On Spot capacity where seconds matter, at-boot provisioning proved wrong.

The solution involved moving provisioning into a golden image: boot an Ubuntu builder, run the provisioning script once, snapshot the disk into an image family, and point instance templates at the family so the newest image auto-selects on next agent boot. Agents now boot in seconds with zero runtime apt.

One detail proved valuable: the image pre-warms the Go module cache and pre-compiles golangci-lint beforehand. Cold Spot agents previously hit lint step timeouts during first runs while modules downloaded; baking this eliminated the last flaky first-build failures.

# Re-baking is a script, not a ceremony: provision once, snapshot to a family,
# recreate the templates on the new image, recycle agents.
gcloud compute images create "jenkins-agent-$(date +%Y%m%d-%H%M)" \
  --source-disk "$BUILDER" --source-disk-zone "$ZONE" \
  --family jenkins-agent      # templates track the family → newest image auto-used

Treating Preemption as Normal

Spot VMs face reclamation at any moment. While the instinct might be to guard against it, the superior approach is to expect it. Since CI builds are pure functions of commits, preempted builds simply re-queue onto fresh capacity — developers observe slightly longer build times, never failed builds.

pipeline {
  agent { label 'spot' }   // fresh single-use Spot VM per build
  options { retry(2) }      // preemption re-queues onto new capacity
  stages {
    stage('build') { steps { sh './ci/build.sh' } }
    stage('test')  { steps { sh './ci/test.sh'  } }
  }
  // No VM cleanup step — the controller deletes the agent on completion.
}

Regarding the actual impact: over the last 30 days, zero preemptions occurred across ~1,400 agent starts. Spot capacity in the region remained stable, so the retry path rarely activated — but including it costs nothing and ensures degradation to slower performance rather than broken builds on bad Spot days.

What It Cost, and What It Proves

Based on GCP list-price estimation: the old always-on 8-vCPU runner cost roughly a couple hundred USD monthly just to operate. The Spot fleet bills exclusively for actual build time — approximately 1,400 builds monthly at ~10–23 minutes each on 2–4 vCPU Spot instances — yielding well under that, a rough 5–10× reduction in CI compute expense. The structural win proved larger: concurrency no longer caps at one machine, eliminating build queue backups.

The pattern generalizes broadly — the same controller also drives AWS Spot agents via EC2 Fleet, demonstrating the "disposable agent, expect preemption" model transcends cloud providers. Nothing here is exotic: a Jenkins controller, the GCE plugin, a golden image, and discipline around disposable agents.

Key Decisions & Tradeoffs

• Spot VMs over on-demand — substantial hourly savings versus possible preemption, which is safe because CI jobs are inherently retry-safe.
• Pre-baked golden image over boot-time installation — at-boot installs raced the agent-launch timeout and intermittently failed; baking enables second-level boots.
• Single-use VMs over long-lived pooled agents — clean environment per run, zero state contamination, zero idle burn.
• Preemption as retry-onto-fresh-capacity, not failure — reclaimed VMs never surface as broken builds.