Bare metal runners

Bare metal runners

Bare-metal runners are physical machines used for CI/CD jobs that need direct hardware, predictable performance, special licensing, GPUs, embedded devices, mobile-device labs, industrial test benches, or network placement that ordinary cloud runners cannot provide. They can be valuable, but they need stronger operating discipline because local state, physical access, firmware, and hardware failure become part of the CI boundary.

This page covers user-facing design, security, monitoring, maintenance, and troubleshooting considerations for bare-metal runner fleets. For general controls, see Runner security hardening, Runner monitoring and support, and Troubleshooting. For platform routing, see GitHub Actions, GitLab runners, Gitea Actions, and Forgejo Actions.

Assistance can help design and operate the runner controls, evidence, maintenance, and incident workflow. The customer remains responsible for physical-site ownership, legal/compliance decisions, safety rules, asset ownership, and business risk unless contracted otherwise.

When bare metal is a good fit#

Use bare metal when the workload requires one or more of these:

• attached hardware such as GPUs, USB devices, lab equipment, mobile devices, HSMs, license dongles, cameras, or embedded boards;
• deterministic CPU, memory, disk, or network performance;
• large local caches that are explicitly approved and governed;
• network placement inside a private facility, lab, factory, or datacenter;
• operating systems or drivers not available on managed cloud runners;
• nested virtualization or kernel-level features that are unavailable in the chosen cloud profile.

Do not choose bare metal only because it feels cheaper. Persistent physical hosts usually increase patching, isolation, hardware replacement, and evidence work.

Ownership and location decisions#

Record these decisions before onboarding:

For EU-sensitive environments, Assistance can provide operational controls and evidence for customer review. Assistance does not make GDPR role determinations, certify the site, or provide legal advice unless explicitly contracted.

Bare-metal security controls#

Physical controls#

• Keep runners in a controlled room, rack, lab, or cabinet appropriate to the data and workload sensitivity.
• Track asset identifiers, serial numbers, location, owner, and support contact.
• Restrict who can access power, console, removable media, attached devices, and network ports.
• Record hands-on access and maintenance actions when the workload is sensitive or regulated.
• Define how lost, stolen, replaced, or decommissioned disks are handled.

Host hardening#

• Use a supported operating system, runner agent, firmware, drivers, and container/runtime stack.
• Enable full-disk encryption where compatible with unattended boot and recovery requirements.
• Disable unnecessary services, users, local admin rights, and inbound network paths.
• Use SSO/MFA-backed administrative access where practical; avoid shared local administrator accounts.
• Keep host firewall, endpoint protection, audit logging, and time synchronization enabled.
• Pin or record driver and firmware versions for hardware-dependent jobs.

Runner isolation#

Bare-metal runners are often persistent. Treat that as a risk:

• dedicate hosts or pools to one trust level: production deploy, non-production build, untrusted test, hardware lab, or regulated workload;
• avoid running untrusted pull requests on hosts with production secrets, private-network access, or sensitive devices;
• clean workspaces, temporary directories, package caches, Docker layers, and device state between jobs;
• consider disposable VM, container, or snapshot layers on top of bare metal where hardware access still works;
• quarantine and rebuild hosts after suspected compromise or unexplained state leakage.

Secrets and private access#

• Store secrets in the CI platform or approved secret manager, not in local shell profiles or files.
• Keep production secrets on restricted runner labels/tags and protected branches/environments.
• Prefer short-lived credentials or identity federation where available.
• Document which private services, lab networks, cloud APIs, databases, or deployment targets the host can reach.
• Use allowlists, VLANs, firewall rules, VPNs, or egress gateways to prevent broad access from a compromised runner.

Monitoring bare-metal runners#

Bare-metal monitoring should include the normal runner signals plus physical and hardware signals.

Alerts should include host name or asset ID, runner labels/tags, location, affected device, business impact, and whether hands-on-site support is needed.

Maintenance windows#

Bare-metal maintenance often needs more coordination than cloud runners. A window should cover:

• operating-system patches, runner-agent updates, firmware, BIOS/UEFI, drivers, GPU/toolkit updates, and container runtime changes;
• physical work such as disk replacement, cable changes, device swaps, dust/thermal work, or rack moves;
• expected impact to labels/tags and repositories;
• validation jobs for hardware, network, secrets, and deployment access;
• rollback plan for drivers, firmware, kernel, or image changes;
• hands-on-site contact and remote-support contact;
• evidence to retain: patch records, access logs, screenshots, job validation, and known issues.

For actively exploited vulnerabilities, leaked runner tokens, suspected compromise, or failing hardware, Assistance may recommend emergency maintenance and temporary removal from CI routing.

Backups and rebuilds#

A healthy bare-metal runner should be rebuildable. Back up or record:

• host inventory, serial numbers, rack/location, attached devices, firmware and driver versions;
• infrastructure-as-code, configuration management, bootstrap scripts, and runner registration process;
• monitoring dashboards, alert rules, and runbooks;
• license files or dongle records where legally and operationally appropriate;
• evidence records and maintenance history.

Do not treat local workspaces, caches, or temporary build outputs as durable backups. Required artifacts should be uploaded to approved artifact storage with explicit retention and region choices.

Common bare-metal failure modes#

See Common runner errors for broader CI symptoms.

Escalation inputs for bare-metal incidents#

Include these details when asking for help:

• CI platform, repository/project, workflow/pipeline/job URL, and runner labels/tags;
• host asset ID, location, hardware profile, operating system, architecture, and attached devices;
• first failure time, last known good run, and whether the issue survives reboot or rerun;
• recent physical access, patching, firmware/driver changes, network changes, or device swaps;
• screenshots or logs for runner agent, system errors, device status, and monitoring alerts;
• whether production secrets, private networks, personal data, customer data, or regulated workloads may be involved;
• hands-on-site contact, remote access method, and approval needed for reboot, replacement, or disk handling.

Decommissioning#

When a bare-metal runner leaves service:

1. Remove it from CI runner groups, labels/tags, and registration records.
2. Revoke runner tokens and any host-specific credentials.
3. Remove network allowlists, firewall rules, VPN access, and monitoring targets that are no longer needed.
4. Preserve required audit evidence and maintenance history.
5. Wipe or destroy disks according to customer policy and data sensitivity.
6. Update asset inventory, runbooks, and support contacts.

Related docs and services#

• Managed runners overview
• Runner security hardening
• Runner monitoring and support
• Troubleshooting
• Common runner errors
• CI/CD audit
• SRE as a Service