Runners

Common runner errors

Common runner errors

This page maps common self-hosted runner symptoms to likely causes and next actions. Exact messages vary by CI platform, executor, and runner version, so use this as a triage guide alongside Troubleshooting, Runner monitoring and support, and the relevant platform page: GitHub Actions, GitLab runners, Gitea Actions, or Forgejo Actions.

Do not paste raw secrets, private keys, full environment dumps, or unnecessary personal data into support tickets. Redact sensitive values and include job links, timestamps, labels/tags, and impact.

Job is queued and never starts#

Typical messages

  • Waiting for a runner to pick up this job
  • No runners matching the specified labels were found
  • This job is stuck because the project doesn't have any runners online assigned to it
  • GitLab job remains pending

Likely causes

  • Label or tag mismatch.
  • Runner group, project, organization, or protected-runner access does not allow this job.
  • All matching runners are offline, busy, paused, or draining for maintenance.
  • Autoscaling cannot create new capacity because of quota, image, network, or provider errors.
  • The job targets a restricted production or private-network runner that requires additional approvals.

What to check

  1. Confirm the workflow runs-on labels or GitLab tags match the documented runner profile.
  2. Check whether the job branch/ref is allowed to use protected runners or restricted environments.
  3. Review runner online status and queue dashboard for the affected label/tag.
  4. Check recent maintenance windows, runner image changes, and provider quota events.
  5. Escalate with job URL, labels/tags, repository/project, first queued time, and business impact.

Runner is offline or not registered#

Typical messages

  • Runner offline
  • Runner not accepting jobs
  • Runner registration failed
  • Authentication failed while registering runner

Likely causes

  • Runner agent stopped or host is unreachable.
  • Registration token expired, revoked, or rotated.
  • CI platform API is unavailable or blocked by network policy.
  • Host clock drift breaks TLS or token validation.
  • Runner was intentionally drained during patching or incident response.

What to check

  • Whether the outage matches a planned maintenance window.
  • Host health, agent logs, and CI platform status.
  • DNS, proxy, firewall, or private egress changes.
  • Runner token rotation records.
  • Whether rebuilding the runner from a trusted image is safer than repairing it.

Checkout, clone, or submodule failure#

Typical messages

  • repository not found
  • permission denied (publickey)
  • fatal: could not read Username
  • The process '/usr/bin/git' failed with exit code 128
  • submodule update failed

Likely causes

  • CI token permissions are too narrow or changed.
  • Private submodules require credentials not available to this runner/job.
  • Network egress to the Git host is blocked.
  • Branch protection or fork behavior prevents token access.
  • The runner image is missing Git, SSH, CA certificates, or known-hosts configuration.

What to check

  • Token scope and whether the job is from a fork or untrusted branch.
  • Repository, submodule, and package registry access.
  • DNS/TLS reachability to the Git host.
  • Recent repository permission or SSO changes.

Tool, shell, or image not found#

Typical messages

  • command not found
  • No such file or directory
  • exec format error
  • image pull failed
  • manifest unknown

Likely causes

  • Workflow assumes tools that are present on hosted runners but not on the managed image.
  • Architecture mismatch, such as ARM64 image on x64 runner or the reverse.
  • Container image tag moved or was deleted.
  • Private registry authentication or egress is missing.
  • Runner image changed without workflow compatibility validation.

What to check

  • Documented runner image baseline and architecture labels/tags.
  • Pinned image tags or digests for critical jobs.
  • Registry credentials, DNS, and egress allowlist.
  • Last successful run and recent image/toolchain changes.

Permission denied in workspace or cache#

Typical messages

  • permission denied
  • operation not permitted
  • cannot remove directory
  • failed to clean workspace
  • EACCES

Likely causes

  • Previous persistent job left root-owned files.
  • Docker, build tools, or package managers changed file ownership.
  • Cache restore created files with unexpected permissions.
  • Cleanup script failed or workspace path changed.
  • Job needs privileges that the runner profile intentionally does not provide.

What to check

  • Whether the runner is persistent or ephemeral.
  • Workspace cleanup logs and disk usage.
  • Container user IDs, bind mounts, and Docker socket usage.
  • Whether privileged mode is actually approved for this workload.

Disk full or no space left#

Typical messages

  • No space left on device
  • ENOSPC
  • write failed
  • Docker build or package install fails during extraction.

Likely causes

  • Cache, Docker layers, artifacts, logs, or workspaces grew beyond planned size.
  • Cleanup task failed.
  • Jobs write large temporary files to the runner instead of artifact storage.
  • Persistent runners retain too much state between jobs.

What to check

  • Disk usage by workspace, cache, container runtime, package stores, and logs.
  • Artifact/cache retention and maximum size.
  • Whether the job should use remote cache or artifact storage.
  • Whether the host should be rebuilt or resized.

Network or private endpoint failure#

Typical messages

  • connection timed out
  • could not resolve host
  • TLS handshake timeout
  • 403, 401, or connection refused from internal services
  • Deployment command cannot reach Kubernetes, database, registry, or cloud API.

Likely causes

  • DNS, firewall, VPN, private link, proxy, or security-group change.
  • Runner source IP not allowlisted.
  • Job landed on a runner without the private-network or deploy profile.
  • Service credentials expired or are scoped to a different environment.
  • Provider or internal service outage.

What to check

  1. Confirm the job ran on the expected restricted runner profile.
  2. Identify exact destination, port, source IP, and environment.
  3. Compare with recent network or secret changes.
  4. Check DNS resolution and TLS certificate errors.
  5. Escalate with destination details, timestamp, runner profile, and whether production is affected.

Secrets missing, unavailable, or exposed#

Typical messages

  • secret not found
  • masked variable is not available
  • environment protection rules were not satisfied
  • A log shows token-like output or private values.

Likely causes

  • Secret is scoped to a different repository, project, group, environment, or branch protection level.
  • Job is running from a fork, untrusted branch, or unprotected ref.
  • Production environment approvals did not run.
  • Secret was rotated or deleted.
  • Workflow prints environment variables, shell traces, or command output that includes credentials.

What to check

  • Secret scope and protected environment settings.
  • Branch/ref trust level and approval state.
  • Recent secret rotation records.
  • Whether exposed values require immediate rotation.
  • Whether job logs or artifacts need restricted retention or deletion according to customer policy.

Treat possible secret exposure as security-relevant. Assistance can help identify operational facts and rotate or revoke credentials when authorized; the customer decides legal and notification obligations.

Cache or artifact upload/download failed#

Typical messages

  • cache not found
  • failed to restore cache
  • artifact upload failed
  • request entity too large
  • 403 forbidden from storage backend

Likely causes

  • Backend outage or credentials issue.
  • Cache key changed or includes incorrect paths.
  • Artifact exceeds size or retention limits.
  • Job has no network route to storage.
  • Retention policy deleted expected data.

What to check

  • Cache key and paths.
  • Artifact size, retention, and storage quota.
  • Backend health and credentials.
  • Whether data in artifacts is approved for the selected retention and region.

Docker, privileged build, or nested virtualization failure#

Typical messages

  • Cannot connect to the Docker daemon
  • operation not permitted
  • privileged mode is required
  • KVM device not available

Likely causes

  • Runner profile intentionally blocks privileged containers or host socket access.
  • Docker service is not installed or not running on the image.
  • Nested virtualization is unavailable in the selected cloud or bare-metal profile.
  • The job needs a specialized runner label/tag that was not requested.

What to check

  • Whether Docker-in-Docker, host Docker socket, privileged containers, or KVM were approved during onboarding.
  • Runner labels/tags for privileged, buildkit, VM, or bare-metal workloads.
  • Security implications for secrets and private-network access.

Bare-metal hardware or device failure#

Typical messages

  • GPU, USB, license dongle, simulator, or hardware test step cannot find the device.
  • Job fails after host reboot or firmware update.
  • Thermal, power, disk, RAID, or NIC errors appear in host logs.

Likely causes

  • Device disconnected, driver changed, firmware drift, thermal throttling, failed disk, or out-of-band management issue.
  • Persistent local state conflicts with a new job.
  • Physical access, datacenter, or office power/network event.

What to check

  • Host inventory, device serial, firmware/driver versions, and last maintenance action.
  • Out-of-band management status where available.
  • Physical access log or hands-on-site availability.
  • Whether the workload can move to spare hardware.

See Bare metal runners for physical controls, maintenance, and escalation inputs.

Escalation checklist#

Include these details when raising a support request:

  • affected platform, repository/project/group, workflow/pipeline/job URL, and job ID;
  • exact error text and first failing step;
  • runner labels/tags, group, pool, executor, architecture, and environment;
  • first failure time, last successful run, and retry behavior;
  • recent changes to workflow, secrets, network, runner image, dependencies, or access;
  • business impact and deadline;
  • whether production secrets, personal data, customer data, or private-network access may be involved;
  • redacted logs or screenshots.