Troubleshooting

Troubleshooting

Use this page when managed runner jobs are queued, failing, slow, unable to reach private services, or suspected of creating a security risk. It gives a practical triage workflow that applies across GitHub Actions, GitLab runners, Gitea Actions, Forgejo Actions, and other self-hosted runner models.

For exact platform behavior, also review GitHub Actions, GitLab runners, Gitea Actions, and Forgejo Actions. For known messages, see Common runner errors. For monitoring signals, see Runner monitoring and support.

Assistance can investigate runner infrastructure, monitoring, platform registration, network reachability, hardening controls, and operational evidence inside the agreed scope. The customer owns application code, workflow intent, release decisions, legal determinations, and approvals for risk acceptance.

Start with severity#

If there is any chance that secrets, personal data, regulated data, or production systems are affected, treat the issue as security-relevant until proven otherwise.

Universal triage checklist#

1. Identify the failing unit — platform, repository/project, workflow/pipeline, job ID, branch/ref, runner labels/tags, runner group/pool, executor, and environment.
2. Confirm impact — who is blocked, production or non-production, deadline, deployment risk, and whether a workaround exists.
3. Compare with recent changes — workflow edits, dependency updates, runner image updates, patching, network/firewall changes, secrets rotation, CI platform changes, or provider incidents.
4. Check platform routing — labels/tags match, runner group/project access is allowed, protected runner settings are correct, and the runner is online to the CI platform.
5. Check runner health — agent process, host CPU/memory/disk, workspace cleanup, container runtime, permissions, and logs.
6. Check dependencies — package registries, artifact stores, source checkout, DNS, private endpoints, cloud APIs, and cache services.
7. Check security boundary — whether the job should have access to the runner profile, secrets, private network, and deployment environment.
8. Recover safely — retry only when the failure mode is understood enough to avoid repeating a destructive or risky operation.
9. Record evidence — timeline, logs, dashboard links, changes reviewed, actions taken, owner decisions, and follow-up items.

Fast failure isolation#

Use this decision table to narrow the problem.

Safe recovery actions#

Prefer reversible, evidence-preserving actions first:

• pause or remove the affected runner from the eligible pool;
• route jobs to a known-good runner profile;
• reduce concurrency or pause low-priority workflows;
• restart the runner agent when no evidence preservation is required;
• rebuild ephemeral runners from a trusted image;
• roll back a runner image, toolchain, or network change using the recorded rollback path;
• rotate credentials when exposure is possible;
• disable private-network egress for suspect pools;
• open a provider or CI-platform support case with collected evidence.

Avoid deleting logs, wiping hosts, or clearing workspaces before capturing evidence when compromise, leakage, unauthorized access, or regulated data exposure is possible.

Escalation packet#

When escalating to Assistance, the customer incident owner, or a platform/provider, include:

• severity and business impact;
• platform and repository/project/group/organization;
• workflow/pipeline/job URLs and job IDs;
• labels/tags, runner group, runner pool, executor, and environment;
• first seen time, last known good run, and retry results;
• recent changes and maintenance windows;
• relevant log excerpts with secrets removed;
• screenshots or dashboard links for queue time, runner status, failures, and host health;
• whether production secrets, private-network access, personal data, or customer data may be involved;
• actions already taken and approvals needed.

For EU-sensitive incidents, also state the agreed data residency, support-ticket location, log retention expectations, and whether customer legal/DPO review is required. Assistance can supply facts and evidence; the customer decides regulatory notifications and legal positions.

Security incident checklist#

If a runner compromise, unauthorized job execution, or secret leak is suspected:

1. Stop new jobs on the affected runner pool or restricted labels/tags.
2. Preserve CI logs, runner logs, access logs, registration records, and relevant artifacts.
3. Identify affected repositories, branches, users, jobs, secrets, environments, and network destinations.
4. Revoke runner registration tokens and remove unauthorized runners.
5. Rotate exposed or potentially exposed secrets.
6. Block or restrict egress to private services until impact is understood.
7. Rebuild runners from trusted images instead of reusing suspect hosts.
8. Document timeline, evidence, containment actions, recovery validation, and follow-up controls.

After the fix#

A runner issue is not closed until the follow-up is clear:

• affected jobs have a successful validation run or a documented workaround;
• monitoring and alerts reflect the recovered state;
• changes, approvals, and rollback details are recorded;
• root cause or best available explanation is documented;
• security or compliance evidence is retained where required;
• recurring causes are added to the improvement backlog.

Related docs and services#

• Common runner errors
• Runner monitoring and support
• Runner security hardening
• Bare metal runners
• Incident response runbook
• Support and escalation