How to delete orphaned GitHub Actions runners with GitHub CLI

How to delete orphaned GitHub Actions runners with GitHub CLI

A quick showcase of removing orphaned self-hosted runners entries from GitHub Actions' settings page with GitHub CLI

There is a high chance you are using GitHub for your projects and are aware of GitHub Actions(GA). They are a GitHub feature that lets you run "workflows," commonly known as CI/CD jobs, on your GitHub repositories.

GitHub offers both managed and self-hosted GA runners. The latter are machines you host on your infrastructure and install the GA agent (software) to communicate with GitHub.

The Context

This article will discuss a specific problem with self-hosted GitHub runners, so let me set some context.

We create a few dozen self-hosted runners (on AWS EC2 spot instances) for some of our workloads once every few hours. These runners (i.e., the underlying virtual machines) get created and removed when we schedule CI/CD jobs on them.

We configure the machines so that they get decommissioned from AWS once the indented workloads have completed their lifecycle. The process will then deregister the runners from GitHub.

All of this is automated and scales up and down on demand.

But sometimes, automation fails. The machines get deallocated reliably (thanks to Terraform) from EC2; inevitably, you see orphaned entries on GitHub's Actions settings page.

screenshot-github-actions

The only ways to clean these up are:

You wait for a cooling-off period (which I believe is 30 days), and GitHub will remove them automatically.

OR

Go to the settings page for GitHub Actions, click on each runner one by one, and delete each runner.

Yuk!

I am a developer and, hence, obsessive about not liking buttons or doing this by hand, so I came up with a quick utility:

Introducing GitHub CLI

GitHub has a nifty CLI, namely the GitHub CLI, or the gh command when you install it.

This tool lets you live happily as a developer — the command line.

You can get many everyday tasks done with the CLI, including working with issues, assignments, PR reviews, etc., you name it. One of the most remarkable features, though, is:

It lets you interact with the GitHub API without needing anything else, like cURL, access-tokens, etc.

WHAAAAAAAAT!!!!

Yes—that's right.

With gh, you already provide GitHub with all the context it needs, so there is no requirement for setting authorization headers or similar things you would need with cURL.

Please RTFM if you want to learn more about the CLI and how to get started.

To get a list of runners, you can run:

gh api -H "Accept: application/vnd.github.v3+json" /repos/OWNER/REPO/actions/runners

And so on.

Use the CLI to remove the orphaned entries

For starters, let's look at the API documentation and get the list of all the runners:

gh api \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners

The above command worked but spat out an unfriendly json. How about using a cleaner format? Luckily, gh provides a jq style query formatting filter.

We need to work with runners that are A) not busy and B) not online. For our needs, I think only the latter would have sufficed, but I went with the below filter anyway:

'.runners[] | {id,status,busy} | select((.busy == false) and (.status == "offline")) | {id} | .[]'

Neat, isn't it? You can fiddle with the filters more on the jq playground.

Now let's put it together:

gh api \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners \
  -q '.runners[] | {id,status,busy} | select((.busy == false) and (.status == "offline")) | {id} | .[]' \
  --paginate

Throw in the -i flag to debug responses and headers if you like.

Moving on, let's revisit the API documentation and, this time look up removing runners:

gh api \
  --method DELETE \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/OWNER/REPO/actions/runners/RUNNER_ID

OK — we have everything.

  1. We can get all the offending runners for the criteria we want.

  2. We have the endpoint that will delete a said runner.

Let's get to the final command:

gh api \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners \
  -q '.runners[] | {id,status,busy} | select((.busy == false) and (.status == "offline")) | {id} | .[]' \
  --paginate | xargs -I {} \
  gh api \
  --method DELETE \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners/{}

That's it.

With a little sprinkle of jq and xargs, we can loop over all those orphaned entries and remove any offline runner still registered with GitHub.

2.png

For most parts, waiting for GitHub to clean things up works fine. Although, more practically speaking, I do this only in times of need. Say when I need to debug our GitHub Actions infrastructure.

This article was just an off-beat demo of GitHub CLI and its capabilities.

I hope you liked this story, until the next one.

Did you find this article valuable?

Support Mrugesh Mohapatra by becoming a sponsor. Any amount is appreciated!