Delete orphaned self-hosted runners from GitHub Actions

Photo by Roman Synkevych 🇺🇦 on Unsplash

Delete orphaned self-hosted runners from GitHub Actions

A quick showcase of removing orphaned self-hosted runners entries from GitHub Actions' settings page with GitHub CLI

Mrugesh Mohapatra's photo
Mrugesh Mohapatra
·Jun 7, 2022·

4 min read

Play this article

There is a high chance you are using GitHub for your projects and are aware of GitHub Actions(GA). They are a GitHub feature that lets you run "workflows," commonly known as CI/CD jobs, on your GitHub repositories.

GitHub offers both managed and self-hosted GA runners. The latter are machines you host on your infrastructure and install the GA agent (software) to communicate with GitHub.

The Context:

This article will discuss a specific problem that I have been with self-hosted GitHub runners, so let me set some context.

We create a few dozen self-hosted runners (on AWS EC2 spot instances) for some of our workloads once every few hours. These runners (i.e., the underlying virtual machines) get created and removed when we schedule CI/CD jobs on them.

We configure the machines so that the machines get decommissioned from AWS once the indented workloads have completed their lifecycle. The process will then deregister the runners from GitHub.

All of this is automated and scales up and down on demand.

But sometimes, automation fails. The machines get deallocated quite reliably (thanks to terraform) from EC2; inevitably, you see orphaned entries on GitHub's Actions settings page.

screenshot-github-actions

The only ways to clean these up are:

You wait for a cooling-off period (which I believe is 30 days), and GitHub will remove them automatically.

OR

Go to the settings page for GitHub Actions, click on each runner, one by one, and delete each runner, one by one.

Yuk!

I am a developer and hence obsessive about not liking buttons or doing this by hand, so I came up with a quick utility:

Introducing GitHub CLI:

GitHub has a nifty CLI, namely the GitHub CLI, or the gh command when you install it.

This tool lets you live in your happy place as a developer—the command line.

You can get lots of everyday tasks done with the CLI, including working with issues, assignments, PR reviews, etc., you name it. One of the most remarkable features, though, is:

It lets you interact with the GitHub API without needing anything else, like cURL, access-tokens, etc.

WHAAAAAAAAT!!!!

Yes—that's right.

With gh, you already provide GitHub with all the context it needs, so there is no requirement for setting authorization headers or similar things you would need with cURL.

Please RTFM if you want to learn more about the CLI and how to get started with it.

To get a list of runners, you can run:

gh api -H "Accept: application/vnd.github.v3+json" /repos/OWNER/REPO/actions/runners

And so on.

Use the CLI to remove the orphaned entries:

For starters, let's look at the API documentation and get the list of all the runners:

gh api \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners

The above command worked, but it spits out an unfriendly json. How about using a cleaner format? Luckily, gh provides a jq style query formatting filter.

We need to work with runners that are A) not busy and B) not online. For our needs, I think only the latter would have sufficed, but I went with the below filter anyway:

'.runners[] | {id,status,busy} | select((.busy == false) and (.status == "offline")) | {id} | .[]'

Neat, isn't it? You can fiddle with the filters more on the jq playground.

Now let's put it together:

gh api \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners \
  -q '.runners[] | {id,status,busy} | select((.busy == false) and (.status == "offline")) | {id} | .[]' \
  --paginate

Throw in the -i flag to debug responses and headers if you like.

Moving on, let's revisit the API documentation and this time look up removing runners:

gh api \
  --method DELETE \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/OWNER/REPO/actions/runners/RUNNER_ID

OK—we have everything.

  1. We have a way to get all the offending runners for the criteria we want.
  2. We have the endpoint that will delete a said runner.

Let's get to the final command:

gh api \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners \
  -q '.runners[] | {id,status,busy} | select((.busy == false) and (.status == "offline")) | {id} | .[]' \
  --paginate | xargs -I {} \
  gh api \
  --method DELETE \
  -H "Accept: application/vnd.github.v3+json" \
  /repos/freeCodeCamp/news/actions/runners/{}

That's it.

With a little sprinkle from jq and xargs we can now loop over all those orphaned entries and remove any runner that has been offline and still registered with GitHub.

2.png

Although more practically speaking, I do this only in times of need. Say when I really need to debug our GitHub Actions infrastructure. For most parts waiting for GitHub to clean things up works fine.

This was just an off-beat demo of GitHub CLI and its capabilities.

I hope you liked this story, until the next one.

Did you find this article valuable?

Support Mrugesh Mohapatra by becoming a sponsor. Any amount is appreciated!

See recent sponsors Learn more about Hashnode Sponsors
 
Share this