Extra CI flexibility with Github Runner on AWS Spot Instances

This article describes how to set up a GitHub Actions self-hosted runner on an AWS spot instance. But first, let’s talk about what a self-hosted GitHub runner is, and why we should use it.

What’s a GitHub Actions runner?

A GitHub Actions runner is an application that runs a job from a GitHub Actions workflow on a machine.
You can use either a ready-made GitHub-hosted runner (Windows, Ubuntu, or MacOSX, each with specific hardware) running on GitHub servers or a self-hosted runner in your own environment.

GitHub Runners can be used for:

• Compilation
• Running tests
• Static Code Analysis (SCA)
• Building a container image

And much more!

We’ve talked previously in this post about how to use GitHub actions with a C++ project and in this post about how to tie it with Incredibuild. Here, we’ll focus on connecting GitHub actions to trigger operations on an AWS spot instance.

Why use a self-hosted runner?

GitHub-hosted runners are a great option if you want to run a workflow quickly and easily. However, there are cases where self-hosted are the best choice for you (as opposed to GitHub-hosted runners):

• You are using a private repository and don’t want to spend money for every minute a job is running.
• You are running a complicated job that requires a machine with better hardware than GitHub-hosted runner machines hardware. 

You’ll find more about self-hosted runners at GitHub documentation.

What’s an AWS spot instance

AWS offers a variety of instance types to deploy on your cloud, like on-demand, reserved, and spot instances.

All three instance types have the same functionality while running, and pricing is the only difference. An on-demand instance is an instance with no long-term commitments where you have to pay a certain per-second rate. A reserved instance is a rented instance for a specific period at a lower per-second rate than on-demand instances.

Unlike these two, A spot instance uses spare AWS capacity and has the lowest price among these instance types (up to 90% lower than the on-demand price). However, there’s a catch. Since you are using unused resources, your instance can be interrupted (with a 2-minute notification).

Try to avoid using spot instances for stateful applications, databases, or workloads containing sensitive information.

For stateless applications that run short-time tasks and can be interrupted such as application testing, CI, data analysis, image rendering, spot instances are the ideal choice.

So what are we going to do?

In this example, the workflow will validate the syntax of Python files in your repository using Flake8 and Pylint (but of course, any other workflow will also do).

You’ll set up an AWS spot instance to run a GitHub Actions self-hosted runner and configure it to redeploy if AWS interrupts it to increase the flexibility of Continuous Integration (CI) jobs at a minimal cost.

This self-hosted runner will execute the workflow jobs.

All the files we will use to deploy the spot instance can be found in this GitHub Repository under the aws-files directory. In this repository, you will also find an example of a GitHub Actions Workflow.

Walkthrough prerequisites:

• AWS account
• Configured AWS CLI
• GitHub repository
• GitHub personal access token (PAT)

Creating and configuring the Spot Instance

To create the spot instance with the correct configuration, you’ll only need 3 files.

1. user-data.sh – a Bash script file contains the commands that will run on an instance at launch. In our case, the script will run on every instance deployed, even on a new one that replaces an interrupted spot machine. It contains a script that installs and configures a self-hosted runner and starts its service. (For more information about the user-data.sh file you can read in AWS documentation).
   

   #!/bin/bash
   
   github-user="Your GitHub Username"
   github-repo="Your GitHub Repository name"
   PAT="Your Super Secret PAT"
   
   # Download jq for extracting the Token
   yum install jq -y
   
   # Create and move to the working directory
   mkdir /actions-runner && cd /actions-runner
   
   # Download the latest runner package
   curl -o actions-runner-linux-x64-2.286.1.tar.gz -L https://github.com/actions/runner/releases/download/v2.286.1/actions-runner-linux-x64-2.286.1.tar.gz
   
   # Extract the installer
   tar xzf ./actions-runner-linux-x64-2.286.1.tar.gz
   
   # Change the owner of the directory to ec2-user
   chown ec2-user -R /actions-runner
   
   # Get the runner's token
   token=$(curl -s -XPOST -H "authorization: token $PAT" https://api.github.com/repos/$github-user/$github-repo/actions/runners/registration-token | jq -r .token)
   
   # Create the runner and start the configuration experience
   sudo -u ec2-user ./config.sh --url https://github.com/<github-username>/$github-user --token $token --name "spot-runner-$(hostname)" --unattended
   
   # Create the runner's service
   ./svc.sh install
   
   # Start the service
   ./svc.sh start

    

2. spot-instance-launch-template.json – a JSON file describes the spot instance’s launch template configuration (Image, Type, SecurityGroups, etc…). For our purposes, it will also contain the user-data.sh file contents encoded in base64.
   

   {
     "ImageId": "ami-001089eb624938d9f",
     "InstanceType": "t2.micro",
     "KeyName": "instance key pair name",
     "SecurityGroups": ["security group name"],
     "UserData": "user-data.sh file content encoded in base64",
     "InstanceMarketOptions": {
         "MarketType": "spot",
         "SpotOptions": {
         "MaxPrice": "0.03",
         "SpotInstanceType": "one-time",
         "InstanceInterruptionBehavior": "terminate"
         }
     }
   }

3. spot-instance-auto-scaling-group.json – a JSON file describing the spot instance’s auto scaling group configuration such as what launch-template to use, the minimal and maximal number of instances, etc. In our scenario, the “auto scaling” is actually used to create a replacement for an interrupted spot machine and not for dynamically flexible growth.This is important because by using an auto scaling group, if a Spot Instance is interrupted and AWS terminates the instance, another spot instance with the same configuration will deploy automatically.

   {
       "AutoScalingGroupName": "spot-instance-asg",
       "LaunchTemplate": {
       "LaunchTemplateName": "spot-instance-launch-template"
       },
       "MinSize": 1,
       "MaxSize": 1,
       "AvailabilityZones": ["your Availability Zones"]
   }
   

Yalla, let’s deploy

1. Clone the repository and change directory to the essential files directory
   

   git clone https://github.com/omermishania/github-runner-on-aws-spot.git && cd github-runner-on-aws-spot/aws-files/

    

2. Create an AWS security group
   Create a security group with these rules:
   Inbound – SSH (port 22)
   Outbound – HTTPS (port 443)
3. Create user-data.sh file
   # Configure the file:Edit the file with your favorite text editor and change the values of the variables to your values when:
   • Github-user = your GitHub username
   • github-repo = your GitHub repository name
   • PAT = your GitHub PAT (personal access token)
     

     vim user-data.sh

   As an example:
   

   …
   github-user="omermishania"
   github-repo="github-runner-on-aws-spot"
   PAT="MY-SECRET-GITHUB-PAT"
   …

   When you finish, save and exit the file.

   # Encode the content of user-data.sh file to base64

    

   cat user-data.sh | base64 -w 0

   # Copy the encoded content, and save it in a safe place.

    

4. Creating spot-instance-launch-template.json file
   # Configure the fileEdit the file and change:
   • ‘ImageId’ value to an EC2 image ID of your choice.
     You can find the EC2 image ID next to the image name (in the red box):
   • ‘InstanceType’ value to your desired instance type
   • ‘SecurityGroups’ value to a list contains the security group name you created minutes ago
   • ‘UserData’ value to the base64 encoded content you copied
   • ‘MaxPrice’ value to the maximum price of the spot instance

   vim spot-instance-launch-template.json

   As an example:

   {
     "ImageId": "ami-001089eb624938d9f",
     "InstanceType": "t2.micro",
     "SecurityGroups": ["gh-runner-spot-sg"],
     "UserData":
   "IyEvYmluL2Jhc2gKCgojIERvd25sb2FkIGpxIGZvciBleHRyYWN0aW5nIHRoZSBUb2tlbgp5dW0gaW5zdGFsbCBqcSAteQoKIyBDcmVhdGUgYW5kIG1vdmUgdG
   8gdGhtUiAvYWN0aW9ucy1ydW5uZXIKCiMgR2V0IHRoZSBydW5uZXIncyB0b2tlbgpQQVQ9ImdocF81SmxTU3V16YXc2ggaW5zdGFsbAplY2hvICJzdmMgaW5zdG
   FsbGVkIiA+PiB0ZXN0LmxvZwoKIyBTdGFydCB0aGUgc2VydmljZQouL3N2Yy5zaCBzdGFydAplY2hvICJzdmMgc3RhcnRlZCIgPj4gdGVzdC5sb2cK",
     "InstanceMarketOptions": {
         "MarketType": "spot",
         "SpotOptions": {
         "MaxPrice": "0.03",
         "SpotInstanceType": "one-time",
         "InstanceInterruptionBehavior": "terminate"
         }
     }
   }

   
   # Run the command:When you finish, save and exit the file.

   Make sure you run this command only after you pasted the base64 encoded user-data.sh

   aws ec2 create-launch-template --launch-template-name spot-instance-launch-template --launch-template-data file://spot-instance-launch-template.json

5. vim spot-instance-auto-scaling-group.json
6. Creating spot-instance-auto-scaling-group.json file
   # Configure the fileEdit the file and change ‘AvailabilityZones’ value to a list containing your desired availability zone(s). You can also change the minimum and maximum instance count if you want to deploy more than one runner.

   vim spot-instance-auto-scaling-group.json

   When you finish, save and exit the file.

   # Run the command

   aws autoscaling create-auto-scaling-group --cli-input-json file://spot-instance-auto-scaling-group.json

   Congratulations, you have just created a self-hosted runner on your spot instance! 

Configure the GitHub Actions workflow to use a self-hosted runner

Our brand new self-hosted runner is up and running!
However, the GitHub Actions workflow is still not going to use it. To make that happen, we need to edit the workflow file and change the ‘runs-on’ value to ‘self-hosted’.

As an example, I used the following workflow to check for valid Python syntax. Your workflow can serve any purpose you would like, you only need to change the value of the runs-on field to ‘self-hosted’ (marked in blue):

name: Python Linting
on:
  push:
  pull_request:
jobs:
  lint-python-code:
       runs-on: self-hosted
       steps:
       - uses: actions/checkout@v2
       - name: Install dependencies
       run: |
       python3 -m pip install --upgrade pip
       pip3 install flake8
       pip3 install pylint
       if [ -f requirements.txt ]; then pip3 install --target=/usr/bin -r requirements.txt; fi
       - name: Lint with flake8
       run: |
      /usr/local/bin/flake8 src --count --select=E9,F63,F7,F82 --show-source --statistics
      /usr/local/bin/flake8 src --count --max-complexity=10 --max-line-length=88 --statistics
      - name: Lint with Pylint
      run: |
      /usr/local/bin/pylint src

And That’s it!

Let’s recap what we have done so far:

• Created and configured an auto-recovering Spot Instance.
• Configured a GitHub actions runner on that instance.
• Configure the GitHub Actions Workflow to use it.

The workflow now runs on your newly created GitHub Runner on a spot instance on your cloud, and in case a spot instance is interrupted and terminated, a new one with a new GitHub runner will be created.

Conclusion

By using a GitHub self-hosted runner on a spot instance on your cloud environment, you have the flexibility to use any custom configurations and tools that will serve exactly your CI purpose and even save unnecessary costs.

In our configuration, the spot instance needs to be alive in order to listen to GitHub runner commands. So we actually pay for a spot instance being live consistently(despite the cheap spot tariffs). One could think of another possible configuration where we use a GitHub hosted runner that only raises the spot instance and lets it do the work. This way we would pay for the GitHub hosted runner only for the time it was invoked and running, and also for the spot instance only for a limited amount of time. However, this requires a more complicated setup beyond the scope of this post.

In order to achieve auto flexibility of spot instances for CI builds without a need to manage them manually, you can use Incredibuild for cloud. Tune in next week to learn how!