Incredibuild Team
reading time:
In 2024, the landscape of software delivery continues to evolve as teams strive to balance speed and reliability in increasingly complex environments. According to Google’s 2024 DORA report, continuous integration (CI) and continuous delivery (CD) have solidified their place as fundamental practices in modern software development. By integrating automation into deployment pipelines, teams are not only accelerating their delivery cycles but also improving the stability and quality of their releases.
As applications grow in complexity, CI/CD pipelines have expanded beyond deployment, playing a pivotal role in automating testing, monitoring, and security processes, thereby enabling teams to maintain a competitive edge in a fast-paced industry. These automated workflows speed up the software development lifecycle, reduce errors, and minimize the need for manual intervention. However, managing CI/CD pipelines can become complex, especially in large organizations with multiple repositories.
This is where reusable workflows, particularly GitHub Actions, can help with CI/CD optimization. Developers can automate their processes right within GitHub repositories with GitHub Actions. It makes CI/CD pipeline creation and management easier, freeing up teams to concentrate more on coding and less on process configuration. The capability of GitHub Actions to establish reusable workflows enables teams to specify a process once and apply it to various repositories, projects, or even entire companies.
This post aims to give DevOps practitioners and developers a comprehensive “how-to” for creating reusable workflows with GitHub Actions. We’ll cover DevOps best practices, typical pitfalls, and cutting-edge techniques for developing modular adaptable workflows. We’ll also examine how incorporating tools like Incredibuild can improve performance even more, especially for resource-intensive or large-scale builds.
By the time you finish reading this article, you’ll know exactly how to implement reusable workflows that optimize your CI/CD pipelines and boost team output.
GitHub Actions is an automation platform that lets developers automate tasks like testing, code deployment, and more. By defining a sequence of actions in a YAML configuration file, specific events can trigger a workflow, e.g., code pushes, pull requests, or scheduled tasks.
While GitHub Actions is valuable, its full potential lies in reusable workflows. These allow teams to define tasks once and reuse them across multiple projects, reducing redundancy and ensuring consistency.
Defining workflows within a repository ensures that your CI/CD pipelines remain consistent and up-to-date. Furthermore, its customizable nature lets you control triggers and actions in your workflows.
Reusable workflows have three primary components:
A solid grasp of these components is crucial for designing flexible, maintainable, and reusable workflows that adapt to a team’s evolving needs.
Reusable workflows are predefined workflows stored in a single location and invoked by other workflows across repositories. This approach centralizes logic, reduces duplication, and ensures consistent implementation of processes like testing, deployment, and linting.
For example, a company with a standardized deployment process can define it as a reusable workflow and call it from multiple repositories. Any updates to this workflow are automatically applied to all dependent repositories.
The primary benefits of this include:
The following recommendations will help companies implement reusable workflows that allow them to establish standardized, effective, and easily maintainable CI/CD pipelines across their repository ecosystem.
For instance, think about dividing building, testing, and deployment processes into distinct workflows that you can combine when necessary. This facilitates changing any one step of the procedure without affecting the pipeline as a whole.
Parameterizing workflows using inputs and outputs adds flexibility. Inputs let you tailor workflows to specific use cases, such as deploying to different environments. Outputs allow workflows to share data.
You can, for example, specify whether the deployment should be made to a production environment or a staging environment and define a reusable deployment workflow.
Example:
on: workflow_call
inputs:
environment:
description: 'Deployment environment'
required: true
default: 'staging'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to ${{ inputs.environment }}
run: ./deploy.sh --env ${{ inputs.environment }}To manage reusable workflows effectively, version control must be maintained. This ensures compatibility across various repositories and facilitates clear communication of changes. A structured plan for versioning is essential.
This popular technique versions reusable workflows (e.g., v1.0.0, v1.1.0, v2.0.0). Semantic versioning helps convey the type and extent of workflow modifications:
Workflows that are simple to debug and monitor require the implementation of strong error handling and logging.
Every workflow stage should result in legible insightful logs, e.g., warnings, errors, and success messages. This helps developers spot problems more rapidly. Implementing robust error-handling mechanisms is also essential, especially in critical scenarios. These can prevent a single error from disrupting the entire workflow.
For non-critical steps, a workflow can continue running even if certain steps fail by using features like continue-on-error. This ensures smoother execution and minimizes the impact of minor issues.
Proper error handling guarantees that the workflow fails gracefully, while thorough logs can assist in promptly identifying the problem when something goes wrong in your workflow.
Follow the steps below in each category for an effective reusable workflow.
In GitHub Actions, you will first have to define a YAML file with the proper syntax and structure to create a reusable workflow. You can store this file in your repository’s .github/workflows/ directory.
Let’s go over the fundamental framework of a reusable process. A basic GitHub Actions workflow consists of:
You will typically define a reusable workflow in a modular way so that it can be called by other workflows across different repositories. Below is an example of a reusable workflow file:
name: Reusable Workflow Example
on:
workflow_call: # Triggers the workflow when called by another workflow
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: '14'
- name: Install dependencies
run: npm install
- name: Run tests
run: npm testA workflow that installs dependencies and executes tests is defined by this structure. By referencing its location, you can reuse this workflow in different repositories. This workflow is meant to be called by another workflow, not by events like pushes or pull requests, as indicated by the on: workflow_call trigger.
A major advantage of reusable workflows is their ability to accept inputs and return outputs. This makes them dynamic and easy to adapt to various use cases.
Inputs are values that can be passed into a reusable workflow when it’s called. You can define required and optional inputs, each with a description and default value if needed.
Example:
on:
workflow_call:
inputs:
environment:
description: 'The environment to deploy to'
required: true
default: 'staging'Here, the input specifies which environment to deploy to. This input is required, and if it’s not passed, the default value of staging is used.
Reusable workflows can return outputs that can then be used by the workflow that called them. Outputs are useful when you want to pass data from one workflow to another.
Example:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Run tests
run: ./run_tests.sh
id: test_results
outputs:
result: ${{ steps.test_results.outputs.result }}In this example, the build job runs tests and outputs the result, which can then be accessed by other workflows.
To call a reusable workflow, you need to reference the workflow file by its path, along with the repository and version (branch, tag, or commit).
Example:
on:
push:
branches:
- main
jobs:
deploy:
uses: my-org/my-repo/.github/workflows/[email protected]
with:
environment: 'production'Above, the deploy job references the reusable workflow stored in the my-org/my-repo repository. It passes the input environment as production.
By calling reusable workflows in this manner, teams can centralize their CI/CD logic and avoid the need to redefine common tasks in each repository.
The following techniques are used to optimize complex workflows and adopt best practices, such as for proper management of sensitive information.
Multiple workflows are often necessary for complex processes. In these situations, nested reusable workflows are a good option, as they enable workflows to be called from within other workflows. They essentially let you divide complicated tasks into smaller parts you can manage more easily.
For example, you can have a reusable workflow for testing and another for building the application, with the test workflow being called by the build workflow.
Example of a nested reusable workflow:
on:
workflow_call:
jobs:
build:
uses: my-org/my-repo/.github/workflows/[email protected]
test:
uses: my-org/my-repo/.github/workflows/[email protected]In this case, the build job calls the build.yml workflow, while the test job calls the test.yml workflow. By structuring workflows in this modular, nested way, they will be easy to update, extend, and maintain.
In some workflows, you might want to run specific jobs or steps only under certain conditions. GitHub Actions provides several ways to handle conditional execution based on predefined variables, inputs, or outputs.
Example of conditional execution:
jobs:
test:
runs-on: ubuntu-latest
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Run tests
run: npm testIn this case, the test job will only run if the event triggering the workflow is a push to the main branch. You can use conditions based on the event type, branch names, or even outputs from other jobs to decide whether to execute certain parts of the workflow.
Handling sensitive information securely is crucial in any CI/CD pipeline. GitHub Actions supports the use of secrets, i.e., encrypted environment variables you can use in workflows without exposing them in the repository.
To reference secrets in your workflow, use the ${{ secrets.<secret_name> }} syntax:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: ./deploy.sh
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}In this example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are stored as secrets in GitHub, ensuring they are never exposed in the workflow file. This keeps sensitive information secure, guaranteeing it is only available to workflows that need it.
Although GitHub Actions is great for automation, its capabilities are further enhanced when combined with other DevOps tools. Teams can improve observability, automate feedback loops, and streamline workflows by integrating with CI/CD monitoring and notification solutions. Consider integrating GitHub Actions with the following:
Including these tools will make your workflows part of a larger automated DevOps ecosystem.
Let’s look at a couple scenarios where reusable workflows are especially beneficial.
In a large organization, multiple repositories might share similar testing needs. Instead of maintaining separate test workflows in each repository, you can create a reusable test workflow and reference it in every repository:
on:
push:
branches:
- main
jobs:
test:
uses: my-org/my-repo/.github/workflows/[email protected]
This approach reduces redundancy, makes updates easier, and ensures consistency across repositories.Another example is using reusable workflows to standardize deployments across different environments. For instance, the same deployment process can be used for staging and production, with inputs determining the target environment:
jobs:
deploy:
uses: my-org/my-repo/.github/workflows/[email protected]
with:
environment: 'production'
This standardization helps reduce errors and guarantees that each environment is deployed using the same process.
Workflow reuse can save time and money, but it requires careful management and upkeep. Here are some basic tactics:
Incredibuild enhances GitHub Actions by accelerating computational tasks through distributed computing. This is ideal for projects that consume a lot of resources, like enterprise software or game development. By integrating Incredibuild, teams can achieve significant performance gains, ensuring faster CI/CD pipelines without compromising quality.
Reusable workflows in GitHub Actions revolutionize CI/CD pipelines by boosting efficiency, maintainability, and scalability. By adhering to best practices like modular design, parameterization, and versioning, teams can build workflows that adapt to their evolving requirements.
Looking ahead, integrating cutting-edge technologies like Incredibuild and exploring advanced automation techniques will be pivotal for DevOps teams to stay competitive in the ever-evolving world of software development.ideo below walks through the fundamentals of optimizing build cache performance with Incredibuild’s Build Monitor. This article serves as both a companion and a reference, providing key takeaways and details for those who prefer exploring at their own pace. We’ll walk through a real-world example and provide actionable strategies to reduce cache misses and enhance overall efficiency. Whether you’re new to build caching or looking to improve your current workflow, this guide has you covered.
Table of Contents
Shorten your builds
Incredibuild empowers your teams to be productive and focus on innovating.
Incredibuild empowers your teams to be productive and focus on innovating.
