Incredibuild Team
reading time:
In 2024, the landscape of software delivery continues to evolve as teams strive to balance speed and reliability in increasingly complex environments. According to Google’s 2024 DORA report, continuous integration (CI) and continuous delivery (CD) have solidified their place as fundamental practices in modern software development. By integrating automation into deployment pipelines, teams are not only accelerating their delivery cycles but also improving the stability and quality of their releases.
As applications grow in complexity, CI/CD pipelines have expanded beyond deployment, playing a pivotal role in automating testing, monitoring, and security processes, thereby enabling teams to maintain a competitive edge in a fast-paced industry. These automated workflows speed up the software development lifecycle, reduce errors, and minimize the need for manual intervention. However, managing CI/CD pipelines can become complex, especially in large organizations with multiple repositories.
This is where reusable workflows, particularly GitHub Actions, can help with CI/CD optimization. Developers can automate their processes right within GitHub repositories with GitHub Actions. It makes CI/CD pipeline creation and management easier, freeing up teams to concentrate more on coding and less on process configuration. The capability of GitHub Actions to establish reusable workflows enables teams to specify a process once and apply it to various repositories, projects, or even entire companies.
This post aims to give DevOps practitioners and developers a comprehensive “how-to” for creating reusable workflows with GitHub Actions. We’ll cover DevOps best practices, typical pitfalls, and cutting-edge techniques for developing modular adaptable workflows. We’ll also examine how incorporating tools like Incredibuild can improve performance even more, especially for resource-intensive or large-scale builds.
By the time you finish reading this article, you’ll know exactly how to implement reusable workflows that optimize your CI/CD pipelines and boost team output.
Introduction to reusable workflows in GitHub Actions
GitHub Actions is an automation platform that lets developers automate tasks like testing, code deployment, and more. By defining a sequence of actions in a YAML configuration file, specific events can trigger a workflow, e.g., code pushes, pull requests, or scheduled tasks.
While GitHub Actions is valuable, its full potential lies in reusable workflows. These allow teams to define tasks once and reuse them across multiple projects, reducing redundancy and ensuring consistency.
Defining workflows within a repository ensures that your CI/CD pipelines remain consistent and up-to-date. Furthermore, its customizable nature lets you control triggers and actions in your workflows.
Reusable workflows have three primary components:
- Workflows define a series of automated tasks and are stored in your repository’s GitHub/workflows directory.
- Jobs are groups of sequential steps within every workflow executed in a specific environment.
- Actions are individual tasks within a job, e.g., installing dependencies, running tests, or deploying applications.
A solid grasp of these components is crucial for designing flexible, maintainable, and reusable workflows that adapt to a team’s evolving needs.
Concept and benefits of reusable workflows
Reusable workflows are predefined workflows stored in a single location and invoked by other workflows across repositories. This approach centralizes logic, reduces duplication, and ensures consistent implementation of processes like testing, deployment, and linting.
For example, a company with a standardized deployment process can define it as a reusable workflow and call it from multiple repositories. Any updates to this workflow are automatically applied to all dependent repositories.
The primary benefits of this include:
- Efficiency: Every time you start a new project, you don’t have to redo routine steps like building the project, running tests, and deploying to the cloud, saving time and money.
- Decreased duplication: Code duplication across repositories is no longer necessary thanks to reusable workflows; this guarantees consistency throughout all projects and lowers the chance of errors.
- Centralized management: It is easier to track changes when you use reusable workflows to centralize tasks like building an image, deploying code, and running unit tests.
- Automatic updates: Any updates/changes made to a reusable workflow automatically benefit all repositories that use it; no manual intervention is required, resulting in easier maintenance.
Best practices for creating reusable workflows
The following recommendations will help companies implement reusable workflows that allow them to establish standardized, effective, and easily maintainable CI/CD pipelines across their repository ecosystem.
Modular design principles
Modularity is one of the most crucial design concepts for reusable workflows. By using a modular approach, you can divide your workflows into more manageable targeted parts that cater to particular tasks. This is essential for scalability because each module can be changed and reused independently without impacting the pipeline as a whole.
For instance, think about dividing building, testing, and deployment processes into distinct workflows that you can combine when necessary. This facilitates changing any one step of the procedure without affecting the pipeline as a whole.
- Single responsibility principle: Each workflow should focus on a specific task, such as testing, building, or deployment, to ensure clarity and reusability.
- Flexible composition: Modular workflows can be combined to form complex pipelines, allowing individual workflows to be updated independently.
Parameterization techniques
Parameterizing workflows using inputs and outputs adds flexibility. Inputs let you tailor workflows to specific use cases, such as deploying to different environments. Outputs allow workflows to share data.
You can, for example, specify whether the deployment should be made to a production environment or a staging environment and define a reusable deployment workflow.
Example:
on: workflow_call
inputs:
environment:
description: 'Deployment environment'
required: true
default: 'staging'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to ${{ inputs.environment }}
run: ./deploy.sh --env ${{ inputs.environment }}Versioning strategies
To manage reusable workflows effectively, version control must be maintained. This ensures compatibility across various repositories and facilitates clear communication of changes. A structured plan for versioning is essential.
Semantic versioning
This popular technique versions reusable workflows (e.g., v1.0.0, v1.1.0, v2.0.0). Semantic versioning helps convey the type and extent of workflow modifications:
- Major version updates: Increment the major version number (e.g., from v1.x.x to v2.0.0) when introducing significant functionality changes or breaking backward compatibility.
- Minor updates and patches: Use minor version updates (e.g., v1.0.0 to v1.1.0) or patch updates (e.g., v1.0.0 to v1.0.1) for backward-compatible changes, e.g., adding features or fixing bugs.
Error handling and logging
Workflows that are simple to debug and monitor require the implementation of strong error handling and logging.
Every workflow stage should result in legible insightful logs, e.g., warnings, errors, and success messages. This helps developers spot problems more rapidly. Implementing robust error-handling mechanisms is also essential, especially in critical scenarios. These can prevent a single error from disrupting the entire workflow.
For non-critical steps, a workflow can continue running even if certain steps fail by using features like continue-on-error. This ensures smoother execution and minimizes the impact of minor issues.
Proper error handling guarantees that the workflow fails gracefully, while thorough logs can assist in promptly identifying the problem when something goes wrong in your workflow.
A step-by-step guide to creating a reusable workflow
Follow the steps below in each category for an effective reusable workflow.
Syntax and structure
In GitHub Actions, you will first have to define a YAML file with the proper syntax and structure to create a reusable workflow. You can store this file in your repository’s .github/workflows/ directory.
Let’s go over the fundamental framework of a reusable process. A basic GitHub Actions workflow consists of:
- Name: An optional but advised name for the workflow
- Triggers: Events that start a workflow, e.g., a custom event push request or pull request
- Jobs: A list of the tasks required to complete the workflow
- Steps: A set of procedures used in every job
You will typically define a reusable workflow in a modular way so that it can be called by other workflows across different repositories. Below is an example of a reusable workflow file:
name: Reusable Workflow Example
on:
workflow_call: # Triggers the workflow when called by another workflow
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: '14'
- name: Install dependencies
run: npm install
- name: Run tests
run: npm testA workflow that installs dependencies and executes tests is defined by this structure. By referencing its location, you can reuse this workflow in different repositories. This workflow is meant to be called by another workflow, not by events like pushes or pull requests, as indicated by the on: workflow_call trigger.
Input and output management
A major advantage of reusable workflows is their ability to accept inputs and return outputs. This makes them dynamic and easy to adapt to various use cases.
Defining inputs
Inputs are values that can be passed into a reusable workflow when it’s called. You can define required and optional inputs, each with a description and default value if needed.
Example:
on:
workflow_call:
inputs:
environment:
description: 'The environment to deploy to'
required: true
default: 'staging'Here, the input specifies which environment to deploy to. This input is required, and if it’s not passed, the default value of staging is used.
Capturing outputs
Reusable workflows can return outputs that can then be used by the workflow that called them. Outputs are useful when you want to pass data from one workflow to another.
Example:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Run tests
run: ./run_tests.sh
id: test_results
outputs:
result: ${{ steps.test_results.outputs.result }}In this example, the build job runs tests and outputs the result, which can then be accessed by other workflows.
Calling reusable workflows
To call a reusable workflow, you need to reference the workflow file by its path, along with the repository and version (branch, tag, or commit).
Example:
on:
push:
branches:
- main
jobs:
deploy:
uses: my-org/my-repo/.github/workflows/[email protected]
with:
environment: 'production'Above, the deploy job references the reusable workflow stored in the my-org/my-repo repository. It passes the input environment as production.
By calling reusable workflows in this manner, teams can centralize their CI/CD logic and avoid the need to redefine common tasks in each repository.
Advanced techniques
The following techniques are used to optimize complex workflows and adopt best practices, such as for proper management of sensitive information.
Nested reusable workflows
Multiple workflows are often necessary for complex processes. In these situations, nested reusable workflows are a good option, as they enable workflows to be called from within other workflows. They essentially let you divide complicated tasks into smaller parts you can manage more easily.
For example, you can have a reusable workflow for testing and another for building the application, with the test workflow being called by the build workflow.
Example of a nested reusable workflow:
on:
workflow_call:
jobs:
build:
uses: my-org/my-repo/.github/workflows/[email protected]
test:
uses: my-org/my-repo/.github/workflows/[email protected]In this case, the build job calls the build.yml workflow, while the test job calls the test.yml workflow. By structuring workflows in this modular, nested way, they will be easy to update, extend, and maintain.
Conditional execution
In some workflows, you might want to run specific jobs or steps only under certain conditions. GitHub Actions provides several ways to handle conditional execution based on predefined variables, inputs, or outputs.
Example of conditional execution:
jobs:
test:
runs-on: ubuntu-latest
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Run tests
run: npm testIn this case, the test job will only run if the event triggering the workflow is a push to the main branch. You can use conditions based on the event type, branch names, or even outputs from other jobs to decide whether to execute certain parts of the workflow.
Secrets management
Handling sensitive information securely is crucial in any CI/CD pipeline. GitHub Actions supports the use of secrets, i.e., encrypted environment variables you can use in workflows without exposing them in the repository.
To reference secrets in your workflow, use the ${{ secrets.<secret_name> }} syntax:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: ./deploy.sh
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}In this example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are stored as secrets in GitHub, ensuring they are never exposed in the workflow file. This keeps sensitive information secure, guaranteeing it is only available to workflows that need it.
Integration with other DevOps tools
Although GitHub Actions is great for automation, its capabilities are further enhanced when combined with other DevOps tools. Teams can improve observability, automate feedback loops, and streamline workflows by integrating with CI/CD monitoring and notification solutions. Consider integrating GitHub Actions with the following:
- Terraform: Automate infrastructure provisioning.
- Docker: Build and deploy containers as part of your CI/CD pipeline.
- Sentry: Monitor errors and exceptions in your applications.
- Slack or Microsoft Teams: Send notifications for workflow events, such as build successes or failures.
- JIRA: Automatically create issues or update tickets based on workflow status.
Including these tools will make your workflows part of a larger automated DevOps ecosystem.
Real-world examples and use cases
Let’s look at a couple scenarios where reusable workflows are especially beneficial.
Use case 1: Multi-repository testing
In a large organization, multiple repositories might share similar testing needs. Instead of maintaining separate test workflows in each repository, you can create a reusable test workflow and reference it in every repository:
on:
push:
branches:
- main
jobs:
test:
uses: my-org/my-repo/.github/workflows/[email protected]
This approach reduces redundancy, makes updates easier, and ensures consistency across repositories.Use case 2: Standardized deployment pipeline
Another example is using reusable workflows to standardize deployments across different environments. For instance, the same deployment process can be used for staging and production, with inputs determining the target environment:
jobs:
deploy:
uses: my-org/my-repo/.github/workflows/[email protected]
with:
environment: 'production'
This standardization helps reduce errors and guarantees that each environment is deployed using the same process.
Best practices for maintaining and updating reusable workflows
Workflow reuse can save time and money, but it requires careful management and upkeep. Here are some basic tactics:
- Regularly review and update workflows: Workflows need to be updated as your technology stack changes or new requirements emerge; versioning lets you manage updates without affecting repositories that rely on previous iterations.
- Modularize workflows: Workflows should be modularized to maintain their focus, small size, and ease of updating; this lowers the possibility of making mistakes when modifications are made.
- Automate workflow testing: Regularly test reusable workflows to know they continue operating as intended; create a specific repository to test reusable workflows before implementing them across several repositories.
- Maintain transparent documentation: Having clear documentation and change logs for your reusable workflows is essential for developers to understand how they work and the inputs/outputs required.
Integrating Incredibuild for Performance Optimization
Incredibuild enhances GitHub Actions by accelerating computational tasks through distributed computing. This is ideal for projects that consume a lot of resources, like enterprise software or game development. By integrating Incredibuild, teams can achieve significant performance gains, ensuring faster CI/CD pipelines without compromising quality.
Conclusion and future trends in CI/CD automation
Reusable workflows in GitHub Actions revolutionize CI/CD pipelines by boosting efficiency, maintainability, and scalability. By adhering to best practices like modular design, parameterization, and versioning, teams can build workflows that adapt to their evolving requirements.
Looking ahead, integrating cutting-edge technologies like Incredibuild and exploring advanced automation techniques will be pivotal for DevOps teams to stay competitive in the ever-evolving world of software development.ideo below walks through the fundamentals of optimizing build cache performance with Incredibuild’s Build Monitor. This article serves as both a companion and a reference, providing key takeaways and details for those who prefer exploring at their own pace. We’ll walk through a real-world example and provide actionable strategies to reduce cache misses and enhance overall efficiency. Whether you’re new to build caching or looking to improve your current workflow, this guide has you covered.
Table of Contents
Shorten your builds
Incredibuild empowers your teams to be productive and focus on innovating.