Hope is not a strategy (Site Reliability Engineering)
Site Reliability Engineering (SRE) is a philosophy that has been steadily gaining followers in recent years, despite being conceptualized in 2003 by Ben Treynor Sloss at Google. With DevOps practices already firmly established in many organizations, is a conflict between the two imminent? Is SRE nothing more than a passing trend? Does SRE complement DevOps or vice versa? Let’s review.
Wikipedia defines DevOps as a “set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality.” According to Site Reliability Engineering, “SREs are focused on finding ways to improve the design and operation of systems to make them more scalable, more reliable, and more efficient.” By simply reviewing the definitions above, one can see that the DevOps definition is more focused on practices, while the SRE definition includes words like design and reliability. It’s true that both covers both, that is: DevOps doesn’t ignore design and reliability and SRE doesn’t go without practices, however when we look at the focus of each approach, we will see that these subtle differences in the definitions of the two are apparent in their actual implementation. We’ll examine some of the more readily apparent distinctions by exploring each in a bit more depth.
DevOps vs SRE – Differences
Culture
Let’s begin by understanding the cultural aspect – both DevOps and SRE are adopted by organizations as a culture rather than a practice area and in this sense, they are the same, even though teams may be designated as primarily responsible for practices supporting that culture. Although each culture is flexible – that is allowing for teams focused on DevOps or SRE practices directly vs embedded practitioners across teams – it is common for organizations to adopt SRE as one or more focused teams while DevOps practitioners commonly tend to be dispersed amongst teams.
For example, and referring to the book once again, in 2016 Google employed over 1000 site reliability engineers with responsibilities that support a culture of SRE across the Conversely, an organization that employs site reliability engineers but does not embrace SRE culture is not likely to be successful with respect to SRE practices. Although more nebulous, the same dichotomy exists for organizations with respect to DevOps culture.
Risk
Next, we should examine the respective approaches to risk – a core attribute of SRE. Both SRE and DevOps seek to minimize risk exposure as well as maximize detection and responsiveness to impact, however, SRE culture views risk management as the goal rather than the more common DevOps approach of managing a symptom. Suppose two nearly identical web application companies – one pursuing DevOps and the other pursuing SRE – wish to implement a new feature as part of their respective applications. Further, suppose that each respective change has the exact same probability of introducing unreliability in the application. Because SRE cultures invest in risk tolerance definitions, immediately this company necessarily has a better understanding of how to react to the proposed change; likely before a line of code has been written to support it. Certainly, DevOps organizations may define and maintain risk tolerance profiles – that is that nothing precludes DevOps from having the same practice – but making risk tolerance definition a core principle of SRE positions the context of risk in the conversation at the outset as a matter of course.
Opportunity Cost
On the opposite side of the scale from risk tolerance can be found opportunity cost. Continuing with the former example, it is likely that the DevOps organization is better positioned to minimize the loss in opportunity cost. This is expected because the SRE organization has specifically and intentionally positioned opportunity cost as secondary to risk management. Restated – the SRE organization accepts lost opportunity costs as necessary in order to achieve the desired risk tolerance. This is not to say that either organization wishes for lost opportunity cost but is to highlight the slightly different prioritization.
Observability
It’s a common saying in DevOps that “if it isn’t being tested then it isn’t working”. Active testing is certainly an important activity for both DevOps and SRE practices, however, the way in which test results are consumed is central to SRE philosophy. Back to our favorite book (chapter 3): “As standard practice at Google, we are often best served by identifying an objective metric to represent the property of a system we want to optimize. By setting a target, we can assess our current performance and track improvements or degradations over time.” Let’s imagine two organizations – one following DevOps and one following SRE. Each organization tracks a Key Performance Indicator (KPI) of failed deployment rate or the frequency with which new deployments result in a negative impact on service. Both organizations may want to target a low value for this KPI – say 5% or less – but the way in which each organization conceptualizes the metric is different: the DevOps team may target a low value as a way to reduce disruption to the service while the SRE team will consume the KPI as a part of calculating the error budget for the service (more on error budgets below).
Error Budget
In order to contextualize observability – both in terms of the value of a metric as well as success in optimizing a metric – SRE teams often concern themselves with a budget. In concept an error budget is no different than a financial one – define how much to spend and track actual spending against that limit. Error budgets are implemented as an agreement between an SRE team and one or more development teams – DevOps or otherwise. This strategy can help to reduce tensions between the two teams, despite their logical opposing priorities of stability (SRE) and velocity (Dev); as long as a development team has not spent their error budget within a timeframe then an SRE team should not need to raise concerns about stability and collectively the teams should maintain the desired velocity.
Postmortem
Despite the best planning and preparation, service interruptions will occur. Handling such interruptions is an important part of both DevOps and SRE practice, but so is what comes after an interruption – a postmortem. Recording the event, actions are taken to correct it, and what actions will be taken to prevent the event from recurring all comprise a postmortem and help to ensure that incidents do not overwhelm a team or an organization. Although any organization may perform postmortem investigations, SRE philosophy specifically prescribes the practice and should be expected from any team pursuing this culture. A DevOps engineer that would fix a failure, would certainly analyze it, document what went wrong in some way, and would, of course, try to avoid that failure in the future. But it is not necessarily required that this DevOps engineer would conduct a full post-mortem analysis of the failure, once fixed. If you are a DevOps engineer and you do perform comprehensive post-mortems, that’s OK, it might just mean that you are implementing the SRE philosophy, whether you know it, or not.
Adopting SRE Culture
There are a few questions to answer about adoption – who, why, and how.
Who?
Although SRE culture is more than simply a set of tools – or even a list of practices – at its core SRE focuses on reliability. An organization that benefits from an increased focus on its technical reliability – not only potential improvements to uptime but also the ancillary gains in improved observability, a greater focus on value-add metrics, and the exercise of error budgeting – is a good fit for the addition of SRE culture.
On an individual level, anyone can become an SRE engineer by adopting the philosophy. Don’t get me wrong, I’m not arguing that you can just become an SRE. I do think, though, that a good engineer can learn to adopt SRE principles and thus pursue the SRE culture. There’s nothing that prevents a motivated individual from learning how to follow SRE practices.
In some organizations, this involves transitioning existing DevOps practitioners to roles focused on SRE practices, either as a single centralized group or embedded within existing teams. Because SRE is largely born out of DevOps philosophy, many of the skills acquired by practitioners will translate in both directions.
It is to be noted that though SRE culture comes from Google, it is not only organizations of Google scale that can benefit from it.
Why?
An increased focus on reliability may benefit many different organizations, however, the primary benefit of SRE comes from its culture – a focus on actionable metrics, viewing a technology stack through error budgeting, incident management practices, and so on. There are many ways – technical and cultural – to improve reliability but pursuing SRE is inherently a mindset as well as a practice.
How?
Because each organization is unique, pursuing an SRE culture will also be inherently distinct to each organization. Some organizations may have internally built DevOps practices, or follow the Information Technology Infrastructure Library (ITIL) framework for delivering information technology services, or adhere to Agile methodologies to organize software development effort – no matter the existing pattern, SRE can be a fit. That said, considering these questions may help to get the ball rolling:
- Should SRE replace DevOps?
Some organizations integrate the two, capturing natural complementary goals and benefits; in fact, some practitioners view SRE as an implementation of DevOps – “One could view DevOps as a generalization of several core SRE principles to a wider range of organizations, management structures, and personnel. One could equivalently view SRE as a specific implementation of DevOps with some idiosyncratic extensions.” (DevOps or SRE – Chapter 1, Introduction – Site Reliability Engineering). Some teams find that embedding one or more SRE specialists within individual DevOps teams is an effective way to ensure horizontal adoption of SRE practices, while some other teams build an SRE center of excellence. - Should SRE replace ITIL?
Much like DevOps, some organizations find that ITIL and SRE are very complementary, with both emphasizing a form of governance around change management and an adherence to SLO. - Is Agile a requirement for implementing SRE culture?
Not at all; while many organizations find Agile methodology to be a good foundation for the iterative processes of SRE and a match for an error budget approach, any methodology can be compatible. SRE is focused on the outcome rather than the production of software per-se, the methodology for producing the software is not directly relevant. As an example, an organization that follows a waterfall methodology would implement a phase of risk assessment and abatement in accordance with SRE’s error budgeting and testing approaches.
Why Not
SRE philosophy may have many benefits for many organizations, but to turn the phrase “for everyone but not everything” – SRE is not for everyone nor for everything.
- Not all organizations benefit from increased reliability or, restated, the incremental cost of increased reliability does not return an equal or greater incremental value for all organizations. For some organizations this is a question of scale; Google’s global scale is not the only one potentially benefiting from SRE, however, a small startup with a few thousand users may not yet justify the investment.
- The reduction in development velocity may not be justified by increased reliability for some organizations. By focusing on practices such as error budgeting, SRE organizations necessarily reduce velocity; and should do so with intent.
- By establishing an SRE practice, organizations can fall into a trap of thinking “reliability is now exclusively an SRE responsibility”; reliability is an SRE responsibility, but it is shared across the organization, just like it was before adopting SRE. While this is not directly a reason to avoid adopting SRE, it can lead to organizations not reaching full SRE potential and even abandoning the adoption.
- Re-labeling teams or team members as “SRE” does not mean that an organization is aligned to the philosophy. For example, an SRE team not implementing error budgeting can hardly be said to be aligned to SRE pillars. Such gaps can lead to organizations viewing SRE as the latest buzzword and dismissing it as a passing fad; at the least such gaps reduce even the nominal value of SRE.
Conclusion
SRE is more than a passing fad but not a silver bullet – it is a culture built on understanding and accepting risk within certain parameters that can be measured.
Organizations wishing to adopt an SRE culture can do so without disrupting their existing DevOps, Agile, ITIL, or other existing strategies – in fact, SRE intentionally fits in. Although all of the ins and outs of SRE are outside of the scope of this article, there are many great resources available on the Internet from SRE practitioners – and even the team who invented it – to help take the next step:
- https://sre.google/
- https://cloud.google.com/blog/products/devops-sre/how-sre-teams-are-organized-and-how-to-get-started
- https://cloud.google.com/blog/products/devops-sre/5-google-sre-resources-to-get-started
- https://www.blameless.com/the-essential-guide-to-sre
