These series are collections of my personal learning process, decision making or point-of-view, specifically from an SRE standing point, throughout my time working in tech. All contents does not contain any confidential information nor any names mentioned. The timeline depicted in the stories are also purposedly made vague to retain anonymousity. Please do not publish or re-share my posts without my permission
Site Reliability Engineering↗
100% is probably never the right reliability target↗
I learned this many times during my career, and somewhat shaped my mentality way before knowing anything about Site Reliability Engineering or Risk Management; that we should always make time to identify potential risks and define a mitigation plan, because we never want to be caught in those kind of situation where we have no clear plan on how to react. A situation that is dire, and the operation people are relying on the IT team to help them soothe the chaos since the system they use broke down and they have to handle the task manually by paper and ink. That works if you’re system is an internal tool that can be temporarily replaced by manual efforts, however, that case does not apply when the core product is technology service.
In this modern day, we all have experienced some sort of system outage, let it be your social media or the web application you use in your office. And when it happens, surprisingly, the reaction of each people differ. People somewhat have some sort of tolerance threshold that decides if they should report it or leave it be and try coming back after several minutes. After that certain threshold, issues does escalate as much as time also passes. BCM certified people, often uses Business Impact Analysis (BIA) and analyzes Impact Over Time which may have different impact & treatments towards the incident, hence these kind of threshold should be defined by the company as it defines whether they remain reliable to their customer.
In one of my former company (I’ll call it A Corp for easier referencing), the COO himself initiated the SRE team and I was the initial member of the team. The initial goal was to find out what was important, meaning the Business Core Function (BCF; New Relic uses the term : key transactions) that should be maintained and be monitored closely since it has direct consequences to the business operation, hence the need for Error Budgets
Most of them are the basic signals for monitoring : availability, latency and error rates, however there are also indicators that are challenging such as “OTP codes should be received by the user within 10 seconds” or “User bounce rate should remain low”. Within the first few meetings, the engineering bargained their way into much more simpler objectives with metrics they have access to and finally reached agreement with the product heads. These SLOs has to be committed by the engineering division hence involving each person to know on what actually matters.
Once I got the information, next I have to translate those indicators into a measurable metrics and define the technical details to do it while also writing it into a Service Level Indicator format1
Aside from my previous company experiences in defining SLOs, I also learned from my other previous company (I’ll call it B Corp) which are dependent on multiple SaaS to operate our daily business. We decided that we want to have a way of listing availability/success rates of each SaaS based on batched transactions. As these SaaS have impact on whether the transaction succeeds, one example of the key indicators that’s data driven is “the amount of users with pending transactions should remain low”.
From those short narration, The phrases such as “Error Budgets” and “pending transactions should remain low” which implies there are certain threshold that we should commit to, and it is not 0%, and probably should never expect to be, because increase of availability is linear to the contribution of cost increase. That doesn’t mean we should surrender our fate of being totally unreliable. By defining an Error Budget for each indicator and committing to it by reacting to it’s alerts of Error Budget Acceleration Rate, we can keep in balance the acceptable threshold from the customer perspective while also keeping the target realistic and achievable for the engineering team
It’s necessary to picture how much time is allowed for a system to go offline and how costly it is increasing for a system’s reliability. Below is the availability table that all SRE are well aware of, that indicates the Error Budget for each availability level 2
| Availability level (uptime) | Allowed unavailability window per year | per quarter | per month | per week | per day | per hour |
|---|---|---|---|---|---|---|
| 90% | 36.5 days | 9 days | 3 days | 16.8 hours | 2.4 hours | 6 minutes |
| 95% | 18.25 days | 4.5 days | 1.5 days | 8.4 hours | 1.2 hours | 3 minutes |
| 99% | 3.65 days | 21.6 hours | 7.2 hours | 1.68 hours | 14.4 minutes | 36 seconds |
| 99.5% | 1.83 days | 10.8 hours | 3.6 hours | 50.4 minutes | 7.20 minutes | 18 seconds |
| 99.9% | 8.76 hours | 2.16 hours | 43.2 minutes | 10.1 minutes | 1.44 minutes | 3.6 seconds |
| 99.95% | 4.38 hours | 1.08 hours | 21.6 minutes | 5.04 minutes | 43.2 seconds | 1.8 seconds |
| 99.99% | 52.6 minutes | 12.96 minutes | 4.32 minutes | 60.5 seconds | 8.64 seconds | 0.36 seconds |
| 99.999% | 5.26 minutes | 1.30 minutes | 25.9 seconds | 6.05 seconds | 0.87 seconds | 0.04 seconds |
Each addition of 9s equals huge commitment in effort & cost involved, we do not want any Error Budget breach to be taken lightly, if that is the case, then there should be further assessment whether top level could understand the risk management behind those mentioned key transactions
I’m quoting Marc Alvidrez’s writing in the SRE Book which summarizes my point of the tradeoffs of increasing reliability more than it should’ve been
We strive to make a service reliable enough, but no more reliable than it needs to be. That is, when we set an availability target of 99.99%, we want to exceed it, but not by much: that would waste opportunities to add features to the system, clean up technical debt, or reduce its operational costs. In a sense, we view the availability target as both a minimum and a maximum. The key advantage of this framing is that it unlocks explicit, thoughtful risktaking.3
Enlightened to Compliance & Risk Management↗
In my early working days, in 2014 I was working as a IT Staff at a factory. My main job is to support certain sites in the factory while also developing internal apps, but later I was also given the responsibility of compiling our team’s Standard Operating Procedure (SOP) & Working Instructions (WI), therefore being the contact person during audits. I never really understood the importance of those papers back then, but I still maintain them and partially memorize the words inside. I modified those documents when necessary, but mostly just because I was told to. I was doing that for around two and a half years, before I moved to another company, even though it was not necessarily about IT or data safety (in this case it was the ISO 45001), that was my entrypoint on how I learned about compliance.
Fast forward into early 2020s, seems funny from where I started back then, until today I am one of the people asking around, creating documentations, collecting evidences for ISO compliance requirements and government regulations. I have been further involved in creating thorough planning of Business Impact Analysis (BIA) and Business Continuity Plan (BCP). It is based on the Business Continuity Management (BCM) SOP that has been signed by our CEO which states that there are several points that has to be documented, one of them is service availability. Initially, my plan was to create runbooks in case of incidents in the scope of Infra team, but later I was introduced to the Risk team (actually it was only one person at the time, if you are reading this, you know who you are xD) which gave me insights of how risk management are mostly done.
Since I am aware of SRE key principals, we shared the same energy and ultimately: concerns. I can say that, in a way, we are paranoids. We acknowledge how systems may fail and cause disruptions into business. However, since we came from different background and different departments, we have different ISO standard to follow (I follow ISO 27001 and my colleague follows the ISO 31000 and ISO 22301), yet we share the same “language”. We had the fire ignited. We started discussing about incident management but eventually sharing many other topics. We learned from each other - Shared stories on what approach is being done in the industry. We knew what we had to do, but was overwhelmed by the amount of it. So we started small and agreed to create BIA and BCP documents for our core services.
Me together with my team devised a detailed, thorough document scoping multiple scenarios - starting from catasthropic into lesser impactful scenarios. At that point we are able point-out several cases in our minds that we think are the component candidates of causing such emergency. And since the Risk Management team didn’t have any personnel specialized in IT mitigation strategies, I was appointed to help in devising ISO compliant documents regarding Disaster Recovery. We were asked to abide into certain templates used throughout the company. One of them is the Risk Assessment Matrix paired with indicators such as Recovery Cost, Regulatory Penalty, Revenue Impact, etc. We have to define severity scale based on it’s impact then the detailed breakdown on how to mitigate them. We spent months to finally came into a well-enough document to present to our consultant to later be reviewed.