rilex.me

Ricky's SRE Markdown Book - Part 2

My personal stories during my professional work with some references included

Story 6 min read

These series are collections of my personal learning process, decision making or point-of-view, specifically from an SRE standing point, throughout my time working in tech. All contents does not contain any confidential information nor any names mentioned. The timeline depicted in the stories are also purposedly made vague to retain anonymousity. Please do not publish or re-share my posts without my permission

SRE Culture Adoption↗

Building culture is hard because culture lives in people unconscious assumptions.1

Why is it important for engineering team to adopt SRE culture ?↗

The name Site Reliability Engineering speaks for itself, it ensures business resiliency as business grows by delivering highly reliable systems. Good SREs both must fully understand why it has to be reliable and how to quickly recover when an incident happens. Once you understand the why, only then you could relate and emphathize with your system’s wellbeing. As I wrote on the previous part, we can figure out the business impact of a service outage using the BIA framework together with Risk Assessment Matrix. The how however comes with different flavors depending on the culture of the company.

On top of the four golden signals of observability2, other SRE team may view MTTR & Error budgets are more dire to be maintained. SREs eventually always will have metrics, SLOs, dashboards and alerting set-up as their main priority when it’s initially bootstraped. In the AI age, tools are mostly already equipped with Analysis Agents to detect anomaly patterns better and deeper.

Since there is no fixated definition on what and how SRE performs their role, the boundaries shifts and continuously evolve. At one time they are responsible for migrating their legacy systems, later they are also given resposibility to reduce costs then in the next month they have to complete series of tasks, preparing evidences for compliance audit. These are the kind of tasks makes the role challenging. Though, those task/role shiftings are more likely to happen in startup companies due to their unpredictable nature, whereas established companies often are more rigid towards changes, at least based on my experience.

How SRE role is different↗

Since they have the vision on why the system’s reliability is paramount and how to quickly recover from incidents, that translates to the ability to view both perspective from business and engineering standing point. Unlike any other role, this unique bird’s-eye view can drive both improvement and mitigation strategies using measurable metrics.3 In a way I see it as a discpiline in maintaining trust - your customer’s trust. Coming from what seems to be a very unlikely-related department, SREs ultimately contributes to the goal of every company, which is to aim toward customer happiness as the end goal4, especially when your company’s reputation relies on that system.

Acknowledging customer point of view↗

Company Culture Is Everyone’s Responsibility.5

In order to build SRE culture, commitment and consistency are key to decide the success of a culture adoption. In many traditional cases where company have none or minimum exposure to SRE culture, resistance will emerge eventually. In A Corp, both me and my COO just started defining our key transactions. We were a small sized startup company, maybe around ~50 people in total and had high engagements between each other. We purposedly involved the engineering and product team to tell us what/which part of our system are important. Since they are already allocated into tribes based on main features, naturally they would name their own processes. Obviously the product owners would lead the race since they know their customers best and are backed by data to prove their claim.

Maybe by now you would start saying “this is not a common SRE practice” or “A weird way on devising SLI and SLO” some might say, but I understood what my COO was trying to do; he’s on a campaign on building a culture by giving an example. He’s involving everyone to care about it, because culture can only be created when everyone participates. As I mentioned previously, SREs and every other engineers also contribute (either directly or indirectly) to customer’s satisfaction. With customer satisfaction being the ultimate goal, then there are several points that has to be addressed :

  1. Which processes that are paramount to their satisfaction ?
  2. What is the range of tolerance on how long those processes to be ?
  3. What other non functional benefit do they care from our service ?

Starting from those high-level questions, we may break them down into smaller and more concise objectives whose indicators are translated into measurable metrics that we are capable of collecting. It is ideal that for each team in the engineering division to have certain degree of ownership and understanding on those part of services to make sure the targets to be achieved are not only considered to be numbers, but as a form of promise or commitment which come from real case scenarios.

A quick detour : The very unfortunate outcome in startups↗

Most of the task engineers had to juggle around with, have valid background reasons and often come suddenly. In startup companies it is very common for SRE/DevOps/Platform Engineers to have these sudden & high-priority tasks due to the unpredictable business environment. They have to juggle between regular operations (often come in ticketing manner), improvement projects and these sudden & urgent tasks.

With that being said, it is common to have different set of KPIs or OKRs each year depending on what the company’s target is on that same year, or even none at all. Engineers are constantly under deadlines and in overwhelmed situation. So the question is, then where is the silver lining to all of these hassles, is there any way to make those tasks more “predictable” ? Unfortunately, that has nothing to do with SRE Culture, that is more due to the unpredictable nature of startup business models. However there are practices that I found useful in navigating through the storm :

  • Set up Special Interest Groups (SIG): you are not alone on the ship, there may be many people having the similar expertise or have the interest to explore and contribute more on top of their current role
  • Series of scheduled sharing and learning sessions: surprisingly on top of all tasks, people always have time for sharing and learning session. ask people if they’re willing to contribute to share their expertise by being the speaker of the sharing and learning session
  • No-work-talk dinners: in the end, engineers are also human that have their own stories and interest, share it and know them better to build trust among team members

Building a culture of excellence involves pushing ourselves out of our comfort zones to inspire and enable the transformation going on around us.6

Footnotes↗

  1. https://viva.pressbooks.pub/managementandob/chapter/culture/ ↩

  2. https://sre.google/sre-book/monitoring-distributed-systems/#xref_monitoring_golden-signals ↩

  3. https://www.elastic.co/blog/site-reliability-engineer-role-evolution#sres-get-to-have-a-bird%E2%80%99s-eye-view ↩

  4. https://www2.deloitte.com/us/en/pages/consulting/articles/sre-its-not-about-numbers-its-about-customer-satisfaction-architecting-the-cloud-podcast-devops-cloud-usability-engineering.html ↩

  5. https://hbr.org/2021/02/company-culture-is-everyones-responsibility ↩

  6. https://www.forbes.com/sites/forbesbusinesscouncil/2023/05/19/three-ways-to-invest-in-company-culture/?sh=56d7687cf477 ↩

filed under
self-improvement story tech sre
More like this

Ricky's SRE Markdown Book - Part 1
Story

Tech Stories
Story