Creating Chaos to Achieve Reliability
How can creating chaos achieve better reliability? Chaos and reliability might seem mutually exclusive, but through the use of Chaos Engineering, SREs can bring about meaningful changes to system resiliency.
October 2, 2023
6 min read
For many people, the first and only times they interact with Executives is during an incident. It can be an intimidating first introduction! While Execs are first and foremost just people too, they tend to require some specific care when it comes to communication, especially when it involves issues that critically impact your business and customers. In this post, we’ll cover the best practices for communicating effectively with Executives during incidents.
You’re in the incident channel rocking yet another incident. Comms are flowing, resolution is in sight, the team is grinding, and you’re feeling good. Then…
Gulp. Suddenly you’re not so confident. Did you miss something? How did they find out about the incident? Should it have been escalated higher? For many people, the first and only times they interact with Executives is during an incident. It can be an intimidating first introduction! While Execs are first and foremost just people too, they tend to require some specific care when it comes to communication, especially when it involves issues that critically impact your business and customers. In this post, we’ll cover the best practices for communicating effectively with Executives during incidents.
Don’t rely on guesswork to know when your Execs want to be informed about incidents. Establishing tripwires—conditions that when met trigger an action—is a great way to demystify when Execs want to be informed. Some examples of tripwires for informing Exec teams could include:
An executive summary is a document that keeps the most pertinent information relating to the incident readily available to your Execs. It should provide a clear, high-level view of:
When escalating incidents to the Exec level, having this report readily available saves time by providing key information to bring your Execs up to speed. From there, they can choose whether to increase their involvement in decision making or to remain informed without increasing involvement. Keep in mind that Execs might also have specific preferences around how they want to receive these updates—whether it's through their assistant, via email, text, Slack, etc. It's good to check early on about how they'd prefer to receive updates.
💡 Save yourself some time and grab our free Executive Incident Summary Template!
Time is precious, especially for Execs who are being pulled in countless different directions. Brevity is key when communicating. Your Exec summary shouldn’t exceed one page in length. Remove information as you update so it stays brief and relevant. Be prepared to answer questions if they have them, but don’t inundate them with information overload upfront. Beware of overly technical language and explanations as well (unless you're working with someone like a Chief Technology Officer), don't assume all Executives will have knowledge of the inner workings of your infrastructure.
It can be easy to fall into patterns like doing things a certain way simply because it’s how it’s “always been done”. As organizations mature, processes can become misconstrued and hardcoded into perceived “rules”. When working with Execs, avoid blindly defaulting to processes and instead think critically about the best path forward. What would you do with no limitations? Execs have the power to allocate resources and set new precedents, which makes having them involved in incidents a huge advantage.
A helpful way to look at levels of abstraction is through this simple framework:
The more granular the decision, the less Exec involvement is needed. Execs rarely need to know about the “how”. When you have an Exec’s time, focus on gaining an understanding of the “Why” and validating that understanding by proposing and securing approval on the “What”.
While it can be intimidating working directly with Executives, especially for the first time, it’s also a great opportunity. Stay present and observe how they think through the situation. You can gain valuable insight into how they approach decision making, and build trust in your incident response program.
Rootly makes it easy to communicate with all of your incident stakeholders, including your Executive team, by consolidating incident information from all your tooling in one place. You can automate escalations and communications, assign roles and tasks, and more, without leaving Slack. Book a free demo today to learn more!