I think everybody who me knows the story, but for those of you who don't, I would like to offer that the worst experience in your life can often be the best over time.
To cut right to it, I'm one of three people (at least that I know) who’ve brought Microsoft down globally. We were making a group policy change, two of us at the same time. There was a bug in the software and voila—nobody cares exactly how it happened, it just happened. We were completely down for 15 minutes. Fortunately, one of my engineering partners Jason Hughes and I had put in some reliability tooling right before this particular change we made and we were alerted promptly. But it was still a global outage, and there’s really no “good” or “short” global outage, especially at Microsoft’s scale. I'm not gonna blow any sunshine up anybody here. We were able to respond quickly and bring Microsoft back up quickly, but it was a bad day.
I sat through the postmortem, and that was memorable to say the least. I've never had that many people in a room who are all concerned about what I was about to say. I think it's probably the most formative event in my life when it comes to being able to manage through chaos and adversity, and now being able to really be there for engineers when things go wrong—because they will. We all come together in those moments and make the situation better, not worse. While it definitely was the only moment in my career I ever thought “shoot, should I just walk out the door right now?” but it was definitely something that I'm glad happened, and it’s certainly made me the man I am today. I'm very thankful for the moment.
In the immediate sense, it was curiosity. I kept thinking “How the F did that happen?” I went home and tried to recreate it. It didn't work. But I was like, the only thing that was anomalous was this one thing, so I was able to go back to the lab and keep trying to figure it out. Thank you to my managers for allowing me to have that crack at it, because I think I would have gone insane without it, but I went back. I found the thing that I thought was anomalous. I kept going back and I tested in the lab and we nailed it immediately. So initially, it was just this burning curiosity, but long term—man, problem solving is fun. That’s sort of what life is about, it’s the reason why I was an engineer in the first place.
Yeah, it’s a lot of learning how to work through these problems in a productive way so that you can motivate your team instead of just crush them. In that moment I talked about, I had an engineering manager at Microsoft who gave me some feedback, and he chose such a harsh moment and delivery in doing so. I viscerally remember it.
But that lesson came to fruition when I was a CTO at a Crypto startup. One of my engineers brought down prod by making a change to dev—and everybody knows environmental separation, we all say it's this golden way—but we never do it right the first time. Anyway, this happened to this engineer and I just looked at him and I remembered what that felt like and that I could make it better or I could make it worse. So I said, “Hey, you brought down a crypto startup with no customers. We're gonna survive. I'm here. I brought down Microsoft globally. Let's push through. Don't worry. All your friends right now, they're just upset they got to work the weekend, they're not upset with you. We're gonna get through it. You're a great engineer. Let’s play on.”
I was grateful to have had that experience earlier in my career because I might not have approached that in the same way had I not.
It really comes down to, you have the opportunity to make each situation for your team better or worse. When you come into any situation, and it doesn't matter if you're the CTO of the VP or the tech lead, you can come in and you use whatever resources or power you have to make it better, that’s a win for the firm.
Keep making iterative progress. I run on a treadmill and one of my favorite trainers I follow always talks about making small deposits every day with your fitness. It's the same way in your engineering career. Just finding little ways to be a better person and teammate every day. Small deposits every day make big rewards over time, especially when you're in a position where—I always laugh saying “VP” because it never has crossed my mind that I'm a VP, I'm just an engineer in my head—but I do understand the gravity of one's words are much more important at that level.
You can learn so much from being just a good teammate. You don't have to sugar-coat or have false honesty, but literally just being there with advice because you've been there before goes such a long way. Trust me, your career is based on relationships. You don't know it when you first start out, but the way you treat people will make a mark on you as you go forward.
I left another place I really loved to join here, and for me it's really about the mission. The idea here is to provide quality health care at a lower cost out of pocket to patients. That’s a mission I can get behind. Plus the people who work here are the reason why I joined. And, since you asked me if I had anything to plug, I've got positions open for site reliability engineering and other engineering positions on my security team and elsewhere in the org. So if you want to come, have some fun and learn a lot while making a difference in the world, come join me at Garner on the Platform Engineering team.
View open roles at Garner at getgarner.com/careers.
.png)
.png){kind=icon}
.png){kind=icon}
