Chris Evans, Co-founder and Chief Product Officer of Incident.io joins us to talk about all things incident-related. Chris, previously the Director for Platform and Reliability at Monzo is a regular conference speaker and author of a number of articles about incident management and observability.
In this episode, we cover the origins of Incident.io, discuss healthy incident management and Chris advises us on the right metrics to be tracking.
Listen to the Humans+Tech podcast on your favourite podcast provider.
Scroll down to read the full transcript of our chat with Chris
Chris' quick fire answers
Chris recommends The Field Guide to Understanding Human Error by Sidney Dekker.
His top tip for keeping up with the industry is to listen to Kelsey Hightower. You can listen to our episode with Kelsey, here.
Chris is inspired by Frank Slootman, of Snowflake computing, formerly CEO of ServiceNow.
We asked Chris to share a nightmare incident story with us. He responded with this one!
In this episode we cover
- The origins of Incident.io, including Monzo's open source tool, Response [00:01:50]
- Building incident management tools as a way of expanding an incident management rota [00:02:38]
- Why existing incident management tools haven't managed to solve the problem of painless incident management [00:08:07]
- Chris' dream of building a tech company from the ground up [00:09:56]
- What it was like to move from Monzo to Incident.io [00:12:47]
- What does good incident management look like? [00:18:20]
- At what point should a company be setting up on call rotas? [00:20:52]
- Metrics and incidents - how to do it well [00:24:06]
- How to make incident management a better dev experience [00:27:20]
- On why we should all be speaking at conferences [00:39:48]
Find out more, and follow Chris
You can follow Chris on Twitter @evnsio and at https://incident.io/
Full transcript
Amy Phillips:0:01
Welcome to the Humans+Tech podcast. I'm Amy Phillips, and this is Aaron Randall.
Aaron Randall:0:06
Hi.
Amy Phillips:0:07
Today, we're very excited to be talking to the one and only Chris Evans. So I first met Chris when he was working as a Technical Product Manager, and leading the Platform team at MOO. He then went on to be the Director for Platform Reliability at Monzo, before becoming co-founder and Chief Product Officer at Incident.io. Chris is also a regular conference speaker, and author of a number of articles about Incident Management and observability. So Chris, welcome to the show.
Chris Evans:0:33
Thank you for having me, I need to pick you up on one thing straight away, which is you said the one and only Chris Evans, and there are literally two very famous Chris Evans's in the world. So you failed straightaway, Amy.
Amy Phillips:0:46
Clearly you are the more better known
Chris Evans:0:49
Of course, of course.
Amy Phillips:0:52
So great to have you here. Now, one of the things that as you know, we like to do with all of our Humans+Tech guests is draw a doodle of them. So we're going to show you yours, and we'd love to get your thoughts on on your doodle.
Chris Evans:1:05
I'm actually really nervous. I'll take it. I like it.
Amy Phillips:1:13
You sound surprised?
Chris Evans:1:14
Well, can I can I be honest about the thing that I think is like was stressing me out, which is that I sort of don't think I have any particularly distinguishing features. Like, you know, if you wear like certain shape glasses, or you've got something that sort of like, yeah, that's the person I think I'm actually quite bland looking. So I'll take it and I like I really appreciate the little flame. Thank you very much.
Aaron Randall:1:38
Yeah, we do struggle with this. Everyone looks like a sausage. So we've struggled to make it's not you it's everyone. My drawings are very bad.
Chris Evans:1:49
Thank you very much.
Aaron Randall:1:50
Hey, you're welcome. So I want to dive straight in and talk a bit more about the idea for Incident.io. So I know this idea came from an internal project you built whilst working at Monzo. And that the project was called Response, an open source tool you built, how did Incident.io come about? And for those who haven't heard about it, what does Incident.io do?
Chris Evans:2:09
Yeah, good question, I guess, I guess, actually, I think I should probably like be very honest and credit, the very, very early inception of like, this kind of tooling to when Amy and I were at MOO, I think it was Ron, who built the very, very first version here, which was like, this super simple tool that used to create a channel for us when we had an incident. And it was like, suddenly, like we had a space to collaborate in and it was like a slash command and was this magical thing. But yes, I guess like fast forwarding to a little bit to Monzo. I, I picked up along with like running platform, the responsibility over the on call team. And as part of that I was sort of trying to trying to make on call an appealing thing for people to join. And part of the reason they weren't joining was because incidents were like quite painful, or quite chaotic, it was a bank, it was very stressful, you know, you could have like MasterCard, not working and people couldn't make card payments. So I wrote the very first version of the tooling there, to basically just try and try and get some folks to join the on call rotation and help out a little bit. And it went sort of from strength to strength within Monzo. So what started as something that sort of created a channel and then pinged out a notification to say, hey, an incident's happening, then sort of got more complex, and it started doing some more interesting things for us, like, you know, pinging data privacy folks when we were mentioning the word 'data breach', and sort of slowly sort of adding these various different like, automations on top of, on top of the incident response. So, you know, nothing like crazy, like go and turn the server off or turn it on again, but just like processey type things. And that was sort of then adopted across the whole of Monzo. And so Monzo went from a place where incidents sort of were things that happened in technology, and they happened, you know, once a once a month, maybe or once a week, you know, at worst. By the time I left Monzo, we were having something in the region of like 10 to 20 incidents a day. And that sounds incredibly scary. And I should sort of like point out that Monzo is not an unsafe place to put your money or doing anything wrong. What they've done is like lowered the bar for incidents. So they've made it that incidents were basically anything that was sort of, you know, not in the normal course of business. So, you know, it could be like a customer operations person posted something in a channel that they shouldn't have done that sort of way to delete a message and clean it up or, you know, we rolled out a thing and had to roll it back. It was basically anything that needed to be like picked up that was reactive with some degree of urgency. And so this was sort of like the aha moment for us. And so Pete, Stephen and myself as the founding team of Incident.io, started working on like an evenings and weekends project to turn what I'd written internally at Monzo into a product that we thought could be, you know, saleable elsewhere and, you know, it was like the pandemic was going on and it will was like, cool, this is something fun to do. And then it sort of snowballed from there. And you know, what was then an evenings and weekends project sort of became more and more, more and more of a sort of time consuming thing. We started selling to customers. And so there was sort of like regular late night or early morning calls to sort of sell this product, start making some traction. And then, yeah, the rest is sort of history we've been growing ever since. And yeah, product's been getting better and better. It's really exciting.
Aaron Randall:5:26
Amazing. And what was that? what point did it dawn on you that this project was actually a company building opportunity? And also a follow up question, I'd love to hear more about how that snowball sort of began to grow. I guess, from the evening and weekend project, to something customers are paying for.
Chris Evans:5:40
Yeah, maybe it's a bit of an unfair reflection to say that it was an aha moment, because I don't think I don't think it wasn't, it wasn't sort of like, you know, we were in the lift one day, and suddenly, it was like, you know what, this is a thing. But I think it was more like, I think the three of us have all kind of, you know, we've all worked in technology, our whole careers. And for me, certainly I know, it's very similar for Pete and Stephen, like I have dreamt of running a technology company, or, you know, building something from the ground up. Like, it's sort of like the dream situation for me to be able to, not just not just about like working for myself, but to sort of just take something from, from nothing to something, you know, we had pretty mild aspirations. In those early days, we were sort of thinking of bootstrapping a thing. And like just being like, well, this could be a good lifestyle business never have to work again. But yeah, so it was more of a sort of a slow, like, hey, this is actually a thing. And it's sort of, you know, came to that. And then I think when it came to like, the snowballing thing, it was basically, we launched the website, when the product was in like, you know, incredible infancy. So it did sort of basic things that Response did, that was Monzo's open source tool. It was a complete rewrite different languages, but sort of did a similar thing of creating some channels, letting some folks know, and a very, very basic web interface. But yeah, we put the website up, and then had like, immediately a bunch of sort of, like, posted on social media, and, like, immediately had a bunch of inbound. And it was like that, that was kind of an aha moment, which is like, okay, cool, we've put a thing out, sort of, like focused on the branding side of things, we bought a domain that was kind of cool. And there was just a lot of like traction from from our network. Basically, if people were like, cool, we have a thing similar our company that does this, or, you know, we have a manual process written down on paper, and this looks great. Like it would help solve that. And so then it became a case of like, starting to figure out how to sell, you know, SaaS software, you know, B2B and like, figure all that side of things out, and it kind of just picked up from there. And yeah, got first customers sort of paying, you know, a few 100 pounds here and there. And it was like, yeah, this is great. This is sort of something's gonna work.
Amy Phillips:7:42
Amazing. What were the sort of things that people, I suppose, like, what what were the problems that you were talking about on social media that people were jumping in and sort of saying, oh, we need that like, because I'm always surprised, like, you know, incident management has been around for a long time. But it's a long way away from being a solved problem. So what were the sort of things you saw that people were saying, like, these are the problems we're having? Come help us.
Chris Evans:8:07
Yeah, that's, I think that's really interesting area, because Incident Management has been around for a long time, most like my whole career, things like systems like PagerDuty have existed, Opsgenie, you know, various different software has existed to sort of help respond to things going wrong. But I think what, what our customers were seeing that was very different. And where we were sort of going was that these software platforms were places you had to go to do things with incident response. So you know, I got paged at 2am, for example, and I wanted to go get someone else I'd have to go and jump back into PagerDuty to go and like, make someone else's phone make a noise. But where I was coordinating, my response was not on PagerDuty, I was in Slack, like most, you know, technology companies. And so you'd be sort of like, the hub of everything going on incident is all founded on good communication and the communications happening in Slack. And you then have to go to PagerDuty to do a thing or Jira to log that, you know, an incident was happening, and then you'd have a doc somewhere else off to the side, which was, you know, where you were tracking action items, or who was doing what. And so that was the thing that really, really resonated and sort of like, I think the message that we sort of landed on was that, you know, we'll meet you where you are. And we, you know, incidents are already really, really stressful. You don't want to be having like more cognitive load on you, you want to be basically doing as effortless, you know, effortless experience doing as little as possible. And so, yeah, it was sort of like, hey, you're already on Slack, we will come in, we will augment that process for you, take a bunch of things off of your plate. If you need to use PagerDuty, you can do that through Slack as well. And you can get hold of the right people and you know, keep context on what's going on. And the only time you should leave that channel is if you're actually going to wherever it is that something's going wrong. So to fix a system or you know, jumping to a terminal sort of thing.
Aaron Randall:9:56
I want to ask you a bit about that point you made about you've always dreamt about building a tech company from the ground up, tell us a bit more about where the kind of the motivation for that came from and how long that has been an aspiration of yours.
Chris Evans:10:09
Yeah, I think in like it probably like university was where I first started thinking about it. And that was mainly motivated by a desire not to have to go and interview places. If I'm very honest, it was just like, hey, I started thing, and no one can interview me. It's my my sort of show.
Amy Phillips:10:27
And here you are now, it's not worked out at all.
Chris Evans:10:29
No. Dammit. So yeah, I think, like, fast forward a little bit like, for me, it was, it was, I think I'm just like, very interested. I mean, I love building clearly. Right. So that's like the first thing. And so that has sort of been something I've been able to do most of my career one way or another with like technology. And I think doing that at a company is like the ultimate form of that. And I think like when I look at my career, when I sort of moved from place to place, it has always been, because I felt like I wasn't, I'd sort of like plateaued on like a learning sort of scale, if you like so had an incredible time at MOO working with Amy, I'd say it's like one of my career highlights, you know, and what I did, there was what we did, there was like, do a massive, massive move of all of MOO's infrastructure from an on premise data center into AWS. And then at the end of that journey, I sort of felt like it plateaued a little bit. And we were back into sort of steady state as a company of like, well, now we're here, how do we optimize and make things better, and, you know, interesting challenges, but I just wanted wanted to be pushed a bit harder. And so for me, the next step was Monzo, which was then the, you know, do do a similar thing to MOO, but on a bigger scale, and in a company that's growing at some astronomical rate. And then I think if you sort of extrapolate all of these kind of, sort of situation, you then get to like building a company where it is just so hard. And the challenge, like I didn't go into it sort of thinking it would be easy. But I knew going into it that I would learn a lot about everything. And so it's really interesting now, like when, when we interview candidates, and we do introductions, and it's like, you know, I'll be like, I'm Chris, I'm working at Incident.io, like, CPO is my title here. But like, really what I do is, and then I just list basically every function of the company. Yes, I'm like doing GTM. I'm doing sales, I'm doing marketing, I'm doing like, all of these things. And it's both like, equal parts terrifying. But also, like, just incredible as is as a learning experience. Like I genuinely, genuinely don't think that there is a sort of more exciting and more sort of fully, like multidisciplinary kind of challenge you can pick up and try to build a company.
Aaron Randall:12:47
I want to ask a lot of questions about that. And we've actually got some a theme on that a little bit later. But before we move on to that, you know, this, this journey has transitioned from Monzo to starting Incident.io and doing your own thing. And obviously, with Incident.io, you went down a VC route, making that transition out of Monzo. What did it mean to you to have the Monzo founders, you know, Tom and Jonas, invest in your new business?
Chris Evans:13:10
Oh, a huge amount, absolutely huge amount. Like I think I think those folks supporting us was just was just massive, and they were so good through the whole process. So like, we started, as I said, we started building this company while we were at Monzo. And that was not something we did behind closed, closed doors, like Monzo has this, you know, very much lived value of like, defaulting to transparency. And so like, early days, we'd spoken to Jonas and been like, Hey, I think by letter of like, our contracts, we're not supposed to, you know, if we start doing work on something during like, Monzo while we're working at Monzo, there's like a risk that you can, you can take that off us kind of thing or prevent us from from doing that. And Jonas was like, Absolutely not like it might be in there. But like, practically speaking, you have my word this is this is fine for you to do was really encouraging. And so, yeah, they were just great through that whole process. They were great when it came to like raising our round when we had to sort of get things from Monzo, you know, to say that this is definitely our IP and stuff like that. And then to have the Tom and Jonas basically backing you to like heavy hitters, when it comes to, you know, this kind of thing. They've they've built an incredible business out of Monzo. And it's just sort of yeah, very buoying to sort of be have those folks sort of feeling like they're willing to put their money where their mouth is, and sort of actually, you know, give you give you that confidence.
Amy Phillips:14:36
It's awesome. It's amazing. So I know it's still you're still very early days, so it feels a little unfair to ask you this but still, we can ask you again in a few years but like what have been your challenges so far? Like what are the highs and lows in Incident.io's history so far?
Chris Evans:14:54
Oh, wow. That's I mean, that's that's like it's like every day there are highs and lows and trying to pick which of these sort of things are
Amy Phillips:15:05
Limit to a week if you want.
Chris Evans:15:07
I mean, like the highs, I would say, I think something that's like really, really been great for me is like selling the product now. And this is sort of like, I guess the context of where this comes from is. So back in the back in the early days, when we used to jump on sales calls and to jump on and like, someone would be like, I want to, I want to see your product. And you'd be like, cool, well, let's do, let's do a demo. And we'll do this, I remember being literally terrified of these like, to the point where like, for like 15 minutes before jumping on the call with them, I'd be like pacing up and down outside my office and sort of like cold sweats and that kind of thing. And then you jump on to the demo. And I'd be like stressing, because I'm trying to like drive, like showing them the product was talking but also listening to them. And then they'd have objections. And I'd be like, I don't know what I'm gonna say here or what to do. And so that that was like, genuinely a massive, massive struggle. And I think where it's now turned around, it's like, literally one of my favorite things to do is to jump on a call with a customer. And I can, you know, give the product pitch basically, in my sleep, I can drive the product and like I'm so in like, basically, at the core of it all, I love the domain, I'm passionate about the domain, I've felt the problems. And so I sort of feel like I got the cheat codes to selling to folks. In so much as like, I just sort of, you know, I can authentically speak about it, basically. And so that's been, I would say, like a personal high of the company, like more general highs for the company, some of the folks that we've managed to get as customers. So we have like some incredible customers and some, you know, well recognized brands and things like that, and increasingly getting bigger companies coming to us who, you know, it sort of astonishes me that they are interested. And then it sort of doubly astonishes me that they see the product, they use it and then we get incredibly, incredibly good feedback from them, and they are buying it. So that's been amazing. Lows, what's what's a good low? I think I think lows for us are maybe some of the people that didn't work out. And I think this is like part of the harsh reality of startups is that they are, they are very different to normal companies. And so and you know, you hire someone, and you're, you know, at the time, you're like, well, our growth trajectory is like, you know, sort of this sort of, not too steep, steeper line, I realize I'm doing this on video, and people can't see this. And then and then you know, things change in your traction picks up more, which is a great problem to have. And then you realize that suddenly you're on a much, much steeper slope. And so some people who you hire, maybe don't, don't end up working out. And that's, that's really hard, because these are people you're in such a small team, and it feels, you know, very close, and you go for dinner together, and you're hustling on deals together. And yeah, so some people have not have not made it. And I think that's that's a really tough thing for us, but a sort of necessary part of keeping a high performance culture. And the great news is that folks who haven't haven't haven't sort of made it all the way through, they have left really amicably and it's sort of, you know, a mutual thing, which is, which is really, sort of, you know, the sort of small silver lining, I guess on a bad situation.
Amy Phillips:18:20
Yeah, for sure. So you mentioned about being passionate about the domain. And I, I've got so many questions for you. Chris and I have spent quite a lot of time, a lot of nights I'd say we've spent on incident calls. But tell us a bit about Incident Management, like what what does good Incident Management actually look like?
Chris Evans:18:46
I think is highly context dependent, which is like the ultimate like way to cop out and hedge before you answer a question I appreciate. But I think it's, it's dependent on like, so many different factors. But I think at the core of it, like, if you look at what an incident is an incident is something bad has gone wrong, and there is some degree of urgency to get something fixed. And I think what makes incidents different to normal work is that urgency puts pressure on people. And I think that, you know, when incidents are certainly treated as the more severe things, what also happens is, you end up sort of bringing together a bunch of people who don't normally work together. So you know, in its simplest form, if you're a customer, if you're a company that does anything with customers, then you're almost certainly going to be putting together customer support folks and people who are dealing with whatever it is that's, you know, that's broken, and those two teams don't normally meet under normal circumstances. And so you've got like, this is the sort of like the core of it, which is, you know, something's gone wrong. Lots of problems. People trying to form a team very, very quickly. And so I think that's quite a unique sort of space to be in essentially. And so I think good Incident Management looks like having like systems in place that allow you to form that team really quickly to, you know, set up some sort of like default rules of engagement and rails for how you'll coordinate that incident. And then essentially taking, you know, a bunch of things off your plate that you know, that have to get done every time and sort of trying to focus on like reducing cognitive burden, so that you can focus on whatever it is that's going wrong at the time, basically, but like, I think at the core of all of this is just like, very, very streamlined communication. That's really what what these things come down to, because it's so hard to prepare anything for an incident, like you have people who have like playbooks for this and that, but they, they sort of, are never going to survive contact with reality fully. And so it's all about, you know, you know, being prepared to be unprepared. Basically.
Aaron Randall:20:52
You run an early earlier stage business, you know, a growing startup but early stage, and there are lots of those around as well. when's the right time for a business, like a tech company or a tech team to think about creating kind of this official on call response policy and the firefighter rota, and all these kind of things that go around incidents?
Chris Evans:21:15
Yeah, I think on call is an interesting one as a sort of like side part of incident management. So I think on call, it's, it's one of those things, I think, when your business is dependent on you being available, that is, you know, and dependent, meaning you cannot tolerate it being offline, that is your trigger for putting in an on call practice. So, you know, if you're doing a social media like site for cats, for example, probably in the early days, when you've got 10 users, it doesn't matter if it breaks at 2am, and you wake up at 8am and then and then fix it. I think as soon as customers are reliant upon something being sort of working, that's your trigger for having on call, that's your, you know, someone will always be available to support this thing. Sort of incident management, I said, I said, I'd sort of put it off slightly to the side. And so much as there's a lot you can do in the early days without overcomplicating things. And so, for example, I would not say that our software is a great fit if you are one team, and you are under 10 people like there is very little that can't be achieved with a bit of string and sellotape and, you know, a Slack channel where you communicate, I think as soon as you get to the point where, you know, either incidents are happening frequently, you've got multiple different teams, you don't spend much time together. That's the sort of trigger for like, cool, well, that it's helpful to have a process there and there's, there's obviously like a huge number of benefits off to the side, we have of starting to have a more official incident management sort of procedure or way of dealing with things. In so much as once you, you know, if you're having things going wrong, or you're being distracted by reactive work often enough, that is, you know, it's good to be able to track that work and incidents or if you know, if your policy is that we will declare incidents for every sort of time that that happens, suddenly you get this visibility into your organization. And so this is something like actually, we live and breathe Incident.io. So we touch wood do not have incidents of like high severity, we have, like I could probably count on, you know, a few fingers, how many times we've actually been offline or unavailable as an application. And when that's happened, we've been fortunate enough that we can sort of rollback or restart something and everything comes back. But what we do have is many, many times when little things go wrong, or we have like little errors in the app, and we treat every single one of those as incidents. So if a customer is like, hey, I tried to do this thing, and it didn't work, they we will treat that as a as an incident. We now have incredible insight as to like how much time are we spending, not doing proactive kind of strategic work? And who's who's the person jumping into all of those incidents? And, you know, what's the impact like on those incidents, are they bad things? Are they small things? And suddenly you get this sort of like insight that you didn't previously have into your organization.
Amy Phillips:24:06
What's the like for companies that aren't currently doing that, like, what's the correct way to start looking for those patterns? So that, you know, it doesn't feel like blamey that like this person did ten incidents, and this other person just did one or, you know, this person broke this server three times last week. How do you actually gather that insight from incidents in a healthy way?
Chris Evans:24:28
Yeah, I think like this, this is like a topic that I am incredibly passionate about is the sort of KPIing, and over metricing incidents. So I think, like, my advice would be to essentially steer clear of any top line number around incidents, there's a really common one that gets thrown around a lot which we have sort of begrudgingly added to our app, primarily because like customers were like, I need this thing. And that's MTTR which is mean time to recovery. And the sort of the theory behind this is is that you go, oh, well, for every incident, I will measure the time from when we sort of deemed it to be the start to deemed it to be over. And then we'll just average that across everything. And we want that to, you know, trend down, or at least not trend up. And like, there's various sort of papers been written about the actual, like, how this is just not an effective thing. And it's very misleading. And at the core of it, it's just like, it's kind of ridiculous. When you think about it, it's like, people are people are not going to take any longer than than they sort of need to in incidents. And no one's choosing when an incident happens. And incidents, almost certainly by their very nature are sort of things that no one no one can predict. And so it sort of just is basically meaningless. And so that, I'd say the same is true of anyone who's trying to sort of go well, Amy had 10 incidents that she reported, I'd be like, That's great. You are, you are all over it. And like I've written blog posts, for incidents.io, which are like why more incidents is a great thing for an organization and you should encourage that. And so yeah, I guess my like overarching advice around like numbers and using incidents for insights is, if you really must use a number, because it's the kind of the way that you can sort of your your entry point, I guess, into looking at something, by all means, but you need to go that level deeper, which is like even if you are addicted to MTTR, when that trends upwards, the first step is not like throw your toys out the pram and table flip, it's like cool this as a starting point for an investigation. And it might be that that investigation leads you down to cool, everything's absolutely fine. We've actually just, you know, changed something with our infrastructure. And that's like, you know, you just got to mean like, it's just like, basically meaningless to treat numbers around this stuff.
Aaron Randall:26:48
One of the things that I've experienced and observed, I guess, firsthand with the tech teams I've worked in is around incidents, is, particularly for newer members of the team that are less experienced with the stack and with the technology you're working with, how stressful responding to incidents is, and how stressful it is joining the firefighter rota for the first time or responding to incidents and being the person on call for the first time. How have you thought about that when you're building incident.io and like, and try to support making it a better dev experience in that regard?
Chris Evans:27:20
Yeah, and there's a few different angles here. So I think the first one is, is that by dealing with incidents like transparently out in the open and lowering the severity, like folks who are looking to ramp up, have an incredible like, back catalogue of, you know, this is how these people deal with incidents. And so the things that we encourage and that I sort of picked up from someone who I used to, I used to work, called Suhail [Patel] at Monzo. So he used to run, he used to do incidents, and he sort of go off on an incident by himself. And it's sort of it was fine, because it was like a low severity thing. And it was just, ended up using these, like incident channels, like debug trails. And so he would just be like talking out loud in a channel. So he's like, Oh, I've seen this thing that's really curious, I'm kind of a bit surprised by that. Here's like, the output of my terminal or this window, I'm gonna go look at this now. And he did this whole thing. And so I, I do that a lot. So I'm on call at Incident.io, when things go wrong, I will create an incident channel and I will just leave my debug trail. And that has like, a number of different benefits, like the first of which, like, if you need to call on anyone else to come and join, you've got this like cool you can catch up in the channel, all the context is there. But as I said, for an onboarding tool, it's incredible for folks to be able to, like browse those things, and you can walk through old incidents, and, you know, go go to debriefs, that's like a fantastic way to learn. But I think then another angle, which is really, really useful, which is when when folks are ready to sort of like, take that first step into being the person who was paged, or first responding to incident is to just do it in [work] hours and do it for low severity things. And you can basically go cool well, either you take it yourself, or you can reverse shadow. So we pull in someone else. And so we have ways of doing that. With incident.io, you can configure the product to basically run these lightweight automations, which would be like cool if an incident is declared, and it's minor, automatically pull in Lawrence, and he will be there to sort of help shadow you through this process. And so that, I think is another really, really good way to level folks up. And then the sort of, I guess, future direction for us is like, we would love to build something that is like a better version of PagerDuty. PagerDuty makes it pretty difficult for folks to shadow people. Like I have spent countless hours at previous companies trying to get like schedules to work and escalation policies to like have two people all at a given time.
Aaron Randall:29:39
I've battled all this stuff. Yeah.
Chris Evans:29:41
It's just so frustrating. It feels like such a basic thing that you want to do as a sort of engineering leader is set this sort of like healthy onboarding, rotation up and you have to sort of hack it and so in the future we will be we will be very much focused on that. I think the general point here is like, we are, the three of us who founded this company, we've all been sort of engineering leaders in one shape or another and have seen these sort of pain points time and time again, and sort of building tools for humans is very much like front and center of what we're trying to achieve.
Aaron Randall:30:11
I love that. I love that, that idea as well of, as you described, like going through this like process of like, posting your terminal screenshots and like talking to yourself, it sounds like an absolute treasure trove of information for other people like to come to come in and see the trail of information you've left for them that something
Chris Evans:30:29
is fascinating. It can be it can be really enlightening, and it's like, genuinely such a good way to learn. Because you'll see people who post things you would like, oh, my gosh, I didn't know you could get that data from that system like that. And it's like, that's, that's honestly, that's one of the biggest, like superpowers I think of incidents is like this concept of tacit knowledge, which is like that knowledge that people just know, and it's sort of tribal knowledge among organizations. And it's really dangerous because people leave and they take that with them and people can't make good decisions if they don't have all of the information. So yeah, the more that you can bring out to the fore the better and, and in fact, this reminds me like my, my, literally, my favorite favorite feature of our product is this really, really like small thing that we built in like an evening, because we still had a little bit of capacity left at the end of a week, which is, at the end of your incident, we will prompt with a little button which says, "How did this incident go? Leave some feedback" and we see customers use this again and again and again, you'll have folks who will be like, Oh, I literally didn't know how Amy found that dashboard, I didn't know that existed. But it was like super useful for finding the way here and it sort of snowballs. And people will just leave all of this feedback. And it's just like you can just see like, like this torrent of tacit knowledge like pouring down. And it's just yeah, a fascinating thing to be able to see basically.
Amy Phillips:31:44
Incredible. So we've, we've talked a little bit around this stuff, but super keen to hear, Chris kind of how you've ended up in this place. Right? So you've gone through lots of changes in your career, starting out as a software engineer, you've been technical product manager, technical director, now co-founder, Chief Product Officer. So what are kind of some of the biggest learnings that you've you've got from this journey and sort of transitioning through these roles?
Chris Evans:32:12
Wow, man, that's that's a big question. What have I, what have I learned?
Amy Phillips:32:19
What about some of these big transitions, right? So like, I'm particularly interested, I suppose, like, how did you move from software engineering into product management? Because that feels like I think it's a popular route but I don't think it's a common route. Like, you know, I think people like the look of it but I don't think that many people actually make that transition. Like, how did you go about doing that? And what were kind of some of the surprises when you did make that move?
Chris Evans:32:44
Yeah, that that is a good question. And one I have a good answer for, which was that, so I spent like seven years as a software engineer at a small company that was developing these like infrared sensors. And I was originally writing like embedded code for them and then later writing like applications the web apps that supported them and won't go into like loads of the details there. But basically, the end of my time there, I ended up ended up like flying like various places around the world to go and like sell our product. And we were selling products to supermarkets, primarily, it was like, counting people in queues and through doors and stuff. And I think for me, I realized that at that point, like, my impact could be greater not as an individual contributor, but as someone who could kind of go and be a bridge between non technical folks and the engineers who were writing software. And, yeah, I think I just sort of, I think carved out a little bit of a niche there in some organizations, which was that, essentially, that translation from, you know, people doing the work on the ground through to, you know, whether it's managers or customers or anyone outside? That's, I think it's quite rare to find that done in a good way. And I think there's a lot a lot of sort of, like benefit of having that that communication streamlined both ways. And so that was what led me into technical product management, which was just, you know, how can I, how can I help in the way that I think I'm sort of most impactful and as it happens, the first jump into technical product management was an absolute, like it was absolute carnage for me and it was like, working at Marks & Spencers in like an enormous, enormous IT organization with my entire team out in India. And whilst they were great, like just all the communication barriers and the like, you know, timezone overlaps and all those sorts of things. It was it was kind of dreadful, and I felt pretty ineffective there. And then it was then when I sort of made the move to MOO that I think it was right, it really clicked for me, which was like big organization-wide projects to do this whole data center move. Basically, I remember just like the the team that we're in sort of felt like felt a little bit like the rock stars of the company because that we've been told like,
Amy Phillips:34:58
Incredible team as well. We should probably give them a shout out because they're probably all listening, I expect?!
Chris Evans:35:03
Well, yeah, this is the thing. So I remember joining and just being like, astonished at the level of talent that that MOO had been able to hire. So we had some, like, just incredible, like platform engineers. And, yeah, and then combined with that, like Meri Williams, who I think, has been on this podcast before, the CTO, and so she was able to basically, I mean, she basically was like, this is the most important thing for me. And really, really repeatedly backed that up and gave, like, all the resources that we needed, whether that was time or money, or additional people or additional weight across, you know, exec team or other teams. And so yeah, I think basically, to summarize, MOO is where I really clicked and was like, yes, this is, this is definitely for me, I really enjoy this whole, you know, one foot in like technical camp, and able to sort of help or at least, you know, help make decisions or, you know, influence those sorts of things. And then the other side of like, helping to, you know, make sure that everyone understood what was going on, and that we were doing things on the right timelines, etc, etc. And then sort of, I think, I think that the, as I said earlier, like the sort of extrapolation of all of this is then being someone who then, you know, I now have like a foot in foot in the engineering camp or foot in marketing camp and a foot in various different camps around the company that I'm in now. And then my sort of stakeholders, for want of a better phrase, are investors and the many, many customers and like the other co founders that I work with. And so, yeah, I think I just found out that those, those are the things that I really enjoy, and I think I'm good at, or at least I haven't managed to, like crash and burn this company yet. So, so far so good.
Aaron Randall:36:42
You mentioned earlier in this discussion about building a company, you used the phrase, it was so hard to build a company, and also around the co founder role, that it's like a truly multidisciplinary role. You mentioned doing GTM plus product plus tech plus everything. How did you find that transition from being an employee Monzo to like running and co founding a business at Incident.io?
Chris Evans:37:07
Yeah. I mean, I remember talking to Jonas, the CTO at Monzo before, before I left and him just sort of like a friendly warning. He was like, this will be like nothing, nothing you've ever done before. I can't remember the exact words, but it was just like, you know, go in eyes open. And I was like, yeah, I got this, this is fine. But I think I think the transition, I think the transition was actually eased by the fact that I had Pete and Stephen around. So we often talk about how, I mean, firstly, just massively grateful we are to be a founding team of three people, and three people who get on incredibly well. And it's been now you know, year and a half and whatever. And we haven't had any, any significant fallouts, which is very nice. But we often talk about like having out of phase like sine waves of you know, when one person's down, it's almost guaranteed that you're not going to have both, you know, the other two not be up or at least, you know, out of phase in some way. And so that, it's been like, incredibly hard starting this company, and so much to do, but we are a tight knit group, and very, very supportive. And so it sort of doesn't, doesn't feel like I've sort of had to face, you know, build a sales org and be like, I have no idea how to do that. It's like, cool, well, let's all go and figure this out. And we'll come back, and we'll regroup and we'll work together on, you know, a master sales deck. And we'll, we'll figure out how we do this and that and like, what does marketing look like? And, yeah, so I think it's sort of, it's just hard. And I think it takes a certain type of sort of run through walls mentality to do it well, which is, you know, you've got to, you've got to be the kind of person that sees, sees a big problem as an opportunity rather than something to kind of like, you know, run away from, and it's hard like that's not an easy thing to do. But I think you sort of just, I don't know, I think the thing that I've developed later in my career actually is just this kind of like attitude of saying yes to things, even if I don't believe it. And like, this is how I got into like, speaking at conferences, and it's like, someone asks you and says, you know, will you do that? Or will you talk at this meetup? And I just sort of was like, Yes. And then I stressed for about three months, progressively, in fact I think it was someone, Amy and I used to work with at MOO, a guy, Mike Harris, who used to describe, I think he described giving conference talks as like reverse hangovers, where you get more and more and more sort of ill feeling towards the event, and then it happens. And then it's like, it's all good again, kind of thing.
Amy Phillips:39:35
Yeah, like the pit of despair. Like, three days before, when you realize you really got to write that talk. And you have deep regrets. And then on the day, it's brilliant. Yeah, afterwards, it's even better and then you sign up for the next one.
Chris Evans:39:48
Exactly. Exactly, but in the build up to the talk, you'll be like, never again, why did I do this? I think the thing the thing that, I can't remember who said it, I can't remember I heard it. I think what I love about conference talking is like I'm a massive introvert and I go to a conference, I really struggle to go and like, just strike up conversation with people around the conference. But if you talk at a conference, people want to just come and talk to you. And it's just like, that's brilliant. I love that. Because I genuinely enjoy talking to people. I just like, it's that like, awkward intro thing, which I think generally British people are just dreadful at. And me especially so.
Amy Phillips:40:27
One thing that's always struck me, Chris, is you have remained very technical. And you've always been quite hands on and even, sort of like when we worked together at MOO, and you were sort of like, I know, you were a technical product manager, but like, you were very much experimenting with things. And "I saw this thing. I've tried this thing out", like, how, how have you managed to stay so in touch with technology as you've gone through all these different roles?
Chris Evans:40:52
Yeah, I mean, I think it's probably a stretch to say I'm really in touch with technology. Like I sort of, I think in the back of my mind, my my, I think for a long time, in the back of my mind, ever since I stopped sort of writing code as a full time full time gig has been that sort of fear of waking up one day and being in a meeting that in a technology company and someone says something, or there's a conversation going on, and you've just been like, I literally have no idea what's going on. And I remember I remember this, like M&S And like, I don't want to sort of talk ill of previous employer. But there were, there were people there who just didn't get get the modern world, they just sort of like in IT orgs. And it's sort of like just seeing out a career. And that has always been in the back of your mind is like, I just don't want to be that person. And as I said earlier, like this sort of niche that I feel like I've carved out as this sort of conduit, I guess, for want of a better phrase between technology, people engineering, people focus on the like, the sharp edge of doing this work, and people outside of that. And I think something else that sort of like is like a barometer for how I'm doing is can I go to a sort of meeting with someone and not have to bring an engineer along to sort of like translate or, you know, be support or anything else like that. So I think I think those things have always factored in my mind. And then like, as to how practically I have sort of, I think stayed in touch with like, technology is like, at my heart. I'm like a massive nerd. And so I see things and like, for me, it is like the solution is just write some software. And so I haven't stopped writing software in sort of some capacity at all, in my mind, whether it's like hacking a Raspberry Pi around to like control the lights in my house or, you know, writing some like open source software over a weekend. I think just being curious and keeping your hand in is sort of the the way that that's that stayed about but yeah, I don't know, I feel like I feel like I've sort of seen the future of where I might draw the line or where my sort of knowledge is. And I think it's crypto in that it's sort of like, I see things and I'm like, I don't understand any of those words. I also have precisely no desire to go and figure it out, either. I'm just like, I'm like, is this what is this what my like, grandparents thought of email and they were just like, it's never gonna catch on, don't worry about it kind of thing.
Aaron Randall:43:09
Definitely that and I feel the same as you as well. Unfortunately, we are running out of time. Before we let you go, as with all of our Humans+Tech guests, we have four quick fire questions we like to ask everyone. So I'm going to send these your way very quickly. And yeah, let's get your rapid responses. So the first question is, what's your top book recommendation?
Chris Evans:43:34
I'm gonna, it's gonna be really, really Oh, no, it's not lame. "The Field Guide to Understanding Human Error" by Sidney Dekker, absolute belter.
Aaron Randall:43:42
Never heard of it. Amy you're nodding your head.
Amy Phillips:43:44
I've never heard of it either. I've never read it. I've never heard of it and nobody has ever recommended it before. So
Chris Evans:43:49
I was about to say Harry Potter
Aaron Randall:43:53
So if you recommend a book Amy doesn't heard of, she's my sort of litmus test for if it's worth reading or not. She's read everything, so this is you've broken the broken that rule. Okay, great. Question number two. What, or who is your number one tip for keeping up with the industry?
Chris Evans:44:08
Former guest on this show, Kelsey Hightower. Absolutely love the guy.
Aaron Randall:44:13
Legend. He's, he's incredible. Yeah. Great. I mean, related question number three, who inspires you in tech?
Chris Evans:44:21
Can I cheat and say, Kelsey Hightower?
Aaron Randall:44:24
Who else inspires you in tech?
Chris Evans:44:28
Who else inspires me in tech? Frank Slootman. I think his name is the former CEO of ServiceNow. And now working at Snowflake, I think he is a a very interesting guy, written a bunch of books, worth checking out and he's been on a bunch of podcasts too.
Aaron Randall:44:46
Amazing. Great. And the final question of the four quick fires. What's the most ridiculous thing about you?
Chris Evans:44:54
Oh, gosh, what's ridiculous about me, ah, I don't know, can I cheat and ask for what other people have said ridiculous about them? Or Amy, what's ridiculous about me?
Amy Phillips:45:08
We can give you an alternative question actually, which will be themed. Tell us about your favorite incident, nightmare story?
Chris Evans:45:17
Oh, I can I can do that. Yeah, so I had my fair share of incidents over over my time. I think the, the very, very worst one that I had is one that I wrote a public post mortem for at Monzo, which was where we were scaling up our database, which is an incredibly routine operation that we had planned for, and we had run through in like, pre prod environments, and done a lot of work for and when we went to production, it all went to shit. And we broke a lot of things for a lot of people and took an entire bank offline. And we, yes, it turned out to be a single, like, boolean flag that was true and should have been false. And so lots of, lots of pain ensued. And, yeah, we wrote in detail about it, so I won't go too far into it right now. But I'm sort of getting the cold sweats just thinking it.
Aaron Randall:46:12
I'm getting sweaty palms and I wasn't even there. Wow. I mean, a great thing to live through well done for getting on the other side. Yeah. Amazing. And finally, where can people find out more about you?
Chris Evans:46:26
I guess either, for me, me personally, follow me on Twitter. I have a ridiculous handle that you can't pronounce. But it's like evnsio or Incident.io. That's a good place to find more.
Aaron Randall:46:40
Amazing. Great, Chris, thank you so much for taking the time to talk to us today. Learnt a bunch and loved, loved hearing your stories. It's been a lot fun. Thank you.
Chris Evans:46:48
Thanks very much.
Aaron Randall:46:51
We'll be sharing all the links in the show notes plus the all important doodle over on Humansplus.tech. I'm Aaron Randall. This is Amy Phillips and you've been listening to the Humans+Tech podcast.
Get the latest posts delivered right to your inbox.