Doing the right thing when things go wrong - The Intercom Blog

Overview

For customers Meet your customers where they already are with the world’s best business messenger for chat, email, voice, social…

Ideas blog Product & Design thoughts from our leadership team

Details

The Ticket podcast Conversations with future-focused leaders at the cutting edge of customer service.

For customers Meet your customers where they already are with the world’s best business messenger for chat, email, voice, social…

Ideas blog Product & Design thoughts from our leadership team

The Ticket podcast Conversations with future-focused leaders at the cutting edge of customer service.

When customers rely on you, minutes matter in an incident. Here’s the process Fin’s engineers follow to detect, mitigate, and learn from every one.

It’s Wednesday at 2pm. A routine deploy goes out. Within minutes, customers cannot open the app. Alarms fire, the pager goes off, and the heartbeat metrics begin to drop.

In some ways, this is close to a best-case incident. It’s daytime, the team is online, and the author of the change is available. There is already a strong hypothesis of what went wrong. And yet, incidents can still spiral without the right process, tools, and shared instincts.

At Fin, shipping is our heartbeat. We have written before about how we prepare for our busiest days, but reliability is not only about handling peak load; it is also about how we respond when something unexpected happens. When customers rely on Fin and Intercom to support their own customers, an incident can interrupt conversations and create uncertainty at exactly the wrong moment. This is why incident management is so important.

Incident management might look like Slack channels, status pages, runbooks, and the latest AI SRE tooling, but they’re only the mechanics. It’s really about the discipline of helping people do the right thing when things go wrong. Our process is built around customer impact: detect problems quickly, bring the right people (our engineers and their Agents) together, reduce impact safely and quickly, communicate clearly, and learn from what happened.

One principle shapes all of this: not every P1 is an incident, but every incident should be a team’s P1.

An incident means there’s a time-sensitive disruption, damage, or degradation to the customer experience, and even a well-run one causes real disorder for the customers affected. So the work to resolve it can’t sit in a backlog while we finish something else. When an incident is declared, the responsible team stops what they’re doing and focuses on getting back to a safe state. That might mean leaving a meeting, pulling in engineers with domain context, escalating to the Incident Command team, or rolling back a change before anyone fully understands why it broke. The longer an incident sits unactioned, the more customers it affects, the more trust it erodes, and the harder it becomes to coordinate.

To understand how we respond to incidents at Fin, we’ll go through how we think about them, as well as the mechanics for how we mitigate and resolve them.

Our incident process usually starts in one of three ways:

Support (and increasingly, Operator) sees a pattern in incoming conversations. Within moments we can open a Git Hub issue and engage the relevant engineering teams through incident.io, our chosen platform for incident response. Frontline support (both our humans and Fin itself) are often closer to customer pain than any dashboard, making their signal extremely valuable when it comes to speed.

Beyond the usual suite of alarms, we have heartbeat metrics for core parts of Fin and the Intercom helpdesk. Rather than only monitoring whether our servers are up, these metrics track whether users can do the thing they came to do, like reply to customer queries. Drops in these metrics can tell us if a core workflow is unhealthy, triggering an incident and a page (a loud notification or phone call generated by incident.io) to the owning team. We do not wait for someone to notice manually.

For example, we actively track the number of responses that Fin is able to generate in conversations. If we see a sudden depression in response generation, we know immediately something is wrong and engage our responders to swing into action.

3. Engineers shipping code are actively monitoring product health

At Fin, all engineers are expected to watch their code rolling out from the time it starts deploying to when it lands in production. Having someone on hand monitoring for unexpected changes in metrics (and testing their changes once live) means that the very instant something unexpected happens, we have the ability to restore a healthy state by rolling back production to an earlier version.

It currently takes a little under two minutes for a rollback to land. That means an engineer, or an Agent monitoring the rollout on their behalf, can often spot and remove any undesired behavior in the product before customers start to notice. When it comes to mitigating customer impact, seconds matter, which is why we’re continually pushing for ever-faster rollbacks in our product.

Clear roles are one of the biggest differences between a chaotic incident and a well-run one. During an incident, everyone should know what they own:

The engineering lead is the first technical owner. They acknowledge the page, assess impact, and pull in help as needed.

The incident commander coordinates the response when escalation is required, making decisions so technical investigators can stay focused.

The business lead handles the customer-facing side, keeping Support and Sales informed.

The resolution owner, usually the team’s engineering manager, ensures follow-ups are completed once the pressure lifts.

These roles exist to reduce cognitive load, not impose hierarchy. They give people a clear way to contribute without having to negotiate ownership in the middle of a high-pressure response.

That structure matters even more outside core engineering hours. To make sure customer-impacting issues are not left waiting until morning, we have an out-of-hours responder group called STRIKE. They cover all engineering pagers across evenings, weekends, and holidays and act as the first line of response when the owning team may not be immediately online. Their coverage helps us avoid one of the most damaging failure modes in incident response: losing time before anyone has taken ownership. Our Senior Principal Engineer, Brian Scanlan, shared more about STRIKE in a presentation at DASH by Datadog, if you’re interested in learning more.

When an incident starts, we follow a four-step process, known as the “incident lifecycle.” We triage, investigate, mitigate, and learn – going beyond one-off resolutions to drive real change. These stages and the associated actions are not prescriptive. Sometimes we need the flexibility to jump around in the process, but they serve as a general structure for our incidents.

Regardless of how the issue is spotted, our paths quickly converge into the same operating model. We run incidents through incident.io, almost entirely from Slack, where teams already coordinate. It takes the toil out of the process by assigning roles, updating severity, and summarizing the timeline for anyone joining midway through. An incident channel becomes the base of operations: one place for the timeline, actions, updates, follow-ups, and escalations.

From there, the first responder acknowledges the problem and sizes up the shape of the issue:

Is there a runbook for this alarm to verify the signal?

If there’s no customer impact, the alarm may need to be reconfigured. In these cases, our responder would decline the incident but create follow-ups to improve the alert for the future. This helps us maintain our position of treating any alarm with the utmost severity and avoiding creating alarm or pager fatigue.

If there is customer impact, they accept the incident. From there, the next job is to understand impact:

Is this one customer, a group, or a core product path?

Impact drives severity, and severity drives how we resource the response. A minor issue might need one or two engineers, but a major incident on a core path could pull in multiple teams and an incident commander from the word go.

At this early stage, clear communication is just as important as the technical assessment. Letting people know something is wrong (or not) often improves our ability to resolve it.

That doesn’t solve the incident, but it gives everyone a shared picture, reduces duplicate questions, and helps people decide whether to get involved.

Once an incident is accepted and triaged, engineers gather evidence, check recent changes, inspect dashboards, and form hypotheses. At this stage, coordination matters as much as technical debugging. Incidents punish ambiguity with extended impact windows, higher stress, and a poorer experience for our customers.

As an incident gets underway, it’s important to use all available resources effectively to improve recovery speed, and this means making sure everyone knows what work is happening and who is doing it. There is no room in an incident for beating around the bush. “Someone should check the queue depth” needs to become a specific and assigned action, not a sentence that floats away.

If the team is stuck, customer impact is broad, or multiple teams are involved, escalation to an incident commander is the right call. They help create structure, clarify roles, decide which mitigation path to take, and handle internal and external communication. This allows technical investigators to focus.

Throughout the investigation, we maintain a bias for action as customers are still affected while we search for certainty. We want to understand what caused the incident, but also how to reduce impact as fast as possible.

As an incident grows, communication scales outward. Clear updates let Support, Sales, and Customer Success answer customer questions with confidence, and when broader external communication is needed, the incident commander coordinates it through the right channels, including our status page, where appropriate.

In many incidents, the fastest path to reducing customer impact is a rollback.

“Incidents are not the place to optimize for elegance. They are the place to optimize for safe recovery”

That is closely connected to how we think about shipping safely at Fin. Speed matters, but speed is only sustainable when recovery is fast too. One of the lessons we repeat in training is: roll back first, fix forward later.

Fixing forward can feel satisfying. You find the bug, code up a fix, and ship it. But incidents are not the place to optimize for elegance. They are the place to optimize for safe recovery.

A rollback usually gets us back to a known state faster than writing and deploying a new fix. It also avoids adding new variables while the system is already unstable. Once customer impact is reduced, the team has more room to understand the issue properly.

However, rollback is not always possible. The issue might be data-related, a feature flag might need changing, a queue might need draining, or a dependency might need bypassing. The point is to be explicit about the options:

Regular updates throughout keep everyone aligned as the picture changes.

When customer impact is mitigated, it is tempting to disappear back into normal work. At Fin, we do not do that.

It’s important we resolve open actions, clean up temporary mitigations, and ensure the issue is truly contained. After that, we can move from recovery to learning.

Good follow-ups are specific and actionable. “Improve monitoring” is not a good follow-up. “Alert when export queue latency exceeds X for Y minutes, with a runbook linked to the owning team” is better.

We look for follow-ups that do at least one of four things:

They also need to be bounded. A good follow-up is something the team can complete in a reasonable timeframe, not a vague aspiration.

For notable incidents, we write an incident review doc and host a meeting with our engineering org. This review process is how a painful moment becomes shared organizational knowledge, and how we make good on our obligation to affected customers to ensure the same class of failure does not return. The people who were there capture what happened, while the wider organization learns from it and future responders get a better system.

A healthy incident culture cannot depend on individual heroics. Customers should not have to rely on one person knowing the system, spotting the issue, remembering the command, and staying calm under pressure.

“Declaring an incident is not overreacting when there is credible risk to customers”

We need something more reliable: trained responders, familiar tools, clear roles, normal escalation paths, and a shared bias toward reducing customer impact quickly.

That also means treating early escalation as good judgment. We would rather mobilize early and stand down than wait for certainty while customer impact grows. Asking for help is not a failure. Rolling back is not an admission of defeat. Declaring an incident is not overreacting when there is credible risk to customers.

In incident management, many things can be corrected. We can change severity, merge incidents, or revise our understanding of the root cause. What we cannot recover is time lost to inaction.

When something goes wrong, customers should not have to wonder whether Fin is taking it seriously. We should be visible in how quickly we mobilize, communicate, and reduce impact. We can deliver this by upholding our incident management process – and learning something every time we use it.

AI is approving our pull requests: Here’s how we made it safe

The safety of speed: How we ship code 180 times per day

Key Takeaways

For customers Meet your customers where they already are with the world’s best business messenger for chat, email, voice, social…
Ideas blog Product & Design thoughts from our leadership team
The Ticket podcast Conversations with future-focused leaders at the cutting edge of customer service
For customers Meet your customers where they already are with the world’s best business messenger for chat, email, voice, social…
Ideas blog Product & Design thoughts from our leadership team