03 / IBM · AIOps · 2021
Anomaly detection & root cause
SRE surface that clustered noisy alerts into narrated incidents — the on-call could see the story, not the storm.
Role
Product Designer
Team
1 designer · 6 engineers · 1 PM · 1 data scientist
Duration
6 months
Platforms
Web application · Slack + PagerDuty integrations
Problem
On-call engineers were drowning in alerts, not signal.
A single service degradation could produce 400+ alerts across the fleet. The on-call spent the first fifteen minutes of any incident just trying to figure out what to look at first.
The AIOps team had built a strong clustering model. What they needed was a surface that let engineers trust and act on it — without hiding the underlying data if they wanted to dig.
Process
Design the narration, not the noise.
01 · Incident-first framing
Reframed the primary object from 'alert' to 'incident.' Alerts became children of the incident, not the top-level noun.
02 · Explainability panels
Every clustered incident gets a 'why we grouped these' panel — features the model weighted, correlated services, timing. Trust follows explanation.
03 · Escape hatches
One click to break a cluster apart into raw alerts, one click to promote a raw alert to a new incident. Never trap the operator.
04 · Playbook attachment
Designed the ritual of attaching a runbook to an incident type — so the second time it happens, the response is a click away.
Results
Mean time to resolution dropped fast.
4.1×
faster MTTR
−78%
alert volume shown to on-call
92%
of clusters marked 'correct' by SREs
The team measured MTTR against a control cohort still on the old console — a 4.1× improvement held over the following quarter. More importantly, alert fatigue survey scores from the SRE org improved meaningfully.
Next case study
Underwriting workbench
IBM · Open Insurance