- main: - text: Incident postmortem SEV-1 - term: Incident - definition: INC-2026-041 - term: Status - definition: Resolved - term: Authors - definition: D. Kaur, R. Idowu - term: Reviewed - definition: 18 Sep 2026 - heading "Checkout was unavailable for 3 hours 47 minutes on 12 September" [level=1] - region "Impact": - heading "Impact" [level=2] - term: Duration - definition: 3h 47m - paragraph: 09:14 – 13:01 UTC - term: Customers - definition: 12,400 - paragraph: 18% of active accounts - term: Failed checkouts - definition: 41,900 - paragraph: 39,600 recovered by retry - term: Data loss - definition: None - paragraph: No double charges - paragraph: A dependency bump raised the payment-provider timeout from two seconds to thirty. When the provider slowed, every stuck request held its connection fifteen times longer and the pool saturated — three lines inside a forty-one file pull request. - figure "Checkout failure rate · 0–60% · 09:00–14:00 UTC": - text: Checkout failure rate · 0–60% · 09:00–14:00 UTC - img "Checkout failure rate rose from under 1% at 09:00 to a peak of 52% at 11:30, then fell below 2% by 13:15. Detection at 09:47 came when the rate was about 6%." - paragraph: "Gridlines 0/30/60%. Dashed: detected 09:47 at 6%, mitigated 13:01. Peak 52% — nine times worse than when the alert fired." - region "Timeline · 12 September, UTC": - heading "Timeline · 12 September, UTC" [level=2] - list: - listitem: - text: 09:02 - heading "Release 2026.9.3 rolls out" [level=3] - paragraph: A dependency bump changes the payment-provider request timeout from 2s to 30s. All six regions by 09:11, no alarms. - listitem: - text: 09:14 - heading "Impact begins Impact start" [level=3] - paragraph: Provider p99 latency rises from 240ms to 4.8s. Each slow request now holds a connection for up to 30s and the 200-connection pool begins to queue in eu-west. - listitem: - text: 09:47 - heading "Detected 33 min to detect" [level=3] - paragraph: "The success-rate alert fires after five minutes below 95%. No latency alert ever fired: the SLO is measured at p50, which never moved." - listitem: - text: 10:26 - heading "Scaling up makes it worse" [level=3] - paragraph: Per the runbook, on-call scales pods from 12 to 40. Each opens connections against the same provider-side ceiling; the failure rate climbs from 28% to 41%. - listitem: - text: 13:01 - heading "Mitigated 3h 47m total" [level=3] - paragraph: A diff against 2026.9.2 surfaced the timeout constant at 11:38 and the rollback completes in the last region. Success rate returns above 99%; a retry job recovers 39,600 of 41,900 checkouts by 13:41. - region "Root cause": - heading "Root cause" [level=2] - paragraph: - text: A timeout raised from - code: 2s - text: to - code: 30s - text: turned a slow dependency into an outage. - paragraph: A pool's throughput is capacity divided by hold time. At two seconds a stuck request freed its connection fast enough that a slow provider cost latency and nothing more; at thirty it holds fifteen times longer, so the pool saturates at a fifteenth of the load it was sized for. The slowdown itself was ordinary — twice last quarter, neither an incident. What changed was our tolerance. - region "Contributing factors": - heading "Contributing factors" [level=2] - list: - listitem: - text: "01" - heading "The latency SLO is measured at p50" [level=3] - paragraph: A p99 blowout raises no alert. Thirty-three minutes passed before a success-rate alert caught it indirectly. - listitem: - text: "02" - heading "A behaviour change hid inside a dependency bump" [level=3] - paragraph: Three lines in a forty-one file pull request, no note in the description, approved in under four minutes. - listitem: - text: "03" - heading "The runbook's first move is the wrong move here" [level=3] - paragraph: "\"Scale up\" is right for CPU saturation and harmful for pool exhaustion. It cost 72 minutes and raised the failure rate." - heading "What went well" [level=2] - list "What went well": - listitem: - strong: Idempotency keys held. - text: No double charges across 41,900 failures; the retry job recovered 94% within 40 minutes. - listitem: - strong: The rollback was clean. - text: Six regions, 83 minutes, no manual steps. - region "Action items · 3 open, 1 complete": - table "Action items · 3 open, 1 complete": - caption: Action items · 3 open, 1 complete - rowgroup: - row "ID Action Owner Due Status": - columnheader "ID" - columnheader "Action" - columnheader "Owner" - columnheader "Due" - columnheader "Status" - rowgroup: - row "AI-1 Alert on p99 checkout latency and on success rate, not p50 alone. Owner M. Halvorsen Due 19 Sep Done": - rowheader "AI-1" - cell "Alert on p99 checkout latency and on success rate, not p50 alone." - cell "Owner M. Halvorsen" - cell "Due 19 Sep" - cell "Done" - row "AI-2 Add a circuit breaker to the provider client, opening at 25% errors over 30 seconds. Owner R. Idowu Due 26 Sep In progress": - rowheader "AI-2" - cell "Add a circuit breaker to the provider client, opening at 25% errors over 30 seconds." - cell "Owner R. Idowu" - cell "Due 26 Sep" - cell "In progress" - row "AI-3 Treat timeout, retry and pool constants as behaviour changes in review — CODEOWNERS plus a required note. Owner Platform Due 3 Oct Not started": - rowheader "AI-3" - cell "Treat timeout, retry and pool constants as behaviour changes in review — CODEOWNERS plus a required note." - cell "Owner Platform" - cell "Due 3 Oct" - cell "Not started" - row "AI-4 Rewrite the checkout runbook to diagnose saturation before scaling, pool metrics first. Owner D. Kaur Due 10 Oct Not started": - rowheader "AI-4" - cell "Rewrite the checkout runbook to diagnose saturation before scaling, pool metrics first." - cell "Owner D. Kaur" - cell "Due 10 Oct" - cell "Not started" - text: Blameless review held 18 September, sixteen attendees from Payments, Platform and Support. Timestamps come from the incident channel and the deploy log; failure rates are the per-minute success-rate metric.