{
  "refs": {
    "e1": {
      "name": "Checkout was unavailable for 3 hours 47 minutes on 12 September",
      "role": "heading"
    },
    "e10": {
      "name": "Mitigated 3h 47m total",
      "role": "heading"
    },
    "e11": {
      "name": "Root cause",
      "role": "region"
    },
    "e12": {
      "name": "Root cause",
      "role": "heading"
    },
    "e13": {
      "name": "Contributing factors",
      "role": "region"
    },
    "e14": {
      "name": "Contributing factors",
      "role": "heading"
    },
    "e15": {
      "name": "The latency SLO is measured at p50",
      "role": "heading"
    },
    "e16": {
      "name": "A behaviour change hid inside a dependency bump",
      "role": "heading"
    },
    "e17": {
      "name": "The runbook's first move is the wrong move here",
      "role": "heading"
    },
    "e18": {
      "name": "What went well",
      "role": "heading"
    },
    "e19": {
      "name": "Action items \u00b7 3 open, 1 complete",
      "role": "region"
    },
    "e2": {
      "name": "Impact",
      "role": "region"
    },
    "e20": {
      "name": "ID",
      "role": "columnheader"
    },
    "e21": {
      "name": "Action",
      "role": "columnheader"
    },
    "e22": {
      "name": "Owner",
      "role": "columnheader"
    },
    "e23": {
      "name": "Due",
      "role": "columnheader"
    },
    "e24": {
      "name": "Status",
      "role": "columnheader"
    },
    "e25": {
      "name": "AI-1",
      "role": "rowheader"
    },
    "e26": {
      "name": "Alert on p99 checkout latency and on success rate, not p50 alone.",
      "role": "cell"
    },
    "e27": {
      "name": "Owner M. Halvorsen",
      "role": "cell"
    },
    "e28": {
      "name": "Due 19 Sep",
      "role": "cell"
    },
    "e29": {
      "name": "Done",
      "role": "cell"
    },
    "e3": {
      "name": "Impact",
      "role": "heading"
    },
    "e30": {
      "name": "AI-2",
      "role": "rowheader"
    },
    "e31": {
      "name": "Add a circuit breaker to the provider client, opening at 25% errors over 30 seconds.",
      "role": "cell"
    },
    "e32": {
      "name": "Owner R. Idowu",
      "role": "cell"
    },
    "e33": {
      "name": "Due 26 Sep",
      "role": "cell"
    },
    "e34": {
      "name": "In progress",
      "role": "cell"
    },
    "e35": {
      "name": "AI-3",
      "role": "rowheader"
    },
    "e36": {
      "name": "Treat timeout, retry and pool constants as behaviour changes in review \u2014 CODEOWNERS plus a required note.",
      "role": "cell"
    },
    "e37": {
      "name": "Owner Platform",
      "role": "cell"
    },
    "e38": {
      "name": "Due 3 Oct",
      "role": "cell"
    },
    "e39": {
      "name": "Not started",
      "role": "cell"
    },
    "e4": {
      "name": "Timeline \u00b7 12 September, UTC",
      "role": "region"
    },
    "e40": {
      "name": "AI-4",
      "role": "rowheader"
    },
    "e41": {
      "name": "Rewrite the checkout runbook to diagnose saturation before scaling, pool metrics first.",
      "role": "cell"
    },
    "e42": {
      "name": "Owner D. Kaur",
      "role": "cell"
    },
    "e43": {
      "name": "Due 10 Oct",
      "role": "cell"
    },
    "e44": {
      "name": "Not started",
      "role": "cell"
    },
    "e5": {
      "name": "Timeline \u00b7 12 September, UTC",
      "role": "heading"
    },
    "e6": {
      "name": "Release 2026.9.3 rolls out",
      "role": "heading"
    },
    "e7": {
      "name": "Impact begins Impact start",
      "role": "heading"
    },
    "e8": {
      "name": "Detected 33 min to detect",
      "role": "heading"
    },
    "e9": {
      "name": "Scaling up makes it worse",
      "role": "heading"
    }
  },
  "snapshot": "- document:\n  - main:\n    - text: Incident postmortem SEV-1\n    - term: Incident\n    - definition: INC-2026-041\n    - term: Status\n    - definition: Resolved\n    - term: Authors\n    - definition: D. Kaur, R. Idowu\n    - term: Reviewed\n    - definition: 18 Sep 2026\n    - heading \"Checkout was unavailable for 3 hours 47 minutes on 12 September\" [ref=e1] [level=1]\n    - region \"Impact\" [ref=e2]:\n      - heading \"Impact\" [ref=e3] [level=2]\n      - term: Duration\n      - definition: 3h 47m\n      - paragraph: 09:14 \u2013 13:01 UTC\n      - term: Customers\n      - definition: 12,400\n      - paragraph: 18% of active accounts\n      - term: Failed checkouts\n      - definition: 41,900\n      - paragraph: 39,600 recovered by retry\n      - term: Data loss\n      - definition: None\n      - paragraph: No double charges\n    - paragraph: A dependency bump raised the payment-provider timeout from two seconds to thirty. When the provider slowed, every stuck request held its connection fifteen times longer and the pool saturated \u2014 three lines inside a forty-one file pull request.\n    - figure \"Checkout failure rate \u00b7 0\u201360% \u00b7 09:00\u201314:00 UTC\":\n      - text: Checkout failure rate \u00b7 0\u201360% \u00b7 09:00\u201314:00 UTC\n      - img \"Checkout failure rate rose from under 1% at 09:00 to a peak of 52% at 11:30, then fell below 2% by 13:15. Detection at 09:47 came when the rate was about 6%.\"\n      - paragraph: \"Gridlines 0/30/60%. Dashed: detected 09:47 at 6%, mitigated 13:01. Peak 52% \u2014 nine times worse than when the alert fired.\"\n    - region \"Timeline \u00b7 12 September, UTC\" [ref=e4]:\n      - heading \"Timeline \u00b7 12 September, UTC\" [ref=e5] [level=2]\n      - list:\n        - listitem:\n          - text: 09:02\n          - heading \"Release 2026.9.3 rolls out\" [ref=e6] [level=3]\n          - paragraph: A dependency bump changes the payment-provider request timeout from 2s to 30s. All six regions by 09:11, no alarms.\n        - listitem:\n          - text: 09:14\n          - heading \"Impact begins Impact start\" [ref=e7] [level=3]\n          - paragraph: Provider p99 latency rises from 240ms to 4.8s. Each slow request now holds a connection for up to 30s and the 200-connection pool begins to queue in eu-west.\n        - listitem:\n          - text: 09:47\n          - heading \"Detected 33 min to detect\" [ref=e8] [level=3]\n          - paragraph: \"The success-rate alert fires after five minutes below 95%. No latency alert ever fired: the SLO is measured at p50, which never moved.\"\n        - listitem:\n          - text: 10:26\n          - heading \"Scaling up makes it worse\" [ref=e9] [level=3]\n          - paragraph: Per the runbook, on-call scales pods from 12 to 40. Each opens connections against the same provider-side ceiling; the failure rate climbs from 28% to 41%.\n        - listitem:\n          - text: 13:01\n          - heading \"Mitigated 3h 47m total\" [ref=e10] [level=3]\n          - paragraph: A diff against 2026.9.2 surfaced the timeout constant at 11:38 and the rollback completes in the last region. Success rate returns above 99%; a retry job recovers 39,600 of 41,900 checkouts by 13:41.\n    - region \"Root cause\" [ref=e11]:\n      - heading \"Root cause\" [ref=e12] [level=2]\n      - paragraph:\n        - text: A timeout raised from\n        - code: 2s\n        - text: to\n        - code: 30s\n        - text: turned a slow dependency into an outage.\n      - paragraph: A pool's throughput is capacity divided by hold time. At two seconds a stuck request freed its connection fast enough that a slow provider cost latency and nothing more; at thirty it holds fifteen times longer, so the pool saturates at a fifteenth of the load it was sized for. The slowdown itself was ordinary \u2014 twice last quarter, neither an incident. What changed was our tolerance.\n    - region \"Contributing factors\" [ref=e13]:\n      - heading \"Contributing factors\" [ref=e14] [level=2]\n      - list:\n        - listitem:\n          - text: \"01\"\n          - heading \"The latency SLO is measured at p50\" [ref=e15] [level=3]\n          - paragraph: A p99 blowout raises no alert. Thirty-three minutes passed before a success-rate alert caught it indirectly.\n        - listitem:\n          - text: \"02\"\n          - heading \"A behaviour change hid inside a dependency bump\" [ref=e16] [level=3]\n          - paragraph: Three lines in a forty-one file pull request, no note in the description, approved in under four minutes.\n        - listitem:\n          - text: \"03\"\n          - heading \"The runbook's first move is the wrong move here\" [ref=e17] [level=3]\n          - paragraph: \"\\\"Scale up\\\" is right for CPU saturation and harmful for pool exhaustion. It cost 72 minutes and raised the failure rate.\"\n      - heading \"What went well\" [ref=e18] [level=2]\n      - list \"What went well\":\n        - listitem:\n          - strong: Idempotency keys held.\n          - text: No double charges across 41,900 failures; the retry job recovered 94% within 40 minutes.\n        - listitem:\n          - strong: The rollback was clean.\n          - text: Six regions, 83 minutes, no manual steps.\n    - region \"Action items \u00b7 3 open, 1 complete\" [ref=e19]:\n      - table \"Action items \u00b7 3 open, 1 complete\":\n        - caption: Action items \u00b7 3 open, 1 complete\n        - rowgroup:\n          - row \"ID Action Owner Due Status\":\n            - columnheader \"ID\" [ref=e20]\n            - columnheader \"Action\" [ref=e21]\n            - columnheader \"Owner\" [ref=e22]\n            - columnheader \"Due\" [ref=e23]\n            - columnheader \"Status\" [ref=e24]\n        - rowgroup:\n          - row \"AI-1 Alert on p99 checkout latency and on success rate, not p50 alone. Owner M. Halvorsen Due 19 Sep Done\":\n            - rowheader \"AI-1\" [ref=e25]\n            - cell \"Alert on p99 checkout latency and on success rate, not p50 alone.\" [ref=e26]\n            - cell \"Owner M. Halvorsen\" [ref=e27]\n            - cell \"Due 19 Sep\" [ref=e28]\n            - cell \"Done\" [ref=e29]\n          - row \"AI-2 Add a circuit breaker to the provider client, opening at 25% errors over 30 seconds. Owner R. Idowu Due 26 Sep In progress\":\n            - rowheader \"AI-2\" [ref=e30]\n            - cell \"Add a circuit breaker to the provider client, opening at 25% errors over 30 seconds.\" [ref=e31]\n            - cell \"Owner R. Idowu\" [ref=e32]\n            - cell \"Due 26 Sep\" [ref=e33]\n            - cell \"In progress\" [ref=e34]\n          - row \"AI-3 Treat timeout, retry and pool constants as behaviour changes in review \u2014 CODEOWNERS plus a required note. Owner Platform Due 3 Oct Not started\":\n            - rowheader \"AI-3\" [ref=e35]\n            - cell \"Treat timeout, retry and pool constants as behaviour changes in review \u2014 CODEOWNERS plus a required note.\" [ref=e36]\n            - cell \"Owner Platform\" [ref=e37]\n            - cell \"Due 3 Oct\" [ref=e38]\n            - cell \"Not started\" [ref=e39]\n          - row \"AI-4 Rewrite the checkout runbook to diagnose saturation before scaling, pool metrics first. Owner D. Kaur Due 10 Oct Not started\":\n            - rowheader \"AI-4\" [ref=e40]\n            - cell \"Rewrite the checkout runbook to diagnose saturation before scaling, pool metrics first.\" [ref=e41]\n            - cell \"Owner D. Kaur\" [ref=e42]\n            - cell \"Due 10 Oct\" [ref=e43]\n            - cell \"Not started\" [ref=e44] [nth=1]\n    - text: Blameless review held 18 September, sixteen attendees from Payments, Platform and Support. Timestamps come from the incident channel and the deploy log; failure rates are the per-minute success-rate metric."
}