Tallisor Incident Review: Card Payments Outage, 9 June 2027

Post mortem presentation example: timeline, cause, fixes

A post mortem presentation has to show what happened, who was affected and why, without blaming a person, and leave the room with fixes that have owners and dates. Tallisor, an invented payments company, reviews a 47 minute card payments outage caused by one mistyped setting, traces it through five links in a failure chain, and commits to seven dated remediations.

Every slide, in order

The whole deck as it renders, one slide after another. Read it the way the audience would.

TALLISOR · INCIDENT

SEV1 · 9 June 2027

Card payments outage, 47 minutes

Blameless post mortem, presented 11 June to engineering and support

Ifeoma Clarke, incident lead

With the payments platform team

Slide 1 · cover · Names the incident, its length and that the review is blameless, before any detail.

SUMMARY

What happened

At 13:55 UTC a configuration change for the card payments API was deployed with the database connection pool set to 20 instead of 200. Seven minutes later, as afternoon traffic in Europe rose, requests began waiting for connections and timing out. Client retries tripled the load. For 47 minutes, from 14:02 to 14:49, up to 62 percent of card payment requests failed. Service recovered when the change was rolled back. No data was lost and no payment was taken twice.

47
Minutes of impact

From the first alert at 14:02 to error rates back to normal at 14:49 UTC.

24
Minutes to the cause

The configuration change was identified at 14:26, after the database was ruled out.

02

Slide 2 · intro · The incident in plain words, so everyone starts from the same facts.

CUSTOMER IMPACT

Who was affected and how

Failed payment requests

18,400

Card payments that returned an error during the window.

Retried successfully

14,900

81 percent went through later the same day.

Payments not recovered

3,500

Customers who did not retry. Merchants were sent the list.

Merchants affected

2,310

31 percent of the 7,450 merchants live that afternoon.

Peak error rate

62%

At 14:21, when retries were at their highest.

Support tickets

640

Opened by merchants during and after the outage.

Service credits

$41K

Paid under the 99.95 percent availability terms.

Data lost

None

No payment duplicated, no record corrupted.

03

Slide 3 · metrics · Customer impact in eight numbers, which is what support and leadership need first.

Detection to recovery, UTC

Times are from the first alert. The longest gap was finding the cause, 24 minutes.

1

Minute 0

14:02. Error rate alert fires. Pager goes to the on call engineer.

2

Minute 4

14:06. On call acknowledges and opens the payments dashboard.

3

Minute 9

14:11. SEV1 declared. Status page updated. Incident lead takes over.

4

Minute 18

14:20. Database checks, begun at 14:11, show it healthy. Ruled out.

5

Minute 24

14:26. The 13:55 configuration change is found in the deploy log.

6

Minute 31

14:33. Rollback starts. Error rate begins to fall at 14:40.

7

Minute 47

14:49. Errors back to normal. Monitoring continues for an hour.

04

Slide 4 · timeline · Detection to recovery minute by minute, so the slow steps are visible.

How one setting became an outage

Config change

13:55 deploy

Pool shrinks

200 to 20

Requests queue

Timeouts at 14:02

Retries pile on

Load tripled

Payments fail

Peak 62%

A one digit change

A pool size of 200 was edited to 20 while another setting was being changed in the same file. Review did not catch it.

No guard on the value

The service accepted any pool size at start up. Nothing compared it with the previous value or a safe range.

Fine at low traffic

Twenty connections were enough for seven minutes, until European afternoon traffic arrived.

Retries without backoff

Two large merchant integrations retried every failed request at once, tripling the load on the queue.

Errors, not slowness

Requests that waited more than ten seconds were rejected, which merchants saw as declined payments.

05

Slide 5 · process · The failure chain from the change to the outage, so the fixes can target each link.

Contributing causes

Five causes, none of them a person. Each one maps to a remediation on the next slide.

Cause
Category
Effect
Fixed by
Pool size changed by mistake
Change process
Started the incident
Actions 1 and 2
No range check on config values
Tooling
Bad value went live
Action 2
Deploy log not on the dashboard
Observability
24 minutes to the cause
Action 3
Merchant retries without backoff
Integration
Tripled the load
Actions 4 and 5
Rollback took seven minutes
Tooling
Longer recovery
Action 6
Agreed by the incident review group on 10 June. Blameless: causes describe systems, not people.

06

Slide 6 · comparison · Contributing causes as a table, each with its category and the remediation that addresses it.

REMEDIATIONS

Seven actions, all dated

1
Second config reviewer

Any change to pool, timeout or limit values needs a second approval. Platform team, done 11 June.

2
Range checks at start up

Service refuses to start if pool size moves more than 50 percent. Platform team, by 30 June.

3
Deploys on the dashboard

Every deploy marked on the payments error chart. Observability team, by 23 June.

4
Backoff in our SDKs

Exponential backoff with jitter in both client libraries. Integrations team, by 7 July.

5
Talk to two merchants

Help the two largest retrying integrations adopt backoff. Partner team, by 14 July.

6
One click rollback

Config rollback in under two minutes from the deploy tool. Platform team, by 21 July.

7
Game day in August

Rehearse this failure in staging with the on call rota. Incident lead, 12 August.

07

Slide 7 · checklist · Remediations with owners and dates, which is what the review exists to produce.

The response itself

How the team handled the incident, apart from what caused it.

Went well

Did not

Alert fired within a minute of errors rising

Status page updated nine minutes in, then every ten minutes

Support had a merchant reply ready by 14:20

Rollback restored service with no data repair needed

List of unrecovered payments sent to merchants that evening

Four minutes to acknowledge the page at a busy hour

Nine minutes spent checking a healthy database

The deploy log was in a different tool from the dashboard

Rollback needed two people and seven minutes

Merchant retries were not visible on any chart

08

Slide 8 · comparison · What went well and what did not in the response, kept separate from the causes.

Questions, then the actions

Every action is tracked to its date. Progress is reviewed at the reliability meeting on 30 June and again after the August game day.

Full report

INC-0609 in the wiki

Action tracker

Reliability board

Follow up

30 June 2027

09

Slide 9 · closing · Closes on where the full report and the actions are tracked, and the follow up date.

The structure

What each slide is doing, so you can reuse the order even with different content.

Slide 1cover
Names the incident, its length and that the review is blameless, before any detail.
Slide 2intro
The incident in plain words, so everyone starts from the same facts.
Slide 3metrics
Customer impact in eight numbers, which is what support and leadership need first.
Slide 4timeline
Detection to recovery minute by minute, so the slow steps are visible.
Slide 5process
The failure chain from the change to the outage, so the fixes can target each link.
Slide 6comparison
Contributing causes as a table, each with its category and the remediation that addresses it.
Slide 7checklist
Remediations with owners and dates, which is what the review exists to produce.
Slide 8comparison
What went well and what did not in the response, kept separate from the causes.
Slide 9closing
Closes on where the full report and the actions are tracked, and the follow up date.

Slide by slide

Slide one names the incident, its severity and that the review is blameless. Slide two summarises what happened in one paragraph. Slide three counts customer impact. Slide four is the timeline from detection to recovery. Slide five shows the failure chain in five steps, from the configuration change to declined payments. Slide six maps contributing causes to fixes. Slide seven lists seven remediations with owners and dates, from a second reviewer on configuration changes to a game day in August. Slide eight separates what went well in the response from what did not. Slide nine says where the full report and tracker live and when progress is checked.

How to adapt this deck

A security incident review should add a slide on data exposure and notification before the timeline, because that is the first question regulators and customers ask. A smaller incident with little customer impact can merge slides two and three and drop the response comparison. For an incident caused by a third party, keep the failure chain but add a column to the causes table showing which links were yours to fix and which were the supplier's.

What incident review guidance says

Google's Site Reliability Engineering book defines a postmortem as a written record of an incident, its impact, the actions taken, the root causes and the follow up actions, and says a blameless postmortem focuses on contributing causes without indicting any person or team. PagerDuty's published incident response process recommends holding the postmortem meeting within three calendar days of a SEV1. This review met two days after the outage.

What makes this deck work

Impact is counted before cause

Slide three gives support and leadership the numbers they need first: 18,400 failed payments, 14,900 of them retried successfully, 3,500 not recovered, 2,310 merchants affected and $41K in service credits. The recovered and lost figures add up to the failed total, so nobody has to ask.

The slow minutes are visible

The timeline on slide four runs from minute 0 to minute 47 and shows that detection took one minute but finding the cause took 24, nine of them spent checking a healthy database. That is why the fix list puts deploy markers on the dashboard rather than adding another alert.

Causes describe systems, not people

The table on slide six lists five contributing causes, such as no range check on configuration values, and maps each to a numbered action on slide seven. A one digit change started the outage, but the review fixes the four things that let it reach customers.

Questions people ask

What should a post mortem presentation include?

A short summary, customer impact in numbers, a timeline from detection to recovery, the chain of causes, contributing causes mapped to fixes, remediations with owners and dates, and what went well or badly in the response. Close with where the written report lives and when actions are reviewed.

How do I keep a post mortem blameless?

Describe causes as properties of systems and processes, not people: no range check on a setting, not someone made a typo. Leave names off the cause slides, and say on the cover that the review is blameless. People share more when they know the meeting is about fixing systems.

How detailed should an incident timeline be?

Detailed enough to show where time went. Minute by minute works for an outage under an hour; hourly works for a longer incident. Mark detection, escalation, diagnosis, mitigation and recovery, and point out the longest gap, because that is usually where the most useful fix is.

When should a post mortem meeting happen?

Soon, while memories are fresh, but after impact is counted. Some published incident processes set three calendar days for the most severe incidents. Share the written draft before the meeting so the time is spent agreeing causes and actions rather than reading.

How many remediation actions should a post mortem have?

As many as the causes need and the team can finish, usually five to eight. Each needs an owner and a date, and at least one should address detection or recovery, not only prevention. Review them at a set follow up meeting so they do not fade once the incident is forgotten.

Can I share the post mortem deck with customers?

Share a customer version, not the internal deck. Keep the summary, impact, timeline and the fixes, and leave out internal tooling detail and team names. A PDF export costs about 1 credit per 10 slides and suits an email to affected customers. A share link always shows the current saved version, which is easier to keep correct.

Build your own in about a minute

The button below opens the generator with this use case already described. Change the wording to match your own, generate, then edit anything you like.

Make one like this

Other presentation examples

Want the steps in the builder? Read Edit your deck with the AI assistant, then Export a presentation to PowerPoint or PDF. For everything this generator can do, see the presentation maker.

Sources

Written and checked by the OneCraft team. Last checked .