Post-mortem report PM-2026-09, booking outage, Slotharbour

Post mortem report template with a timeline, five whys and owned actions

A post mortem report has to explain what happened, what it cost, why the system allowed it, and what will stop it happening again, without blaming the person who made the change. This four page blameless review of a 107 minute booking outage gives the impact in four figures, a timed timeline, five whys down to a root cause, and six actions with owners and dates.

Create a document with OneCraft4 A4 pages, editable, then download as a PDF

The document, page by page

Every page as it renders and as it prints, with nothing summarised. Read the wording before you reuse it.

Slotharbour · Post-mortem PM-2026-09 · Blameless review · InternalPage 1 of 4
Slotharbour · Post-mortem PM-2026-09 · Blameless review · InternalPage 2 of 4
Slotharbour · Post-mortem PM-2026-09 · Blameless review · InternalPage 3 of 4
Slotharbour · Post-mortem PM-2026-09 · Blameless review · InternalPage 4 of 4
Post-mortem: 107 minute booking outage
Incident
PM-2026-09, severity 1
Date
Tuesday 22 September 2026, 18:04 to 19:51, local time
Review meeting
Friday 25 September 2026, 14 attendees
Author
Tomasz Brennan, incident commander
Status
Published Monday 28 September 2026
This is a blameless review
It looks for the conditions that let a reasonable change cause an outage, not for a person to blame. People are named by role. Anyone in the same seat with the same information could have made the same change.
1. Summary

At 17:58 a configuration change doubled the number of application workers from 24 to 48 to handle the evening booking peak. Each worker opens its own pool of 20 database connections, so the platform tried to hold 960 connections against a database limit of 500. From 18:04, new requests waited for a connection and timed out, and customers could not make, change or cancel bookings. The change was rolled back at 19:31 and service was fully restored at 19:51.

2. Impact
107
Minutes down
3,140
Failed bookings
412
Venues affected
286
Support tickets

The outage fell in the busiest two hours of the week for restaurant bookings. Of the 3,140 failed booking attempts, 1,870 were retried successfully after 19:51, leaving 1,270 that did not return. Availability for September fell to 99.75 percent against a monthly target of 99.9 percent; 107 minutes is more than twice the 43 minutes of downtime the target allows in a 30 day month. Venue plans with a service level credit will receive one automatically on their October invoice.

3. Timeline
Time
Event
17:41
Change to raise workers from 24 to 48 approved in review; the diff shows only the worker count
17:58
Change deployed; new workers start opening connection pools
18:02
Connection saturation alert fires and goes to the team email inbox
18:04
Database reaches its 500 connection limit; booking requests start timing out
18:19
Error rate alert pages the on call engineer, 15 minutes after impact began
18:27
Incident declared at severity 1; incident commander and database engineer join
18:46
Status page updated; database restarted, connections fill again within two minutes
19:12
Team links the timeouts to the 17:58 deploy from the connection count graph
19:31
Change rolled back to 24 workers; connections fall below the limit
19:51
Error rate back to normal; queued notifications sent; incident resolved

Detection took 15 minutes and diagnosis took a further 53 minutes from the page to linking the deploy. The restart at 18:46 was a reasonable first step but cost time, because it treated the symptom while the cause, the worker count, was still deployed.

4. Five whys
1
Why did bookings fail?
Requests could not get a database connection within the 5 second timeout, so the API returned errors.
2
Why were no connections free?
The platform was trying to open 960 connections against a limit of 500, after workers doubled from 24 to 48 at 20 connections each.
3
Why did the change not adjust the pool size?
Worker count and pool size are set in two different configuration files, and the review showed only the file that changed. Nobody saw the total.
4
Why did testing not catch it?
Staging runs 4 workers against its own small database, so total connections never come near a limit there.
5
Why was the warning missed?
The connection saturation alert fired at 18:02 but was routed to an email inbox rather than to the pager, a setting left over from when the alert was noisy.
1
Why did bookings fail?
Requests could not get a database connection within the 5 second timeout, so the API returned errors.
2
Why were no connections free?
The platform was trying to open 960 connections against a limit of 500, after workers doubled from 24 to 48 at 20 connections each.
3
Why did the change not adjust the pool size?
Worker count and pool size are set in two different configuration files, and the review showed only the file that changed. Nobody saw the total.
4
Why did testing not catch it?
Staging runs 4 workers against its own small database, so total connections never come near a limit there.
5
Why was the warning missed?
The connection saturation alert fired at 18:02 but was routed to an email inbox rather than to the pager, a setting left over from when the alert was noisy.
1
Why did bookings fail?
Requests could not get a database connection within the 5 second timeout, so the API returned errors.
2
Why were no connections free?
The platform was trying to open 960 connections against a limit of 500, after workers doubled from 24 to 48 at 20 connections each.
3
Why did the change not adjust the pool size?
Worker count and pool size are set in two different configuration files, and the review showed only the file that changed. Nobody saw the total.
4
Why did testing not catch it?
Staging runs 4 workers against its own small database, so total connections never come near a limit there.
5
Why was the warning missed?
The connection saturation alert fired at 18:02 but was routed to an email inbox rather than to the pager, a setting left over from when the alert was noisy.

Root cause. Total database connections are a product of two settings that live apart, and no check stops a deploy that pushes the total past the database limit. The missed alert lengthened the outage but did not cause it.

5. What went well, and what did not
Went well
Did not go well
Incident declared and roles filled within 8 minutes of the page
Saturation alert went to email, so 15 minutes passed before anyone looked
Status page kept venues informed from 18:46
Restart tried before checking what had changed
Rollback cleared the timeouts at once when applied
No dashboard tied deploys to connection count
6. Actions
No.
Action
Owner
Due
1
Route the connection saturation alert to the pager
On call lead
Done 23 Sep
2
Add a deploy check that fails if workers times pool size exceeds 80% of the database limit
Platform engineer
Fri 9 Oct 2026
3
Move worker count and pool size into one configuration file
Platform engineer
Fri 16 Oct 2026
4
Put a connection pooler in front of the database
Database engineer
Fri 13 Nov 2026
5
Show deploy markers on the database dashboard
Observability engineer
Fri 2 Oct 2026
6
Add "what changed in the last hour" as step one of the incident checklist
Incident commander
Wed 30 Sep 2026

Actions are tracked in the engineering board under PM-2026-09 and reviewed at the weekly reliability meeting until closed. Action 4 removes the cause class entirely; actions 2 and 3 stop a repeat until it lands.

7. What customers were told

The status page carried four updates between 18:46 and 19:58. Every affected venue received an email on Wednesday 23 September with the times, the cause in plain terms and the service level credit where its plan has one. Venues were also sent a list of the booking attempts that failed and did not return, so they could contact those diners.

Time
Status page update
18:46
Investigating: bookings are failing for some venues
19:15
Identified: a configuration change is the likely cause; rollback under way
19:36
Monitoring: bookings recovering after the rollback
19:58
Resolved: bookings working normally since 19:51; a full review will follow

Section by section

What each section is for, so you can keep the ones you need and drop the rest.

Reference block and blameless note
Incident number, date, review meeting, author and status, and how the review treats people.
1 and 2. Summary and impact
The change and the connection arithmetic, then four impact figures and availability against target.
3. Timeline
Ten timed events from approval to resolution, with detection and diagnosis times.
4. Five whys and root cause
Five numbered whys down to two separated settings and no total check.
5. What went well and what did not
Three of each, side by side.
6 and 7. Actions and customer communication
Six actions with owners and dates, then status page updates and the venue email.

How to adapt this document

First, for a non technical failure, such as a missed delivery run or a payroll error, keep the same skeleton and swap the system terms for process steps; the timeline and five whys work just as well. Second, for a smaller incident, merge the summary and impact into one paragraph and drop the communication section, but never drop the actions table. Third, if your organisation has service level targets, always state the availability for the period against the target, because that number tells leadership how much the incident cost in terms they already track.

What makes this document work

The arithmetic of the failure is on page one

The summary states the sum that broke the system: 48 workers at 20 connections each tried to hold 960 against a limit of 500. Anyone reading it understands the failure without knowing the code, and the deploy check in action 2 tests exactly that product.

The timeline shows where the time went

Ten timed rows show the alert firing at 18:02 into an email inbox, the page at 18:19, a restart at 18:46 that treated the symptom, and the link to the deploy at 19:12. The note under it turns that into 15 minutes to detect and 53 to diagnose.

The whys stop at a condition, not a person

The third why lands on two configuration files that live apart and a review that showed only one. The root cause is that no check stops the total passing the limit, which is fixable. A why that ends at a name fixes nothing.

Questions people ask

What does blameless mean in a post mortem?

It means the review assumes people acted reasonably with the information they had, and looks for the conditions that let a normal action cause harm. People are named by role, not singled out. Blame makes people hide mistakes, which hides the next outage's cause. Blameless does not mean nobody is accountable for the actions that follow.

When should a post mortem be written?

For every incident above an agreed severity, and for near misses that could have been serious. Hold the review within a few working days, while memories and logs are fresh; here the outage was on a Tuesday, the meeting on Friday and the report published the following Monday. Waiting weeks loses detail and momentum.

Is five whys always five?

No. Five is a rule of thumb that pushes past the first, obvious answer. Stop when you reach a condition the organisation can change, which may take three whys or seven. Watch for branches too: the missed alert here is a separate chain that lengthened the outage, so it gets its own action.

Who should attend the post mortem meeting?

The people who responded, the owner of the system that failed, someone from customer support, and a facilitator who was not deeply involved. Fourteen people attended this one. The meeting checks the timeline, agrees the root cause and the actions; the written report records what was agreed, with owners and dates.

Should a post mortem be shared with customers?

Many software companies publish a shorter public version: what happened, the impact, the cause in plain terms and what is changing. Keep internal details such as system names and staff roles out of it. Customers mostly want to know it will not happen again and whether they are owed a credit, as the venue email here explains.

How do you make sure the actions get done?

Give each one a single owner and a date, track them where engineering work is already tracked, and review open actions at a regular meeting until closed. Rank them: here action 1 was done the next day, while action 4 removes the whole class of failure but takes weeks, so actions 2 and 3 cover the gap.

Build your own in about a minute

The button below opens the generator with this use case already described. Change the wording to match your own, generate, then edit anything you like.

Make one like this

Other document examples

Want the steps in the builder? Read Create a document with AI, then Document builder components. For everything this generator can do, see the document maker.

Sources

Written and checked by the OneCraft team. Last checked .