Runbook: roll back a bad deploy

Written before it was needed, which is the only time worth writing one.

Purpose

A deployment is serving something broken and the fastest correct move is to put the previous build back. This is not for a broken build that never deployed; that one is a red pull request, not an incident.

Preconditions

You can reach the hosting dashboard for the project.
You know which deployment was good. The status page records a commit per verified host, which is often enough to find it.
Someone is writing down what is happening. If nobody is, that is step zero.

Steps

1. Record the start time

date -u +%Y-%m-%dT%H:%M:%SZ

Expected: a UTC timestamp you have written down.

Rollback: none needed. Knowing when this started is free and you will want it in an hour.

2. Confirm the failure is real, from outside

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://{{host}}/

Expected: a status that is not 200, or a time that explains the report. If you get 200 quickly, the problem is not the deployment and this is the wrong runbook.

Rollback: none. You have not changed anything yet.

3. Identify the last known good deployment

Find the deployment that was serving before the current one, and note its commit.

Expected: one commit sha you can point at.

Rollback: none.

4. Promote the previous deployment

Promote the known-good deployment to production in the hosting dashboard.

Expected: the dashboard reports the older deployment as current.

Rollback: promote the deployment that was current when you started. You recorded it in step 3, which is why step 3 exists.

5. Verify from outside again

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://{{host}}/
curl -sSI https://www.{{host}}/ | head -n 2

Expected: 200 on the apex, and 301 from www to the apex. Check both: a rollback that restores the app but loses the redirect is half a rollback.

Rollback: if this still fails, the deployment was not the cause. Stop rolling back and start investigating.

curl -sS -o /dev/null -w '%{http_code}\n' https://{{host}}/favicons/og-image.jpg

Expected: 200.

Rollback: none. This is a check, not a change.

This step exists because of a real bug. qr-forge served a 404 on /favicons/og-image.jpg after its share-image and icon set merged. The set was corrected and re-verified, and the path now returns 200. (F-1, source: app-inventory sheet, qr-forge row notes, verified 2026-09-21)

7. Open the fix

The rollback bought time; it did not fix anything. Open a pull request against the branch that broke, with the failing check reproduced first and passing after.

Expected: a pull request whose CI is green.