The 2am Page: Operating Model Tested

Part 9 of “The Landing Zone That Survived”, a year in the life of a New Zealand platform team, told from fakey.xyz. Fictional organisation, aggressively fake people, realistic problems.

Recap. Increment four paid Serge’s first ENFORCE wave and Penny’s four questions. The estate still runs on 1.6 FTE of platform capacity (Robbie at 1.0, Fakey at 0.4, the security engineer at 0.2; Tessa’s 0.1 sits with finance), plus contractor Max Overhead, whose conversion waits on the annual plan Barry promised in Part 7. The plan has not landed. This is the post where every artefact gets tested at once, at 2am, on a Friday, by a P1.

Every document in your platform programme is a hypothesis about a future bad night. The runbooks, the escalation table, the two way contract, the decision tree, the exception register, all of it is a bet that says: when the bad night comes, these artefacts will be enough for the person standing in the dark, who is tired, and who may not be us.

At 2:11am on a Saturday in November, Wellington time, the bet paid out. Mostly.

2:11am: The Page

The sequence began at 2:08am with an Azure Firewall Premium health event in the hub, a control plane failover that Azure’s own SLA describes as transient and self healing. In most hub deployments, that is a footnote. In fakey.xyz’s, it triggered a cascade the team had modelled but never lived.

The firewall failover dropped and re-established its data plane. During the ninety seconds of instability, BGP sessions from the Petone colo VPN flapped. The VPN re-established cleanly, but the failover’s route table convergence left one of the two colo sites’ routes preferencing a stale path. The surviving path traversed the remaining colo tunnel, which has a lower MTU the converged routes did not account for. Traffic did not stop. It degraded, packet fragmentation, retransmits, latency that swung between acceptable and terrible depending on the flow.

By 2:09am, Netty Latency’s fraud scoring service, which draws transaction context from the colo side transaction stream during the migration transition, was watching its p99 breach its SLO for the first time in production. Her alerting, tuned to a razor’s edge by a team that does not tolerate ambient anxiety, fired.

Detection is 2:08. The SLO breach is 2:09. The pages, and Max’s ack, are 2:11.

By 2:11am, three pages fired in sequence:

1. Azure monitor → platform health stream → the pager → Max Overhead.
2. Netty’s SLO alert → her team’s on-call, her data scientist, the former workstation operator, now properly on the workload rotation, who, per the two way contract from Part 7, checked the platform status channel first, found nothing posted, and paged the platform P1 path.
3. The composite platform alert (firewall health + colo VPN error rates + workspace ingestion anomalies) → the platform P1 path → Max again, now with severity confirmed by a second signal.

The triple-signal design worked. No single page could have told Max what was happening. Three of them, arriving within four minutes of the 2:08 detection, gave him a picture before he’d finished making tea: firewall event, VPN flap, SLO breach on a migrated workload, residency-relevant data in the degraded path.

That last clause is why this incident mattered. The degraded path carried payment transaction data in flight between the colo and the cloud. Which meant, potentially, the Privacy Act clock.

2:11am to 2:40am: What “the Operating Model Was Real” Actually Looks Like

Here is the timeline, reconstructed from the incident log, and the reason each step worked is annotated, because the why is the stealable part.

2:11, Max acknowledges the page. Three minutes from the 2:08 detection. Within the 15-minute commitment. He is awake because the pager is loud and because contractors who take on-call take it seriously. He opens the platform runbook titled “Hub connectivity degradation”, written by Robbie Deployment, tested once in a game day, and, this is the sentence from the post incident review that made Fakey’s month, “runnable top to bottom without improvisation until step 7.”

2:14, Max posts to the platform status channel. First public communication, three minutes after ack. The two way contract’s promise, communicate during platform-caused or suspected-platform incidents, was kept before anyone asked. Netty’s on-call engineer, watching the channel instead of isolated in his own incident, stood down from investigating his own service and began capturing SLO evidence. The boundary between “platform broke” and “workload coped”, rehearsed in the DNS retro, held under pressure at 2am. This is what a two way contract is for. Nobody had to negotiate roles in the dark.

2:19, Max reaches step 7 of the runbook: the step that requires judgement. The route table fix is documented, but the runbook honestly flags it: “Applying the corrective route during traffic may cause a second, briefer flap. If workload SLO impact is active, coordinate before applying.” Max did not improvise. He did what the runbook told him to do at step 7: he called Netty’s on-call engineer. Direct, human to human, at 2:19am. They agreed: the scoring service’s circuit breaker was holding in degraded mode; a 60-second second flap was survivable; the degradation as it stood was not, because p99 was drifting toward the SLO boundary. Second flap approved. Applied at 2:23. Routes converged by 2:25. Firewall stable. Colo traffic clean by 2:31.

2:26, Netty’s on-call, unprompted, posted the sentence that the entire pilot partnership investment had been building toward:

“SLO recovering, p99 back inside budget, circuit breaker reopening to full feature set. Whatever you did, do more of it. Also: logging the SLO excursion window for our post incident review, flagging that the degraded path processed transaction context, so your privacy assessment may want this.”

Read that again. A workload engineer, at 2:26 in the morning, voluntarily flagging the compliance dimension of his own incident to the platform team. That reflex, the instinct to hand the privacy question to the people whose job it is, did not exist at fakey.xyz nine months ago. It exists because the Part 6 retro put security in every retro, and because the decision tree from Discovery Week gave everyone a shared map of what “might be notifiable” means.

2:34, Max, following the runbook’s final section, invoked the incident-to-Privacy-Act decision tree. It was 2am. He was a contractor three months in, still waiting on the annual plan that was supposed to convert him. And the artefact worked exactly as designed for exactly this person: a competent stranger, tired, with no tribal knowledge. The tree asked: was personal information exposed? accessed? processed in an unauthorised location? His answers, from the monitoring estate: in flight data was encrypted end to end (TLS the whole way, the fragmentation degraded performance, not confidentiality); no unauthorised access in the logs; processing locations unchanged. Preliminary assessment: no notifiable event; assessment recorded; escalation to Serge Secure flagged as required in business hours.

He filled in the assessment form. He paged nobody at Serge’s house. He wrote the incident timeline while it was fresh. At 2:47am he posted: “P1 resolved. Recovery complete 2:31. Privacy preliminary: non notifiable, recorded, Serge to review Monday. Writing it up now.” Then he went back to bed.

Total platform impact: 22 minutes of degraded colo transit, zero data exposure, zero customer-visible failures, the scoring service’s degradation architecture and the platform’s fast recovery together kept the SLO excursion inside Netty’s design tolerance.

The runbook, the contract, the decision tree, the paging path, the status channel, the monitoring signals: every artefact from Parts 4 through 7 did its job on the same night, most of them simultaneously, most of them for a person who had never lived through the events that created them.

That is the whole argument of the year, in one Saturday morning.

The Almost

It went well. Almost entirely. And the almosts, found in Monday’s review, run by Fakey with her no blame rule and Serge in the room, are the next layer of the operating model, because a system’s second-order gaps only appear after its first-order gaps are fixed.

Almost #1: The escalation table had a hole where “the platform team is the incident” should be. Max’s P1 escalation path was: runbook → judgement step → call the workload → resolve → escalate to Fakey after the fact. What it lacked: a threshold for waking Fakey up. The route fix had worked first try, but if it had not, Max’s documented options at 2:30am were “keep troubleshooting” or “wait for business hours.” For a P1 with residency implications and a live SLO breach, neither was right. The fix: an explicit second escalation trigger in the runbook: “if the corrective action fails, or if impact exceeds 30 minutes, wake the secondary on-call regardless of hour.” Fakey’s comment on the change: “Waking me for nothing costs one bad night’s sleep. Not waking me when you are stuck costs the platform’s whole credibility. Err toward waking me.”

Almost #2: Serge opened Max’s assessment at 9:40am Monday. That was the counter-signature: forty minutes after business hours started, on first read. Eight hours too late for Serge’s comfort, and years too early for the process to be trusted without a second pair of eyes. The preliminary assessment said non notifiable, and it was right. What his review surfaced was the real issue: a contractor had made a preliminary Privacy Act determination alone, at 2am, and the process had no verification step. The assessment was correct this time. The process did not guarantee it. The fix, proposed jointly by Serge and Max, which is worth noting, is a second-signature rule: preliminary non notifiable assessments made under time pressure get a counter-signature from the privacy function within 12 business hours, every time, no exceptions. Serge did this one because he is Serge. The next one must not depend on him checking Slack.

Judgement calls made in the dark get daylight reviewed.

Almost #3: The status channel was great for the people watching it, and invisible to everyone who was not. Netty’s team watched it because the DNS retro had taught them to. Vera Transaction’s team, whose payments API rides the same colo transit, found out about the incident on Monday. Their workload happened to be unaffected (their traffic takes the direct path, not the corridor), but they could not have known that at 2am, and the contract promises communication to affected workloads. The platform had no way to say “affected: X, Y; assessed and unaffected: Z.” The fix: the status channel gained a reach matrix, a living map of which workloads depend on which platform components, maintained as part of vending, so the first communication of any incident includes an honest affected/unaffected call. Fakey’s note: “In an incident, silence is information too. Make sure the silence says what you mean.”

Almost #4, the small human one: Max, four minutes into the incident, had spent ninety seconds looking for the on-call contact numbers, Netty’s on-call, Fakey’s secondary, because the runbook referenced “call the workload on-call” without saying how. The numbers existed in three places, one of which was out of date. The fix took twenty minutes (a single contacts page, generated from the on-call tooling, linked from every runbook) and the out of date entry was, of course, a number that would have mattered.

The Post Incident Review: The Artefact That Ties It Together

The PIR was published five business days later, per the two way contract’s promise, and it was structured the way the series has taught: timeline, five findings, owner and date on each fix, and a section that no PIR template in the industry includes and all of them should:

“What this incident validated.” The two way contract (roles held under pressure, zero negotiation). The decision tree (first 2am use, correct, now with second-signature). The paging path (3-min ack). Cross-redundant signals (the composite alert caught what no single monitor would have). The workload team’s degradation architecture and compliance instinct. The runbook’s honesty about its own limits, step 7 worked because it admitted step 7 was a judgement call.

The final line of the PIR, from Fakey:

“Nine months ago this incident was impossible, not because the network was better, but because there was no platform, no pager, no contract, and no Netty to design a circuit breaker. The incident happened because the platform is real. The outcome happened because the operating model is too.”

Serge Secure’s contribution to the PIR, filed into the CPS 234 evidence pack alongside it, the control whose operating effectiveness this evidences, the one the risk committee scoped Serge to in Discovery Week, for MegaCorp Megalodon’s Australian regulated counterparties:

“Incident detected by automated monitoring at 2:08. Acknowledged at 2:11. Workload impact contained by workload resilience. Privacy assessed same-night with daylight counter-signature at 09:40 Monday. Post-incident report within five business days. This is the control operating as intended, evidenced.”

CPS 234 evidence, it turns out, is not a quarterly document. It is a bad night, handled well, written down.

Fakey closed the increment-four review with a line that does not belong in a PIR and does belong on the next agenda: Max is still a contractor. Barry’s annual plan promise is past its trigger. That conversation opens Part 10.

The Increment Review Metrics

The Steal This Checklist

• Treat every artefact as a bet on a bad night, runbooks, contracts, decision trees. Then schedule the night: game days find second-order gaps before real 2ams do.
• Runbooks must mark their own judgement steps, and the judgement step’s protocol is coordinate with the affected workload, human to human, not improvise alone.
• A second escalation trigger in every P1 runbook, “if the fix fails or impact exceeds X, wake the secondary regardless of hour.” Waking someone for nothing is cheap. Not waking them when stuck is not.
• 2am preliminary privacy assessments get daylight counter-signatures, every time. Judgement made in the dark gets reviewed in the light.
• An incident reach matrix, which workloads depend on which platform components, maintained at vending time, so your first status post says who is affected and who is not.
• Communicate before you are asked, the first channel post inside 5 minutes buys you patience you will otherwise never get back.
• PIRs include “what this incident proved”, not just what it broke. Your controls’ first real test is evidence, and evidence is the product.
• Verify in flight encryption assumptions in your logs, not your architecture diagrams, the residency/confidentiality assessment at 2am depends on it.

Next in the Series

Part 10, “The Second Team, and the Difference Between a Platform and a Custom Build.” Increment four’s ENFORCE wave is already live. Hugh Documents’ production cutover is where those controls meet the strictest data classification in the company, and he immediately demands things the golden path never imagined. The exception register has to work for a difficult ask, not only for a darling. Some asks you build, some you route, and some you decline, in writing, with love. Hugh still wins something real in the exchange. Barry’s funding promise comes due in the same increment.

Has your platform had its 2am yet, the night the runbooks either held or did not? What did the morning after retro find that no game day had?