Onboarding the Pilot Team for Real

Part 6 of “The Landing Zone That Survived”, a year in the life of a New Zealand platform team, told from fakey.xyz. Fictional organisation, aggressively fake people, realistic problems.

Recap. Part 5 left a working vending path, policies in audit mode, and Serge’s written deal: the pilot’s applicable ENFORCE controls activate this increment, before production. This post is the cutover that tests whether that deal holds.

There is a difference between a customer who is using your platform and a customer who is depending on it. The first is a relationship. The second is an obligation, with an on-call roster on one end and a production incident on the other.

Increment Two is where fakey.xyz’s platform team crossed that line, and, this is the honest part, they crossed it without noticing, which is how it happens everywhere.

The Moment They Crossed the Line Without Noticing

It happened on a Tuesday, in a routine conversation that Fakey McFakerson later reconstructed in the post incident review as “the moment we became accountable for uptime, via email.”

Netty Latency’s team had finished their fraud scoring engine’s first phase and were planning the production cutover. Netty posted in the platform channel:

“Plan: production environment next sprint, cutover the scoring service the week after. One question for the platform crew: what is your change freeze over Christmas, so we can plan around it?”

Nobody on the platform team had a change freeze, because nobody on the platform team had formally considered that other people’s production workloads would now be frozen around theirs. The platform had gone from being a thing teams deployed into, to being a thing their production depended on, a firewall in the traffic path, a workspace their logs flowed to, a vending pipeline their DR plan would someday reference.

Netty had assumed the platform had production SLAs, change management, and a freeze calendar. The platform had none of those things. It had a sticky note and a really good vending form.

Fakey’s response was to call a meeting titled, honestly, “We Are Now Part of Production”. Serge Secure brought the Part 5 commitment into the room. Estate-wide enforcement waves still begin in increment four. Netty’s applicable ENFORCE set cannot wait that long if production lands this increment. He signs a dated exception: those controls activate before cutover, named, owned, reviewed. It is the second contentious entry in the exception register, right after the corridor. The increment-four date holds for everyone else. The promise is not broken. It is scoped.

The first whiteboard item was everything the platform team was now implicitly promising:

• Hub network availability. Netty’s scoring traffic transits it. Down hub equals down fraud scoring. The 8ms SLO dies in front of customers.
• Firewall and the low latency corridor. Part 4’s measured exception is now load bearing production infrastructure.
• Central Log Analytics. Serge Secure’s BS11 evidence, the CPS 234 pack for MegaCorp Megalodon’s Australian regulated scope, and the Privacy Act detection capability all live here. Its failure is a compliance incident, not just a technical one.
• The vending pipeline. DR rebuilds, new environments, and the migration waves all flow through it.
• The platform team itself. 1.6 FTE of platform capacity. Who is paged? Paged for what? Paged from where?

Lesson 1, The platform becomes production grade the moment the first production workload plans its cutover, not when you finish your checklist. Find that moment deliberately, or it will find you in an incident.

The Corridor Meets Reality, or: Measured Once, Measured Again

The low latency corridor from Part 4, the policy gated, logged, quarterly-reviewed exception for the fraud scoring stack, survived its first production test. Barely, and not the way anyone expected.

The proof of concept had measured firewall hop latency: ~1.9ms per hop. What the POC had not measured was the corridor’s behaviour under real production traffic patterns, specifically, Netty’s scoring engine’s behaviour when the transaction stream bursts (peak: 11:45am daily, when NZ’s payment volumes spike before lunch).

The first week of production-load rehearsal surfaced two findings:

Finding one: the corridor itself was fine. Direct peered traffic between the scoring subnets and the transaction stream subnet held p99 well inside budget. Part 4’s decision to measure, gate, and monitor the exception was vindicated. Serge Secure noted this in the exception register’s first quarterly review with the words “exception performing as designed”, four words that did more for the exception register’s credibility than any policy document could have.

Finding two: everything around the corridor was not fine. The scoring engine’s telemetry side-channel, model metrics, feature-store stats, health probes, was configured to flow through the hub firewall to the central workspace, and at peak load that telemetry volume saturated the firewall’s throughput enough to add jitter to other spokes’ inspected traffic. The corridor had protected Netty from her own firewall; it had not protected everyone else from Netty.

The fix took two shapes, and both are the instructive part:

1. Architectural: telemetry moved off the firewall path entirely, diagnostics flow over Azure backbone to the workspace via private endpoints, never via hub-inspected routing. This is now a documented platform standard: observability traffic never rides the inspection path.
2. Process: the exception register’s quarterly review template gained a line item, “adjacent impact assessed?”, because an exception can be healthy while quietly degrading its neighbours.

Netty’s contribution to the review meeting, delivered with her characteristic inability to soften anything:

“Your firewall is a shared road. My corridor is a bus lane. This week my bus was fine and I congested your road for everyone else. If the register only checks the bus lane, you are measuring the wrong thing.”

Fakey stole the metaphor on the spot. It is now in the platform documentation.

The Cutover, and the Difference Between a Runbook and a Plan

The production cutover for the scoring service was scheduled for a Wednesday evening, out of NZ business hours, with a rollback plan. What follows is not the incident, that is Friday’s story. Wednesday went almost smoothly, and the almost is the lesson.

The cutover ran: DNS shifts, traffic ramp, the 11:45am Thursday peak as the real acceptance test. The scoring service held its SLO through the peak, p99 inside budget, corridor stable, telemetry flowing. Netty declared it “production, tentatively, until something surprises us.”

Then a quieter surprise, recorded here because it explains Friday: the scoring service’s scale-out hit a regional quota in Azure NZ North that nobody had checked. The service scaled to its ceiling at 2am Thursday and threw alerts. Netty’s on-call engineer was paged, because the workload team had sensibly set up their own alerting. The 2am page went to Robbie’s personal phone, which is precisely the finding Friday later formalised: that was not a system.

Quota raised by 6am. Service healthy. No customer impact. Fakey’s post cutover review produced a one-line addition to the vending pipeline’s checklist:

“Quota and capacity check for the requesting team’s stated scale at vending time, not at 2am.”

The deeper lesson, which Fakey wrote into the design document’s open questions:

A runbook tells you what to do. A plan tells you what you should have checked before you needed the runbook. The cutover had a rollback runbook. What it lacked was a pre cutover checklist owned by both teams, platform and workload, covering the shared-fate items (quota, scale limits, support entitlements, regional capacity) that neither team owns alone.

The joint cutover checklist, platform sections and workload sections, each team signing the other’s, became increment two’s most-reused artefact, and the template the migration waves will follow.

Friday, 4:52pm: The Incident

And then came Friday.

At 4:52pm, after most of Wellington had mentally clocked off, a pull request merged to the platform repo and the pipeline deployed a change to the DNS private resolver configuration in the hub. The change was small, well reviewed, and correct against its intent. It was also, because inbound endpoint subnet references were evaluated in a dependency order the plan did not capture, applied out of order, briefly removing DNS resolution for two private zones, including the zone the fraud scoring service’s feature store lived on.

For eleven minutes, the scoring service could not resolve its feature store. Netty’s alerting fired at 4:58pm. Robbie was walking out the door. The page hit his personal phone, the same informal path as 2am. He identified the cause by 5:03pm and rolled the change back by 5:09pm. Full recovery. No customer-visible impact. The scoring service’s circuit breaker degraded gracefully to cached features, which is exactly what Netty’s team had built it to do, and is a story for their blog post, not this one.

Eleven minutes. Here is what made those eleven minutes the most valuable of the increment, the retro on the following Monday, run by Fakey with a strict no blame rule, produced five findings that reshaped the platform:

1. The blast radius of platform changes is every workload, simultaneously. A workload team’s bad deploy breaks their app. A platform team’s bad deploy breaks the platform’s promise to everyone. The pipeline now requires platform infrastructure changes to declare, in the PR template, their blast radius, and anything touching DNS, network routing, or identity requires a second reviewer outside the platform team. The reviewer pool: Netty, Vera Transaction, and Hugh Documents. Your customers reviewing your changes is the operating model working, not the operating model failing.

2. “Deployed via pipeline” and “deployed correctly” are different guarantees. The change was exactly what the repo said. The repo was wrong about how it would apply. Terraform’s ordering behaviour earned its own entry in the platform’s known gotchas document, and the pipeline gained a post apply smoke test: resolve a canary FQDN in each critical private zone, every deployment, automatically.

3. Nobody outside the platform team knew how to reach the platform team. Netty’s engineer’s first instinct was the platform channel. Fine. Their second thought was “if this had happened at 3am, who do I even page?” Thursday they already had the answer: Robbie’s personal number. That is not a system. This finding feeds the operating model work that becomes Part 7’s crisis. Its first output was immediate: a platform contact policy. Channel for business hours. A cheap SaaS pager, stood up in a day, for platform P1s. An on-call rotation that, given 1.6 FTE of platform capacity, was formally declared to be “Robbie, primary; Fakey, secondary; both of us hoping the bus factor holds until hiring.” Fakey flagged the rotation’s fragility to Barry Bigboss in writing the same week. It stayed on the record. The platform now had a sticky note, a vending form, and a pager.

4. The workload teams’ grace under fire proved the platform’s design. The scoring service’s circuit breaker degradation, Netty’s alerting firing within six minutes, the clear separation between “platform broke” and “workload coped”, every investment the pilot partnership had made paid for itself on Friday evening. Fakey’s note in the retro, underlined twice: “The pilot team made us look good in our own incident. That is what a design partner is for.”

5. Serge asked the compliance question, and it was the right one. In the retro, Serge Secure’s only input was: “Are these eleven minutes notifiable under the Privacy Act?” The answer, no, no personal information was exposed, only unavailable, and the service degraded without data loss, took twenty minutes to establish confidently, because the platform’s logging covered the failure but the breach-assessment reasoning lived in nobody’s head. The fix: a one-page incident-to-Privacy-Act decision tree, drafted by Serge and Locksley Keymaster, now attached to the platform’s incident process. Detection capability, Discovery Week’s requirement, had been built. Assessment capability had not. Now it exists, and it exists because security was in the room for the technical retro, which is why security is in the room for every retro.

The Increment Review: New Metrics for a New Relationship

The increment review with Barry Bigboss carried a new metrics table, because the platform’s obligations had changed:

Hugh Documents’ onboarding is the first repeat customer, the first test of whether vending generalises beyond the design partner. Their identity-document workload landed in Azure NZ North, with the classification question from Discovery Week filled in by someone who had now seen the scheme. No corridor exception. No 8ms budget. The residency and tagging path from Part 3 held. That is the quieter win of the increment, and it belongs in the table because it is the first time the platform was used by a team that did not help design it.

Barry Bigboss looked at the “11 minutes” line for a long moment, and then asked, and this is the reason Barry is the CEO in this story and not a villain, “Which of these numbers is the next one to get worse before it gets better?”

Fakey’s answer: “The on-call rotation. Two people on a pager is not a rotation, it is a countdown. I need it funded in the annual plan, and I need you to hear that the platform’s reliability is now coupled to one person’s continued employment.”

Barry wrote it down. Whether it gets funded is Part 7’s problem.

The Steal This Checklist

• Identify the moment you became production, the first workload planning a cutover around you. Formalise the implicit promises that moment creates.
• Observability traffic never rides the inspection path, and your exception register assesses adjacent impact, not just the exception itself.
• Joint cutover checklists, platform and workload teams each sign the other’s; shared-fate items (quota, capacity, support) checked at vending time.
• Platform PRs declare blast radius; network/DNS/identity changes get a reviewer from a workload team.
• Post apply smoke tests in the pipeline, canary DNS resolution, connectivity probes, every deployment.
• A paging path that exists before the first 3am incident, plus a named, written-down rotation, with its fragility escalated in writing.
• Security in every retro, the “is this notifiable?” question gets asked by someone whose job it is to ask it.
• A tabletop tested incident-to-Privacy-Act decision tree, detection is not assessment.
• Report incidents to the exec honestly, with the “which number gets worse next” answer ready.

Next in the Series

Part 7, “The Operating Model, or: Who Is Watching the Watchers.” The platform now has production obligations, an on-call rotation of two, a monitoring estate generating real alerts, and no formal operating model. When Tessa Spreadsheet’s cost alerts, Serge’s detection alerts, and genuine platform health alerts all land in the same silent inbox, fakey.xyz discovers the difference between having monitoring and having an operator, and Barry Bigboss finally has to choose between funding the platform and testing Fakey’s resignation letter draft.

When did your platform cross the line from used by production to depended on by production, and did you notice at the time, or in the retro?