Reliability on a growing platform

Business context

A multi-tenant product had moved from a single deploy script to several squads sharing one production account. Incidents were taking too long to classify, and releases clustered on evenings because nobody trusted the daytime path.

Technical challenge

Create a delivery and operations baseline that squads would actually use, without pausing the roadmap or inventing a platform team that the company could not staff.

Constraints

• No freeze on feature work.
• Existing cloud vendor and Git host stay in place.
• On-call is a duty for product engineers, not a separate NOC.

Approach

• Map the unofficial release path, including the two people who still held production keys.
• Introduce infrastructure as code for the environments that already existed, rather than a greenfield redesign.
• Split CI into build-once artefacts and an explicit promote step.
• Define three user-facing symptoms as the only pages that wake someone at night.
• Add a tenant identifier to logs and traces so support can isolate a customer without a production query.

Technologies in play

Named only at the category level. We do not list products as if they were a certified stack for a fictional client.

• The organisation’s existing cloud and Git platform
• Infrastructure as code already familiar to the team
• The observability stack already licensed

Implementation notes

• Week one produced an accurate diagram of the current path — including the embarrassing parts.
• Shared modules covered network, identity, and the runtime used by all services.
• A canary on the highest-traffic read path was rehearsed, including rollback.
• Runbooks were written from the last three real incidents, not from a template.

Outcomes

No measured client result is shown here. In a live engagement we would only publish restore-time, change-fail, or adoption numbers that the customer has approved.

Lessons

• A platform nobody uses is just another system to maintain. The golden path has to be the easy path.
• Alerting is a product. If it cries wolf, engineers will mute it.
• Tenant-aware observability is a support feature, not a nice-to-have for enterprise deals.