Where Security Meets Reliability: Lessons from Google

It Started With a Bus WiFi Password

On September 27, 2012, a routine company-wide email at Google — announcing a changed WiFi password for campus shuttle buses — triggered a cascade of failures so absurd, so human, and so instructive that it ended with an engineer in Australia drilling open a safe with a power drill.

The email went out to thousands of employees. The resulting traffic spike overwhelmed Google's internal password manager, a tool originally built for a small team of sysadmins. The primary replica crashed. The load balancer failed over to the secondary. That crashed too.

The on-call engineer had never handled this service before. Restarting it required a hardware security module (HSM) smart card — stored in safes at various global offices. The combination to the safe? Stored in the now-offline password manager. A colleague in California had the combination memorised and retrieved a card — but the service still wouldn't start. The Australian team eventually drilled open their own safe to grab another card. Same result.

After hours of chaos, the team discovered the smart card had simply been inserted upside down. The green light on the reader meant nothing useful. Flipping the card restarted the service. Outage over.

This story is the opening of Google's open book, Building Secure and Reliable Systems, and it perfectly illustrates the book's central thesis: security and reliability are deeply entangled — and optimising for one without considering the other can be catastrophic.

Reliability vs. Security: Two Different Assumptions

The core distinction between the two disciplines isn't about what can go wrong — it's about why.

Reliability engineering assumes things will go wrong by accident: a bad deployment, a hardware fault, an unexpected traffic spike.

Security engineering assumes someone may be actively trying to make things go wrong.

That difference changes everything about how a system should respond to failure. A door lock designed for reliability might fail open during a power outage — letting people escape safely. A lock designed for security must fail closed, or an attacker simply cuts the power.

This creates real design tradeoffs:

• Redundancy improves reliability, but every redundant path is a new attack surface. An adversary only needs to find a vulnerability in one of them.

• Incident response benefits from many eyes in a reliability crisis — but a security incident demands need-to-know secrecy, so the attacker isn't tipped off to the recovery effort.

The CIA Triad: Three Properties, Two Lenses

Both security and reliability converge on three fundamental system properties — Confidentiality, Integrity, and Availability (the CIA triad). But they approach each from a different angle.

Confidentiality — A stuck push-to-talk microphone in a cockpit can broadcast private pilot conversations with no adversary involved. A simple hardware flaw creates a breach.

Integrity — In 2015, Google SREs discovered cryptographic check failures caused by cosmic-ray-induced single-bit memory flips. Not an attack — but completely indistinguishable from one until investigated.

Availability — When a magnitude 4.5 earthquake struck the San Francisco Bay Area in 2019, the resulting search traffic spike looked exactly like a distributed denial-of-service (DDoS) attack on Google's infrastructure. Context is everything.

What Security and Reliability Share

Despite their differences, both disciplines are emergent properties — they don't live in a single module or config file. They arise from the interaction of every part of a system. You cannot bolt them on later. They must be designed in from day one.

Here are the key principles they share:

1. They're Invisible Until They Break

Both properties are easy to defund because they're invisible when working well. But failures are expensive. In 2017, disclosed data breaches contributed to Yahoo's acquisition price dropping by $350 million. That same year, a power failure caused Delta Airlines to cancel nearly 700 flights and delay thousands more.

2. Simplicity Is a Superpower

A simpler design is easier to audit, easier to reason about, and harder to exploit. Every unnecessary component is both a potential failure point and a potential attack surface.

3. Systems Always Evolve — and That's Risky

In 2006, a developer removed two lines of code from OpenSSL to silence a debugging warning. The change reduced the random number generator's entropy so drastically that cryptographic keys became brute-forceable. The bug wasn't caught for nearly two years.

In 2018, a small change to a YouTube logging library — which looked fine in review and passed all tests — caused servers to run out of memory under production load, taking YouTube offline globally for over an hour.

4. Design for Resilience, Not Just Prevention

Systems must be built to absorb failure. From a reliability standpoint, that means handling load spikes and component failures through redundancy and distinct failure domains. From a security standpoint, it means assuming breach and limiting blast radius — through least privilege, compartmentalised permissions, and defense in depth.

Google uses multi-party authorisation for sensitive operations, ensuring no single insider (malicious or simply mistaken) can cause catastrophic damage alone.

5. Logging Is Double-Edged

Complete logs accelerate recovery — but they're also a target for adversaries. Logs must never contain credentials or personally identifiable information. And logging itself can introduce reliability problems: the YouTube 2018 outage started with a logging library change.

6. Practice the Crisis Before It Happens

Google models incident response on FEMA's Incident Command System — clear chains of command, documented playbooks, and regular drills via their Disaster Recovery Testing (DiRT) program. When crisis hits, it is too late to design the response.

The Takeaway

Every element of the bus WiFi incident — the underpowered service, the security measures that blocked recovery, the engineer with no runbook, the safe whose combination was locked inside itself — made sense in isolation. That's the point.

Security and reliability failures rarely come from obvious negligence. They come from the compound interaction of individually reasonable decisions, made without considering the other discipline.

The principles are the same regardless of what you're building — whether it's a payment platform or a personal blog. Build simple. Design for failure. Assume an adversary. Practice recovery before you need it.

And label which way the smart card goes in.

Based on Chapter 1 of Building Secure and Reliable Systems by Adam Stubblefield, Massimiliano Poletto, and Piotr Lewandowski — published open-source by Google.