Chaos Monkeys
In 2010, Netflix engineers did something that would get most of us sacked: they deliberately crashed production servers. On purpose. During business hours.
They called it Chaos Monkey.
The name stuck, but the principle behind it was deadly serious. It wasn’t about being clever or edgy. It was about accepting the harsh truth that things WILL break. The only question is whether you’ll be ready when they do.
Traditional thinking says protect everything, wrap it in cotton wool, create elaborate change control processes that take six months to approve a DNS update. We’ve all been there. Those 3am calls when something that “couldn’t possibly fail” inevitably does.
Netflix flipped the script. Instead of pretending they could prevent all failures (that delightful fantasy we sell to boards worldwide), they assumed failure was inevitable and asked a different question: “How do we make sure we can handle it?” Their Chaos Monkey randomly terminates instances in production. Not in staging. Not in test. In production.
The first time they ran it, things broke. Of course they did. But here’s the genius bit - they fixed those weaknesses, ran it again, found new ones, fixed those too, then Rinsed and repeated until their systems could lose entire availability zones without anyone noticing.
Fast forward to 2025, and every major tech company has some version of chaos engineering. Yet we still see organisations treating their production environments like sacred temples where mortals fear to tread. Recently, a FTSE 100 company told SPG they couldn’t possibly test failover scenarios in production. “Too risky,” they said. Three days later, they had a 6-hour outage because their untested failover…failed over.
The lesson isn’t that you should immediately unleash chaos on your infrastructure (please don’t - your board might not appreciate the philosophy). It’s this: The systems that survive aren’t the ones wrapped in the most protective layers. They’re the ones that have been battle-tested, stressed, and proven resilient through controlled adversity.
We spend millions on insurance policies for systems we’re too afraid to properly test. That’s not risk management - it’s risk denial.