When datacenters lose power, generators should kick in. That didn’t happen for Google in London
October outage caused by bad PLC config exposed at a time maintenance work reduced resilience
This outage highlights the critical importance of robust backup systems and diligent maintenance, especially for services as vital as Google's data centers. It's a stark reminder that even the most tech-savvy companies need to keep their hardware and configurations transparent and accessible to ensure reliability.
Panel script
A darkened industrial warehouse at night, with a lone technician in a hard hat and yellow safety vest adjusting a physical switch labeled 'Maintenance Override' on a wall-mounted control panel.
When the PLC configuration went wrong, the backup generator stayed silent.
Rate this comic
Script
Panel 2
Inside a Google datacenter control room. Rows of blinking servers and screens. A technician in a hard hat is adjusting a large panel of switches and cables. TECHNICIAN (muttering)
TECHNICIAN: Great, another power outage... Why don’t these things just work automatically?
Panel 3
The same control room, but now a large diagram of the power grid is displayed on a screen. A maintenance worker is pointing at a section of the diagram, explaining something to a colleague. MAINTENANCE WORKER (pointing) COLLEAGUE (skeptical)
MAINTENANCE WORKER: See this? We were supposed to test the PLC configuration last week. But it got delayed.
COLLEAGUE: So you’re telling me that’s why the generators didn’t kick in?
Panel 4
Inside the datacenter generator room. A large, industrial generator sits idle. The technician from Panel 1 is standing next to it, looking bewildered. TECHNICIAN (to generator) GENERATOR (voiceover, dryly) The generator remains silent, with its cover plate slightly ajar, hinting at a misconfiguration.
TECHNICIAN: We really need you right now. Why didn’t you go, 'I’m ready!'?
GENERATOR: Yeah, I thought 'next Tuesday' was the day.
Comments