- Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million.
- The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline.
- Bastian, speaking from Paris, told CNBC’s “Squawk Box” on Wednesday that the carrier would seek damages from the disruptions, adding, “We have no choice.”
General practices aside, should they really not plan anybackups system though? Crowd strike did not cause 500 million in damages to delta, deltas disaster recovery response did.
Where do we draw the line there though I’m not sure. If you set my house on fire but the fire department just stands outside and watches it burn for no reason, who should I be upset with?
Well, in your example you should be mad at yourself for not having a backup house. 😛
There’s a lot of assumptions underpinning the statements around their backup systems. Namely, that they didn’t have any.
Most outage backups focus on datacenter availability, network availability, and server availability.
If your service needs one server to function, having six servers spread across two data centers each with at least two ISPs is cautious, but prudent. Particularly if you’re setup to do rolling updates, so only one server should ever be “different” at a time, leaving you with a redundant copy at each location no matter what.
This goes wrong if someone magically breaks every redundant server at the same time. The underlying assumption around resiliency planning is that random failure is probabilistic in nature, and so by quantifying your failure points and their failure probability you can tune your likelihood of an outage to be arbitrarily low (but never zero).
If your failure isn’t random, like a vendor bypassing your update and deployment controls, then that model fails.
A second point: an airline uses computers that aren’t servers, and requires them for operations. The ticketing agents, the gate crew that manages where people sit and boarding, the ground crew that need to manage routine inspection reports, the baggage handlers that put bags on the right cart to get them to the right plane, and office workers who manage stuff like making sure fuel is paid for, that crews are ready for when their plane shows up and all that stuff that goes into being an airline that isn’t actually flying planes.
All these people need computers, and you don’t typically issue someone a redundant laptop or desktop computer. You rely on hardware failures being random, and hire enough IT staff to manage repairs and replacement at that expected cadence, with enough staff and backup hardware to keep things running as things break.
Finally, if what you know is “computers are turning off and not coming back online”, your IT staff is swamped, systems are variously down or degraded, staff in a bunch of different places are reporting that they can’t do their jobs, your system is in an uncertain and unstable position. This is not where you want a system with strict safety requirements to be, and so the only responsible action is to halt operations, even if things start to recover, until you know what’s happening, why, and that it won’t happen again.
As more details have come out about the issues that Delta is having, it appears that it’s less about system resiliency, although needing to manually fix a bunch of servers was a problem, and more that the scale of flight and crew availability changes overloaded that aforementioned scheduling system, making it difficult to get people and planes in the right place at the right time.
While the application should be able to more gracefully handle extremely high loads, that’s a much smaller failure of planning than not having a disaster recovery or redundancy plan.
So it’s more like I built a house with a sprinkler system, and then you blew it up with explosives. As the fire department and I piece it back together, my mailbox fills with mail and tips over into a creek, so I miss paying my taxes and need to pay a penalty.
I shouldn’t have had a crap mailbox, but it wouldn’t have been a problem if you hadn’t destroyed my house.