We’re betting you noticed that it felt like half the internet went offline the other week. Microsoft instances were bricked, airlines fell apart, public transport was impacted, and everything all went a bit nuts because of something called “Crowdstrike.”
So what’s the deal with this Crowdstrike thing anyway?
We waffle on about all kinds of things – a lot of the stuff we talk about you probably hadn’t given any thought to or even considered. But, seeing as this was plastered all over the news, and you might even have been affected in one form or another, we think it makes for a topical blog post.
First of all, let’s talk about who Crowdstrike is and what they do.
Who is Crowdstrike?
Crowdstrike is a cybersecurity company. They specialise in threat intelligence and cyber attack response. They provide specialised protection for devices like computers, servers, and mobile devices, and help other organisations understand the tactics and techniques that cyber criminals use to orchestrate attacks. They have services that respond to attacks and help mitigate impact on organisations, and they assist in recovery during and after incidents. Crowdstrike also offers managed security services, as well as cloud security and protection.
Well, they certainly sound amazing on paper, don’t they? That’s a pretty impressive list of services. And up until very recently, they were well renowned.
So what happened?
In a nutshell, a software update was pushed out that wasn’t ready. Falcon, one of Crowdstrike’s services, received an update. The intent was to identify newly observed malicious activity by cyber criminals and attackers. However, the update itself applies to something that communicates at an extremely low level in Windows computers. Something that lives in the same space as the Kernel*. Kernel-level processes are the most privileged level on a Windows PC; it’s more or less a direct interface with hardware. Naturally, this “logic error” (as Crowdstrike says it was), ended up causing almost immediate crashes within seconds of systems booting up.
Unfortunately for Crowdstrike and those affected, this update propagated far and wide affecting shedloads of systems. Most notably plenty of US airlines, healthcare, banking, you name it.
What’s worse is this wasn’t going to be a quick fix either. Each affected system had to be restarted and booted in safe mode so the offending file could be removed from the PC. Oh, did we mention this has to be done using the command line no less, and if drives were encrypted, as most business drives are, a 48-character recovery key needed to be entered first to get access to the drive?
IT professionals and system administrators are not having a nice time of it right now, and they’re probably still working on restoring some less critical systems out there.
*The Kernel is a core piece of software at the heart of a computer and generally has control over everything in the computer’s system. Wee bit important, then.
It wasn’t Microsoft’s fault.
Rather unfairly, Microsoft seemed to be getting the blame for the outage. Whilst Azure (Microsoft’s cloud computing platform) did go down around the same time, it was offline whilst most us Brits were in bed – and it was back up by the time we were in the office at 9am on 19th July.
Windows 3.1 and 95?
Those of you (like Graham) who are old enough to remember Windows 3.1 might be astounded to learn that a couple of US airlines were completely unaffected by the Crowdstrike situation. It turns out that their systems were running on Windows 3.1 and Windows 95!
3.1 was released in 1992 and 95 was released in 1995. Who’d have thought that not applying your security updates for more than 30 years would actually be a smart move? (Sidenote: we don’t recommend doing this unless you’re a massive company with a million dollar IT budget who can address insecurities and problems if your 30 year old system goes wrong.)
What’s next?
Well for one thing there are probably plenty of systems out there that still need recovering. We’re also still seeing further outages that may or may not be related to the Crowdstrike bug – Microsoft is still investigating so keep an eye on the tech news for updates.
Crowdstrike’s stock price took a bit of a dive, as did their reputation and, as you’d expect, companies are now seeking damages. While the full impact probably still hasn’t been assessed just yet, you can bet your bottom dollar that this is all going to turn very litigious over the coming weeks and months.
What lessons can we learn from this situation?
Don’t jump on day-one updates. Those of you who have sites with us may notice that just because the latest version of a plugin, theme, or WordPress has been released doesn’t mean you necessarily get it immediately. We always take the time to check the latest releases for conflicts, bug reports, and hotfixes before we go ahead with major version updates.
We’d like to stress that we, of course, always apply emergency security patches where appropriate. But we never jump on the latest and greatest update without a healthy amount of suspicion. It’s all too easy to introduce breaking changes.
The usual advice always applies, such as ensuring to make backups of critical systems and data (more than one, and store them in more than one place too). Always have some form of disaster recovery plan. If your laptop died tomorrow, how would you recover? If you lost access to your primary email account, how would you get a hold of your contacts?
You’ll never be prepared for all eventualities, but you can have plans and systems in place that cover you when something goes wrong. Want help making sure this doesn’t happen to your website? Drop us a line to find out how we can help.