Friday, July 19, 2024

Major Global Outages Affecting Microsoft and Crowdstrike Customers

A limited number of RSI customers may have been impacted by the flawed update from IT security provider, Crowdstrike.  Our engineers were given patches by Crowdstrike to resolve any issues.  The fixes were implemented in the early morning.  All systems should be operation.

According to Reuters

Cybersecurity firm CrowdStrike has deployed a fix for an issue that triggered a major tech outage that affected industries ranging from airlines to banking to healthcare worldwide, the company's CEO said on Friday.

The technology glitch was caused by a faulty update from CrowdStrike, a U.S. cybersecurity technology company based in Texas, in a single content update for Windows hosts.

It resulted in Windows computers and tablets crashing and displaying a blue screen, known informally as the "Blue Screen of Death." Over half of Fortune 500 companies use CrowdStrike software, the firm said in a promotional video this year.

Microsoft said separately it had fixed the underlying cause for the outage of its 365 apps and services including Teams and OneDrive, but residual impact was affecting some services.

A massive IT outage was disrupting operations at companies across multiple industries on Friday, with major airlines halting flights, some broadcasters off-air and sectors ranging from banking to healthcare hit by system problems.

"We're deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our company," Kurtz told NBC News' "Today" program.

"Many of the customers are rebooting the system and it's coming up and it'll be operational," Kurtz said. "It could be some time for some systems that won't automatically recover."

CrowdStrike's "Falcon Sensor" software was causing Microsoft Windows to crash and display a blue screen, known informally as the "Blue Screen of Death," according to an alert sent by CrowdStrike earlier to its clients and reviewed by Reuters.

"The issue has been identified, isolated and a fix has been deployed," George Kurtz, the president and CEO of CrowdStrike posted on X. "We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website."

An American Airlines Boeing 737 Max takes off from Miami International Airport on Dec. 29, 2020 in Miami, Florida. (Joe Raedle/Getty Images / Getty Images)

The travel industry was among the hardest hit with airports around the world reporting delays and issues with their system network, while banks and financial institutions from Australia and India to South Africa warned clients about disruptions to their services.

Where did the problem originate?

The company’s popular Falcon Sensor software appears to be the origin of the issue. Falcon is an antivirus platform used to secure “endpoints” like laptops, servers, mobile devices and point-of-sale systems. In order to monitor these endpoints for malicious software and suspicious activity, CrowdStrike software has deep-level access to the device’s operating system.

This is known as kernel-level access, referring to the core level of a computer’s operating system that facilitates interactions between software and hardware. Cybersecurity software often needs this highly privileged access so that it can access any part of a computer’s system that hackers may target.

The update that CrowdStrike pushed appears to have impacted the kernel-level driver that CrowdStrike uses to monitor devices for malware, according to IT analysts. The faulty code appears to be interacting with the Windows operating system and causing computers to crash.

These affected devices then get stuck in a cycle called boot looping, in which the computer fails to complete its regular boot up sequence and then reboots in a seemingly endless cycle.

What’s the solution?

CrowdStrike says it has deployed a patch to fix the faulty software update but that won’t immediately resolve the outage.

This is because the computers impacted by the outage cannot boot up and get online to receive the fix. Instead, IT admins around the world will have to physically go into a machine’s system and delete the faulty driver.

CrowdStrike provided the following workaround steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. Boot the host normally.

Others have found success in simply rebooting affected computers over and over again, in the hopes the CrowdStrike update gets pushed through the network before the machine hits the Blue Screen of Death.