CrowdStrike apologizes for IT outage, defends access to Microsoft kernel

Cybersecurity giant CrowdStrike is apologizing for “letting customers down” after a flawed Falcon sensor update disabled millions of computers on July 19.

Adam Meyers, vice president of counter-adversary operations at CrowdStrike, appeared before a U.S. congressional committee on September 24 to answer questions about a bug that caused an estimated 8.5 million Windows computers to freeze and display Microsoft’s infamous blue screen of death (BSOD).

The U.S. House of Representatives Homeland Security Committee on July 22 requested public testimony from CrowdStrike CEO George Kurtz.

Kurtz promised to do so as soon as the incident was fully resolved, but the company ultimately decided to send Meyers.

CrowdStrike Falcon Sensor Fault Explained

Before Congress, Meyers said the “perfect storm” was caused by the update having a “mismatch between input parameters and predefined policies.”

“On July 19, 2024, new threat detection configurations were validated through standard validation procedures and sent to sensors running on Microsoft Windows devices. However, the configurations were not understood by the Falcon sensor rules engine, leading to the failure of the affected sensors until the affected configurations were replaced,” Meyers explained.

Read more: Cybercriminals are taking advantage of the chaos caused by the CrowdStrike outage

CrowdStrike’s efforts to reboot affected systems

Meyers also detailed how CrowdStrike helped restore affected systems.

On July 22, the company introduced automated techniques to speed up remediation efforts.

CrowdStrike employees were then dispatched to assist customers in recovering their systems.

The problem with this outage is that in order to restart the failed machine, physical access to it is required.

“As of July 29, virtually all of our customer systems were back online,” Meyers confirmed.

CrowdStrike continues to face multiple lawsuits following the July outage.

These include both the company’s shareholders and Delta Airlines.

Delta has accused CrowdStrike of “negligence” and says it lost $500 million in the outage, which led to thousands of flights being canceled.

Read more: CrowdStrike Windows Crash: What We Can Learn

CrowdStrike’s preventive measures to prevent similar incidents

Meyers spoke about the actions CrowdStrike is taking to ensure an incident like this never happens again.

These improvements include:

• Validation: CrowdStrike has introduced new validation checks to help ensure that the number of inputs expected by the sensor and its predefined rules match the same number of threat detection configurations that were provided
• Testing: The company has improved existing testing procedures to cover a wider range of scenarios
• Customer Control: CrowdStrike customers now have more control over how configuration updates are deployed to their systems
• Deployments: CrowdStrike now takes a phased approach to deploying threat detection updates, meaning customers do not have to deploy updates immediately
• Security: The company has added additional runtime checks to the system to ensure that the data provided is as expected by the system before any processing begins
• Third-party reviews: Two independent third-party security software vendors have been hired to conduct further reviews of the Falcon sensor code and comprehensive quality and release process reviews.

Microsoft Kernel Access Discussed in CrowdStrike

Congressmen asked Meyers whether software like the CrowdStrike Falcon sensor should have access to Microsoft’s kernel.

Kernel access refers to the ability of a software program or process to interact directly with the operating system kernel. The kernel is the core component of an operating system, responsible for managing hardware resources, processes, and memory.

While most applications run in user space, several critical applications, including antivirus, endpoint detection and response (EDR) solutions, and other security products, are installed in the Microsoft kernel.

This level of access is essential for many cybersecurity solutions to effectively monitor and protect systems. However, it also raises concerns about potential unauthorized access or misuse, especially in the event of a security breach.

The CrowdStrike incident has prompted Microsoft to consider moving antivirus software updates and other threat detection tools to user mode to reduce the likelihood of more serious incidents, sources said.

Meyers, however, opposed the decision, arguing that without access to the kernel, CrowdStrike’s security products may be less effective.

He said products like Falcon have “visibility into everything that’s happening on a given operating system,” which helps prevent threats and helps “provide protection against tampering.”

Meyers mentioned that Scattered Spider, the group responsible for hacking into Las Vegas casino networks, often uses “new techniques to escalate their privileges to regularly disable security tools.”

Meyers said CrowdStrike “will continue to leverage the operating system architecture.”