Summary
The global computer outage of July 19, 2024, serves as a reminder of the weaknesses inherent in our complex technological ecosystems. This incident, triggered by a faulty software update released by CrowdStrike, reverberated across multiple industries, causing widespread disruptions and highlighting the critical need for robust incident response plans, continuous improvement, and vigilance – a blunt reminder that strategic preparedness and rapid incident response are more crucial than ever in our age of ever-evolving cyber threats.
The Incident Unveils
It was 0147 EDT, overnight and into Friday.
The unusual stillness for a bustling Thursday evening and the paradox of the tech world came into play—the approaching weekend against the eerie thoughts of updates being released into production systems.
As I was about to call it a night, the silence was shattered by the shrill ring of systems alerts and monitoring dashboard lights changing status, followed by ringing phones. The almost full moon cast a silvery glow over the misty mountain, starkly contrasting the chaos unfolding in the tech world.
This Friday would be unlike any other in our IT history. Our interconnected world was on the brink of an event that would underscore our systems’ fragility and highlight our false sense of security—a reminder of the delicate balance we maintain in our digital lives.
Incident Overview
For the ones unsure, CrowdStrike is a cybersecurity company with a significant market presence and serves various industries globally. One of their main products, Falcon, provides endpoint protection through real-time threat detection, prevention, and response, using advanced Artificial Intelligence and Machine Learning to identify and mitigate security threats.
A software update from CrowdStrike containing a critical error caused nearly 9 million Windows computers to crash worldwide, leading to widespread system failures and affecting numerous industries globally, including nearly 700,000 direct enterprise customer relationships between CrowdStrike and Microsoft.
The fragility of our interconnected systems was demonstrated when airlines faced delays and cancellations, financial institutions experienced disruptions in online banking and ATM services, and media companies saw interruptions in broadcasting.
This incident not only disrupted day-to-day operations but also highlighted the potential for a single point of failure to cascade into a global crisis.
A Technical Overview
CrowdStrike updates are installed automatically into production environments to ensure that the systems are protected with the latest security measures against emerging threats. This practice helps maintain high security but can lead to widespread outages if an update contains a flaw, as seen in the recent outage.
The faulty software update from CrowdStrike introduced a critical error in the .sys channel file, which is integral to system operations and contains a logic bug that disrupts the normal functioning of the Windows system. Security tools like CrowdStrike’s Falcon use these files to monitor system behaviors at a granular level, providing real-time protection against threats and protecting system activities at a deep level.
These .sys files act as drivers or kernel modules, facilitating communication between the operating system and hardware components, operating at a low level, or providing necessary services to the computer system, often interacting directly with the hardware or core system functionalities to manage resources, perform input and output operations, and ensure system stability.
An Insight into the Invisible Enemy
As we experienced firsthand, the impact of critical systems outages can be as disruptive and devastating as traditional warfare. An invisible enemy, such as a bug or a malicious code, can disrupt nations and large operations worldwide with resources at a fraction of the cost of conventional warfare methods. For instance, deploying a cyberattack is significantly inexpensive and less conspicuous than an electromagnetic pulse (EMP) attack, which would escalate tensions and likely provoke retaliation on a larger scale.
The interconnectedness of modern IT systems means that a failure in one component can have cascading effects, impacting numerous sectors simultaneously. The ability to disrupt critical infrastructure, financial systems, and communication networks can cripple a nation’s economy and security without firing a single shot.
The CrowdStrike fiasco illustrates the power and efficiency of digital warfare, which has rendered me many sleepless nights as the contrast between the invisible threats posed by software and the visible, traditional means of waging war.
Unlike physical attacks that require significant infrastructure, personnel, and logistics, cyberattacks can be executed remotely with little risk to the attacker. The stealth and anonymity provided by the internet allow nefarious actors to launch attacks that are difficult to trace and attribute, thus avoiding immediate retaliation.
This scenario underscores the importance of viewing cybersecurity not just as a technical issue, but as a matter of national security, highlighting the need for robust and comprehensive incident response plans, and the continuous improvement of security measures to protect against the invisible enemy that can strike without warning.
Strategic Insights
The CrowdStrike incident offers several strategic insights into the vulnerabilities of complex systems. First, it highlights the necessity of rigorous testing and validation of updates before deployment. A single faulty update can have far-reaching consequences, disrupting multiple sectors simultaneously.
Second, the incident underscores the need for comprehensive incident response plans. Organizations must conduct regular vulnerability assessments, implement advanced threat detection tools, and ensure continuous monitoring of their systems. Effective communication protocols are also essential to inform stakeholders and manage the crisis transparently.
My account as an Incident Response Leader
What I’m about to share underscores the critical importance of a robust incident response plan and the necessity of continuously enhancing security measures.
However, I must emphasize that, through the exceptional efforts of our world-class IT team (Application, Infrastructure, and Security), whose dedication I had the privilege of witnessing firsthand, we achieved close to total restoration of systems across multiple data centers and continents before CrowdStrike’s released a statement about the incident – even before many users had their first cup of coffee for the day. This remarkable achievement proves the extraordinary caliber of Haystack’s IT group’s understanding of the landscape and commitment to excellence. The team’s relentless efforts and hands-on approach transformed the recovery into an exemplary model of efficiency and excellence.
On that fateful night, as I prepared to call it a night and started investigating and assessing the cause of my systems going offline across data centers globally, I revisited the “Planning Considerations for Cyber Incidents” guide from my previous experiences with the Cybersecurity and Infrastructure Security Agency (CISA), an arm of the US Department of Homeland Security.
The CISA guide provides a structured framework for assessing the situation, activating the incident response team, and implementing effective containment and recovery strategies. By using the procedures in the CISA document as a guideline, in conjunction with my team’s Business Continuity, Disaster Recovery, and Incident Response plans, we were able to execute a coordinated and effective response in a structured manner.
Once the incident response protocol was initiated, we were tasked with restoring access and reestablishing timely availability for the affected platforms. We found the pattern within the system outage, identified and organized the affected systems, and initiated a detailed analysis to identify the root cause. We coordinated with internal teams and external stakeholders; we communicated regularly to inform everyone of our progress and manage expectations accordingly.
Our focus on system restoration involved rolling back faulty systems, composing a structured recovery procedure based on rigorous testing, outlining team members’ responsibilities, and distributing the vetted recovery steps with the group. We followed post-restoration, comprehensive validation, and user acceptance testing to ensure all systems were fully operational.
What Nefarious Actors Can Learn from This
While CrowdStrike has stated that the incident was not a cyberattack, the scenario described aligns with known patterns of nation-state supply chain attacks aimed at long-term espionage. These attacks require advanced capabilities and are designed to remain undetected while extracting valuable data over extended periods.
Nefarious actors, including rogue nations, state-sponsored hackers, and organized cybercriminal groups, can glean several insights from the CrowdStrike outage, as far as creating a sense of instability and uncertainty, making it easier to execute larger-scale attacks when defenses are perceived as unreliable while eroding trust in security controls and providers.
During such events, attackers can observe how quickly and effectively organizations respond and recover, gaining an understanding of the strengths and weaknesses of incident response plans. This knowledge allows them to craft more effective attacks in the future.
The incident also underscores the potential impact of targeting critical infrastructure components. By compromising security tools like CrowdStrike, attackers can create a ripple effect, disrupting multiple industries and sectors. This is particularly attractive for state-sponsored actors aiming to cause widespread disruption.
Strategic Recommendations
In light of the outage caused by CrowdStrike’s system update, organizations can focus on enhancing their incident response and cybersecurity measures. Here are some specific recommendations:
- Schedule and conduct a cybersecurity incident response simulation within the next quarter. Use real-world scenarios to test current plans’ effectiveness and identify improvement areas.
- Cybersecurity should be included as a regular agenda item in board meetings. Review the organization’s security posture, recent incidents, and ongoing improvement initiatives.
- Create a flexible, modular, scalable security architecture that allows centralized policy orchestration but decentralized enforcement. This will help protect all endpoints and systems more effectively.
- Perform a comprehensive risk assessment of all third-party vendors and partners. Ensure they adhere to strict cybersecurity standards and have robust incident response plans.
- Implement a zero-trust security model, which assumes that threats can come from outside and inside the network. This involves continuous verification of user identities, strict access controls, and real-time monitoring of all network activities.
- Leverage artificial intelligence and machine learning to detect anomalies and potential threats in real-time. These technologies can analyze vast amounts of data and identify patterns that might indicate a cyberattack, enabling quicker and more effective responses.
Practical Advice for Individuals and Businesses
In addition to organizational recommendations, individuals and small businesses can also take steps to enhance their cybersecurity posture:
Cybersecurity Steps
- Implement regular data backup procedures to ensure that critical information can be recovered in case of a cyberattack.
- Post-incident analysis and continuous improvement are vital for enhancing resilience against future threats.
- Conducting thorough post-mortem reviews helps identify gaps in the response process and provides valuable insights for refining incident response plans.
- Training staff based on lessons learned from incidents ensures the organization is better prepared for future crises.
- Install and maintain robust firewall and antivirus software to protect against common threats.
- Conduct regular cybersecurity training sessions for employees to raise awareness and ensure they understand best practices.
Cyber Hygiene Tips
- Ensure that all devices and software are regularly updated to patch known vulnerabilities.
- Use strong, unique passwords for different accounts and enable multi-factor authentication (MFA) wherever possible.
- Be cautious of phishing emails and links. Verify the source before clicking on any links or downloading attachments.
By recognizing these items, we can better prepare for the evolving landscape of cyber warfare, ensuring that our systems are resilient and capable of withstanding the sophisticated threats posed by those who seek to exploit them or vendor negligence.
#DerettiTalks - Takes on Security, Privacy and Technology 