On Friday the 19th, a Rapid Response Content update from CrowdStrike inadvertently caused global IT outages, affecting critical infrastructure, travel, healthcare, and more. Such incidents highlight the complexities of maintaining mission-critical infrastructure and underscore the importance of robust cybersecurity measures.

Overview of the Event

The unintentional update by CrowdStrike led to significant disruptions across various sectors, exposing vulnerabilities in IT infrastructure worldwide. This event serves as a stark reminder of the challenges faced in securing essential systems and the need for vigilant cybersecurity practices.

How Ctrl Handled It

At ctrl:cyber, the primary focus is the protection and stability of our clients’ data and operations. Minimal impact from this incident was had, thanks to a strategic approach to deployments and minimal reliance on the affected technology, allowing for swift issue resolution.

Minimal Impact on Clients: Thanks to our associated parties having progressive deployment strategies, which involves introducing changes to small customer groups first, most of our clients avoided widespread disruptions. The risk of such incidents is significantly reduced by not pushing kernel-level changes globally.

Immediate Response for Affected Clients: For the few clients impacted due to the technology in question, our team promptly addressed the issues, ensuring minimal downtime and maintaining operational stability.

Within the first fifteen minutes of the incident, Ctrl was alerted to the update and was able to determine the common denominators involved.

Promptly, the few clients that were determined to be affected were contacted and informed, with advice provided in relation to resolving the issue based off the information had at the time and how to minimise the risk to their environments.

Any clients that were able to avoid using their computers were informed to do so if not necessarily due to the unprotected nature of the digital environment. The clients that were not affected were also contacted, with the potential of their 3rd parties being at risk and leading to exposure.

Proactive Measures

To safeguard against similar incidents, several proactive measures are implemented:

Extensive Internal Testing: Before any update is rolled out, it undergoes rigorous internal testing to ensure it does not compromise system stability or security.

Swift Rollback Capability: In case an update causes issues, there is the capability to roll back changes quickly, minimising impact on our clients.

Strategic Partnering: Collaborating with trusted partners, we emphasise the following principles:

No Kernel Updates: Only user-mode components are updated, avoiding risks associated with kernel-level changes.

Controlled Deployment: Updates are released gradually to ensure system stability.

Customer Control: Clients can manage Live Security Updates via a simple toggle.

Transparency: Detailed release notes and auditing information accompany every update.

How to Handle Such Incidents in the Future

To manage incidents like this effectively in the future, organisations should:

Diversify Their Portfolio of Partners: Relying on a single technology provider can create single points of failure. Diversifying ensures resilience and reduces risk.

Implement Strong Governance, Risk, and Compliance Measures: Regularly review and update your cybersecurity policies to adapt to emerging threats.

Stay Informed and Prepared: Maintain an awareness of potential vulnerabilities and have a response plan in place.

Implement Business Continuity and Disaster Recovery: Implement Business Continuity and Disaster Recovery plans for your business and ensure there is regular resting in place after any major changes to your digital environment or the plan itself.

Vendor-Agnostic Approach

At Ctrl maintaining a vendor-agnostic stance is essential. This approach helps avoid single points of failure and ensures clients benefit from the best solutions in the market. Diversifying the portfolio of partners enables the provision of robust, resilient cybersecurity services.

Ctrl offers comprehensive business continuity services and expert advice on governance, risk, and compliance. Stay informed, stay secure. For any questions or assistance, contact us today.