Security Automation: Walking the Line Between Triumph and Turmoil

🖊️Author: Jason Allen, CTO at Digital Hands

In the continuously evolving world of security operations, finding the perfect balance between automation and human involvement is an ongoing debate. However, one thing is certain: some amount of automation is necessary to handle the large amount of security data that analysts work with every day.

Automation and human intuition each have their strengths. Machines excel at speed and precision, while humans possess judgment, intuition, and gut instincts. However, both humans and automation can make mistakes, so my journey with automation has been long and insightful.

It was in the early 2000s I first experienced the power of automation at scale. My government employer transitioned from relying on a single Unix supercomputer for data analysis to a distributed architecture of several thousand Linux servers. This shift created an urgent demand for automation as our small team faced the massive task of overseeing an ever-expanding number of mission-critical systems. We quickly developed a set of modest yet effective set Perl scripts and C applications to address the challenge. These primitive automation tools enabled our small team of seven to efficiently manage a huge server farm while still getting a good night’s sleep.

Today, the cybersecurity landscape is experiencing a similar transformation. Security data volumes continue to grow almost exponentially, while security teams tend to remain the same size or even shrink. This ever-changing environment and the resulting flood of alerts would utterly overwhelm even the best SOCs without the assistance of automation to keep pace with these dynamic demands.

At Digital Hands, we've leveraged automation to empower all aspects of our security operations, encompassing everything from threat intelligence propagation to equipping our security analysts with the ability to perform tasks with remarkable speed and precision. Although the possibility of automation failure remains a constant consideration, we've instituted a robust set of best practices that have effectively minimized the occurrence of automation-related issues.

To effectively manage automation complexities, I've found it essential to address three crucial challenges and establish countermeasures. These rules apply to automating security, infrastructure, and other technologies. These rules are universal and have proven to stand the test of time.

Unpredictable Integration Point Failures:
Automations often integrate with various devices or APIs, which can fail in bizarre and unpredictable ways. At Digital Hands, our strategy involves a graceful two-step: retries for initial API call failures and notifications to keep us informed after a preset timeout. We also record these missteps in our reporting platform, enabling us to investigate and manually follow up when necessary.
Forgetting Manual Processes:
Complex automation can lead to people forgetting how to manually perform the tasks they automate. To counter this, ensure proper documentation of the automation process, well-formatted code with useful comments, and, if possible, visual representations of the automation. At Digital Hands, we use our proprietary SOAR platform called Maestro to provide clear visualizations of our automation jobs.
Bad Automation Design
Poorly designed automations can fail in spectacular ways. It's crucial to treat automation like software and use good software development practices. At Digital Hands, we conduct manual code reviews, use test suites, employ CI/CD pipelines for test execution, and implement robust error handling and notifications.

Additionally, our reporting platform plays a vital role in monitoring security operations automation. By conducting regular manual assessments of important metrics, we are able to stay ahead of any potential failures in the automation process. This proactive approach helps us to identify and address any issues before they escalate.

In the face of this complex dance between humans and machines, it's crucial to maintain a delicate balance. As we tap into automation's transformative capabilities, we must not forget the unique strengths of human intuition, judgment, and adaptability. By striking the right harmony, we can harness the best of both worlds, creating a powerful synergy that propels the evolution of security operations.

The key to successful automation lies in its ability to augment human expertise rather than replace it. By carefully crafting and maintaining our automation code, we ensure that it serves as an extension of our team's capabilities, amplifying our ability to detect, respond, and adapt to the ever-changing security landscape.