Overburdened SOC Analysts Shift Priorities

Yet another red flag that the security operations center is burning: Most SOC analysts now say their primary role is to close alert investigations or the number of alerts as quickly as possible rather than to study and remediate security threats, according to a new survey.

Last year, some 70% of SOC analysts surveyed by CriticalStart actually said the reverse: They saw their main role as analyzing and fixing security threats and issues. But in the managed detection response firm's newest survey — published this week — only 41% say that's their priority. And most (more than 70%) handle 10 or more security alerts per day, an increase over last year, when 45% were operating at that workload.

SOCs also get hit with false-positive security alerts at a rate of 50% or higher, according to the report, and nearly 40% say when SOC analysts can't keep up with alert, they either turn off some alerting functions or hire more analysts.

"Security teams are just trying to survive without the required resources and head count. They often filter alerts using email rules to send notifications to an alert folder, where they are ignored. Some teams even turn off security-system alerting so there is no record of security events not being monitored," says Rob Davis, CEO of CriticalStart, which recently surveyed 50 SOC professionals in enterprise organizations, managed security services providers, and managed detection and response providers. "It is very common for analysts to increase the thresholds for creating security events to reduce volume."

Turning down the volume on alerts to keep up with them puts organizations at risk of a real attack, says Chris Calvert, co-founder and vice president of product strategy at Respond Software, a security automation vendor. "If I am being attacked constantly and I have real vulnerabilities to manage but only a small team, how do I prioritize? In today's environment, detection and remediation is just as important as prevention and we often don't have the budget to cover everything," he says.

This latest study is yet another in a string of recent SOC reports this year underscoring the growing problem of an overwhelming and impossible volume of security alerts to sift through for that needle in the haystack, and the lack of people to fill the seats in the SOC. CriticalStart's report shows how these stresses are leading to heavy turnover, with more than 75% seeing SOC turnover rates of more than 10%, and close to half seeing turnover rates of 10% to 25%.

A Ponemon Institute study last month revealed that more than half of IT and security pros consider their SOC inadequate to thwart security threats and some 65% were thinking of leaving their positions due to alert overload, long hours, and incomplete visibility into their IT infrastructures.

"Turnover continues to increase and retention is a major issue for companies," notes CriticalStart's Davis. "With virtually no unemployment, the best analysts are constantly recruited. Executives fear the expertise from completed security projects will walk out the door and ruin their investment in tools."

It's a vicious cycle: Much of the stress in the SOC comes from analysts surrounded by too many security tools that don't work well together or that they don't have time or resources to fully master, as more alerts bombard their screens every day. They just don't have the time or expertise to master the tools, or stay on top of the alerts these systems pump out.

"More security sensors and log sources containing more signatures of potentially malicious activity combined with exponential IT growth — and a dramatic increase in malicious attacks," Calvert explains. He says SOCs should measure the time and effort spent on false positives and automate the process where they can.

The noise and overload of tools and alerts can escalate quickly, according to Larry Ponemon, president of the Ponemon Institute. "A lot of research studies find the whole issue of interoperability and scalability is largely ignored and as result, the technologies don't actually work together, and you have more [tools] than you need," Ponemon says.

An overwhelmed SOC can result in dangerously long times to resolve and remediate an attack. Some 42% of the SOC analysts in Ponemon's report, sponsored by Devo Technology, say it takes months or years on average to resolve a hack. That mean time to resolution, as it is called, occurs at 22% of organizations in a matter of hours or days.

CriticalStart's report found that most SOC analysts (78%) take 10 or more minutes to investigate each alert they see.

Security experts recommend outsourcing some or most SOC operations to managed or cloud-based providers, training up existing SOC analysts, and automating low-level tasks.

"The SOC of the future will look very different and provide the advantages of an in-house SOC with the lower cost of outsourcing," Davis notes. "A SOC must quickly detect attacks and respond before a breach occurs. This requires resolving every alert instead of filtering or ignoring security events. If you aren't resolving every alert, then you can't detect every attack."

He says that because most alerts are common across organizations, an outsourced SOC model can tackle "known good" false positives more efficiently and cheaply.

Related Content:

Check out The Edge, Dark Reading's new section for features, threat data, and in-depth perspectives. Today's top story: "Fuzzing 101: Why Bug-Finders Still Love It After All These Years."