SOC Wins & Losses

Security operations centers (SOCs) have gained more prestige, profile, and, in some cases, budget in the organization. But even well-resourced SOCs suffer many of the same woes that struggling SOCs do: an incomplete view of all devices connecting to their networks and an overload of redundant and underutilized security tools spitting out more data and alerts than they can handle or grok.

More alarmingly, many still struggle to quickly resolve security incidents. In some 40% of SOCs, the mean time to resolution (MTTR) is months to years, according to a study by the Ponemon Institute and commissioned by Devo Technology that published this week. Around 37% resolve incidents within weeks and 24% within hours or days.

With the exception of the most mature SOCs, that slow resolution rate is typical, notes Julian Waits, general manager of cybersecurity at Devo. "Their program is still immature, they don't have playbooks in place, and so much is still happening manually," he says.

It takes about a week for Texas A&M University's SOC to resolve incident, according to Dominic Dertatevasion, associate director of IT at Texas A&M's SOC. That MTTR is based on what A&M SOC tools can actually see, he notes. "Within a week, they should be able to identify where the host or user is and clean it up or educate the host to reset passwords" and other controls, Dertatevasion says.

Texas A&M's SOC is actually watching not only the network on its massive flagship campus in College Station, Texas, but it also provides SOC services for 11 universities under the A&M system as well as a half-dozen government state agencies on its network. "We're only seeing what we can see and what you can give us access to. I'm 100% sure we're missing stuff," Dertatevasion says of the other campuses his team services.

Some 40% of security pros say their SOCs have too many tools. Devo's Waits says it's not surprising that SOCs end up with too many tools that often overlap or produce redundant data. "A new technology gets brought in and many of the older technologies [overlap] ... another thing gets added on the stack, and there's not thought on how to optimize them," he says.

The most common overlapping tools are endpoint detection and response products and network detection tools, he says. And the consolidation among security vendors also inadvertently results in redundancy in the SOC. He points to the example of next-generation firewall vendor Palo Alto Networks, which now also has endpoint technology.

Different tools can be generating alerts on the same IP address but are run by different SOC analysts, he notes.

Dertatevasion says Texas A&M adds a new tool every one to two years in a slow and deliberate strategy. The goal is to allow analysts to gain expertise in the tools and ensure they fit well into the SOC ecosystem and operations before adding anything new.

"We might be different than private industry in that we have SOC-managed tools we run and our constituents have tools they purchase and might bring along. We've always had to adapt to the tools other people bring along and try not to overpromise or overdeliver on that," he says. "We don't want to be in a jack-of-all-trades-but-master-of-none type of situation."

Given that the security landscape is constantly evolving, he says, the university can't afford to keep any insufficient tools, anyway.

Automation has been the battle cry for streamlining and eliminating the high volume of alerts tools generate in the SOC. More than 70% say they want more automation in the SOC, especially to help relieve the manual labor of alert management, incident evidence-gathering, and malware defense, the study found.

But, yes, there is such thing as too much automation, where the SOC analyst ends up being relegated to more of a help desk role that doesn't tap his or her skills. As Sean Curran, a partner with West Monroe Partners, describes it, too much automation can turn SOC analysts into robots that can't properly pivot when an incident pivots from script. He points to a case where SOC analysts disabled a legitimate alert because it didn't fit the runbot.

"They didn't know what to do with it," so they assumed it was a false positive and disabled it, he recalled during a recent Dark Reading panel discussion on SOCs and incident response.

"They're just shuffling tickets" in that scenario, Dertatevasion says. "I aim for my organization to automate the boring stuff. If we're seeing something three times a day, and every time we see this set of IOCs we know it's benign and we're not going to escalate it, then we automate it."  

COVID-19
Meanwhile, there's been a well-documented high burnout rate among SOC analysts, leading to turnover. The Ponemon-Devo report – based on a survey of IT and IT security professionals in organizations with SOCs and taken between March 11 and April 5, at the start of the pandemic – found that 78% of SOC analysts describe the SOC "very painful" to work in, an increase from 70% last year. Around 60% are looking to jump ship and change jobs.

A recent study from Exabeam found that 64% of SOC analysts on the front line were leaving their jobs because they saw no career path for them there.

"We did this research before we really knew the reality of COVID-19," Devo's Waits notes. The stress levels likely have escalated, with the teams sent to work from home who weren't accustomed to it, and the underfunded SOCs are even more challenged without the face-to-face work support, he notes.

It's often "more chaotic" working from home, especially with family and other personal distractions, he notes. "Now SOC analysts may have something slip through the cracks" more easily, he says.

Related Content:

Learn from industry experts in a setting that is conducive to interaction and conversation about how to prepare for that "really bad day" in cybersecurity. Click for more information and to register for this On-Demand event.