The SOC Gets a Makeover

Blame it on the success of the SIEM. For many security operations center (SOC) managers, the security information and event management system was both a blessing and a curse: It was a way to consolidate and correlate security alerts from firewalls, routers, IDS/IPS, antivirus software, and servers, for example, into a centralized console. But with the recent wave of new security tools, threat intelligence feeds, and constantly mutating threats, SOCs are drowning in anywhere from thousands to a million security alerts daily. 

"A lot of companies have tool fatigue right now. There are a lot of tools that are partially implemented and not getting the care and feeding they need," says DJ Goldsworthy, director of security operations and threat management at Aflac.

The flood of alerts and out-of-tune tools, compounded by the industry's persistent talent gap and high turnover rate for junior-level SOC analysts, have forced some organizations to rethink and retool how they organize and run their SOCs.

In many cases, the evolution is being spurred by another tool: The new generation of security orchestration and automation tools that streamline and automate some tasks with automated playbooks is replacing some of the more manual tasks of clicking through each and every alert, looking for that deadly needle in the haystack.

SOC operators are reorganizing their people power, too, collapsing the virtual hierarchical walls between Tier 1, 2, and 3 SOC analysts and fostering a more cooperative and collaborative team operation where analysts work together to analyze and troubleshoot an event – and don't simply pass the baton up the chain after completing their task.

Josh Maberry, director of security operations for managed security services provider Critical Start, says once security alerts from multiple tools started flowing into the SIEM way too fast and furiously, the floodgates opened, and SOC analysts became overwhelmed. "There's only so much you can manually get to one in day, only so much you can see," says Maberry, whose firm built its own event orchestration platform, Advanced Threat Analytics. "So [SOC] analysts began to drown ... The whole thing became an events-to-bodies ratio, and that's no way to win."

Organizations struggle to staff their SOC operation today, both due to that losing events ratio and also because there just aren't candidates to fill the jobs. Some 80% of organizations don't have enough analysts to run their SOC, according to a new report from security event orchestration vendor Demisto. It takes eight months, on average, to train SOC analysts for readiness; meantime, there's a two-year turnover for one-quarter of all SOC analysts, the study shows. The fallout: It takes organizations an average of 4.35 days to resolve a security incident.

More than half either don't have incident response process playbooks in place, or they do but don't bother updating them, the study found. "One relatively overlooked side effect of the alert fatigue and day-to-day firefighting that security teams face is the stagnancy of security processes. When analysts are strapped for time, they find it difficult to capture the gaps in current processes and update them on an ongoing basis," the report states.

Taking Back the SOC
One of the larger SOC operations is that of security vendor Symantec, with six SOC locations worldwide. Symantec's Herndon, Va., site houses both the company's internal SOC as well as a SOC that manages security event and response services for its customers. A team of more than 500 security professionals make up Symantec's global SOC operation, which handles more than 150 billion security logs per day.

Figure 1: Symantec SOC in Herndon, Va. --- Courtesy of Symantec

Symantec's internal SOC analysts are designated by level of experience and seniority (think: tiers), but they often work as a team when a security event occurs. Tony Martinez, cybersecurity operations lead for the Joint Security Operations Center (JSOC) in Herndon, says all SOC analysts – even Tier 1 analysts – are encouraged to handle an incident "end to end," meaning from detection to resolution/response. "They don't have to just throw the ticket over the fence" to a senior analyst, he says. "They ask a senior analyst to assist them."

Symantec's JSOC recently added Splunk's Phantom security orchestration and automation platform to consolidate security tools and alerts. On the managed security services side of the house, SOC analysts sit in close proximity to foster more collaboration during their shifts, and Tier 1/entry-level SOC analysts undergo three months of intense training plus a timed "queue" test that simulates incoming incidents in the SOC. Not only do they have to solve each issue correctly in the queue test, but they also must explain why they chose a specific answer, says Steve Meckl, director of managed security services at the Symantec SOC.

Junior SOC analysts ultimately get to take on more regular calls with customers and attend on-site customer visits for face-to-face meetings.

"We don't work in silos at all," Meckl says. All SOC analysts get to work on a problem from start to finish, with the junior analysts getting input from a senior one.

This more advanced and hands-on role for entry-level SOC analysts is becoming a trend: The Tier 1 SOC analyst role is expected to evolve into more of a Tier 2 role, where he or she can analyze a flagged alert themselves rather than passing it over to a Tier 2 SOC analyst.

In most SOCs, Tier 3 analysts are the more skilled analysts who can investigate a threat or malware more deeply, and do forensics. As more of the Tier 1 analysts work gets picked up by automation (think: security orchestration and automation tools), that provides Tier 3 the bandwidth to conduct more proactive operations like threat hunting. Tier 1 and 2 take on more of the investigative duties. 

Brian Genz, a senior engineer at Northwestern Mutual and an expert in security orchestration and automation, response, and threat hunting, says the insurance company decided to fashion its SOC as slimmer and more collaborative to better thwart rapidly evolving threats.

Northwestern Mutual's implementation of a new security orchestration and automation tool has helped shape that transformation with automated playbooks for handling events as the come in. "Our junior SOC analysts are becoming more engaged in the what and why now  –  not just 'close this tickets to hit your metrics,'" Genz says.

Genz says the insurer, which uses an MSSP for around-the-clock security ops, actually considers its SOC an IR analyst operation. "So we don't typically use the term 'SOC,'" he says. "We try to put people in the shoes of an incident response analyst."

Homegrown
Sometimes it takes a little homegrown technology to streamline SOC operations. Take Aflac, which in its SOC runs a centralized SIEM with a behavioral analytics platform that handles a terabyte of security log data each day. The system is streamlined with a proprietary risk algorithm created by Aflac that aggregates and filters alerts. Aflac has reduced its alert count by about 70% with this combination of automation, analytics, and risk scoring.

It has also changed the role of Aflac's entry-level SOC analysts. They're not manually clicking through raw alerts and either ignoring or escalating them like traditional Tier 1 analysts. "This has automated Tier 1 to an extent," Aflac's Goldsworthy says.

The junior analysts at Aflac examine the insurer's pre-vetted alerts: "They analyze them, do a smell test, and if they think it's worthy of investigation, they send it to a Tier 2 analyst," he says.

Even so, the SOC still operates in a traditional tiered manner, but each level has more advanced duties than in the old days. Tier 2 analysts handle preliminary incident investigation and then hand off confirmed events to the incident response team. Tier 3 analysts provide forensics investigations and typically have deep endpoint and network analysis skills, Goldsworthy says.  

Another tool that's changing Aflac's SOC operation is deception technology – a sort of next-generation honeypot – to further minimize its false-positive alerts. Goldsworthy calls its Attivo Networks deception tool "an insurance policy for the unknown."

Goldsworthy says deception decoys give SOC analysts a "unique perspective" about attackers and their methods, which they then can share with other members of the team and, in turn, respond accordingly with proper defenses. "Deception also allows our security team to collaborate with and enable the business by allowing for more rapid adoption of new technologies because deception can be deployed wherever the business needs IT to go," he says.

(Next Page:  SOC Analyst Mashup)

(Continued from Page 1)

Some SOC operators are looking to cut head count as much as possible. One approach is to completely automate Tier 1 tasks, incorporating them into orchestration and automation platforms, for instance. "You cut the number of people," says one CISO from a large company who requested anonymity. Tier 2 analysts become more of the skilled first responder who can move and adapt, and Tier 3 can tackle more proactive threat hunting and IR.

"So you bring in more skilled people," the CISO says, who can be trained to perform more advanced tasks like penetration testing, and operate with a smaller team.

Tier Mashup
At managed security services provider MKACyber's SOC in Fairfax, Va., data and detection are organized by use cases and attack types that can help guide junior analysts through the SOC process. Mischel Kwon, founder and CEO of MKACyber, says the goal is to operate less as a tiered operation and more as a collaborative one. The analysts have "gates" that lead them step-by-step through the process and release them to the next step, for example, she says.

So when a phishing email generates an alert, for instance, the process flow guides them through a review to confirm whether or not the alert is actually an incident. "They then upload all of the artifacts, and a [senior] analyst reviews their work and approves it for the next level."

By integrating the approvals process and IR actions, analysts of all levels work together, she says.

Kwon predicts SOCs will be more cloud-based in the next five to 10 years. "I'm hoping there's a change for the future with more integration of vulnerability management and remediation in the SOC" as well, she says.

Data-handling in the SOC also will evolve. "In the next few years, I see us doing different types of handling data ... getting more into tagging and more data science-type research as opposed to the SIEM [approach]," Kwon says. "I see us moving away from the SIEM model."

Related Content:

Black Hat Europe returns to London Dec 3-6 2018  with hands-on technical Trainings, cutting-edge Briefings, Arsenal open-source tool demonstrations, top-tier security solutions and service providers in the Business Hall. Click for information on the conference and to register.