Rashōmonitoring

There’s something to be said for pure, unprocessed data: You know it doesn’t come with any assumptions.

Here’s a simple example: Logs show use of an application from an executive’s phone in Maryland. They also show some failed login attempts from an unknown device in Tokyo, within two hours of the other events. Now, some analytics would assume that the second set of logs was an attack from Asia, and the APT ALARM would go off, with "hacking-back" teams, energy drinks, and virtual chest bumps all around.

But suppose the executive really was in Tokyo and had left her phone at home, where her 6-year-old picked it up and started playing with it. And because she’d left the phone at home, she was borrowing someone else’s iPad -- and, it being late at night after a liquid dinner, the login process just wasn’t working as well as it usually does.

Security products are featuring more analytics these days to help automate and speed the interpretation and response process -- and that’s good because humans are both (relatively speaking) slow and expensive. But any rules, algorithms, or interpretations of the data can also reflect the perspective and assumptions of whoever created them.

These perspectives can clash, as shown in Akira Kurosawa’s classic film "Rashōmon," in which the main characters all relate their versions of the same story. In the same way, analysts can put their own interpretations on security events, depending on their own states of knowledge and even the order in which they see the data. Here are some assumptions that you may want to take into account when using automated or manual analysis:

"

"

In order to avoid falling victim to unconscious (or undocumented) assumptions, make sure you know the models behind your analytics. Are you using a product from a company that started in the defense sector? Is the statistical analysis intended to detect fraud in financial transactions, not overdue library books? Are you using statistical baselines that are out of date and don’t reflect your current application traffic? How are historical events weighted in analyzing new ones?

I’m not saying that you should distrust your SIEM. But I am saying that you shouldn’t stop questioning it, or yourself. Once in a while, take a fresh look at your unfiltered data sources, shake up your reporting, and have a different person interpret the alerts in your SOC. Make sure that you haven’t become complacent in your everyday monitoring because what you see tends to become what you expect to see.

(I would like to thank Sandy "Mouse" Clark at the University of Pennsylvania for her discussions on this topic; she’ll be coming out soon with new research around how assumptions affect security.)

Wendy Nather is Research Director of the Enterprise Security Practice at the independent analyst firm 451 Research. You can find her on Twitter as @451wendy.