Open Source Tool Looks for Signals in Noisy AWS Cloud Logs

When investigating a potential attack on cloud services, Daniel Bohannon frequently has to contend with the verbose logging of Amazon Web Services (AWS), a problem that can allow an attacker to hide in an avalanche of data.

While AWS typically produces only a single event for every programmatic API call, accessing the management console through the Web leads to an exponential increase in events, says Bohannon, a principal threat researcher with cloud identity service provider Permiso Security. In one session, for example, 81 clicks on users, workloads, and roles resulted in 5,370 events recorded in the AWS log file.

The sheer volume of events results in so much noise that it becomes hard to determine what a user is actually doing in the AWS console, the threat researcher says. For that reason, Bohannon and fellow threat researcher Andi Ahmeti plan to release an open source tool at the Black Hat Asia conference in Singapore to help security managers and incident responder consolidate cloud log events into a record of user actions.

"The idea is that you input your raw logs — and we [show] 100% of those raw logs — but then we enrich the data on top of that, which ... contains all the information about the events that led to those events," he says. "We all have that full signal information, which ... contains the summary, the labels, all that kind of stuff."

Historically, the volume of data in log files have made it difficult to determine the events that led up to a compromise. Sometimes the problem is that the cloud service does not log enough events to determine what happens, such as last year's criticism that Google Cloud Platform (GCP) fails to log adequate data when a user accesses a storage instance.

In other cases, the specific ways that cloud services communicate information to their customers can result in a lack of visibility, especially with the differences that companies face with multicloud use. More than half of companies have open ports undermining their security and take some two months to close the vulnerabilities. 

AWS Produces Avalanche of Events

For businesses using AWS, the number of events produced in a log while using the Web console can be significant. Just clicking on a list of users produces 18 events for only three identities, and that's a mild case of AWS's verbosity, says Bohannon. A user who clicks in the AWS console to view the users in the identity and access management (IAM) console will see more than 300 events produced in the logs for CloudTrail, Amazon's auditing capability.

"That number can actually get as high as 100, 300, or even 700 events, depending on the settings you have in your Web browser — all just for one click," he says. "So at a core level, every single action you take produces at least one event, but often it's dozens or sometimes even hundreds of additional events associated with it."

The researchers' open source tool, Cloud Console Cartographer, aims to turn the list of events captured by CloudTrail into a succinct timeline of actions taken by the user. The program adds comments to the cloud log that categorizes a series of captured events into signals — actual user actions.

"We want to show all of our mapping to remove as much noise as we can but still keep all the raw events," Bohannon says. "So anything that's unmapped is great, it's still evidence, and defenders can make the most sense of it."

No Plans for Other Clouds

Cloud Console Cartographer, which will be available on GitHub, produces an enriched log of events and has a Web interface that lists the signals in a table. Currently, 240-plus rules for classifying collections of events into user actions — that is, signals — have been created that will be used to enrich log files.

The two threat researchers intend to keep working on expanding the number of classifiers and hope that others will do the same.

Bohannon and Ahmeti may move on to developing the tool for other cloud platforms, but because different cloud providers have different ways of logging, what works for AWS will not work for Microsoft Azure or Google Cloud Platform, they say. AWS is verbose, but Azure is the opposite — its logs are so terse as to be unhelpful, Bohannon says.

"I feel like every platform, every cloud platform has unique challenges that will have to be addressed in different ways," he says. "So we might find in the future that we can integrate other cloud platforms into this, but [for now] we at least have plans for additional GUIs related to AWS that we are going to be working on after the initial release."