'Very Noisy': For the Black Hat NOC, It's All Malicious Traffic All the Time

BLACK HAT ASIA – Singapore – When you're in an environment where the overwhelming majority of network traffic is classified as posing a severe cybersecurity threat, deciding what to be concerned about becomes not a needle in a haystack situation, but a needle in a needlestack problem.

That's the word this week at Black Hat Asia, where Neil Wyler, global lead of active threat assessments at IBM X-Force, and Bart Stump, senior systems engineer for NetWitness, took to the stage to give attendees a look inside the event's enterprise-grade network operations center (NOC). The duo oversaw the NOC's design and led the security team for the show, which ran from May 9-12. The multi-vendor network supported attendee Wi-Fi access; internal operations such as registration; the needs of business hall stands; and the communications requirements of technical trainings, briefings, keynotes, and vendor demonstrations.

"When we discuss the traffic, try to explain to others that at Black Hat it's bad all the time — all or most of the traffic is malicious," Wyler explained. "That sounds scary, but for this crowd that traffic is normal. There are people demoing attacks, there are red teams trainings going on, etc., and that means that we don't really block anything. We let that traffic fly because we don't want to take down a demo on stage or on the expo floor. Unless we see a direct attack on our infrastructure, say the registration system, we let it go."

So, in order to ferret out the actual bad, bad traffic, the NOC relies on a number of dashboards that allow a real-time view of everything flowing through the network, with the ability to capture stats on everything from device profiles to which cloud apps attendees are connecting to. It also captures raw packet data so NOC analysts can go back and rebuild sessions in the event something seems abnormally suspicious, to look at "every single thing someone is doing with every packet, in a way we can't using just logs," Wyler noted.

One of the more unusual dashboards put in place for the event offered a heat map of where Wi-Fi, Bluetooth, and even peer-to-peer wireless connections were being used, offering a quick look at where people were congregating and where there might be cyber issues afoot.

"It's an interesting perspective," explained Stump. "The bottom left corner of the map is actually the show floor, and after the business hall opened up, that got more red. You can see when breaks are happening and when they put the beverages out because people migrate. And overall, it's a quick visualization for us to see where potential issues might be coming from, where we should focus our attention."

A heat map of where devices connected to the network.

In all, the NOC tracked 1,500 total unique devices connecting to the network across mobile phones, Internet of things (IoT) gear, and other endpoints, with DNS queries at their highest for the event since 2018. About three-quarters (72%) of that traffic was encrypted — a refreshingly high amount, the researchers noted. And interestingly, a domain called Hacking Clouds hosted the most user sessions — more even than the show's general Wi-Fi network for attendees.

In terms of the apps being used, TikTok made an appearance in the Top 10 for the first time, the team observed. Other top apps included Office 365 (no surprise there), Teams, Gmail, Facebook, and WhatsApp.

Interesting NOC Happenings

A few interesting incidents emerged from the data during the event, the duo noted. In one case, an individual was generating so much malicious activity that all of the NOC systems alerted at once.

"One particular person was so noisy that every NOC vendor partner saw their activity at the same time," Wyler said. "We're talking SQL injection on public-facing websites, WordPress compromises, lots and lots of scanning for vulnerabilities and open ports. It was like they learned something this week and went, 'Let me see if it works. I've heard about Log4j, let me see what's out there.' They took a training class and now they're spreading their wings and flying."

After the person moved from attacking restaurant chain websites to probing payment sites, it was clear the activity wasn't demo-related, so the team pinpointed the person and sent the individual a cease-and-desist email.

"We figured out they were sitting in the hallway looking out at the Bay, just attacking company after company after company," Wyler said. "We explained that it's still illegal to do what they're doing, so please discontinue attempting to execute vulnerabilities on public-facing websites. This is a violation of the Black Hat Code of Conduct and we will come find you if it doesn't stop — love, the NOC. They got that and everything stopped."

Other incidents involved VPN issues, including one that was transmitting the user's location information in clear text. The team captured the data, plugged it into Google Maps and generated a view of exactly where the person had been during the day.

A VPN leak allowed the team to create a map of the user's location.

Yet another issue involved an endpoint detection and response (EDR) vendor that was sending all of the usage data it was collecting on the endpoints of its users in clear text back to its servers; one antivirus vendor was found sending unencrypted SMTP emails containing pricing quotes and other information in an unencrypted fashion, along with login credentials — allowing easy harvesting.

"An attacker could have pulled down quotes, changed quotes, gathered internal work information and customer information, definitely not good," said Stump. "It could be used to craft phishes or to manipulate pricing."

In all cases, the team worked with the problematic entities to resolve the issues. The NOC, quite simply, is on the case, according to Stump.

"People often say that at Black Hat, you shouldn't even get on the network because it's dangerous," said Stump. "But our goal is actually to leave attendees more secure than when they arrived. And that's why we do things like letting people know they're sending passwords in clear text, or when we see cryptomining activity, we'll alert them. We're committed to that."