Lessons From the Black Hat USA NOC

At Black Hat USA, the network operations center (NOC) and security operations center (SOC) are one in the same — reasonable for a network that exists to serve a huge gathering of security professionals. While the network that exists for a high-intensity week is unique in many ways, in others it is a concentrated example of what is possible when professionals with different areas of expertise — and different vendors — work together.

Neil Wyler, known to most as "Grifter," made sure that working together is what happened in the NOC. He described the scope of the Black Hat network operation, explaining that the team used the physical cable provided by the facility (in this case, the Mandalay Bay Convention Center) but brought in everything else. From demarc to access points, everything was deployed in less than a week, then was expected to operate without flaw for another week before it disappeared.

In many respects, the network echoed the design of an enterprise network that would support tens of thousands of active users, according to Steve Fink, the chief architect of the Black Hat network. The major difference, he said, was that at the end of the conference, the team presented a "State of the NOC" report that laid out the entire architecture of the network, from device manufacturers and model numbers to IP address range. That last detail, he said, means that the network must be redesigned and completely redeployed at every conference.

As lights in the NOC blinked to signal activity, and screens changed to show flows and conditions, security and network professionals from an array of vendors watched and discussed changes with one another. Where paleontologists Niles Eldredge and Stephen Jay Gould used "punctuated equilibrium" to describe their model of how species develop, "punctured specialization" might be an apt description of how professionals worked together in the Black Hat NOC. Each vendor sent specialists (often, quite high-level specialists) to support the equipment they supplied to the network, but in the NOC you saw these specialists visiting other vendors' work areas, talking about both how their equipment was working together to protect the network and its users, and about how the equipment itself worked — sharing information among curious professionals eager to increase their knowledge in the field.

Some of those professionals documented their time in the NOC in their own blog posts, while others shared experiences in video form. In both cases, the information shared was an echo of the data that moved between team members for the two weeks of setup and operation for the network.

That model might be one of the most valuable things to come out of the Black Hat NOC. We hear experts talk about specialization and generalization, and how one or the other is in ascent or eclipse, but the fact is that most security professionals have areas of specialty. When a culture and management join together to encourage those specialists to be curious about other specialties and share their own expertise, the overall knowledge level of the organization can increase, and managers might find that job satisfaction goes up, burnout rates go down, and turnover is reduced.

A Black Hat NOC happens three times each year, in Las Vegas, London, and Singapore. Much of the staff repeats at the different sites, though new members are always coming in as vendors change and the staff within vendors move from position to position. Punctured specialization allows new team members to come in and be quickly welcomed for the knowledge they bring and the curiosity they share with their fellow team members.

That's a lesson that security managers can draw from, regardless of the equipment deployed on their networks or the size of their teams. And it's a lesson the Black Hat NOC team will keep teaching at every conference.

Related Content:

Black Hat Europe returns to London Dec 3-6 2018  with hands-on technical Trainings, cutting-edge Briefings, Arsenal open-source tool demonstrations, top-tier security solutions and service providers in the Business Hall. Click for information on the conference and to register.