Pen Testers Who Got Arrested Doing Their Jobs Tell All

When they first scanned the cardkey to the front entrance of the Dallas County Courthouse in Iowa, red-team experts Gary De Mercurio and Justin Wynn didn't hear the requisite click of a lock disengaging. It was after midnight on Sept. 11, 2019, the last leg of their penetration-testing engagement for the state of Iowa's Judicial Branch, and they got their first big surprise of that now-infamous evening.

"Justin grabs the door and we look at each other, and I said, 'Did it work?' and he's like, 'No, it's open,'" recalls De Mercurio, a senior manager at Coalfire. "The door was locked, but they hadn't latched it all the way."

So the two social engineering and physical pen-test experts could get a more accurate take on the entrance security, Wynn closed the door and they started all over again with the cardkey, this time with the door locked. De Mercurio then slid a plastic cutting board retrofitted with a handy notch into the doorjamb and used it to unlatch the door. The pair figured they had somewhere between 20 to 30 seconds from then until the building alarm would sound, so they executed the usual next step in the physical testing process: checking the strength of the alarm's passcode settings by first typing in the system's default code as well as easy-to-guess combinations.

Once the alarm sounded, the pair went back to work looking for other potential vulnerabilities in the courthouse while waiting to see if the authorities would respond. In three other facilities they had tested for the state agency, building alarms had not dialed out to law enforcement — a significant security hole. "I had my fingers crossed, hoping this one dials out and gives the client a softball win because everything else was pretty abysmal that we had encountered" security-wise, says Wynn, a senior security consultant at Coalfire.

It did, and that's when the second big surprise came: an arrest, followed by felony charges, a night in the slammer, and nearly five months of a hellish legal quagmire driven mainly by a power struggle between state and county officials in Iowa over who had legal jurisdiction over the courthouse building they had entered. De Mercurio and Wynn, who were fully exonerated in January after all charges against them were dropped, today at Black Hat USA Virtual will publicly share the full story of their harrowing experience and how it's shaped new pen-testing engagement protocols at their company — and their advice and recommendations for fellow physical pen testers so they can avoid a similar backlash to their social engineering and physical pen-test engagements.

"They Disavowed You"
It took just a few minutes after the alarm blared for officers to arrive at the turn-of-the-century county courthouse structure in the city of Adel, which sits across the street from the Dallas County Sheriff's Department. De Mercurio, a former Marine, and Wynn, knew the drill: make verbal contact, show your hands, and be very cautious in your interaction with responding officers. As the officers stood outside the door preparing to enter, De Mercurio and Wynn stood at the top of the staircase in the courthouse and shouted out their names and who had hired them, explaining that they were performing a security audit on behalf of the Iowa Judicial Branch's State Court Administration.

When they were met with silence, they waited a few more minutes and then descended the stairs, carefully approaching the door with their hands out until the deputy motioned for them to come outside.

Although Coalfire had contracted with the State Court Administration in various engagements since 2015, this was the first time Wynn and De Mercurio had worked for this client. This engagement was a full-scope red-team project, including external and internal testing, application penetration testing, social engineering, and a physical pen test. Aside from their physical pen-testing toolkits, the pair were armed with what they call a "get out of jail free" letter, a written authorization signed by the judicial branch that proved they were working on behalf of the state.

The officers checked their IDs and verified their story, and determined they were legit and free to go. De Mercurio says the interaction with the officers was professional and ultimately amicable — that is, until the county sheriff arrived. "As soon as the sheriff shows up, everything changes. People start to disgengage, they start to back off the steps, and it quickly becomes us versus them" with some of the officers, he says.

He says Dallas County Sheriff Chad Leonard berated De Mercurio and Wynn for thinking that the courthouse was under the state's jurisdiction. "He basically tells us we should feel pretty stupid that we didn't know the courthouse belongs to the county, not the state," recalls De Mercurio.

De Mercurio and Wynn were handcuffed and marched across the street to the sheriff's office, despite one of the responding officers vouching that the men had been cooperative and could just walk over with them rather than be led in handcuffs. "But it's obvious that he's [Leonard] mad because the state has sent us and he doesn't think the state has jurisdiction — that they're just stepping on his toes."

And when the pair's arraignment six hours later occurred in the very same courthouse they had broken into and been arrested just hours earlier, the irony wasn't lost on them. "The judge took it personally; clearly she had not been filled in," Wynn says. "All she has been told is we caught these two guys last night breaking into the courthouse, and then she kind of loses it ... and eventually raises our bail by 10 times the norm."

The judge set their bail for $50,000 each, rather than the usual $5,000, for felony charges of burglary and possession of burglary tools, they say. "We're charged at this point with felony arrests," De Mercurio says. Coalfire paid their bail, and after nearly 20 hours in custody, the two were free to go home. But their case took several new, more complicated twists. One such problem surfaced while they were jailed: The state officials who had hired them were now saying the two weren't supposed to be testing their systems. "'They disavowed you,'" De Mercurio recalls his boss telling him in a call from the jail.

For the sheriff, it was a "power play," Wynn says, and for the state officials, it was all about covering themselves from any blame. "So, you had these two powers at play and everyone's trying to cover themselves."

The two men had been on-site in Iowa for several days, conducting physical and logical pen tests after-hours at three other state buildings. The day before their arrest they had captured the primary flag for the project: successfully gaining access to the Iowa Judicial Branch's network. They had set up a drone device at anther courthouse — the Polk County Courthouse — where they plugged in to a network switch that would ultimately provide remote access to the network. "We verified it was on, but we never really got to work with it. It was connected to our servers," but after the arrest, officials in Polk County removed it. "The Polk County Courthouse wasn't even aware it was in their location until the Dallas County police picked us up."  

Painful Lessons
It took a state legislative hearing to lead to Dallas County officials finally dropping all charges against De Mercurio and Wynn in late January 2020.

Meanwhile, the nearly five months between arrest and the exoneration had its share of drama: The Polk County Sheriff's office "threatening" Coalfire with legal action, according to De Mercurio, and state officials ultimately conceding in the hearing under questioning that the Coalfire pen-test and physical engagement had been sanctioned by them, he says.

When asked by Dark Reading for comment and an interview with state officials about the case, a spokesperson for the Iowa Judicial Branch referred to an Oct. 4, 2019, statement by the late Iowa Supreme Court Justice Mark Cady to the state's Senate Government Oversight Committee apologizing for "mistakes" that were made in the case; a report from outside counsel on the case; and an Oct. 10 Supreme Court ruling that ordered the state to obtain legal review of all information security contracts, to get state court-administrator approval for pen-testing engagements and coordinate with local and state law enforcement, and to ban after-hours access to courthouses and physical "break-ins" by pen testers. The ruling also called for contracts to "distinguish between 'physical testing' and 'penetration testing.'"

The Coalfire pen testers maintain that there was no miscommunication with the state officials who hired them for the gig. They say they discussed all of the attack scenarios and vectors in a two-hour phone meeting about the engagement, including floor-by-floor plans for each building they would test. "They had no doubt," Wynn says.

But they didn't record the call, which they now regret. "Always record your phone calls, at least with physical engagements," De Mercurio advises.

They also recommend ensuring an engagement contract is reviewed by lawyers. "Try to make your contract as ironclad and succinct as possible," De Mercurio says.

Their experience has made them advocates for protecting physical pen testers, and they are pushing Coalfire to update its policy to reflect the risks. They say getting legal involved prior to the sales pipeline is key, and that deals are well-vetted beforehand.

"I always thought this [arrest] could be possible, but absolutely the charges would be pressed against the company and I'd have personal protection," Wynn says. "So I will push that out to other testers: Verify that your company has blanket protection" for you, he says.

In reality, he says, Coalfire legally could have left him and De Mercurio out to dry and washed their hands of the case. But Coalfire's CEO Tom McAndrew stood by them.

"Tom did the right thing. The legal team said, 'they're not charging Coalfire — leave them there and let them figure it out,'" but McAndrew resisted, De Mercurio notes. "He went to the executive committee and said find me a lawyer and bail them out."

Among some of the policy changes Coalfire is considering is replacing some of the physical security testing tasks that could potentially trigger legal troubles or misunderstandings. According to De Mercurio, that means some possible new services that allow them to do that work without putting them at risk. They say they're also exploring how to create a community bail fund.

Being able to freely tell their story is liberating; they faced plenty of press scrutiny and even some from the security community about possible missteps that could have led to the legal troubles they experienced. "It's good to be able to come forth with the details and to be able to validate [our actions]," Wynn says. "On the [legal challenges on] the question of scope, the entire industry rallied around us. Our infosec family took care of us, and we appreciate it."

Register now for this year's fully virtual Black Hat USA, scheduled to take place August 1–6, and get more information about the event on the Black Hat website. Click for details on conference information and to register.