Airline Security: The Technical Task Of Connecting Dots

In his investigation into the government's bungled handling of the would-be Christmas Day bomber, President Obama was miffed to learn that sufficient information existed to reveal the plot, but that intelligence experts failed to "connect the dots." The President will soon find that connecting the dots is the hardest part of this highly complex intelligence undertaking.

"This was not a failure to collect intelligence; it was a failure to integrate and understand the intelligence that we already had," Obama said of the intelligence breakdown. "The information was there." He has ordered a review of the government's aviation screening technology and procedures and its terrorist watch list system, and preliminary findings are due as early as today.

There are many pieces to this puzzle, including intelligence data on known terrorists and suspects, information gleaned from passports and visa applications, ticket purchases, airport screening systems and procedures, airline passenger lists, video surveillance, information generated by acquaintances of terrorists and suspects, phone records, and even clues on social media sites.

Pulling those data streams together--from federal agencies, law enforcement, foreign governments, and private sector companies--and getting that information to the right people quickly and in useable format are huge technical challenges. While there were obvious missed opportunities in the case of Umar Farouk Abdulmutallab, including failure to take action with information in hand, it would be a mistake to underestimate the end-to-end data integration effort required as one of, simply, "connecting the dots."

Many corporate IT departments struggle daily with similar data integration issues in their customer service, supply chain, manufacturing, and other operations. They may have invested millions of dollars in data analysis infrastructure, but be caught off guard when a disgruntled customer jumps to a competitor.

For TSA, Homeland Security, and other government agencies on the front lines of the fight against terrorism, the consequences of missed signals are potentially much more serious, but the technical challenges of assimilating and analyzing data from myriad sources are familiar. Intelligent Enterprise's Doug Henschen, an authority on business intelligence, says the feds face a classic information management challenge, one of sifting through many terabytes of structured data in disparate databases and unstructured data in the form of documents and e-mail, all in hopes of finding a few fragments of potentially life-saving information.

The information is "there," as the President says, but how do tidbits of data in a dozen different places get transformed into actionable insight? Some of the technologies and practices that come into play include enterprise content management, master data management (i.e. data governance), data cleansing, complex event processing, text mining, identity resolution, data integration middleware, data mining, BI tools, relational databases, and data warehouses. U.S. intelligence agencies have already implemented many of these capabilities, so the question is whether they've done so effectively and what more can be done.

President Obama has ordered that corrective steps be taken immediately, which means agency heads and CIOs are already busy evaluating what to do differently as they try to stop the next bomb-clad terrorist. Emerging technologies could play a role. In-Q-Tel, the CIA's tech investment arm, has recently invested in social media analysis software and open source search capabilities, in two examples of the possibilities. As InformationWeek's Alex Wolfe reports, Obama's security push is also spurring scanner patents.

Of course, technology alone won't do it. Security lapses in the Dec. 25 incident weren't merely a matter of technical limitations or failings, as the forthcoming White House report will demonstrate. Solutions will require attention to people, process, and communications.

Federal CIO Vivek Kundra should play a central role in the push for better government intelligence. Kundra knows how to scrutinize IT architecture to unearth data and how to rally government CIOs around a common cause, as he's doing in support of Obama's government transparency initiatives. Kundra must tackle homeland security and the fight against terrorism with the same focus and urgency.

There's a saying in the BI crowd that hints at the intractable nature of the problem: "If only we knew what we know." (There's a book on knowledge management by that title.) President Obama is understandably frustrated over the government's inability to aggregate and share critical, time-sensitive information, and he's right to insist on improvement and accountability. Getting it right won't be fast or easy, but government intelligence can and must be better.

Government Technologist is a regular column by John Foley, editor of InformationWeek Government. You can follow him at Twitter.com/jfoley09.

Federal CIO Vivek Kundra is our Chief of the Year. Find out his plans for executing on his many goals as well as the many challenges ahead. Download the report here (registration required).