User Awareness Training: Gauging Success

As with most endeavors that require time and money, security awareness and training programs have their champions and their naysayers. Among the naysayers are those who believe there is no way to measure the effectiveness of employee-focused cybersecurity education. The champions believe that there are always proven ways to gauge success and return on investment. Whatever camp you fall in — for, against, or neutral — it’s worthwhile to consider metrics that can demonstrate how effective (or ineffective) your security awareness training programs are.

Think Beyond Click Rates
Because phishing attacks bring significant pain to organizations of all sizes, infosec awareness teams tend to focus on the use of simulations when implementing an anti-phishing training program. There is no argument that phishing tests are a great tool for assessing end-user vulnerabilities. But the click/no-click measurements offered by these types of assessments tell only part of the story.

Our recently released Wombat Security Beyond the Phish™ Report offers a good example of the types of insights that can be gleaned from question-based knowledge assessments and questions asked and answered within training modules. The report shows that, year-over-year, end users exhibit a better understanding of the tactics social engineers use to deliver phishing attacks, and the ramifications of falling for a phishing email. The report also showcases the differences between click rates and knowledge assessment metrics. In comparing data points from the same industries, the report showed that click rates alone can give organizations a false sense of security because they imply a knowledge level that is not borne out within question-based assessments.

At the end of the day, one phishing email example is just that: one phishing email example. Cybercriminals are crafty, using different themes, tools, and tactics to fool end users. That’s why relying strictly on go/no-go click metrics to gauge understanding can leave you only marginally informed. It should go without saying, but in case it doesn’t, basic data on completion of training does not give you adequate insight into end-user knowledge levels. So when you are evaluating training tools, look for those that can provide actionable data about user strengths and weaknesses.

Think Beyond Training Metrics
You don’t need to rely strictly on your awareness and training tools to deliver data that speaks to the effectiveness of your program and the return you are achieving on your outlay. In fact, you can tap into many of the security events that you already are (or should be) tracking to help measure the success of your awareness and training efforts.

The following types of metrics — and the changes you see over time — can help you make the case as to whether your program is paying off (or whether your efforts are misplaced):

The simple reality is that if you are running an effective security awareness training program, you should see positive changes in these types of metrics over time. And if you are not, you should rethink what you’re doing. Most of the metrics noted above are at least indirectly — if not directly — linked to security spending. Though you may not be inclined to tie all improvements to your security awareness training program, it is nevertheless reasonable to say that more informed, better educated users make fewer mistakes, which in turn puts less of a strain on infosec teams and security budgets.

Gretel Egan is the Brand Communications Manager for Wombat Security, a leading provider of cybersecurity awareness and training software that helps change employee behaviors. She has extensive experience researching and writing technical and non-technical content for a variety of industries.