Posted
In the first blog of this series, I took a 30,000 foot view of the seven tenets of TI Ops defined in the Dawn of Threat Intelligence Operations paper. In this blog, the last in the series, I’m discussing Tenet 7 – Creates measures of effectiveness and success for produced and consumed threat intel that are understandable and relevant to the business.
CTI teams struggle to measure the effectiveness of their efforts, which affects their ability to measure return on investments, measure success, and ultimately communicate the value of the CTI function.
Jon Oltsik found in his recent cyber threat intel survey that – “Seventy-one percent of security professionals say it is difficult for their organizations to measure ROI on its CTI program. Given that many organizations don’t know what to look for, are overwhelmed by CTI volume, or treat threat intelligence programs like graduate school, this one comes as no surprise. CISOs suffering from one or several of these problems will find it difficult to pinpoint measurable benefits from CTI dollars.” Cyber threat intelligence programs: Still crazy after all these years | CSO Online
Obviously,this is not the situation CTI teams or CISOs want to be facing. However, this is not an intractable problem. I believe there are three activities that will aid TI Ops teams and leaders to mature their measurement capabilities, allowing them to determine their level of success and to quantitatively demonstrate the value of their CTI investments. But be aware! Like any other measurement exercise, it takes effort, time, and discipline. It’s not something that you will create overnight and it’s not a one-and-done exercise. Andrew Jacquith said it well in his book “Security Metrics” (the picture is of my well-used copy from 2007).
“Metrics do not suddenly appear fully formed like Athena from the forehead of Zeus. Metrics are created through an evolutionary process that gains consensus and ensures regular and thorough reviews both before and after publication.” (p. 244)
So here is your plan of attack knowing the journey ahead:
Align to the Evolved Threat Intel Lifecycle, paying particular attention to the Planning and Direction, and Feedback and Validation phases.
Planning and Direction is where the engagement with stakeholders occurs, understanding, agreeing, and documenting their requirements. This may seem daunting, and based on the survey findings from various studies, it’s still a challenge for CTI teams, but it’s not hard to build intel requirements. Having intelligence requirements allows you to create metrics and measure how well requirements are being met. You can measure how well the requirements were met and identify areas for refinement and improvement.
Feedback and validation is also important. One of the critical functions of metrics is to facilitate discussions and debate. Measuring the performance of intel requirements naturally brings stakeholders into the conversation, ensuring they are engaged and participating, and enabling refinement and improvements, which benefits the intel producers and consumers.
Have a plan to identify, agree and document metrics, measure the CTI function, validate, and report the success, areas of improvement, and value of the CTI team.
This is nothing more than good project and process management, but it does take dedicated resources, time, and investment from cohorts, leaders, and stakeholders. Back to Security Metrics, I like the approach Andrew provides (p.244) with a modification – closing the feedback loop.
- Identification
- Definition
- Development
- Quality assurance
- Production deployment
- Visualization of results
- Analysis of results (including validation)
- Implement changes / improvements (including actioning feedback)
I like to map this model to the Deming cycle – Plan, Do, Study/Check, Act,
Having a process mindset and methodology makes the metrics and measurement lifecycle more approachable and sustainable.
Define metrics and measurement alongside the investments and maturation of the CTI function.
When you start your CTI journey, don’t forget about metrics. The key is to realize that in the early phases of implementation, you should focus on metrics that may not have the highest value, but are reasonably low effort to collect (lower left quadrant in the 2×2 below). This gives you a starting point that becomes a baseline for measuring and it also doesn’t put an unreasonable burden on the TI Ops team. This will likely be more operationally oriented metrics like:
- Number of intelligence requirements being tracked
- Number of feeds being ingested
- Number of Indicators in the library
- Number of reports produced
- Number of incidents worked
Over time, the plan should be to expand into the lower-right quadrant, where there is more value but the effort will be slightly higher. This would include metrics like:
- Indicators observed
- New incidents from CTI
- False-positive ratio
- Counter-measures implemented
And ultimately aiming for the upper-right quadrant, where metrics like these exist:
- Mean time to detect and respond
- Incident criticality impacted by CTI
- New intelligence created
- Person hours effort saved through task and process automation
The tl:dr is to start measuring! Pick a few metrics and start measuring, preferably before you kick off a new CTI project. This was one of the topics some of my cohorts, and I discussed in this blog on how to get started with a TI Ops program.
How the ThreatConnect Platform Helps
When you are collecting data and knowledge in a central location, generating metrics gets easier. The ThreatConnect Platform has a variety of tools and means to help. For example:
Leverage Dashboards
The highly flexible and customizable Dashboards in the platform make it easy to generate and display real-time metrics like intelligence requirements being tracked, feeds being ingested, Indicators in the Library, reports produced, and incidents being worked or completed.
Built in ROI measurement
Playbook automation has the ability to measure the ROI when automating tasks and processes. You provide a few parameters, and the Platform does the calculations. These can then be used in Dashboards and Reports.
Thanks for reading!
I hope you enjoyed reading this blog series as much as I did writing it. If you’d like to learn more about Threat Intelligence Operations and the ThreatConnect TI Ops Platform, please reach out to one of our experts.
Check out the full list of tenets in the original blog post of the series:
