I’ve been thinking about Threat Intelligence (TI) and Threat Intel Platforms (TIP) lately. What makes a platform useful? What role should Threat Intel play? Does TI even matter? Just so we’re on the same page, Threat Intel is not just Indicators of Compromise (IOC). It includes IOCs, but it’s also the knowledge, context and evaluation of those elements that inform decisions and action.
What makes a TIP? We can break it down to the essential functions:
Collection – TIP is the repository for all collected data. There should be an ETL process that accepts all formats and can output all that data into common formats like JSON, STIX, etc.
Contextualization – TIP should enrich existing data and assist in the evaluation of new data. Combining multiple data sets to create a new dataset is synthesis, this is a core function of a TIP. It should be create high quality data from multiple less useful data sets. Different perspectives of the same data increases the fidelity of data in the system. This is an often overlooked part of a TIP, aggregation of data can reveal new information. This is why TIPs exist.
Integration – Data from the TIP should be API accessible for other systems and devices. A TIP is only a part of a larger toolset. It can serve multiple users and use cases. TIPs can help Hunt Teams, Security Operation Teams, Incident Response Teams and Compliance Teams. Anyone that needs context for an investigation is a user.
Correlation – The TIP should make it easy to compare data and recognize trends. Everything with networking and security boils down to link analysis. Can we link infrastructure to an actor? Can we spot patterns in the data? A TIP should hold massive amounts of data and the result should be a trove of valuable intelligence.
Analysis – Any work done on the data outside the TIP should be easy to store in the TIP. Ideally a TIP would have the built in ability to automate analysis, but they can’t always keep up with new techniques. Using an API, a user should be able to pull data from the TIP, analyze it and push a new data set back into the system. This is key for keeping up with evolving threats.
Action – A TIP should have the ability to trigger actions based on rules defined by users. These actions can be as simple as sending an email or as complex as triggering ACL changes.
If it isn’t obvious…that’s all super complicated. A system that can do all those things well is nearly impossible. Trying to store any data type, while still being fast and searchable… good luck! Throw in the element of time and one could go insane building that platform. Users want a TIP that can answer queries on data that’s ingested in real time, but they also want a system that stores historical data forever. If you’ve ever built a system that deals with timestamps, you now these two paradigms can’t easily exist in the same data store. A datastore that can ingest data in real time and allow near instant queries against it, is not particularly well suited to storing historical data. This is one of the many difficult problems a TIP has to solve.
A platform that consumes TI should:
Have reliable data – without reliable data a TIP will become a liability. Delivering false information leads to bad analysis and unnecessary actions. If my TIP takes me on wild goose chases, I won’t rely on it for analysis. When data goes into the TIP, I should be confident that it’s displayed correctly and accurately.
Save analysts time – Instead of using ten tools to gather information about a threat, they have a single place to view and evaluate information.
Streamline analysis – A TIP needs to make my job easier. If I can manipulate data faster in a spreadsheet or a text file, then I’m not going to use the TIP.
Integrate with my existing tools – If the data can only live in your TIP, it’s of limited use. The TIP is a data repository that allows direct access to the data, but also enriches other tools. The TIP should make everything better and not attempt to replace existing tools. A robust API is essential to getting that valuable information to where it can be actioned. Feeding it to security devices, feeding proxies, and feeding security teams. The data needs to be usable to be useful.
Evolve – A TIP should have a long life. It should be instrumental in automating repetitive tasks. Saving an analysts time is just the start, once the analyst defines a process, that process should be replicated and automated. For example, identifying particular threats that impact the enterprise and sending alerts. Evolution is important. Novice analysts will perform the same query daily. Experienced analysts figure out how to script that query so results are delivered automatically. Expert analysts turn those queries into alerts that can be acted upon. It’s a maturity cycle that will determine how useful your tool will be. Analysts will constantly grow. If the tool doesn’t grow with them, it’s a poor investment. The tool you choose needs to assume your analysts will become experts and assist them as they advance.
Be Trustworthy – This is one of the most important features of a TIP. The user should have confidence the data they are evaluating is accurate and trustworthy. If the tool has data discrepancies, it becomes difficult for the analyst to trust what they are seeing. This slows analysis while the user cross checks the results. Once a tool has demonstrated that it can’t be trusted, it becomes a challenge for that tool to regain that trust. Losing trust in a security tool means certain failure for that tool. Infosec workers tend to share experiences with other analysts. It won’t take long for word to spread if your tool has bad data and is unreliable.
Everyone wants a TIP. Everyone attempts to build a TIP. As soon as you collect IOCs and put them in a database to compare to other IOCs, you’re creating a TIP. It’s basic, but it’s using correlation to create knowledge and intelligence from collected data. There are endless homegrown TIPs being created. In many cases, the homegrown TIPS are sufficient. It’s when the amount of data being collected becomes overwhelming, that those TIPs start to show their limitations. The most common fix is to purge old data. Purging can work, but there is much context to be learned by including previous activity into our analysis.
I’ve often wondered if the razor blade model would be good for a TIP. Give away the TIP for free and sell data feeds that are plug and play. A free TIP could provide the basic features and users could purchase analysis plugins that performed advanced tasks on the existing data. Plugins could be created and open sourced just as easily. With a robust free platform, the user base would expand quickly.
A free platform would be the razor and the razor blade business would be booming.
