Posted
There are many advantages to having a centralized Threat Intelligence Platform (TIP) to aggregate, analyze and act on your own threat intelligence. Among them, is empowering the threat analyst to interact with new threat data as it is aggregated by providing a direct interface to speed up their workflow. This makes collaboration easier and essential to the threat analysis process. Analysts using ThreatConnect can take a set of raw indicators from any source, be it human or machine generated, and use the features of the platform to breathe new ‘life’ and relevancy into their raw data set. What do I mean by ‘life’? I mean, having a collection of indicators that are just as dynamic as the adversary who leverages them. The indicator auto-enriches and is associated with relevant details tying them to the key infrastructure components used in a malicious campaign or attack. Indicators that are “alive” are not stagnant; they are not sitting idly on a spinning disk waiting to become obsolete, and with ThreatConnect, you can put them to work for you. Consider the time an analyst saves by having ThreatConnect answer a few questions that should be asked of any new indicator:
- Who in my community, or among my internal ThreatConnect Organization, has seen these same indicators?
- Has anyone already associated these indicators with another incident, threat or adversary?
- For my host and IP indicators, what DNS and WHOIS lookup data can I obtain and how frequently has it changed?
- Can I be alerted if this indicator updates or if someone else has seen it?
Having the ability to quickly and easily get answers to these questions is just part of what makes a community-driven approach to threat intelligence so powerful.
So, let’s begin with the basics of importing content into an incident:
Create an Incident:
One of the key features of ThreatConnect is being able to quickly organize your data, group it and associate it, developing an ever increasing amount of context over time. In this example, we will create an Incident but other groupings are available to us depending on what our analytic usecase may be.
Step 1: Create
Step 2: Categorize and Enrich
We can apply as many system level or custom attributes to an Incident as we need to capture all of the relevant details. We can also apply Security Labels and custom Tags (a feature we will address later on in the context of indicators).
Now that we have an Incident created, it is time to populate it with raw information or finished analysis, depending on your preference/individual use case. Let’s take a look at how we import data within ThreatConnect.
Importing indicators: Structured vs. Unstructured
Within ThreatConnect, structured data imports allow for maximum levels of user control over the data before it is imported and require less overhead afterwards. Since finding and aggregating indicator data in a structured format is not always possible, ThreatConnect also provides a powerful feature for parsing indicator data from unstructured sources such as text, PDF and other document formats. It also maintains analytic creature comforts such as “find and replace” features that can defang malicious URLs, domains, or IP Addresses that have been modified so they cannot be clicked on. This means you can grab any text from your favorite analysis blog or the latest PDF write up provided by <insert your favorite security vendor name here> and load it directly into ThreatConnect for indicator parsing and extraction. This dramatically speeds up analyst workflow and threat discovery by allowing analysts to completely bypass what would otherwise be a lengthy and unwieldy process of manual extraction and data massage. It also allows you to include indicators in your analysis that may otherwise have fallen by the wayside in unread technical whitepapers or blogs.
ThreatConnect Research TIP: There are times you may want to just use only Structured or Unstructured imports, other times, you may want to use both. It all depends on how the data is presented to you. Consider using the filtering feature to assign common attributes (Descriptions, Sources etc.) or ratings and confidence values. These can always be updated or changed manually or programmatically via the ThreatConnect API.
What follows is a quick, step-by-step guide on how to import a structured set of indicators:
Step 1: Import
To import data from within my ThreatConnect Account, I simply select import indicators from the top right menu bar:
I’ll want to select ‘STRUCTURED CSV’ since I am using a spreadsheet. For more information on how to structure your CSV, please reference the tool tip within the STRUCTURED CSV option.
Note that my CSV contains the correctly formatted indicator TYPE with the appropriate indicator VALUE. I have included a default DESCRIPTION and SOURCE attribute for additional context and referencing. Of the populated fields, the last two contain a ‘1’ and are optional as they tell ThreatConnect to enable automatic historical WHOIS and DNS lookup information for HOST indicators. It is important to note, that if your WHOIS and DNS results are not immediate post-import, don’t worry – ThreatConnect has scheduled them and they will be auto-enriched.
ThreatConnect Research TIP: It is a good rule of thumb to enable the WHOIS and DNS tracking feature because you want ThreatConnect to update any infrastructure or registration changes. ThreatConnect will also make associations to overlapping WHOIS metadata and DNS resolutions, revealing any non-obvious relationships over time.
By default, the target destination you will import your data into will be your private individual or organization account. However, you have a choice to import data directly into a community to which you may belong – such as the Common Community. Keep in mind that these communities may have their own anonymity and data sharing policies and code of conduct, so it is important to first understand the differences between private imports and community imports.
Step 2: Validate
In this step, I simply need to validate that all eight fields were correctly identified. Thankfully, ThreatConnect helps you during this validation process. If you happen to have errors at this point, just double check the structure of your CSV and make sure it conforms with the format outlined in the import tooltip. After clicking next, I can then validate that all ten of my indicators were found.
Step 3: Confirm
In this step, I will confirm which of my indicators are new and which already exist within my communities. Here, I can see that all ten of my indicators already exist. This is good news because it means somebody else within my organization or community has already done some initial analysis and it ensures that nobody has to do the same work twice. I also have the option to view existing indicators to ensure they are not already associated with the Incident, Threat or Adversary I am currently working. If your indicators already exist in the system, re-adding the existing indicators, will append any new Description and Source to any existing indicator Attributes you have access to, capturing new information regarding indicators that may be “repeat offenders”.
Step 4: Security Labels and Tagging
Once confirmed, I want to select Security Labels appropriate for my intended audience. Security Labels allow me to set custom security controls around the Indicators themselves as well as associated context within the Attributes. I may also want to tag my indicators. Tagging is a powerful way to make it easier for other analysts to quickly identify and categorize my data and associate them with similar intelligence themes.
ThreatConnect Research TIP: Security Labels are very handy when classifying Indicators, Groups or Attributes. Security Labels allow you to convey your intent as to whether the content can or cannot be shared, as well as how the information can be used. Examples that ThreatConnect Research have created and used are CLIENT CONFIDENTIAL or APPROVED FOR RELEASE.
Step 5: Create New Associations and Save
Finally, in this final step, I will associate my Indicators with an existing corresponding Grouping, e.g., Incident, Threat, Email or Adversary, I created and hit save. Note that Indicators can also be associated with specific adversaries, threats, signatures, emails, tasks and documents.
Unstructured Data Import
When importing an unstructured set of Indicators, I will mostly follow the same process. The key difference only applies to steps 1, 2 and 3, so these next instructions will focus only on what is different during an unstructured import.
Step 1: Import
I begin by importing a PDF file that I obtained from a great whitepaper on the “Inception Campaign” from the Blue Coat Labs blog. This write up contains a plethora of indicator data that would take far too long for me to manually parse myself, but I want to immediately capture it, enrich and go hunting for some of the content within my defensive integrations.
Step 2: Validate
When importing from unstructured data, it is important to note that the parsing engine will extract anything and everything that looks like an indicator. This will often include hostnames, email address, URLs and IP addresses that are not malicious but referenced in the document. Analysts will need to validate that the indicators are of interest and prune those not of interest before the import to ensure that the desired content is captured.
Step 3: Confirm
After validating the data, I have the option to add Description, Source, Rating and Confidence. Note: Any change I make here is applied to all of my Indicators. Since these values will vary with each indicator, I will choose to skip this step and apply the necessary changes to each indicator individually, after I finish my import.
Conclusion
Unfortunately, for far too long, the state of the art for many Threat Intelligence Teams was email and spreadsheets. Keeping track of multiple dynamic threats over extended periods of time no longer scales to the features of basic office automation. Content and context are everywhere, from analyst inboxes to download folders. With ThreatConnect, analysts can now resurrect what would otherwise be forgotten threat intelligence, allowing analysts to put that data to work within a Threat Intelligence Platform. Getting data into a Threat Intelligence Platform should be as quick and efficient as possible so that time is spent analyzing threats, not munging and massaging data. This is why ThreatConnect gives our users several interface options to Import and populate their Individual or Organization accounts. In future “How-Tos”, we will cover the ThreatConnect API and explain how to automate the ingestion of threat intelligence.