Perspectives on Threat Intel

Source: detect-respond.blogspot.com

A while back, I tweeted, saying that "threat intel has it's own order of volatility".  That tweet got one RT and 2 favorites, and at the time, not much of a response beyond that.  Along the way, someone did disagree with me on that, stating that rather than an "order of volatility", threat intel instead has a "shelf life".

Thinking about it, I can see where both are true.

To begin with, let's consider this "order of volatility"...what am I referring to?  Essentially, what I'm talking about was detailed in 2002, in RFC 3227, Guidelines for Evidence Collection and Archiving.  In short, the RFC states that when collecting evidence, "you should proceed from volatile to the less volatile".  What this means is that when collecting "evidence", you should collect that evidence that is most likely to change first, or soonest.  This is a guiding principle that should be used to direct collection methodologies.

As such, the "order of volatility" itself is guidance, as the definition appears in section 2.1 of the RFC, whereas the discussion of actual collection does not begin until section 3.

The term "shelf life" refers to the fact that threat intel indicators have a time period during which they are useful, or have value.  Notice that I don't say "specified time period", because that can vary.  Those who conduct these types of investigations have seen where a single file, with the same name and stored in the same file system location on two different endpoints, placed in those locations minutes apart, may have different hashes.  Or, during an IR investigation, you may find RATs on two different endpoints that are essentially the same version, but with different C2 domain names.  When doing historical analysis of indicators such as collected hashes and domain names, we see how these are useful for a limited amounted of time; they have a "shelf life".

Several years ago when I was doing PCI investigations, we ran across a file named "bp0.exe".  We'd seen it before, and happened to have a copy of the file that we'd seen 8 months prior to the current investigation.  Both files have the same path within the file system (on two completely different investigations and endpoints, of course), and they had different MD5 hashes.  Using fuzzy hashing, we found that they were 98% similar (using the phraseology somewhat loosely).  This is a good example of how MD5 hashes have a "shelf life" and tend to remain valid for a limited time.

When you consider David Bianco's Pyramid of Pain from a network- or malware RE-perspective, the technical indicators at the lower half of the pyramid (hash values, C2 IP addresses, domain names) definitely have a shelf life; they are only 'good' (valid) for a specified period of time.  These indicators can change between campaigns, or as is often the case, during a campaign.  Those who perform threat intel collection and analysis are also aware that C2 IP addresses and domain names can also change quickly (often within hours or even minutes), and there are organizations that continually monitor this sort of thing and track them (i.e., what IP addresses various domain names resolve to, etc.) over time.

So, when you look at "threat intel" from a perspective that is external to a compromised infrastructure, using open source intel collection (and including analysis of samples from sites such as VirusTotal...), then the indicators do, indeed, have a shelf life.

However, when considering these same threat intel indicators from the perspective of the endpoint systems, we can see how they have an order of volatility, as well, and that delays in detection and response will lead to some of these endpoint indicators becoming more difficult to recover, and even unavailable.  If an installer is run on a system, and deletes itself after infecting the endpoint, then the file (along with the MD5 hash of that file) is gone.  In some cases, this happens so quickly that the installer file itself may not be written to physical disk, so there is literally nothing that can be recovered, or "carved".  When malware reports out to a C2 domain, how long does that persist on the endpoint?  Well, it may depend on the particular API used by the malware to perform that off-system communications.  If the malware was written to create it's own network sockets, the domain name may exist in memory for only a very short time, and persist on the network (this does not include any logs on other endpoints) for an even shorter period of time.  The domain name may be found in the pagefile, but again, this does not mean that the domain name is recorded on the endpoint indefinitely...even a C2 domain name or URL will only persist in the pagefile for so long.

If you consider the Pyramid of Pain from the perspective of endpoints on a compromised infrastructure, the indicators begin to have an "order of volatility", in the manner described in RFC 3227,  Some indicators can persist for varying periods of time on those endpoints; some will persist for only a short time, while others will persist for quite some time.

For example, some indicators may be present in the Security Event Log, and in most cases (that I've dealt with), event records of value have been obviated by the normal operation of the system, simply due to the fact that the Security Event Logs have "rolled over", as older event records have been overwritten by newer ones.  I've received images of systems and the Security Event Log was 22MB in size, and when parsed, contained maybe...maybe...2 days worth of events.  The specific event I was interested in occurred weeks (or in some cases, months) prior to when the image as acquired.

Some data sources...RecentFileCache.bcf file, AppCompatCache data, Prefetch files, for example...can be obviated by the passage of time and the normal operation of the system.   I've seen IR team response procedures that included running multiple tools on a live system, which caused the Prefetch files of interest to be deleted, as new programs were run and new Prefetch files created by the operating system.

Ultimately, it seems that the discussion of "shelf life" versus "order of volatility" depends upon your perspective.  If you're not considering endpoints at all, and only looking at the "threat intel" that we usually see collected, discussed and shared (MD5 hashes, C2 IP address and domains) through open sources, then yes, I believe that it is well understood that the indicators do have a "shelf life"; that is, a C2 IP address or domain may be valid at the time that it was found, but there's nothing to say that it will continue to be valid 6 weeks or 6 months later.

However, from the perspective of the endpoint, it's pretty clear that indicators have an order of volatility all their own, and that order can be impacted (in some cases, significantly) by external stimulus, such as IR data collection procedures, or actions taken by an adversary (or an admin).

Full Disclosure
Like many other organizations, my employer provides threat intelligence to clients, some of which is network- and malware-based, and collected through open sources.  In my role within the company, as an incident responder and digital forensic analyst, I tend to be both a consumer and producer of threat intel that is based on analysis of endpoints.  What I wanted to point out in this post is that there are different perspectives on the issue, and that doesn't mean that any one is wrong.

Addendum, 17 Mar: Looking back on this post, it occurs to me that the "shelf life" description applies to indicators in the bottom half of the pyramid, from a malware RE perspective, and the "order of volatility" description applies to indicators at all levels of the pyramid, from a host or endpoint perspective.