Thursday, August 21, 2014

What does that "look like"?

We've heard this question a lot, haven't we?

I attended a conference about 2 1/2 years ago, and the agenda for that conference had about half a dozen or more presentations that contained "APT" in their title.  I attended several of them, and I have to say...I walked out of some of them.  However, hearing comments from other attendees, many folks felt exactly the same way; not only were they under-whelmed, but I heard several attendees express their disappointment with respect to the content of these presentations.  During one presentation, the speaker stated that the bad guys, "...move laterally."  One of the attendees asked, "what does that look like on systems?", and the speaker's response was to repeat his previous statement.  It was immediately clear that he had no idea...but then, neither did anyone else.

Corey has asked this question in his blog, and he's also done some great work demonstrating what various activities "look like" on systems, such as when systems are exploited using a particular vulnerability.

What I'm referring to in this post isn't (well, mostly...) something like "...look at this Registry key."  Rather, it's about clusters or groups of artifacts that are indicative of an action or event that occurred on a system.

Clustering
As we share and use artifacts, we can take a step back and look at where those artifacts exist on systems.  Then we begin to see that, depending upon the types of cases we're working, artifacts are clustered in relatively few data sources.  What this means is that on drives that are 500GB, 1TB, or larger, I'm really only interested in a few MB of actual data.  This means that during incident response activities, I can focus my attention on those data sources, and more quickly triage systems.  Rather than backing up a van full of hard drives and imaging 300 or more systems, I can quickly narrow down my approach to the few systems that truly need to be acquired and analyzed.

This also means a significant speed-up for digital analysis, as well.  I don't maintain tables of how long it takes to acquire different hard drives, but not long ago, I had one hard drive that took 9 hrs to acquire, and another that was 250GB that took 5 hrs to acquire.  Knowing the data sources that would provide the biggest bang for the buck, I could have retrieved those after I connected the hard drive to the write-blocker, but before I acquired the hard drive image.

Reliability
As we share and use artifacts, we begin to see things again and again.  This is a very good thing, because it shows us that the artifacts are reliable.

Like others, including Corey, I don't see sharing artifacts on any sort of scale.  Yes, there are sites such as forensicartifacts.com, but they don't appear to be heavily trafficked or used.  Also, I generally don't find the types of artifacts I'm looking for at those sites.  I've been achieving reliability, on my own, for various artifacts by using them across cases.  For example, I found one particular artifact that nailed down a particular variant of a lateral movement technique; once I completed my analysis of that system, I went back and searched the entire timeline I'd developed for that system, and found that that artifact was unique to the event I was interested in.  I've since been able to use that artifact to quickly search successive timelines, significantly speeding up my analysis process.  Not finding that artifact is equally important, as well, because (a) it tells me that I need to look for something else, and (b) in searching for it, I can show a client that I've done an extremely thorough job of analysis, and done it much quicker.

Oxidation
A great reason for sharing artifacts is the oxidation of those artifacts.  Okay, so what does this mean?

The type of artifacts I'm referring to are, for the most part, not single artifacts.  Rather, when I talk about what something "looks like" on a system, I'm generally referring to a number of artifacts (Windows Event Log records, Registry keys/values, etc.) clustered "near" each other in a timeline.  How "near" they are...ranging from within the same second to perhaps a couple of seconds apart...can vary.  So, let's say that you're doing some testing and replicating a particular activity, and you immediately "freeze" your test system and find six artifacts that, when clustered near each other, are indicative of the action you took.

How many times as incident responders and digital forensic analysts do we get access to a system immediately after the initial intrusion occurred?  What's more likely to happen is that the initial response occurs hours, days, or even weeks after in the initial incident, and is the result of an alert or a victim notification.  Given the passage of time, these artifact clusters tend to oxidize due to the passage of time, as the system continues to run, and to be used.  For example, if a system becomes infected by a browser drive-by and IE is used in the infrastructure, some artifacts of the drive-by may be oxidized simply due to the normal IE cache maintenance mechanism.  Logs with fixed file sizes roll over, the operating system may delete files based on some sort of timing mechanism, etc.  All of this assumes, of course, that someone hasn't done something to purposely remove those artifacts.

Someone may share a cluster of six artifacts that indicate a particular event.  As others incorporate those artifacts into their analysis, the reliability of that cluster grows.  Then someone analyzes a system on which two of those six artifacts have oxidized, and shares their findings, we can see how reliable the remaining four artifacts can be.

I specifically chose to use the term "oxidize", because the term "expire" or "expired" seem to imply that the lifetime of the artifact had passed.  Sometimes, specific artifacts may not be part of a cluster, due to specific actions taken by the intruder.  For example, we've all seen files be deleted, time stomped, and other actions taken that force artifacts to be removed, rather than allowing them to timeout.

Format
At the moment, there are many questions with respect to sharing artifacts; one is the format.  What format is most useful to get examiners to incorporate those artifacts into their analysis?  Because, after all...what's the point of sharing these artifacts if others aren't going to incorporate them into their analysis processes?

During July, 2013, I posted a number of articles to this blog that I referred to as "HowTos"...they were narratives that described what to look for on systems, given various analysis goals.  Unfortunately, the response (in general) to those articles was the Internet equivalent of "cool story, bro."

A great indicator that I've used comes from pg 553 of The Art of Memory Forensics.  The indicator I'm referring to is pretty easy to pick out on that page...I've highlighted it in green in my book.  The authors shared it with us, and I found it valuable enough to write a RegRipper plugin so that I can incorporate that information directly into my own timelines for analysis...and yes, I have found this artifact extremely accurate and reliable, and as such, very valuable.  Having incorporated this indicator into my work, I began to see other artifacts clustered "around" the indicator in my timelines.  I also found that the indicator maintained it's reliability when some of those other artifacts were oxidized due to the passage of time; in one instance, the *.pf file was deleted due to how Windows XP manages the contents of that folder.

Final Thoughts
A good deal of the indicators that I'm referring to can be abstracted to more general cases.  What I mean is, an artifact cluster that is indicative of targeted threat (or "APT") can be abstracted and used to determine if a system was infected with commodity malware.  Other artifact clusters can similarly be extrapolated to more general cases...it's really more about how reliable the artifacts in the cluster are, than anything else.

8 comments:

Joachim Metz said...

So first I'm not sure if you are aware of this but GRR now has basic "artifact" support and for plaso it is on the roadmap. You might or might not seen the talk at blackhat about the GRR artifacts.

I'm going to address only the surface of a couple of the challenges.

1. A lot of the information is locked up in books, tools or blog posts. The information is often poorly documentation and validated. Recent examples in the book your are referring to made this very clear again. If people writing the books, programs and post can't get it right how do you suppose others will?

So the first challenge is making sure to create an authoritative set of information that is validated that is agreed upon.

2. So our take on it was. Let's put the basic knowledge into the tools and have contextual information on a public accessible source, e.g. forensicswiki. Now the challenge becomes what an "artifact" and how to define it pro-grammatically. You also mention "indicator", another conceptual term where various attempts exists to define them [http://forensicswiki.org/wiki/Cyber_Threat_Intelligence].

My take on it keep-it-simple first and let's try to share something between a limited set of tools. Once we have a foundation that proofs to work for 80% cases we already have made significant progress and start scaling up.

3. There exists a lot of indicators in reports, but the problem is that these were never tested against a larger sample set. So the false positive/negative ratio is unknown and without the original context pretty useless. This is a "friction field" between forensic analysis and malware analysis. If you want to tackle APT cases successfully you'll have to master them both.

4. Regarding clustering; I like to think about it beyond 1 system. A typical Windows 7 system will have roughly 4 to 5 VSS, so 5 to 6 copies of the same system in time. A case with lateral movement will have spread across multiple systems. Do our current tools allow us to analyze this at scale? IMO largely no. At least one refreshing take on this is: timesketch.org

Last but not least, I see a lot of people talking about it, but very little people doing actual work on it. So if you're one of these people that would like to see this happen, drop me a mail.

Harlan Carvey said...

Joachim,

A lot of the information is locked up in books, tools or blog posts. The information is often poorly documentation and validated.

That's kind of what I'm getting at. For me, I keep my case notes for the individual cases I work, and have a separate document into which I put indicators. That way, I have one place to go to find them, and validate them as they apply to the case.

Many of the indicators that I have and use have been successfully validated across multiple cases.

Let's put the basic knowledge into the tools and have contextual information on a public accessible source, e.g. forensicswiki.

Into which tools do we put this basic knowledge?

Another concern that I've heard expressed is, what happens to the contextual information when someone is in a situation where they cannot access the Internet?

...various attempts exists to define them.

I tend to believe the message gets lost of we spend too much time and effort in defining things.

For the most part, I agree with what you've shared. For me, asking for input from the community at large hasn't worked...maybe this is another instance where someone just needs to start taking steps in a particular direction, making adjustments along the way, and wait for others to follow.

Regarding #4, you and I have different experiences regarding Windows 7 systems, but I tend to think that's a good thing. With respect to scalability, what I've done for the past 5 or so years when looking at multiple systems in targeted threat cases is to simply merge my timelines...this is surprisingly revealing, and very often reveals TTPs.

I see a lot of people talking about it, but very little people doing actual work on it.

I agree, but only partially. I really don't see many people talking about it, but when I bring it up, I tend to find agreement...but everything ends there. I also tend to see a lot of excuses/reasons for why people don't share artifacts and indicators.

I've also found over time that providing examples of how to share artifacts without exposing specific case information, and having discussions, does nothing to get analysts to start sharing.

I've also found that when you do share artifacts, not many analysts incorporate what you've shared into what they do. I've been told by a number of analysts that they'll just save the email, or put the document into a folder, and when they feel that they need to revisit what was said (assuming that they remember it...), they'll search for it. If information isn't used and incorporated into analysis processes, then it doesn't get validated, nor does it get extended.

For example, I've included a number of indicators in the tools I released with WFA 4/e. The file eventmap.txt allows a number of Windows Event Log records to be "tagged" so that they are easier to understand when included in a timeline. The malware detection checklist includes additional indicators...but at this point, I have no idea if anyone is even using them, to say nothing of validating them.

My final concern is those who will take and use the indicators, without contributing indicators nor validation back to the project.

Joachim Metz said...

> Into which tools do we put this basic knowledge?

For context:
https://www.blackhat.com/docs/us-14/materials/us-14-Castle-GRR-Find-All-The-Badness-Collect-All-The-Things-WP.pdf

Though others have been talking about similar ideas. Sorry for not directly mentioning those, but can't find the links atm.

> Another concern that I've heard expressed is, what happens to the contextual information
> when someone is in a situation where they cannot access the Internet?

In the current information age sharing without Internet connectivity is a hassle, but IMO this problem can be easily overcome.

> I tend to believe the message gets lost of we spend too much time
> and effort in defining things.

Can't agree more ;) but for sharing pro-grammatically you'll have to have some standards. Preferable ones that don't make matter worse.

> Regarding #4, you and I have different experiences regarding Windows 7 systems,
> but I tend to think that's a good thing.

Definitely a good thing the more different perspectives we have the better coverage we can attain.

> what I've done for the past 5 or so years when looking at multiple systems in targeted
> threat cases is to simply merge my timelines..

That is exactly where timesketch comes in useful ;)
but focusing on supporting multiple analysts at the same time as well, directly sharing interesting search filters among one and other. Take that idea one step further, what about sharing this information with e.g. trusted third parties? Though now things become more challenging, largely from non-technical aspects.

> I agree, but only partially.

That is fine ;)

> The file eventmap.txt allows a number of Windows Event Log records to be "tagged"
> so that they are easier to understand when included in a timeline.

Not sure I'd not heard about it yet, have not read WFA 4/e so likely not. Do you have a direct link for me?

Though it sounds similar to a mix of where plaso is heading with filtering and tagging and what I'm trying to add with winevt-kb [https://code.google.com/p/winevt-kb/] by creating a redistribute-able database of event messages. I know this was done before by grokevt, but alas their code and documentation did not fulfill my needs.

> My final concern is those who will take and use the indicators,
> without contributing indicators nor validation back to the project.

Can't agree more. Though in my experience giving back to a project sometimes can be not possible, e.g. if it is no longer maintained, or people feel unsure or slightly intimidated by the idea. There are definitely, for a lack of a better term, "free loaders" out there.

Harlan Carvey said...

> For context:...

I printed the PDF out...it's interesting that there are thoughts there like what Corey Harrell and I have discussed, such as artifact categories.

> ... IMO this problem can be easily overcome.

I completely agree.

>> The file eventmap.txt allows a number of Windows Event Log records to be "tagged"
>> so that they are easier to understand when included in a timeline.

> Not sure I'd not heard about it yet, have not read WFA 4/e so likely not. Do you have a direct link for me?

http://windowsir.blogspot.com/p/books.html

Go to the line for WFA 4/e, click on the "Book Materials" link.

Anonymous said...

>> great indicator that I've used comes from pg 553

For those of us with the Digital version (no page number) can you give us the title of the section. I purchased the Kindle Edition of the book.

Joachim Metz said...

> For those of us with the Digital version (no page number) can you give us the title of the section.
> I purchased the Kindle Edition of the book.

Not sure what so "great" about the indicator and which one Harlan is referring to specifically, that is something that Harlan needs to answer. However the page discusses strings in gs.exe [gsecdump]. I think the bottom line message is that, if you understand the behavior of tools the attackers use, you can use secondary behavior e.g. Registry keys being updated as indicators.

Note that it is better to strings with -t option to see the actual location of the string in the executable and check before hand if the executable is packed. Also know that strings in an executable have a high false positive rate by itself. It is very easy to hide tools in legit looking executable.

> Go to the line for WFA 4/e, click on the "Book Materials" link.

Thanks for the link. I had a short look at eventmap.txt.

I understand that tagging them will give you "event log entries of interest", though it would be nice to take this idea a bit further. The problem with solely using tagging is the noise/signal ratio, e.g. if you rely on an event log entry in the security log to be present as an indicator you must be very lucky to find it, if the environment you're analyzing is poorly administrated.

So in plaso we are working towards something we dubbed it "anomaly detection", which is one of the possibilities the analysis plugins allow us to do. The clustering you mentioned earlier is a subset that could be possible with the analysis plugins. Alas atm I'm heavily revising this part of the codebase and other priorities allow me only slow progress. Also see:
http://plaso.kiddaland.net/analysis-technique/analysis-plugins

Harlan Carvey said...

Not sure what so "great" about the indicator...

It was great because the Volatility folks shared it, and others were able to validate it and find it reliable, making one of the points of my blog post.

...you can use secondary behavior e.g. Registry keys being updated as indicators.

This is something I've been suggesting for a LONG time. Seeing ripples at the edge of a pond, I don't need to see that someone threw a stone in the water to know that something happened. I don't need to actually see a deer to know that one lives in the woods, if I see droppings, prints, etc. Chris Pogue referred this as "expert eyes".

... though it would be nice to take this idea a bit further.

I agree. However, I have not received a single comment on this tool since the book was released. I had attempted to do something similar by building alerts into RegRipper 2.8...similarly, I have no idea if others find it useful, or if others have thoughts on how it might be improved.

Harlan Carvey said...

So in plaso we are working towards something we dubbed it "anomaly detection",...

I think that "we" is the key part of the above sentence. You appear to be part of a team working on this, whereas I've been looking to the community for input.