HowTo: Add Intelligence to Analysis Processes

How many times do we launch a tool to parse some data, and then sit there looking at the output, wondering how someone would see something "suspicious" or "malicious" in the output?  How many times do we look at lines of data, wondering how someone else could easily look at the same data and say, "there it is...there's the malware"?  I've done IR engagements where I could look at the output of a couple of tools and identify the "bad" stuff, after someone else had spent several days trying to find out what was going wrong with their systems.  How do we go about doing this?

The best and most effective way I've found to get to this point is to take what I learned on one engagement and roll it into the next.  If I find something unusual...a file path of interest, something particular within the binary contents of a file, etc...I'll attempt to incorporate that information into my overall analysis process and use it during future engagements.  Anything that's interesting, as a result of either direct or ancillary analysis will be incorporated into my analysis process.  Over time, I've found that some things keep coming back, while other artifacts are only seen every now and then.  Those artifacts that are less frequent are no less important, not simply because of the specific artifacts themselves, but also for the trends that they illustrate over time.

Before too long, the analysis process includes, "access this data, run this tool, and look for these things..."; we can then make this process easier on ourselves by taking the "look for these things" section of the process and automating it.  After all, we're human, get tired from looking at a lot of data, and we can make mistakes, particularly when there is a LOT of data.  By automating what we look for (or, what we've have found before), we can speed up those searches and reduce the potential for mistakes.

Okay, I know what you're going to say..."I already do keyword searches, so I'm good".  Great, that's fine...but what I'm talking about goes beyond keyword searches.  Sure, I'll open up a lot of lines of output (RegRipper output, web server logs) in UltraEdit or Notepad++, and search for specific items, based on information I have about the particular analysis that I'm working on (what are my goals, etc.).  However, more often than not, I tend to take that keyword search one step further...the keyword itself will indicate items of interest, but will be loose enough that I'm going have a number of false positives.  Once I locate a hit, I'll look for other items in the same line that are of interest.

For example, let's take a look at Corey Harrell's recent post regarding locating an injected iframe.  This is an excellent, very detailed post where Corey walks through his analysis process, and at one point, locates two 'suspicious' process names in the output of a volatile data collection script.  The names of the processes themselves are likely random, and therefore difficult to include in a keyword list when conducting a search.  However, what we can take away from just that section of the blog post is that executable files located in the root of the ProgramData folder would be suspicious, and potentially malicious.  Therefore, a script that that parses the file path and looks for that condition would be extremely useful, and written in Perl, might look something like this:

my @path = split(/\\/,$filepath);
my $len = scalar(@path);
if (lc($path[$len - 2]) eq "programdata" && lc($path[$len - 1]) =~ m/\.exe$/) {
  print "Suspicious path found: ".$filepath."\n";
}

Similar paths of interest might include "AppData\Local\Temp"; we see this one and the previous one in one of the images that Corey posted of his timeline later in the blog post, specifically associated with the AppCompatCache data output.

Java *.idx files
A while back, I posted about parsing Java deployment cache index (*.idx) files, and incorporating the information into a timeline.  One of the items I'd seen during analysis that might indicate something suspicious is the last modified time embedded in the server response be relatively close (in time) to when the file was actually sent to the client (indicated by the "date:" field).  As such, I added a rule to my own code, and had the script generate an alert if the "last modified" field was within 5 days of the "date" field; this value was purely arbitrary, but it would've thrown an alert when parsing the files that Corey ran across and discussed in his blog.

Adding intel is generally difficult to do with third-party, closed source tools that we download from someone else's web site, particularly GUI tools.  In such cases, we have to access the data in question, export that data out to a different format, and then run our analysis process against that data.  This is why I recommend that DFIR analysts develop some modicum of programming skill...you can either modify someone else's open source code, or write your own parsing tool to meet your own specific needs.  I tend to do this...many of the tools I've written and use, including those for creating timelines, will incorporate some modicum of alerting functionality.  For example, RegRipper version 2.8 incorporates alerting functionality directly into the plugins. This alerting functionality can greatly enhance our analysis processes when it comes to detecting persistence mechanisms, as well as illustrating suspicious artifacts as a result of program execution.

Writing Tools
I tend to write my own tools for two basic reasons:

First, doing so allows me to develop a better understanding of the data being parsed or analyzed.  Prior to writing the first version of RegRipper, I had written a Registry hive file parser; as such, I had a very deep understanding of the data being parsed.  That way, I'm better able to troubleshoot an issue with any similar tool, rather than simply saying, "it doesn't work", and not being able to describe what that means.  Around the time that Mandiant released their shim cache parsing script, I found that the Perl module used by RegRipper was not able to parse value "big data"; rather than contacting the author and saying simply, "it doesn't work", I was able to determine what about the code wasn't working, and provide a fix.  A side effect of having this level of insight into data structures is that you're able to recognize which tools work correctly, and select the proper tool for the job.

Second, I'm able to update and make changes to the scripts I write in pretty short order, and don't have to rely on someone else's schedule to allow me to get the data that I'm interested in or need.  I've been able to create or update RegRipper plugins in around 10 - 15 minutes, and when needed, create new tools in an hour or so.

We don't always have to get our intelligence just from our own analysis. For example, this morning on Twitter, I saw a tweet from +Chris Obscuresec indicating that he'd found another DLL search order issue, this one on Windows 8 (application looked for cryptbase.dll in the ehome folder before looking in system32); as soon as I saw that, I thought, "note to self: add checking for this specific issue to my Win8 analysis process, and incorporate it into my overall DLL search order analysis process".

The key here is that no one of us knows everything, but together, we're smarter than any one of us.

I know that what we've discussed so far in this post sounds a lot like the purpose behind the OpenIOC framework.  I agree that there needs to be a common framework or "language" for representing and sharing this sort of information, but it would appear that some of the available frameworks may be too stringent, not offer enough flexibility, or are simply internal to some organizations.  Or, the issue may be as Chris Pogue mentioned during the 2012 SANS DFIR Summit..."no one is going to share their secret sauce."  I still believe that this is the case, but I also believe that there are some fantastic opportunities being missed because so much is being incorporated under the umbrella of "secret sauce"; sometimes, simply sharing that you're seeing something similar to what others are seeing can be a very powerful data point.

Regardless of the reason, we need to overcome our own (possibly self-imposed) roadblocks for sharing those things that we learn, as sharing information between analysts has considerable value.  Consider this post...who had heard of the issue with imm32.dll prior to reading that post?  We all become smarter through sharing information and intelligence.  This way, we're able to incorporate not just our own intelligence into our analysis processes, but we're also able to extend our capabilities by adding intelligence derived and shared by others.

MBR Infector Detector

Now and again, I get those analysis gigs where someone suspects that a system may have been infected with some sort of malware, but they aren't sure, and don't really have anything specific (Event Log entry, AV alert, etc.) to point to.  I know that others get these sorts of gigs as well, and like them, I have a process that I go through when examining images of these systems.  This usually starts with checking for installed AV products (MRT, etc.) to review their logs, as well as checking for AV having been run before the system was taken offline...if logs are available, they can tell you a lot, particularly the product and version run.  From there, I also mount the image and scan it with other AV tools.

One of the steps on my list is to also look for MBR infectors.  What's an "MBR infector", you ask?  Read on...
F-Secure "Hippie" Description (1996)
SecurityVibes - Mebroot (2008)
F-Secure - Mebroot (3 Mar 2008)
Symantec - Mebroot (30 July 2010)
Sunbelt - TDSS/TDL4 (15 Nov 2010)
F-Secure, 17 Feb 2011
MMPC - Sinowal, aka Mbroot, Mebroot (8 Feb 2011)
MMPC - Win32/Fibebol.A (7 Mar 2011)

If you read through the above links, particularly those that are AV vendor descriptions of MBR infectors, you'll notice some commonalities...in particular, when the MBR is infected, other sectors prior to the first partition (usually, sector 63) contain something...a copy of the MBR, code to be injected into the system, something.  Now, this doesn't mean that this is the case for ALL MBR infectors, just those that have been mentioned publicly.

Usually, what I would do is load the image into FTK Imager, and scan through the sectors manually...but why to do that, when you make the computer do it?  That's right...I wrote a (wait for it!) Perl script (mbr.pl) to do this for me!

So, what the script does is scan through a range of sectors from an image file; by default, it will scan through sectors 0 through 63 inclusive, but the analyst can set different sectors to be scanned.  When a sector that does NOT contain all zeros is found, the script will flag it.  By "flag it", in summary mode, the script will just list the sector number.  In a more detailed mode (which is the default), the script will print out the contents of the sector to STDOUT, in a hex viewer-like format.  This way, it's real easy for the analyst to see, "hey, this sector just contains some strings associated with Dell installs", or "Hey, this sector is the start of a PE file!"  Because the output goes to STDOUT, you can pipe it through "more" or redirect the output to a file.

Also, using another switch, the analyst can dump the raw sectors to disk.  This allows you to generate MD5 or ssdeep hashes, run ssdeep hash comparisons, submit the raw dump to VirusTotal, etc.

Overall, it's pretty cool.  I usually run mmls against the image anyway, and many times I'll see that the first partition starts at sector 63.  Other times, I've found the starting sector for the first partition by searching the image via FTK Imager for "NTFS".  Regardless, with the output of mmls, I can then run mbr.pl as part of my malware detection process, and just like other parts of the process, if nothing unusual is found, that's okay.  If something is found, it's usually correlated against the output of other steps in the process.  The overall goal is to as thorough a job as possible.

Forensic Analysis Process/Procedures

I've seen posts recently on some of the lists regarding processing forensic data...in most cases, the original question seems to center around, what is (are) the first thing(s) you do with your forensic data?

I thought I'd approach a response from a couple of different perspectives...

Goals
The VERY first thing I start with, regardless of what type of work I'm doing...IR, data collection, CSIRP development, forensic analysis...whatever...are the customer goals. Always. Every time.

Customer goals are documented early on in the engagement, most often from the very first call. They're also revisited throughout the engagement, to ensure that the on-site responder or the analyst is on track, and also ensure that the customer's expectations for the engagement are managed properly. It's a pretty bad feeling to have no communication with the customer, and deliver a report, only to have them say, "...uh...I thought you were going to do X..."

The thing about customer goals is that I can go on-site and back up a Ryder rental truck and acquire images of 300 or 500 systems...but for what? At what expense? If the customer needs an immediate answer, by telling them, "hold on...I've gotta image all of these systems first...", I've already done my customer a huge disservice. The customer's goals dictate everything about the engagement...how many responders are sent on-site, which ones, which analysts will be engaged, how long the engagement will take, etc.

Another thing about goals is that an analyst can easily consume 40 to 80 hours just analyzing an acquired image, and never answer the questions that the customer asked. Not only does the analyst need to be sure to keep the customer's goals in the forefront of their minds, but they also need to ensure that the goals are clear, understandable, and achievable. One popular issue is the customer who ask the analyst to, "...find everything suspicious...", and the analyst takes that and starts analysis...without ever determining what constitutes "suspicious". I've analyzed systems used by pen testers, and systems used by administrators...for these systems, the existence of tools like MetaSploit, pskill, psexec, etc., wouldn't necessarily be "suspicious". What if you find password cracking tools on a system, and spend considerable time determining and reporting on their use...only to find out that the user was an admin whose job it was to test password strength?


Makin' Copies
Pretty much the first thing I do once I have my images in the lab is make working copies and secure the originals. This is a basic step, I know...but for me, all I need to happen is to get caught once with going on-site and then not being able to access my data. I'm one of those guys that tries to learn from the mistakes of others, rather than my own. I'm not always quite as successful as I'd like to be, but I was almost caught by this once, so I learned my lesson. I had just copied and verified an image from a USB external wallet drive, and began working on the copy. After a day of processing, I went to get a file off the USB ext HDD, and Windows asked me if I wanted to format the drive. For some reason, something had happened to the drive...I have no idea what it was. The point was, however, that I had made my working copy...I had copied the image file, verified the file system, and used Jesse's md5deep to ensure that the image file had completely and correctly copied.

Documentation
Before addressing the actual analysis of an acquired image (or any other data), I make sure that my documentation is in order. At this point in my case management, this usually means that I've started my case notes...the first thing at the very top of my case notes is usually the customer goals. This helps me focus my analysis efforts...I usually even outline an analysis plan to get myself started at this point.

Documentation is a consistent aspect of engagements, beginning to end. Documentation keeps me on track, and also allows me to pass off the work to someone else in case of an emergency, without having them start over completely. It also allows an analyst to pick up a case 6 months or a year later and see how things went...particularly if they need to go to court. Documentation should be clear, concise, and maintained in a thorough enough manner that the analysis is repeatable and can be verified by another analyst.

Analysis
After all that, the very first thing I like to do before doing any actual analysis is to extract specific files from the image...Registry hives, Event Logs, etc...and to look for specific items (i.e., AV logs, MRT logs, etc.). Depending upon the goals of my analysis, I may even run TSK's fls.exe to generate a bodyfile, ultimately for inclusion into a timeline for analysis.

I tend to do this sort of thing if I have something that may take a while...AV scans of a mounted image, keyword searches, etc...because I don't want to go back to the customer and say, "sorry it took so long, but I was running a scan and couldn't do anything else." To me, that's simply unprofessional...analysts are hired as "experts" or at least relied upon as professionals, and should conduct themselves as such. That, in part, means doing analysis tasks in a parallel, rather than serial, fashion should simply be part of what we do. So, if I have a process that I need to run that's going to take some time, I'll extract pertinent files first so that I can continue with my analysis with the other task is running.

Of course, this all ties directly back to the goals of the engagement. In fact, depending upon the goals, I may make two working copies of the image file...one to work on while the other is being scanned. Or, I may not even run AV scans...I've found the malware that the customer described without having to run any scans.

So...what are your first steps?

When a tool is just a tool, pt II

Okay, this is part II for this post, because I posted an awesome rant to a thread in one of the forums, and I wanted to include that here, as well, because it kind of applied...and it's my blog, I can do what I want. ;-)

The thread can be found here, and the post I'm referring to is on the third page, in response to someone mentioning, "...don't forget that if you wind up working civil or criminal cases your tools are going to get challenged in court. Open source tools are more vulnerable to attack by the opposition than commercial tools that are standardized."

My rant, if you want to call it that, had to do with what I see as a gross misconception with respect to court cases; specifically, some commercial tools are used primarily because the analysts themselves are familiar with them, and perhaps as a result, the players in the court system have also become familiar with them. That is to say that some commercial tools are recognized within the court system, and therefore, not a great deal of additional explanation is required.

As such, it isn't the tools that are challenged in court...it's the analyst and their processes.

Also, I think another huge issue that doesn't appear to be considered when folks are making statements such as the one I quoted above is that an analyst just doesn't decide one day to walk into court, take the stand, and testify. It just doesn't happen that way.

Instead, the attorney you're working with or for (prosecution or defense) is gonna want to know your answers before he asks you questions on the stand, and the fact that you're testifying and what you're testifying about are going to be part of the discovery process...so the other side is going to have a chance to cross-examine you. As such, if there's anything that would lead the attorney you're working with to suspect that you being cross-examined would sink the case, they're not going to put you on the stand. The same is true if he or she simply doesn't feel that the results of your analysis are pertinent to their case.

With me so far? I guess what I'm saying here is that there's a heck of a lot that goes on before an analyst ever gets to the point of approaching the stand in a court of law.

Now, can we agree that an acquired image is nothing more than a stream of bits, 1s and 0s, in a file on a disk? If we can agree to that, and if the integrity of that data, that stream of bits, can be verified and validated, then why does it matter what tool I use to extract data? What does it matter if an analyst determines that an illicit image is in the image using some commercial tool's Gallery View, or by mounting the image read-only with ImDisk and viewing the image file through Windows Explorer? Regardless of the tool used, the image was there, and that doesn't change. The same is true with other data...credit card numbers, other sensitive data, etc. One tool doesn't necessarily magically make it visible where some other free and/or open source tool wouldn't be able to extract the same data.

Now, don't get me wrong...I'm not against using commercial tools. I've used them myself (and I'm seeking therapy...just kidding) when the need has arisen. But the fact of the matter is that commercial forensic applications are just like any other tool, with their own inherent strengths and weaknesses. In some cases, I've found that processes using open-source and free tools, such as timeline creation tools, have allowed me to structure data for analysis in ways not possible through the use of commercial tools. In other cases, I've found short-comings in using commercial tools, just as I've found short-comings in using open-source and free tools. That doesn't mean that commercial tools shouldn't be used...it just means that all tools should be considered just for what they are...tools.

What should matter most is the process used and documentation created by the analyst. If you thoroughly document what you've done, then why shouldn't you be able to testify about it on the stand, regardless of the tools used? I know a few analysts who've documented their work such that someone else (i.e., LE) could validate their findings via commercial tools (because that's what the LE analyst was most comfortable with) and then testify about the "findings".

So, what do you think? Are open-source tools "more vulnerable to attack"? Why does it matter if I extracted a Registry hive file from an image, and then extracted the LastWrite time from a specific key using a Perl script? Or a hex editor? Or if someone else did the same thing, but through EnCase or FTK? The fact of the matter is that if you go to that location on the disk or within the hive file, extract 64-bits, everyone who does so should arrive at the same answer...right?

Or should I just curl up in the fetal position in the corner of my office, and rock myself to sleep, chanting, "I'm a pretty girl" over and over again?

When a tool is just a tool, pt I

A tool is just a tool...that's it. A tool by any other name would smell so sweet...no, wait...what? Who let Willy Shakes in here? ;-)

I know what you're thinking..."fount of wisdom, dude." Sweet. But think about it...think about what we do, and how we do it. If you hear someone say, "yeah...I do forensics. But I need <insert commercial tool name here>", then back away slowly, don't make direct eye contact and make no sudden movements.

It's long been known that subversive tools and techniques, colloquially referred to as "anti-forensics" tools, haven't been directed at subverting other tools...no, tools such as timestomp aren't meant to subvert EnCase or even NTFS. What's being targeted here is the analyst and their training.

Not sure what I mean by this? Check out Simon's post over on Praetorian Prefect that discusses, in part, that whole COFEE/DECAF yawnfest. Simon had a previous post that addressed COFEE and some of the hype surrounding the tool set being "leaked".

When you really think about it, DECAF is meant for one thing...to subvert the use of COFEE. If the responder is a one-trick pony, and ALL they have is that COFEE package...game over, and DECAF has...no, wait...I wasn't gonna say "done it's job". No, what I was gonna say was the DECAF has demonstrated the shortcomings inherent to types of responders that rely solely on the use of one tool, such as COFEE.

Have you ever heard, "...I ran these tools, some of them didn't work...but here ya go."? I was working a fairly (in)famous engagement back in Dec '06 and one of the analysts from the primary on the contract ran the Windows tools from Helix 1.9 against a live system. I was handed the data the next day, and found that a little over half of the tools didn't have output from the tool available...they just had that "XXX is not recognized as an internal or external command" message. In this case, the issue wasn't due to something being loaded on the system, rather it was an issue of someone really knowledgeable in Linux trying to do IR on a Windows system. The one tool they had didn't work...but the system could ONLY be accessed at 2am, and the analyst had no idea that anything had gone wrong. So, even though I was tasked with doing "emergency" IR, I (and everyone else) had to wait another 24 hrs to get the data we needed.

Okay, that was three years ago...but in a lot of ways, for a LOT of responders, this really hasn't changed too terribly much. Take a malware detection gig, for example...someone has a system that they think has malware on it, so the responder acquires an image of the system, and maybe memory. They take the data back to the lab, in-process the data, then mount a working copy of the acquired image read-only and scan it with one AV scanner...and find nothing, and that's ALL they do, and they report their findings. But wait...did they check to see if the scanner they used was the same one already installed on the system? If it is, what have they really done at that point? What about the fact that a great deal of malware (depending upon what you read and who you believe, this could be as much as 40-60%) isn't detected by commercial AV the first couple of months that it's in the wild?

Are you beginning to see my point here? Look at Conficker...remember Conficker? One of the things I found most interesting about it was that it took advantage of standard business processes (ie, file shares, thumb drives) to spread on internal networks. Imagine responding to a customer site and telling them, "you're gonna have to shut down your file servers until we get this cleaned up." What the...?? A great many calls came in over the next couple of months as organizations with up-to-date AV scanning engines and signatures got p0wnd'd by variants of Conficker, Virut, and other nastiness. That's right...variants. As in, stuff that the AV couldn't see, but eventually got classified as being pretty close to the stuff that the AV could see...just not close enough for it to see it. See what I mean? Not the stuff the AV could detect, but the new stuff...stuff that did the same thing as the other stuff, but just "looked" different.

The problems and p0wnage that folks faced with this stuff had everything to do with the fact that they relied on a tool, rather than a process, to protect them. "Hey, I've got AV in place...I'm good." Who said that...the dudes from AIG? If your fallback plan was to call in an IR team...well, the malware continues to own you for 72 hrs or more as you go through contract negotiations, waiting for analysts to arrive, and finally getting them spun up on your infrastructure and the situation.