Saturday, February 21, 2009

Looking for "Bad Stuff", part I

Searching for unknown issues within a Windows image is always a tough thing...a great deal of the incident response and forensic analysis that I do is preceded by a triage worksheet, interviews of key personnel, etc. Sometimes, I will even ThunderDome two people who give me disparate information, simply because it's a good interro...I mean, interview technique. Anyway, the purpose of all this is to narrow down the issue as much as possible to help me identify what an issue, what the source might be, etc.

Even in the face of all this, we all sometimes still get the directive to find "all the bad stuff" on a system, either on a live system or within an image. This is like looking for something in a hay stack, only you don't know what the something is. Generally, you try to do things like ask, "when did you first notice this activity?", or "what kinds of things alerted you to this issue?"; it may be network traffic, sluggishness on the system, etc.

Recently, a friend asked how to determine if a system might be compromised. We see this a lot with incident response, as's a system that we think might be compromised...can you tell us if it is? So, let's assume that you have nothing but an acquired image of a Windows system to analyze...what can you do to determine what happened on the system, with very little to go on?

Mounting The Image
One of the first things we can do to make our analysis somewhat more efficient is to gather some tools. As such, we'd like to mount our image as a read-only file do so, we can look to commercial apps such as ASRData's SmartMount, or you can use freeware tools such as ImDisk or VDKWin. The VDK executable will let you get the partition table information from within the acquired image, as will the GUI-based Partition Find and Mount (discussed at the SANS Forensic Blog)...however, Partition Find and Mount does not appear to have the ability to mount a partition read-only; it will reportedly allow you to mount a potentially corrupted partition, so this may be an order to recover data for analysis, mount the partition, and then acquire an image of it.

Also, as long as we're discussing mounting images, we HAVE to talk about F-Response Enterprise Edition and the Management Console that Matt has released! When working in a live scenario when an enterprise capability is required, the FEMC makes it SO easy to connect to a live system and collect data, and F-Response, by its very nature, makes it SO safe.

Let's not forget all of the great work Rob Lee has done with SIFT! "SIFT" is the SANS Investigative Forensic Toolkit Workstation, a Linux-based VM that Rob has assembled for use, not only in training, but also in practical application. Rob even recently posted to the SANS Forensic blog, demonstrating how you can use SIFT and the Linux "mount" capability to mount your acquired image as a read-only file system in Linux, and then access it via Windows.

Getting yer Scan On!
Okay, choose whichever means or method works for you...but at this point, we can assume that we have an acquired image mounted on our analysis system as a read-only file system/drive letter. A great way to get started on "looking for bad stuff" is to start with some AV/anti-spyware scans. Now, running these scans does not mean we're means we're smart; we're making use of our available resources to perform a modicum of data reduction.

To that end, here are some links to free AV scanners:
Free AV Scanners
Portable AV/Malware Security Tools
PCTools - Free AV

One of the reasons why I mention these free AV tools isn't just because they're free...part of it has to do with their limited capability. What I mean by this is that most of the free AV tools are just order to get real-time protection, you need to pay for the additional capability. Yeah, but...we don't want the real-time protection...all we want is to be able to update the tool (even manually...including the fact that we did so in our documentation), and run a scan when we decide to...even if it means as a Scheduled Task. One of the great things about these tools, particularly the ones that are portable, is that you can keep them all on a thumb drive, update them regularly, and run them right from the thumb drive. I've has some great success with many of these, including ClamAV and a2Free.

Targeted Analysis
Okay, getting some of our bulk data reduction out of the way is a great way to get started, but nothing is going to keep us from the need to roll up our sleeves and get deep into the weeds with our analysis. Also, in the face of a lot of recent activity, one of the things that is obvious is that the AV industry, by it's nature, is usually one step behind the malware authors. Therefore, don't be surprised if you've scanned the mounted image and not found anything specific that would point to the activity in question. This is where a more surgical approach is required...besides, this is more fun, too!

Log files
If you're read (or heard from someone who has read) my book, you'll know that Windows systems are just chock full of log files! There are a number of places you can look for information about that status or state of the system, determining such things as attached devices and installed software. One useful log file is MRT.LOG, which is used by the MS Malicious Software Removal Tool. This is essentially a microscanner (similar to McAfee Stinger) in that it protects a system against a very limited set of threats. This can help you narrow down what might have been on the system.

Don't forget that AV applications keep their own logs, as well. Mandiant's recent release of the APT Forensics M-unition Pack includes a PowerPoint presentation that lists, for example, the location of Symantec AV OnAccessScan logs. AV products generally tend to keep logs as to when the product was last updated, as well as the status of various scans that have been run, either automatically or on demand.

Event Logs
The Windows Event Log is something of a specialized log, as on Windows XP and 2003, they are kept in a binary format (Vista and beyond use an XML format). While the actual configuration of what is and isn't audited is controlled, in part, through the Registry, some applications will record information in the Event Logs. For example, AV applications will record information in the Event Log.

Parsing the Event Log into something readable can be difficult, unless you have the right tools. I tend to start with my own; evtrpt and evt2xls.

Note: Evt2Xls does NOT have a 64K limit on the number of rows it will process...that limit is imposed by some versions of Excel. The spreadsheet app in OpenOffice apparently does not have that same limit.

Registry Analysis
This can be a pretty comprehensive subject, one which I've already started down the road on...I won't go into any more detail here, but will instead leave this for later discussions.

Specialized Tools
There are also a number of specialized tools you can use to help narrow down any issues that might come up...I'll list them here without a great deal of explanation in hopes of generating questions or discussion:

WFPCheck - not a released tool, but something I wrote (and discussed in WFA 2/e) that I use to help me determine if there's been something on the system that disabled WFP and infected or replaced "protected" files
Yara and Scout Sniper - you should really check this out!

Practical Application
The more data you have, the better you will be able to narrow things down. I recently performed an examination of an image, and had some external network-based sources that I was able to use to determine the source of certain behaviour that had been observed on the network. Service start times within the Event Log correlated very closely to the times that network-based applicance logs showed certain activity starting on several days; from this, I was able to narrow down the presumed malicious activity to the interaction between two applications on the system.

Note: I don't care what Webroot customer support tells you...even if Spy Sweeper is not configured to protect the hosts file, it will still parse it and issue DNS queries for the domains found in that file.


Anonymous said...

Harlan and I have discussed this at length - there is no "Find All Evidence" button in any of the forensic utilities! Performing a general sweep like he discusses here underscores the importance of having a concrete understanding of how Windows (or any operating system for that matter) normally works. How many lsass.exe processes should be running? From where? What should the etc/hosts file look like under normal conditions? These questions and many others should be researched and understood well before you have an incident.

Additionally, a tool that I use in instances like this is GMER. It is a stand alone binary which can be run locally, from CD Rom, or from a thumb drive which shows you running processes, loaded dlls and lots of other neat goodies. Oh also highlights hidden processes and files in red - which sounds like a "so what" kind of thing until you show it to a customer and they think you are Bill Nye.

I think the key component to take away here is asking questions...forget asking the "right" questions - just ask a bunch of inevitably gather good intel. Once you have your answers, write them down and work through each of them - checking them off as you go (or virtually using Case Notes or other note taking utility).

Great Post Harlan!

Chris Pogue (aka - I have assimilated the pooh!)

Claus said...

Harlan - Do you have any recommendations for good (and/or freeware) raw-disk sector viewers?

Windows OS supported would be preferred, but any that I might find on a Linux LiveCD disk (Helix?) would be fine as well.

At our "shop" there is discussion regarding the method/efficacy of disk-encryption being used. I am booting a whole-disk encrypted system via a WinPE disk (Linux LiveCD's would also work) then view the raw disk sector info. I've been doing this already with a few utilities and can pretty much demonstrate the visible difference between a WDE system and one that is not.

However, I'm always on the lookout for new and better utilities, and imagine that raw-disk sector viewing is a technique you are quite versed in and figured you might know the best of the best.

I've got a collection of Win-based ones I will be posting soon.

Any recommendations would be appreciated and valued.


Claus V.

H. Carvey said...

@Claus - any hex editor should do. I use UltraEdit, but X-Ways Forensics gives you this capability as well.

As far as something on Linux, I'd suggest contacting Rob Lee after he gets back from Dubai and asking him about what may be in SIFT.

I've been focused more on the means for getting WDE keys from RAM.

@Chris - You're right, in that asking a bunch of questions will end up showing you the way to starting your examination. Many times, asking the same questions during both the initial triage call and then again when you're on-site can reveal some interesting information.

Thanks for the comments, guys...keep 'em coming!

Ken Pryor said...

Excellent post and very timely. I can put this info to use right away. Looking forward to part II.

H. Carvey said...

Look no II was posted this morning!

Ken Pryor said...

LOL discovered that right after posting my comment. I followed the link to part 1 from google reader and part 2 hadn't shown up there yet. After posting the comment I returned to your main page and saw part 2 had arrived.