Working with Volume Shadow Copies

To begin with, let me say right up front that most of the information in this post, particularly the latter half, is not something that I developed myself...consider this more of me being a secretary(albeit unpaid) for Rob Lee, Troy Larson, and apparently Jimmy Weg...apparently, these guys all knew about what I'm going to present here well before I started down this road.

That being said, away we go...

Based on something I saw in Troy Larson's presentation at DCC2009 regarding Volume Shadow Copies, I thought I'd try something...I wanted to see if I could mount an image of a Vista system from a Vista system, and access the Volume Shadow Copies within the image.

I started with a Vista Home Edition system and an image of that same system on a USB external HDD. I connected the USB external HDD to the live Vista system and mounted the acquired image with each of several tools. I used ImDisk, SmartMount v1.0.5, the 14 day trial copy of Mount Image Pro, and P2 Explorer.

In each instance I mounted the image as a drive letter, verified that I could access the volume, and ran vssadmin list shadows. In none of the instances did vssadmin recognize the mounted drive as a source of Volume Shadow Copies. Well, I take that back...I didn't even get that far with P2 Explorer...it automatically kicked off its MD5 hashing, and once that was done, reported that the image was corrupt.

Now, Troy had mentioned in his testing that only EnCase PDE will mount an image in a manner through which vssadmin can access Volume Shadow Copies within the image. Okay, well, that's not something I have available at this point.

Now, Troy, Jimmy, and Rob mentioned something in one of the lists recently that seemed interesting...basically, to summarize what was said...if you have a VMWare guest of Vista, for example, and you have an acquired raw/dd image of a Vista system, you can generate at .vmdk file for the image and add it to the Vista VM as a hard drive, and then you can 'see' the Volume Shadow Copies in the acquired image.

So I set out to see if this was something I could replicate. I used ProDiscover to create a .vmdk file for the acquired image (again...Vista Home OS), and I opened VMWare Workstation 6.5. I went to the settings for my Vista Ultimate VM and added the new .vmdk file to the properties for the VM as a hard drive. When I booted the Vista VM and logged in, I could see the acquired image right here as E:\. So far, so good.

I then ran vssadmin, like so:

C:\>vssadmin list shadows /for=e:

Lo and behold, I saw a list of Volume Shadow Copies for the E:\ drive! And yes, the entries for "Originating Machine" corresponded to the name of the system from which the image had been acquired.

The next step was to see if I could create symbolic links using mklink...the short version is that I could, but I could not access them, as I kept getting "The parameter is incorrect" messages. Suffice to say, I even created symbolic links for Volume Shadow Copies from the C:\ drive, and got the same message. It turns out that the issue with mklink is that the trailing \ is absolutely required (something that was also mentioned on the SANS blog). So the command looks like:

C:\>mklink /d C:\shadow \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy18\

With this, you can then run tools such as RegRipper against the hive files, or copy out selected files for analysis (or better yet, just run your tools to collect the information), etc. Once you're done, you can remove the symbolic link with:

C:\>rd C:\shadow

Before I go on, let me remind you...you MUST have a \ at the end of the Volume Shadow Copy in the mklink command.

Moving on, I downloaded George M. Garner, Jr.'s FAU tools and ran the following command:

C:\tools>dd if=\\.\HarddiskVolumeShadowCopy6 of=g:\shadow6.dd --localwrt

HarddiskVolumeShadowCopy6 is one of the identified Volume Shadow Copies from the E:\ drive. I wanted to acquire an image of the Volume Shadow Copy to an attached USB external drive (G:\), hence the use of the --localwrt switch. For about 10 min, I let the command run, and kept running "dir g:\" from another command prompt, and kept seeing that shadow6.dd was 0 bytes. I stopped the imaging (Ctrl-C) and found that the output file was over 2.5GB! So I then re-ran the command, and just let it run...and it will run for a while, as I'm acquiring from a USB ext HDD to a USB ext HDD.

Here's the results of the 'dd' command:

C:\tools>dd if=\\.\HarddiskVolumeShadowCopy6 of=g:\shadow6_2.dd --localwrt
Copying \\.\HarddiskVolumeShadowCopy6 to g:\shadow6_2.dd
Output: g:\shadow6_2.dd
146526953472 bytes
139738+1 records in
139738+1 records out
146526953472 bytes written

Succeeded!

Now that the acquisition is complete, the next step is to verify the acquired image. Opening the image in FTK Imager, I was able to verify that I had a complete, readable file system. At this point, I can do everything with this image that I would with any other acquired image.

Again, let me remind you that this isn't something I came up with...apparently, others have known about this, I'm just writing it down.

Summary
1. Start with a raw/dd image of Vista or above
2. Create a .vmdk file for the image
3. Add the .vmdk as a hard drive to a VM of a like OS (if image is Vista, use a Vista VM)
4. Boot the VM, use vssadmin to locate VSC's on the image drive (or use WMI to get concise info)
5a. Use mklink to 'mount' the VSC's you're interested in, or...
5b. Acquire the full VSC using dd

Resources
Troy's Vista Forensics Slides (one version, anyway)
Shadow Copies on Wikipedia
Shadow Copy Client
Win32_ShadowCopy WMI Class

It's about time...

Sometimes you need a tool, and you just don't have one available to do the thing you need done.

Recently, I was doing some analysis, and as part of that was running regslack to see what I could find in the unallocated space of some Registry hive files. Within the hex dump of unallocated space, I could see what appeared to be Registry keys (signature "nk"), but these were in the unallocated space dumps, not as part of deleted keys that had been extracted by Jolanta's regslack tool. I could also see, right there in my editor, what appeared to be a FILETIME date/time stamp. So I had a string that comprised the date/time stamp, but no way to translate it quickly.

So I wrote a tool.

I called this tool Decode64, as I wanted to decode the 64-bit FILETIME objects...but from the starting point of a binary value, the way RegRipper does it. No, I needed to be able to start with a string. So I wrote Decode64 to take a string representing the date/time stamp (analyst pastes it into a textfield...how easy is that??) and with the push of a button, translate that to both a Unix epoch time, as well as to a human-readable time, in GMT format.

Now, if you look closely at the graphic with this post, you'll see that the string entered into the first field is not indicative of a 64-bit timestamp...nope, it's one of those 128-bit timestamps that MS uses (i.e., in the Scheduled Task .job file format, and in various Registry values beginning with Vista). Consider that an Easter egg. ;-) So, I'll be using this to not only determine if something falls within my incident timeframe, but to also enter information into a timeline via the TLN tool that I wrote, and included with the timeline tools in the Win4n6 group.

Speaking of time, when I talk to folks about analyzing images for indications of malware infections, I will usually start with asking what AV product was installed. What most folks don't realize is that many Windows systems will have the MS Malicious Software Removal Tool (MRT) installed, and that the MRT maintains a log of activity, to include anything it finds and removes during a scan. I thought, cool...I can include that information in a timeline, IF I can parse it out. Now I can. ;-)

C:\tools>mrtparse.pl -f mrt.log -s YoUrMoM
1257686548|MRT|YoUrMoM|| - Backdoor:Win32/Rbot in file://C:\WINDOWS\system32\ss ms.exe (sigseq 0x000016677E6E8F8A)
1257686548|MRT|YoUrMoM|| - Win32/Rbot and Removed!

As you can see, mrtparse.pl is a pretty simple script. Right now what it does is parse through the mrt.log file looking for any segment of the log file that contains a return code between 6 and 13 (all I've see so far is 0 and 6). Once it finds such a segment, it parses out what was found. Pretty simple.

I only have a couple of log files for testing, so the functionality of the script is limited at this point. Also, I don't have anything definitive that describes the format that the time is maintained in (i.e., local system time or GMT). So there is definitely area for improvement, but those are both easy fixes, as well.

Some other thoughts I had were to include all return codes of "0", as well. My thought that it would be a good idea if you could see something like a malware file being added to the system, and the results of tools such as MRT and AV right in there, as well.

NOTE: Keep in mind that MRT is NOT an AV solution (...and you're going to ask me, "...but what is?", aren't you??) - it's more akin to a microscanner (similar to Stinger) in that it only protects against a very specific list of malware.

A possible next step would be to look at adding support for Windows Defender logs...

Resources
A note on Defender settings
Default Scheduled Tasks in Vista
Defender not working and IE homepage set to "about:blank"?
AV recommendations for Windows (links to log files)

Some Analysis Coolness

TimeLine Analysis

The most recent issue of Hakin9 is available now...my second article on timeline creation and analysis is in this one; it's a hands-on walk-through of using the tools I put together, and use on a regular basis. You know...eat your own dogfood, as it were.

What do I like so much about this analysis method? Well, it's fast, it's relatively easy, and it lets an analyst (i.e., me) see a bunch of stuff all together in one place. It's pretty cool to see things like a remote login, creation of the PSExecSvc service, see that service start, then see a bunch of other files being created...to include the data files created by the malware.

Another thing I like about timeline creation and analysis is this...let's say you've got an analyst (or a team) on-site working an engagement, and they're stuck with something; determining the avenue of infection or compromise...whatever. Now let's assume that it's an engagement involving sensitive data, and they're trying to scope everything AND do collections. You can have those analysts dump the file system metadata, extract selected files from the system or image, zip all of that up and send it to someone for analysis. Not only do you run your analysis in parallel...you're not sending that sensitive data out! That's right, folks...you can increase your response efficiency and effectiveness using off-site staff, without further exposing sensitive data!

The version of the tools used in the article are available for download from the Win4n6 Yahoo group. The tools are all separate, standalone tools for right now because, to be honest, I don't always use them all together. Sometimes, it's good to see activity in a different format...in others, it's good to see a limited subset of activity (say, just your Event Log records) all at once, before moving on. By having separate tools, the analyst can intelligently select what they want added to the timeline in order to build it out.

File and Document Metadata

When I used to present at LE-oriented conferences more often, I'd talk about a nifty little tool out there called MergeStreams. This is a great little tool that essentially allows you to "hide" an Excel spreadsheet inside a Word document. This only applies to pre-Office 2007 document formats, however. I'll say that again...it only works on versions of MSOffice that use the OLE compound document format. What I'd show is someone pasting pictures (re: illicit images) into a Word document and then merging those with an Excel spreadsheet. Name the file "myspreadsheet.xls" and you would see the Excel spreadsheet. Rename the file, giving it a .doc extension, and you'd see the Word document.

While we're talking about Office document metadata, now is a good time to revisit some tools for extracting metadata; for pre-Office 2007 documents that use the OLE structured storage format, I've used the tools from my book, oledmp.pl and wmd.pl quite effectively, and there's OffVis from MS; for Office 2007 documents, try cat_open_xml.pl.

Didier Stevens has come up with something similar for PDF documents. All I can say about this is...wow. Really. This takes me back to '99, when I was sitting in the EnCase Introductory Training course in Leesburg, VA, and we were talking about file signature analysis. Gone are the days where we can simply scan for file signatures and compare that to the file extension...in order to do a decent job, we now have to go deeper. Just because a file begins with "MZ", is it really a Windows PE file? Is the PDF or Word (pre-2007) document really just a document, or is it a container masking/hiding something else?

Remember, a lot of the anti-forensics techniques out there target the analyst and their training.

Speaking of files, have you seen this new plugin from Bit9 called FileAdvisor? It's apparently a shell plugin for Windows, so if you find a suspicious file on your system, you can right-click it, and hash it and submit it for analysis. To view results, you'll need to register at the site with your name, email address, and a password. I don't necessarily see this on every user's desktop, but I do see responders and analysts possibly having it installed on a system somewhere.

Memory Parsing/Analysis
Jeff Bryner has put together a Python script for extracting FaceBook artifacts from a memory dump called pdfbook. For Windows systems, the script parses memory dumps from pd...I wonder if you could do the same thing using a full memory dump, extracting just the memory used by the process? Jeff has also released yim2text, a Python script for extracting Yahoo chat artifacts. Very cool.

In The News

The Register is reporting that bot masters have hidden a control channel in the Google cloud via AppEngine. Interesting article, take a read. The article also points out that both Facebook and Twitter accounts have been seen being leveraged as control mechanisms. Quoted from the article:

And that may be another reason why black hats are flocking to the cloud.

"Going to a company as big as Google and saying 'Can we get an image of that server,' that's a pretty high barrier," he said.

I'd suggest that that would have to do with the implementation. Cloud is being sold as the next big thing...but what is it? Well, it depends on who you're talking to.

Something else that's been making its rounds is spilled COFEE...Dark Reading picked it up, as well. Folks, the only reason this is getting the press it is, is because this was originally released only to law enforcement. Other than that, it's really not that big of a deal. Hogfly weighed in on this, as well...he apparently felt so strongly about this...dude, his last post was in August! ;-)

FTK 3 has "explicit image detection" capabilities (PDF here). This looks to be very useful for finding images, but I'm not sure that that's really the issue at hand these days...I may be wrong. I mean, I thought that it wasn't so much a matter for LE to find the images (although the coolness factor might be that in the video, Erika Lee uses the term "trained", implying a neural network of some kind...), but it was more a matter of addressing the Trojan Defense. I mean, once you find the images, you have to then demonstrate that the user in question intentionally downloaded and viewed them, and possibly shared them. This is were browser/web history, P2P, and Registry analysis come into play. Know anyone who knows anything about "Registry analysis"?

Speaking of which...

I ran across this AP article regarding the "Trojan Defense" hosted at the Fox News site. This is an interesting article to me, because this is something I've been discussing with LE for a number of years now. One of the key aspects of the analysis performed can be seen here:

A technician found child porn in the PC folder that stores images viewed online.

For most examiners, this refers to the browser cache; for IE, the Temporary Internet Files subfolders. Now, I'm not about to disparage any analysts skills or capabilities...all I'm going to do is point some things out. First, those TIF subfolders aren't created by IE, they're created by the use of the WinInet APIs, which IE uses. Now, this means that another app that uses the same APIs would also create the subfolders, and if it were running in the context of the logged on user, the folders would be created in the user's TIF directory.

Where did I get this? Well, I got a little help from my buddy Robert "Van" Hensing...check out his blog post from 2006. This was valuable to me, as I had conducted an exam for a customer, and one of the oddities I found was that the Default User's web history (I was using ProDiscover in my examination, and there's an extremely useful function to search for and parse web history...) had been populated. I tracked that back to a copy of wget.exe running with privileges elevated to System level...but I digress.

So, it's entirely possible to get just about anything on a system and make it look like the user did it. Why do that? Perhaps to discredit the user or law enforcement...I don't know, I'm not this guy.

My point is that we can't simply look at the folder the files are located in and their date/time stamps, and think we've got it wrapped up. There are a number of other places on the system that we can look...Prefetch folder, Registry, etc...in order to answer the question of did a Trojan do it? before it's asked.

Happy Birthday, VMI!

I know, another off-topic blog post, but this should the last one for a while!

The Virginia Military Institute was established on 11 Nov 1839, having originally been an armory manned by unruly troops. The armory was turned into a school, with the idea of having the students/cadets (or "keydets") guard the armory. The daily changing of the guard existed while I was there, and exists today.

Something old is shed and something new is added with each era. VMI started as a single barrack, and when I was there, I lived all four years in "New Barracks". This passed April, while visiting during my 20th reunion, I got a look into the "third barracks", which stands on the spot where LeJeune Hall, the cadet canteen, stood while I was there.

I did note that the cannon balls were still present in the wall of Old Barracks nearest Mallory Hall. To see other changes, check out Brother Rat, filmed in 1938, and starring Ronald Reagan...it was filmed, in part, on post at VMI.

VMI has a great tradition and a number of notable alumni; Chesty Puller attended, as did Patton. Dabney Coleman attended VMI. Perhaps VMI's most notable graduate is George C. Marshall, 1901, who went on to become the Chief of Staff of the Army (5-star general), Secretary of Defense, and Secretary of State. He is also a recipient of the Nobel Peace Prize, and the "Marshall Plan" for the restoration of Europe after WWII is named for him.

Happy Birthday, VMI, and Happy Birthday, Brother Rats!

Happy Birthday, Marines!

This is usually a technical blog, but I wanted to take a moment to recognize the service and sacrifice of all current and former Marines on what is the 234th Birthday of the Corps.

For 234 years, Marines have served just like is says in the Marine's Hymn, exemplified by the motto, Semper Fidelis (Always Faithful).

In my short service in the Corps, I celebrated this date on land, in foreign lands, and even at sea. I was even the commander of the cake detail once (not nearly as ominous as it sounds). I've also seen some pretty humorous things happen during the ceremony...like the time four 2ndLts from TBS comprised the cake detail... 4 2ndLts + 1 Cake + 1 ramp to the stage = a true Benny Hill moment!

That being said, today is a day to thank any and all Marines you know...and by "thank", hug or buy 'em a beer...or both.

Semper Fi!

p0wnage

A little over a month ago, I purchased a Dell Inspiron 1545 from the Dell Refurbished shelf. Most of the systems I've purchased from Dell have been procured through this route, and I've been pretty happy with the systems.

Until now. Tonight, I was p0wned by MS.

See, I'd purchased the laptop to do Windows 7 forensic (and in particular Registry) research. You know, use it like a user would and then see what the system "looks like" from a forensic perspective. Do what a user would do, then do like a forensic nerd would do.

Well, it seems that the laptop I purchased is running an Intel T4200 processor. It has 410 Million transistors, but does not support hardware virtualization.

Okay, my thought was that I was going to get an almost-brand-spanking-new system...no way it wouldn't support hardware virtualization. Well, it doesn't. What this means is that this laptop doesn't support XP Mode. Wow, so much for that rather critical portion of research.

So, the lesson learned here is, don't assume that the latest and greatest box, even one birthed in the past year, is going to have the necessary functionality to support what you want to do. In fact, as far as XP Mode is concerned, if that system you've got your eye on has an Intel processor, assume that it doesn't until proven otherwise. Wonder where I got that? My favorite forensics tool, Cory Altheide, found this at the Parallels site...notice what it says at the bottom of the page about AMD microprocessors.

Addendum: It seems that while the laptop I just purchased does not have a processor that supports hardware virtualization, the Dell Latitude D820 that I purchased in 2006 DOES! All I need to do is enable it in the BIOS...

More Linky Goodness, plus

I attended one day of the NetWitness User Conference last week, and ran into Richard Bejtlich there. Richard posted some of his thoughts on the keynote address that day. Hey, can you pick out in the picture which one is Richard? ;-)

As a side note, I'll be presenting at the WFO-ECTF meeting later this month, on Registry and timeline analysis.

Check out Claus's blog post on spilled COFEE. Claus also has an very good post full of links to useful tools...I know this is kind of circular, me linking to Claus's set of links (in this post by Claus, they come back to this blog...), but Claus's set is comprehensive enough, why bother retyping all that? Claus has always had some excellent links to portable apps, particularly AV tools that can make IR a bit easier, and in his most recent blog points out some remote desktop tools that IT admins may find useful, as well as Wireshark for Windows 7!

Brett and Ovie have got a couple of new CyberSpeak podcasts up...the 7 Nov podcast not only has a news story where a system infected with a keystroke logger led to the conviction of the suspect, but there's also an interview with Matt Shannon of F-Response.

Speaking of Matt, you have GOT to check out his latest press release about TACTICAL and the CyberSpeak podcast! Regarding TACTICAL, this is what Matt has about it on his site:

F-Response TACTICAL is the newest F-Response software product. TACTICAL has been uniquely designed to streamline live analysis, collection, and authentication. TACTICAL is built around the best of the F-Response Field Kit and Consultant edition, it was designed from the bottom up to be easier to use, faster, smaller, and more efficient.

TACTICAL uses a unique dual dongle/storage device solution that allows an investigator to bring their favorite tools to bear on Windows, Apple, and Linux targets.

This sounds very cool! Check out Matt's excellent CyberSpeak interview for more information! I cannot wait to see TACTICAL in action...

The 1 Nov CyberSpeak podcast has an interesting discussion on data breach notification...check it out.

Hey, anyone remember Clippy? If you do, check out the version of Clippy for VIM...

I ran into an interesting issue recently that I thought I'd bring up here. I've been doing some analysis, and as part of that analysis, I've been using regslack to get deleted keys and unallocated space from Registry hive files. When I find a deleted key, there's a LastWrite time associated with it, but sometimes I see Registry keys in the unallocated space of the hive files. What this means is that the key found in unallocated space no longer meets the conditions for or definition of a "deleted key" (more on that later). However, in the hex found in unallocated space, I can clearly see the FILETIME data for the key's LastWrite time, and that's something I might like to translate into something usable. So let's say I see the FILETIME data:

be 31 72 dc d4 94 c9 01

Using Perl, I can run that through a conversion routine and get something a bit more usable:

Sun Feb 22 10:03:50 2009

Pretty simple. Pretty cool.

Speaking of FILETIME objects, its a brave new forensics world, folks! Windows Registry keys have FILETIME LastWrite times, and some values contain FILETIME "objects" within their binary data. Others contain *nix epoch (32-bit) timestamps. Still others contain 128-bit representations of date/time stamps - this format is also found in Scheduled Task .job files. I was parsing some Symantec AV logs recently, and it turns out that they have their own format for date/time stamps! Interesting stuff...and there's SO much more!

Link-alicious

Thanks to Rob Lee's tweets, I read about AccessData's new Registry Summary Reports recently. These seem like baby RegRipper plugins...not quite there yet. I contacted some marketing guys from AD and they said that Registry Viewer has had this capability for years...I never knew about it when I was using FTK v1.71 or 1.8 (and I was too busy pulling my hair out just trying to install v2.0!!)...guess I just missed it.

ImagineLan has an SRP Explorer utility that looks quite interesting, allowing you do to do diffs of Registry hives across Restore Points. Interesting.

From the MMPC, a post regarding the MSRT October release, with a case study.

Rifiuti, the tool from FoundStone for parsing Recycle Bin INFO2 files, has a version available for Vista Recycle Bins called rifiuti2. This is actually a rewrite of the original code, according to the Google Code page. And yes, there is a version available for Windows.

Anyone caught the most recent edition of Captain Forensics??

Addendum: Chris Novak had an excellent (albeit short) article published on the Fundamental Failures With IR Plans...essentially, not having/testing one. Chris makes an excellent point...although war stories are sexy (re: fascinating) and writing is boring, the fact is, you have to have an IR plan. I was talking to someone recently about a company that stores CVV data, and does their own PCI self-assessment...and passes. I bet they also check "yes" when they get to the PCI DSS requirement 12.9...

I found some interesting stuff on Ophacki recently, starting with this analysis from Joe Stewart, and then moving on to the ISC write-up. Folks, if nothing else, this sort of thing demonstrates the lengths malware authors can and will go to in order to keep their stuff on systems. This bit of malware is a link hijacker, which essentially means that it's relatively harmless...however, more and more customers are asking folks like me (and the malware experts I work with) to determine if the malware has the capability to steal data, and if so, what did it take. First, this ties in to the IR plan article...if you have no plan and do nothing but shut the system off (i.e., capture no volatile or network traffic data), my ability to help you answer those questions is limited. Your IT staff are closest to the incident...it may take me 24-48 hrs to get there, depending on flights (and contract negotiations). If you have a plan and trained response staff, you can at least collect the data...

Second, if you haven't hugged your malware analyst today, do it now. I mean, seriously...look at how complex Ophacki is...it reportedly hides in memory by destroying the PE header. It doesn't decrypt the strings it needs in memory until it needs them, and then it deletes them when it's done. It also deletes the SafeBoot key, and reportedly hooks the APIs to delete Registry keys so that you can't delete the keys it uses for persistence. It also removes Zeus.

In short, there is a lot of sophistication for a simple link hijacker...which tells me that there's some kind of economy at work here...someone REALLY wants to make money!! What if the intent were different? What if the author really wanted to steal data? What if the malware was used to determine the type of system it was on...if a home user's system, look for indications of web-based purchases (or just sniff the keyboard), look for tax returns, and then turn the system into a zombie.

My point is that malware is getting more and more sophisticated, and malware authors are responding not only to standard business practices (i.e., Conficker spread via network shares) but also to some of the response procedures - some malware does not write to disk, but instead remains persistent in memory because the systems are rarely rebooted. Some folks recommend simply nuking the box from orbit...clean and reinstall...but malware authors know that the box will likely be put right back in place with the same holes and vulnerabilities, because that part of the re-installation plan is very often missed or skipped, because there is no IR plan.