Tuesday, January 06, 2009

Memory Collection and Analysis Tools

Recently, there was a post on the SANS Forensics blog about memory collection and analysis tools, but for some reason, it seems that folks are STILL having trouble with this process; I'm seeing posts in forums (forii??) saying that "...this tool doesn't work..." or "...that tool crashed when I tried to use it...". My suspicion is that the tools aren't being used correctly as folks may not understand their limitations.

Collection
I tend to separate collection and analysis...I've found that it's better to do each one right separately than it is to try to do both at the same time and split your resources.

XP/2003
There are a number of tools that allow for collection of physical memory on Windows XP and 2003 systems. The first and perhaps best known was a version of dd.exe modified by George M. Garner, Jr. This tool is no longer available, but it was used as the exemplar tool for collecting physical memory for the DFRWS 2005 Memory Challenge. Another tool is dcfldd.exe, written by Nick Harbour (currently with Mandiant). However, this blog post is one of several locations where it seems that someone's discovered that attempting to dump the \\.\PhysicalMemory object doesn't work, but dumping /dev/mem (yes, on Windows) appears to have an affect. Nigilant32 is a GUI-based tool that you can use to dump the contents of RAM, as well.

These tools are also available on the Helix CD. In addition, HBGary released their fastdump tool for dumping physical memory.

The delineation mark for the tools came with Windows 2003 SP1, as that's when MS removed user-mode access to the PhysicalMemory object. This, in turn, required a change in tools to collect physical memory.

2003 SP1 and Above
The tools used to collect the contents of physical memory for Windows 2003 SP1 and above (Vista) systems can also be used on XP and 2003 systems.

The first tool available to dump (and analyze) the contents of Physical Memory from Windows 2003 SP1 systems and above was the KntTools from George M. Garner, Jr. Then came ManTech's MDD, Matthieu Suiche's win32dd, GSI's winen (and winen64, for 64-bit systems), and Mandiant's Memoryze. All of these are great tools, but like any tool, each has it's strengths and weaknesses.

For example, right on the main page for the MDD tool, it states, "The software can copy up to 4 GB of memory to a file for later analysis." I'll leave it to smarter folks to tell you why, but I think that most tools have this same limitation. Matthieu's win32dd will dump memory in a raw, dd-style format as well as a crash dump-compatible format in case you'd like to use MS Debugger tools to analyze the dump. GSI's winen dumps memory in the EWF-style format (requires additional support for most tools to analyze). While most of the other tools download as an EXE, Memoryze downloads as an MSI file, and creates quite a footprint on the system.

One tool that must be mentioned here is F-Response. By itself, F-Response does not provide the capability to dump the contents of physical memory, but the most recent version of F-Response provides you with remote, read-only access to physical memory on Windows systems; you can then use tools like FTK Imager, or any other acquisition tool (dd.exe, dcfldd.exe, etc.) to acquire your memory dump. The figure I've posted here shows what physical memory on a remote system "looks like" on your investigation system...in this example, it shows up as \\.\PhysicalDrive2, and can be accessed by tools such as FTK Imager. Go here to see how to install F-Response EE remotely on Windows systems.

All of these tools work best if tested and deployed before an incident occurs, before the tool itself is actually needed. Tools such as mdd.exe and win32dd.exe can be deployed on CD or a thumb drive, and F-Response can be deployed remotely (even in stealth mode) over the network, allowing you to accelerate your response.

Note: I don't want to leave out hibernation files. While I don't recommend hibernating a system as the default process for collecting the contents of physical memory, there may be times when its the only option. For example, if a system already has a hibernation file on it, IMHO, I'd prefer to use that file for historical analysis, to establish or validate a timeline of activity on the system. However, if the system is capable of hibernating, there may be times when forcing the system to hibernate may be the only way to collect the contents of physical memory. You can use powercfg.exe to configure the system to hibernate, and then use run32dll.exe to force it to hibernate.

Analysis
There are some frameworks available that provide a means of analysis on memory dumps from several versions of Windows, such as Andreas's PTFinder Collection. Andreas's Perl scripts are primarily used to list processes (and pool tags) from within Windows memory dumps.

About the only tools I can think of that provide more than what Andreas's scripts provide for analyzing Windows 2000 memory dumps are my own scripts, which are still available on SourceForge, as well as on the Windows Forensic Analysis DVD (and they will be provided on the DVD with the second edition of WFA).

When it comes to analyzing memory dumps from Windows XP SP2 and 3 systems for an incident response perspective, the Volatility Framework is the way to go. Yes, I know it's CLI, requires Python (which is free), but to be honest, this framework gives you unprecedented access to the contents of the memory dump. I say "contents" because not only can you get things like the active process list and active network connections at the time of the dump, but you can also run a linear scan through the dump for exited processes and expired network connections. The DFRWS 2008 Rodeo includes a memory dump you can try this out on. The output of the various commands can be easily redirected to files, and then correlated using Perl or Python scripts (such as vol2html.pl).

Volatility includes the capability of working with hibernation files (incorporating Matthieu's work; note that Matthieu has also released a standalone hibernation shell, in alpha) and crash dump-format memory dumps, in addition to the raw, dd-style format dumps. Also, Volatility is extensible, and others have taken the opportunity to create modules that can be easily installed and used with the framework (ie, Jesse Kornblum's cryptoscan and suspicious modules, and Moyix's Windows message queue module).

Addendum: I've recently seen the release of additional plugins; malfind from MNIN Security, and moddump from Moyix. I've gotta take a look at and try these plugins...man, it's it AWESOME the way Volatility can be extended??

Note: This is one of those places where I have seen folks say, "It doesn't work." As of now, Volatility works on XP SP 2 and 3 memory dumps. Please feel free to try it on other memory dumps, but as a courtesy, please refrain from stating "it doesn't work" (most of us already know that), as well as asking "why doesn't it work?" Thanks!

Mandiant's Memoryze is an XML-based tool that started life as part of the Mandiant Intelligent Response (MIR) product, and will allow you to collect and analyze memory dumps from Windows XP, 2003, and Vista systems. To install Memoryze, download the MSI file and the installation wizard will guide you through the process. Memoryze ships with a number of batch files that make it a bit more user-friendly, and Mandiant also include the Audit Viewer, a wxPython-based GUI tool that makes viewing the output of Memoryze a bit easier. Posts on the M-unition blog that describe how to use and extend Memoryze.

Also, please feel free to use tools such as strings or grep to extract information from memory dump files. However, keep in mind that if these tools are run against the entire dump file, the information you find may be without context. However, using any of the available tools to dump process memory for a memory dump file, and then searching it using strings/grep will provide you with a modicum of context.

It wouldn't be fair of me not to note HBGary's Responder product. I had an opportunity to try out the Responder Pro product a while ago, and found it be very useful for malware analysis. I haven't had an opportunity to try the Field edition.

13 comments:

Anonymous said...

I'll guess that our recent dialog on memory tools was an instigator for your blog. ;-) Since we spoke, I chatted with a ManTech rep who, thanks to Jesse K., noticed some of the threads on a forum. ManTech is studying the 4GB limit, as it's a critical issue as 64-bit systems are becoming the norm. I also learned that ManTech is working on an apparent conflict with Nvidia drivers, which might have been the reason why I had trouble with MDD on an XP64 VM. I don't have a machine that doesn't have Nvidia display drivers!

It's important to distinguish the 64-bit machines and, perhaps, Vista. First, Matthieu's tool, as its name implies, is 32-only. Matthieu graciously offered some comments and pointed out that driver signing costs are an impediment to developing a Vista 64 tool. Driver signing also prevents MDD from operating on Vista 64. (I think that Vista 32 also requires driver signing, but I'm not certain.) Does F-Response allow acquiring Vista RAM remotely?

The hibernation approach may be the only alternative if its essential to acquire memory on, for example, a Vista 64 machine with >4GB. Applied by a qulaified person, the invasiveness may be about the same as running a tool on the local system. As your second section suggests, acqusition and analysis must work together.

Jamie Butler said...

Harlan,

Thanks for mentioning Memoryze and Audit Viewer. I would like to quickly point out two things. Yes, Memoryze does have a larger footprint than win32dd or MDD; however, if you do not mind writing about 4 or 5 MB to disk (for installation), I do not think you loose anything when analyzing a running system's memory. I say this because Memoryze uses the paging file(s) so if a process or data was forced out of memory Memoryze should still be able to analyze it. Of course this only applies when doing analysis. Memoryze does not acquire the paging file(s) when it acquires a snapshot of memory. You must do the data collection/analysis on the target machine. Second, to provide more context than running strings across an acquired image Memoryze provides a "-strings" parameter that will collect all the strings in the context of the owning process.

Again, thanks for the writeup. Can't wait to get your new book!

Jamie

Matthieu Suiche said...

Harlan > Did you forget to mention the most famous post-mortem debugging tool in the Analysis part? WinDbg? :-) win32dd aims to avoid to reinvent the wheel regarding the analysis part. I don't really think a single man could beat the awesomeness of WinDbg developper team and their several years of works and experience. That's why I strongly suggest this tool -- I assume investigator doesn't analyze memory snapshot on the target machine. It looks Memoryze philosophy is the opposite since Jamie said "You must do the data collection/analysis on the target machine.".

Jimmy > In fact, driver signing is only mandatory on 64-bits Microsoft Windows O.S. from Windows Vista and later. This donnot apply to 32-bits version of Windows Vista, 2008 and Seven.

H. Carvey said...

Great comments all, thanks.

No, Matthieu, I didn't forget anything...I wanted to leave something for smart people like you to talk about! ;-)

H. Carvey said...

Jimmy,

No, the credit for instigating the post goes to someone in the ForensicFocus forum...sorry, dude.

Anonymous said...

Harlan,
One minor update to memory acquisition tools vs. Volatility; There's currently a known bug in EnCase Enterprise and Winen that causes the first 256K of an Acquired (but not Unerased) memory image to get nulled out. You can preview the memory image in EE just fine, but it gets mangled during acquisition. If you Unerase it rather than Acquiring it to an image file, the problem does not arise. See the following Guidance Forum thread for more details:

https://support.guidancesoftware.com/forum/showthread.php?t=33851

I've heard that the fix for this is currently scheduled for inclusion in 6.13.

John McCash

Jamie Butler said...

Matthieu,

Our philosophy is a little different from yours for two reasons.

1. Memoryze is the memory acquisition and analysis portion of our enterprise product MIR as Harlan pointed out. Because it was built with the enterprise in mind, it is not practical for us to acquire and bring back 15,000 or 20,000 memory images to analyze offline.

2. The risk of doing the analysis on the host is similar to the risk associated with acquiring memory on the host. A.) The attacker can block access to physical memory or B.) as Darren Bilby pointed out in his talk at Ruxcon06, the attacker can intercept the calls to map the view of physical memory. Both are possible when doing acquisition or analysis. If attack A is carried out, Memoryze or win32dd will error out. If attack B is used, neither tool will be effective. Joanna demonstrated the equivalent of attack B even when doing hardware acquisition.

BTW, I really think your work with the hibernation file is awesome.

Jamie

H. Carvey said...

John,

One minor update to memory acquisition tools vs. Volatility...

While I'm aware of the reported issue with winen, I'm not sure how the rest of your comment applies with respect to the first sentence...I tried to clearly point out that Volatility is an analysis tool, NOT an acquisition tool. Sorry for the confusion.

Anonymous said...

Harlan,
Apologies for not being clear. I didn't complete my comment. The relevence is that the nulled out first 256K of the memory image, as caused by the EnCase/Winen bug, results in a failure of Volatility to correctly identify the type of the memory image. It breaks some of the other Volatility commands as well.
John

Stu said...

Excellent information on your blog. Good job. My company at http://www.progressiveoffice.com does computer network support. So, this information was very helpful. Thank you for the work you did on this.

Matthieu Suiche said...

Jamie,

1. This is a very interesting point. As far I've seen people in Forensics industry have several different opinions regarding this point. Yesterday I talked about this with NFI people they said they need to know how much memory is used by tool because this is very a very important point in front of the court. Moreover, they want to use the smallest program as possible to avoid to erase evident in unallocated pages. I don't know how exactly Windows Memory Management deals with exited and killed programs, then I don't know if this way of thinking is right or not.

2. This is the reason why win32dd is going to be less and less dependant of Windows Kernel API to avoid such attacks. In a unreleased version of win32dd I was able under Windows 2003 and below to reuse the crash dump address space (fixed address at 0xFFBE0000), used during the BSOD, with my own functions to map physical address I wanted. Behavior is a bit different under Vista and later, Microsoft Kernel developper rather used MiReserveSystemPtes() function instead of a fixed virtual address. I'll publish an update in the next version of win32dd to add an option in addition of the two actuals (\Device\PhysicalMemory and MmIoMapSpace).

Thanks, I really appreciate this feedback from the author of "ROOTKITS, Subverting the Windows Kernel" book :)
Matthieu

John H. Sawyer said...

Hey Harlan,

I just received the following press release from HBGary about their new FastDump Pro that handles 64-bit and 4+ GB RAM.

"Sacramento, California – January, 13, 2009—In a significant industry milestone, HBGary, Inc., (http://www.hbgary.com) a leading provider of computer forensic, incident response, security assessment solutions and services, announced FastDump Pro, the first memory acquisition software to offer 32- and 64-bit support for all Microsoft ® Operating Systems from Windows ® 2000 and up with more than 4 gigabytes of RAM. For the first time, organizations and investigators can preserve and analyze physical memory snapshots of 32- and 64-bit Windows® Operating systems including Windows(R) 2000, Windows ® XP, Windows ® Server 2003, Windows ® Vista, and Windows ® Server 2008 and all their service packs.

“Based on feedback from Fortune 100 and government customers, computer intrusions into physical memory were one of the top security concerns in 2008. Some malware is not visible anywhere on the computer but in physical memory. While hackers continue to code malware in 32-bit, this malware is migrating to infect workstations on 64-bit Windows platforms that have more than 4GB of memory” said Greg Hoglund, CEO of HBGary, Inc. “As a result, we needed to develop and offer 64-bit versions of our memory acquisition software to help customers find the artifacts needed to solve their computer intrusion and forensic investigations. As an analyst or investigator, you never know what piece of information will help you solve the crime. Today if you’re not preserving and analyzing physical memory offline, then you’re not performing a complete computer investigation.”

FastDump Pro is the next generation version of HBGary’s flagship product, FastDump. FastDump was first released in April 2008 as a free download. Since its release, Fortune 100 corporations and 20 out of the top 30 government agencies have downloaded the product.

Other Key Features
FastDump Pro enables investigators and security analysts to easily "freeze the live memory" on workstations and servers. In addition to 64-bit support, FastDump Pro also provides process probing, compression, speed upgrades, and nearly 100% reliable memory-page queries for systems with more than 4GB of RAM.

Pricing and Availability
FastDump Pro is available now. Please contact sales@hbgary.com for purchasing. Current HBGary customers will receive this version with their Responder license at no cost while the product also can be purchased separately at $100.The original flagship product, FastDump which provides 32-bit support, remains as a free download at http://www.hbgary.com/download_fastdump.html"

Andrew Tappert said...

And for those interested in memory acquisition and analysis on non-Windows platforms, please check out Second Look for Linux memory forensics from my company, Pikewerks.