Thursday, June 16, 2005

RAM, memory dumps, and debuggers...oh, my!

One of my recent posts on obtaining the contents of physical memory using userdump.exe attracted the attention of some folks as MS, and I ended up having a long conversation with Robert Hensing on the topic of obtaining and analyzing the contents of (physical) memory.

The long and short of it is this...the tools and techniques you use all depend upon what you want to do. How's that for a direct answer? No, that wasn't Robert's response...he provided much more information than that...I'm just summing it up for you. I'm going to give you a short and sweet explanation, opening this topic up for discussion.

If you want to grab the contents of physical memory, you can do so with dd.exe...keeping in mind that tools like this, as well as LiveKD, don't lock kernel memory prior to performing their dump. Therefore, what you get is a smear, of sorts, as the contents of physical memory are changing all the time, while you're dumping those contents. Also, keep in mind that dd.exe does not produce output that is compatible with MS debuggers.

Now, if you're looking to get the memory used by a process, then the way to go is to download the MS debugging tools, and run those from a thumb drive or CD. The debugging tools have a plethora of switches, but Robert was nice enough to write a VB script that's included in the tools called "adplus.vbs" that will let you easily dump the memory contents of multiple processes. The debugging tools do this by first suspending the process, and after the dump is complete, you can then either resume or kill the process. The output can then be analyzed using the debugging tools.

I have to say that this makes a lot more sense than using just pmdump.exe and strings. Robert even went so far as to point out that this can even be used when hunting for rootkits, as some rootkits will insert code into memory used by all processes...when attempting to perform analysis of the memory dump, the debugger will crash when it attempts to reference memory that doesn't exist or is hidden.

Now, what if you want to capture a snapshot of the system? Well, the only real way to do that is with a crash dump (ie, Blue Screen). And when I say "real", I'm referring to the most forensically sound method of doing this...that being said, being able to do this requires some preparation. I'd highly recommend that if you have critical systems that you're really concerned about, that you put a lot of thought into considering these options. First, take a look at KB254649 for an overview of dump file options on 2000, XP, and 2003, and then KB244139 for a feature that allows a memory dump file to be created from the keyboard. Go through the second KB article thoroughly and consider adding the recommended modifications to critical systems. I'm told that some folks out there have done this.

I'd also consider setting up testing systems with the same options, so that when you're testing malware, you can generate dumps of process memory or even crashdumps as part of your testing and then use the MS debugging tools for your analysis.

Robert added that there are debugger plugins that allow you to view and analyze very useful information, such as TCP/IP connection information, from a full crashdump, as this information is handled by the kernel.

So...the long and short of this is that some recommendations that work on Linux systems aren't exactly advisable on Windows boxen, depending upon what you want to do. Sure, you can use dd.exe to image memory, but the output isn't compatible with the MS debugging tools, and since there don't seem to be any tools available for really parsing and analyzing this information, it's of limited use.

Other useful resources include MS KB articles on debugging.


Robert Hensing said...

Actually - I didn't write the current versions of ADPlus (which I think is 6.something). I wrote versions 1-5. :)

Keydet89 said...

Either way, dude, thanks for the talk, and the info...I'm re-working some of the stuff I've had set up already, to include recommendations for an analysis box, how to do the analysis, etc.

One thing that's come up is that some malware knows when it's running in a VM session or in a debugger, so something like memory (or full-out crash-) dump analysis would be key for that sort of situation.

John H. Sawyer said...

Great post. I like the idea of using something a little more designed for the situation than just strings against dd grabs. Pmdump is definitely useful when dealing with malware that has encrypted executables that gets decrypted in memory, however, the debugging tools sound perfect for digging into those processes more thoroughly. Another good tool (on the unix side) for digging through memory and the pagefile is foremost as you can carve out quite a few files than just analyzing with strings.

I can just hear my sysadmins laughing when I start talking to them about setting up premeditated crash dumps on their production servers. They might actually go for it as their previous I'll_never_be_hacked mentality has been shattered. ;-)

Keep up the good work!

John H. Sawyer said...

Oddly enough, when I went to check for an updated version of foremost, there is now a beta port to Windows released on 6/12/2005.

Keydet89 said...


Good link...I've tried foremost and found it to be somewhat lacking. I modified the config file to only look for certain types of files (JPG, MOV, AVI, GIF, etc) and got quite a few files recovered, even though I had never viewed such files on the system in question (XP Pro in a VMWare session). All of the files that weren't 0 bytes could not be opened in the default viewers for each file type.

However, getting into pagefile carving is a little off-topic.

Djazz said...

Indeed an very interesting read.
One downside is that the crash dump method will only work if is te system is already prepared. :-(
How often will will that be, when you respond to an incident!

the first questions that come to my mind is:
Is it possible to use an UNC path for the location of the memory dump? Of course securing this location will be very imporant.

It is good to know the downsides of using dd. But I think it still will be an option till there is some method to lock the memory.

When examining an dd memory dump, could it be possible to use information from the PE headers the determine the length of executable and carve it from the dump?

Well only questions but no answers.

Anonymous said...

Check the address below to download most popular debugging tools