Sunday, February 01, 2009

Thoughts on memory footprints...

I was chatting with a friend the other day, and we got to talking about the issue of the memory footprints of various tools. We agreed on the need to capture the contents of physical memory first during incident response activities, before doing anything else. I'm also a proponent of grabbing the portions of memory (active process list, network connections, etc.) as well, as this information can be used to do a modicum of differential analysis, as well as give you a quicker view into what's going on...but be sure to grab as pristine a copy of memory first. As Nick and Aaron said at BlackHat DC 2007, so you "...can answer new questions later." Sage advice, that.

So we got to talking about the effect that just the first tool will have on memory...whichever tool you choose to use to dump the contents of physical memory, be it F-Response, FastDump, mdd, win32dd, winen, Memoryze, or if you choose to kick it old school on XP, an old, unsupported copy of dd.exe. Regardless, when you launch the tool, the EXE is loaded into memory to be run, establishing a presence in memory and a "footprint". This is something that cannot be avoided...you cannot interact with a live system without having some effect on the system. Responders should also be aware that the converse is true...during incident response for something like a major data breach (not naming names here...Heartland), the system will continue to change even if you don't touch the keyboard at all. Such is the nature of live response.

When the EXE is loaded into memory, it consumes pages that have been marked as available; these pages are not currently in use, but may contain remnants that may be valuable later in the examination, much like artifacts found in unallocated space on the hard drive. This is somewhat intuitive, if you think about it...if Windows were to launch a new process using pages already in use by another process, then you'd have random crashes all over the place. Depending on the amount of memory in the system, the memory manager may need to write some of the pages being used by other processes out to the pagefile/swap file in order to free up space...but again, the new process will not consume pages currently being used by another process.

I know that this is an overly simplistic view of the whole process, but its meant to illustrate a point...that point being that there's a LOT of discussion about "memory footprints" of various tools, but when I listen to the discussion, I'm struck by the fact that it hasn't changed a great deal in the past 5 years, and that it doesn't incorporate either the experts in the field, or actual testing. In fact, what strikes me most about these conversations is that the primary concern about "memory footprints" in many cases comes from the folks who "don't have the time" to conduct the very research and quantitative analysis that they seem to be asking for. I think that for the most part, many of use accept the fact that there, yes, Virginia, there is a Santa...I mean, there will be memory footprints of our live response actions, but that it is impossible to quantify the content that is overwritten.

I think that we can agree that memory pages (and content, given pool allocations, which are smaller than a 4K memory page) are consumed during live response. I think that we can also agree that from a technical perspective, while some pages may be swapped out to the page file, the only pages that will actually be consumed or overwritten with new content, and their old content will no longer be available (either in physical memory or in the page file) are those marked as available by the operating system. From an analyst's perspective, this is similar to what happens when files are deleted from a disk, and sectors used by those files are overwritten. This will lead to instances in which you can "see" data available in a RAM dump (through the use of such tools as strings or grep), but as the content may be in pages marked available for use, that data will have little context (if any) as it cannot be associated with a specific process, thread, etc.

2 comments:

Ritch said...

Harlan,
Let me start by saying thanks for all of your work in the forensics field and for maintaining this always informative blog. Although I am not an official 'blog follower', I do frequent your blog as well as many others.
This will be my first blog post so, here goes.

Regarding incident response, I have in the past been a proponent of first dumping a systems RAM and then (second) running some additional tools (which I have scripted) to grab a variety of different volatile and non-volatile system information. During some recent testing however, I may be re-thinking this order.

I was recently testing a variety of RAM dump tools including FastDumpPro, FTK Imager, Mdd, dd, and Winen. During my testing I found that Winen (64 bit version) would continually crash my 64 bit Windows 7 system causing it to re-boot. That made me think of previous testing that I had done (a couple of years ago) where other RAM dump tools had also caused my system to either freeze (causing me to force a re-boot) or crash and re-boot.

I remember two recent search warrants where I had was tasked with collecting running Windows systems. In the first case, the system crashed during the RAM dump and on the second case, the systems USB ports were so bad that it took me 10 minutes to just get the USB cable working.

That being said, during my testing I have not (yet) had a system crash while collecting volatile / non-volatile data. Perhaps I have just been lucky but so far, but I have never had any complications (other than trying to maintain a USB connection) while collecting system information.

So, I have started questioning my methodology. While realizing that capturing system information first (before the RAM dump) will certainly affect the RAM causing a much larger 'foot print', is it better to first capture the system data which is not as likely (in my experience) to cause a catastrophic system failure. A catastrophic failure would obviously cause the loss of all volatile data.

Working in Law Enforcement, I do not (usually) have the privilege of walking into a nice clean office to collect computers. More often than not, I am crawling around on a filthy oil stained carpet in some tweekers bed room (trying to keep the flees off of my pants) while attempting to determine just what type of system I have and how to process it. Many of these systems are only partially assembled and are lucky to have functional USB ports.

So, for now I am not sure how I will proceed. I will likely developed my methodology on a case by case basis and hope for the best.
Thanks,
Ritch

H. Carvey said...

Ritch,

Sometimes, you just have to do what you gotta do...

Don't worry, others have seen the same things you have...unfortunately, when it comes to dumping memory from Windows systems, you get what you pay for. Part of the problem is that there are areas of memory, that even if you just read them, you can crash the system.

If you don't have the necessary memory dumping tool, one that you've tested, then I'd recommend going with the batch file instead. Sometimes, the risk or cost of tipping the box is just too great.

Also, this is a perfect example of why I so strongly recommend that cops talk to us in the corporate side. You wouldn't believe the numbers of stories I've heard from LE, where some of us (not always me, believe me...) will say, "oh, yeah...we've seen that...".

For the most part, I think it's a trust issue, and a lot of LE simply aren't willing to climb that mountain. I know Maj Newell at the Broken Arrow PD in Tulsa, OK, did it...and she's reaping the benefits of it!