Tools

Memory Analysis
When I've had the opportunity to conduct memory analysis, Volatility and bulk_extractor have been invaluable.

Back when I started in the industry, oh, lo so many years ago, 'strings' was pretty much the tool for memory "analysis".  Thanks to Volatility's strings plugin, there's so much more you can do; run 'strings' (I use the one from SysInternals) with the "-o" switch, and parse out any strings of interest.  From there, using the Volatility strings plugin lets you see where those strings of interest are located within the memory sample, providing significant context.

I've run bulk_extractor across memory samples, and been able to get pcap files that contained connections not present in Volatility's netscan plugin output.  That is not to say that one tool is "better" than the other...not at all.  Both tools do something different, and look for data in different ways, so using them in conjunction provides a more comprehensive view of the data.

If you do get a pcap file (from memory or any other data source), be sure to take a look at Lorna's ISC handler diary entry regarding packet analysis; there are some great tips available.  When conducting packet analysis, remember that besides WireShark, you might also want to take a look at the free version of NetWitness Investigator.

Carving
Like most analysts, I've needed to carve unallocated space (or other data blobs) for various items, including (but not limited to) executable images.  Carving unallocated space, or any data blob (memory dump, pagefile, etc.) for individual records (web history, EVT records, etc.) is pretty straight forward, as in many cases, these items fit within a sector.

Most analysts who've been around for a while are familiar with foremost (possible Windows .exe here) and scalpel as carving solutions.  I did some looking around recently to see if there were any updates on the topic of carving executables, and found Brian Baskin's pe_carve.py tool.  I updated my Python 2.7 installation to version 2.7.13, because the pip package manager became part of the installation package as of version 2.7.9.  Updating the installation so that I could run pe_carve.py was as simple as "pip install bitstring" and "pip install pefile".  That was it.  From there, all I had to do was run Brian's script.  The result was a folder with files that had valid PE headers, files that tools such as PEView parsed, but there were sections of the files that were clearly not part of the file.  But then, such is the nature of carving files from unallocated space.

Addendum, 1 Feb: One of the techniques I used to try to optimize analysis was to run 'strings' across the carved PE files, in hopes of locating .pdb strings or other indicators.  Unfortunately, in this case, I had nothing to go on other than file names.  I did find several references to the file names, but those strings were found in the files that were clearly part of the sectors included in the file that likely had little to do with the original file.

Also, someone on Twitter recommended FireEye's FLOSS tool, something you'd want to use in addition to 'strings'.

Hindsight
Hindsight, from Obsidian Forensics, is an awesome tool for parsing Chrome browser history.  If you haven't tried it, take a look.  I've used it quite successfully during engagements, most times to get a deeper understanding of a user's browsing activity during a particular time frame.  However, in one instance, I found the "smoking gun" in a ransomware case, where the user specifically used Chrome (was also using IE on a regular basis) to browse to a third-party email portal, download and activate a malicious document, and then infect their system with ransomware.  Doing so by-passed the corporate email portal protections intended specifically to prevent systems from being infected with...well...ransomware.  ;-)

Hindsight has been particularly helpful, in that I've used it to get a small number of items to add to a timeline (via tln.pl/.exe) that provide a great deal of context.

Shadow Copies
Volume shadow copies (VSCs) and how DFIR analysts take advantage of them is something I've always found fascinating.  Something I saw recently on Twitter was a command line that can be used to access files within Volume Shadow Copies on live systems; the included comment was, "Random observation - if you browse c$ on a PC remotely and add @TIMEZONE-Snapshot-Time, you can browse VSS snapshots of a PC."

An image included within the tweet chain/thread appeared as follows:

Source: Twitter

I can't be the only one that finds this fascinating...not so much that it can be done, but more along the lines of, "...is anyone doing this on systems within my infrastructure?"

Now, I haven't gotten this to work on my own system.  I am on a Windows 10 laptop, and can list the available shadow copies, but can't copy files using the above approach.  If anyone has had this work, could you share what you did?  I'd like to test this in a Win7 VM with Sysmon running, but I haven't been able to get it working there, either.

Addendum, 1 Feb: Tun provided a tweet to Dan's blog post that might be helpful with this technique.  Another "Dan" said on Twitter that he wasn't able to get the above technique to work.

As a side note to this topic, remember this blog post?  Pretty sneaky technique for launching malware.  What does that look like, and how do you hunt for it on your network?

Windows Event Logs
I recently ran across a fascinating MSDN article entitled, "Recreating Windows Event Log Files"; kind of makes you wonder, how can this be used by a bad guy, and more importantly, has it?

Maybe the real question is, are you instrumented to catch this happening on endpoints in your environment?  I did some testing recently, and was simply fascinated with what I saw in the data.

Sniper Forensics, Memory Analysis, and Malware Detection

I conducted some analysis recently where I used timeline analysis, Volatility, and the Sniper Forensics concepts shared by Chris Pogue to develop a thorough set of findings in relatively short order.

I was analyzing an image acquired from a system thought to have been infected with Poison Ivy.  All I had to go on were IPS alerts of network traffic originating from this system on certain dates...but I had no way to determine how "close" the clocks were for the system and the monitoring device.

I started by creating a timeline, and looking for the use of publicly documented persistence mechanisms employed by the malware in question (Windows service, StubPath values within "Installed Components" Registry keys, etc.), as well as looking for the use of NTFS alternate data streams.  All of these turned up...nothing.  A quick review of the timeline showed me that McAfee AV was installed on the system, so I mounted the image via FTK Imager and scanned it with ClamWin 0.98 and MS Security Essentials (both of which were updated immediately prior to the scan).  MSSE got a hit on a file in the system32 folder, which later turned out to be the initial installer.  I searched the timeline, as well as the Software and System Registry hives for the file name, and got no hits beyond just the existence of that file within the file system.

The system was a laptop system and had a hibernation file, which was last modified the day that the system itself had been shut down.  I exported the hibernation file for analysis, and then downloaded the Win32 standalone version of Volatility version 2.3.1, and used that to convert the hibernation file to raw format, then identify the profile (via the imageinfo plugin) and get a process list (via the pstree plugin).

I got in touch with Jamie Levy (@gleeda), who pointed me to this post, which was extremely helpful.  Based on the output of the malfind plugin, I was able to identify a possible process of interest, and I searched the timeline for the executable file name, only to find nothing.  However, I did have a process ID to correlate to the output of the pstree plugin, and I could see that the process in question was a child of services.exe, which told me not only where to look, but also that the installation process required at least Administrator-level privileges.  Correlating information between the RegRipper samparse.pl and profilelist.pl plugins, I was able to see which user profiles (both local and domain) on the system were members of the appropriate group.  This also provided me with the start time for both processes, which correlated to the system being started.

A fellow team member had suggested running strings across the memory dump, and then searching the output for specific items; doing so, I found what appeared to be the configuration information used by Poison Ivy (which was later confirmed via malware RE performed by another team member).  As a refinement to that approach, I ran the Volatility vaddump plugin, targeting the specific process, and dumped all of the output files to a folder.  I then ran strings across all of the output folders, and found a similar list of items as when I'd run strings across the entire memory dump.

I validated the configuration item findings via the Volatility handles plugin, looking for mutants within the specific process.  I also dumped the process executable from the memory dump via the procmemdump plugin; the first time I did it, MSSE lit up and quarantined the file, so I disabled MSSE and re-ran the plugin (I set my command prompts with QuickEdit and Insert modes, so re-running the command line as as simple as hitting the up-arrow once, and then hitting Enter...).  I was able to ship the dumped PE file and the initial wrapper I found to a specialist for analysis while I continued my analysis.

I then ran the svcscan plugin and searched the output for the executable file name, and found the Windows service settings, to include the service name.  I also ran the dlllist plugin to see if there were any DLLs that I might be interested in extracting from the image.

I also ran the netscan plugin, and reviewing the output gave no indication of network connections associated with the process in question.  Had there been a connection established associated with the process of interest, I would've run Jamaal's ethscan module, which he talked about (however briefly) at the recent OMFW.

I had downloaded the Poison Ivy plugin, which according to the headers, is specific to one version of the PIvy server.   The plugin ran for quite a while, and because I had what I needed, I simply halted it before it completed.

Timing
Something important to point out here is that the analysis that I've described here did not take days or weeks to complete; rather, the research and memory analysis was all completed in just a few hours on one evening.  It could have gone a bit quicker, but until recently, I haven't had an opportunity to do a lot of this type of analysis.  However, incorporating a "Sniper Forensics" thought process into my analysis, and in turn making the best use of available tools and processes, allowed me develop findings in an efficient and thorough manner.

In short, the process (with Volatility plugins) looks like this:
- convert, imageinfo, pstree
- malfind - get PID
- run vaddump against the PID, then run strings against the output
- run handles against the PID
- run procmemdump against the PID (after disabling AV)
- run dlllist against the PID
- run netscan; if connections are found associated with the PID, run ethscan

At this point, I had a good deal of information that I could add to my timeline in order to provide additional context, granularity, and relative confidence in the data I was viewing.  File system metadata can be easily modified, but Windows Event Log records indicating service installation and service start can be correlated with process start times, assisting you in overcoming challenges imposed by the use of anti-forensics techniques.

It's also important to point out that Registry, timeline, and memory analysis are all tools, and while they're all useful, each of them in isolation is nowhere near as powerful as using them all together.  Each technique can provide insight that can either augment or be used as pivot points for another.

Resources
A couple of interesting items have popped up recently regarding memory acquisition and analysis, and I thought it would be a good idea to provide them here:

• Jamie also pointed me to this post, which, while it wasn't directly helpful (the image I was working with was from a Win7SP1 system), it does show the thought process and work flow in solving the challenge
• From MoonSols, some new features in DumpIt
• Mark Baggett's ISC Diary posts on winpmem and its uses: here, and here
• Page_brute - check out how you can incorporate the pagefile into your analysis, until Volatility v3 hits the streets

New Volatility Plugins

Andreas has posted a couple of new Volatility plugins recently that look really interesting, in that he provided the capability to link file objects to processes, as well as scanning a memory dump for driver objects.

Memory Analysis, for real!

A bit ago, Rich over at HBGary was nice enough to provide me with a dongle for Responder Field Edition, so that I could take a look at it and work with it from an incident response perspective. Having that kind of access to that kind of tool is really pretty amazing, as it provides a perspective to the overall environment that you don't get through reviewing web sites and testimonials.

First, some quick background...if you haven't been by the redesigned HBGary site, go by and take a look. Also, be sure to add the Fast Horizon blog to your RSS feed. The Fast Horizon blog gives some interesting insight into technical aspects of the Responder product (i.e., Digital DNA, etc.) that you won't get even by working with the tool. Also, as I'm working with the Field Edition, I don't have access to Digital DNA, so I won't be able to comment on that.

Okay, to make things even better when using tools like Responder to perform memory analysis, Hogfly has started a memory snapshot project, and has posted a memory snapshot from a VM guest running Windows XP (part II is posted here), with actual malware running. Pretty cool, and a great idea at that! This not only provides access to memory dumps with something to actually look for, but it also provides a great example of why you'd want to dump memory in the first place!

I'll be taking a look at Hogfly's first memory snapshot (.vmem file) with Responder and Volatility, as the snapshot is of a Windows XP system. I won't be using Memoryze, as I downloaded the MemoryzeSetup.msi file and ran it several times, and had nothing installed. I tried specifying a path for the installation, as well as using the default path. I even tried re-downloading the MSI file. Nothing worked.

Responder Field Edition
Responder provides a very easy to use GUI interface for working with memory dumps or snapshots. You simply select the memory snapshot to import, select a few options for what to do during processing, maybe add some keywords to search for, and then you let the tool do its thing.

A couple of very interesting things that Responder will do is look for "keys and passwords" (a string search for components that might include passwords) as well as for "Internet History" or URLs. When locating both of these, the column display in Responder includes a field for the offset where the string is located; this is useful as you may be able to map something interesting to a location in memory to give that string some context. In reviewing the recovered URLs from Hogfly's memory snapshot, I found a LOT of Microsoft- and MSN.com-related entries, as well as some others, such as:

http://192.168.30.129/malware/
http://*:2869/
http://www.iec.ch
http://%s:%u http://[::1]:8080/

There are a number of others (too many to list here), and there were also references to tunes.com and musicblvd.com.

For incident response purposes, one of the things I really like about Responder is the ability to quickly get the memory snapshot open and locate the issue. In this case, for me, the "offending" process stood out like a sore thumb...svhost.exe, PID: 1824, with a command line of "C:\Windows\msagent\svhost.exe". From there, you can expand the tree listing for that process in order to view specific information about and from that process. For example, we can see that the process has a couple of open file handles, to include the "foo" user's TIF index.dat file, and that the process also has an open TCP socket from port 1059 to a remote IP address on port 9899. This information can be extracted quickly once the memory snapshot has been collected, and the information available can then be used to correlate with other information, such as network packet captures, firewall or network device/appliance logs, etc.

From an incident response perspective, Responder is a great tool to really get you started in quickly identifying issues. I would like to be able to dump the binary, if possible...I selected the process in the Processes tree view (left-most pane of the UI), right-clicked and selected Package, View Binary from the context menu...some dialog boxes flashed, but I didn't end up with a binary, nor did I get anything stating that I couldn't have it. This may be due to the fact that the memory pages for the process executable had been paged out; in cases such as this, using FastDump Pro would've been the way to create a memory snapshot, incorporating the pagefile, as well, providing a more complete picture.

Over all, however, Responder lets you very quickly process information in a graphical format, providing speed and agility for something for which just a couple of years ago, there was no real process or methodology at all available.

Addendum: I heard from Rich with respect to parsing strings from binaries with Responder, and its actually pretty simple. If you select a particular process in the Processes tree view, you then expand the process listing for the Modules (see the above image) and select the executable image, which is also listed as a module. Expanding that tree lets you see Bookmarks and Strings. So with a couple of mouse clicks, you can view the strings extracted from any particular binary. Similarly, viewing, analyzing, or saving a copy of the binary is simply a matter of right-clicking on the name of the EXE file in the Modules tree view, choosing Package, and selecting the option that you want (see figure).

Volatility
Analyzing the memory snapshot with Volatility was somewhat different, as Volatility is a purely command line interface (CLI) tool. However, that does not make Volatility any less powerful when it comes to memory analysis. I won't go through everything Volatility can do, as it's been discussed before (and is rather nicely documented), but suffice to say that I ran the following commands against the memory snapshot:

ident
datetime
pslist/psscan2
connections/connscan2
sockets
files
regobjkeys

The great thing about these commands is that they can be included in a batch file and run all at once, and finished off with tools such as JL's vol2html to tie everything together into a single report (examples here and here). Don't forget that you can also include other modules, such as malfind, moddump, or Moyix's Registry modules. JL's code is a great example of what you can do with these tools, showing the flexibility available through Volatility.

Correlating output like this can make it easier to identify suspicious processes. From there, once you identify a process of interest, you can then use the "procdump" and "memdmp" commands to collect as much of the executable image and process memory, respectively, as you can. In this case, the executable image data appears in the local directory as "executable.1824.exe" and the memory dump appears as "1824.dmp". You can then view the contents of these files in a hex editor, or simply extract the strings. When running the procdump command, however, I received a number of "Memory not accessible" responses, indicating that the memory pages had been paged out.

As you can see, various memory analysis tools have strengths and weaknesses, and should be considered just as they are...tools to do a job. Rather than advocating one specific tool, I'd advocate understanding and using them all. Responder is more OS-complete than some of the other available tools, while Volatility provides a level of granularity not seen in the other tools. At this point, if someone asked me, "which tool do you recommend...Responder, Volatility, or Memoryze", I'd have to say "yes".

Another thing to remember is that looking at these tools in isolation only provides part of the answer, and only taps a small portion of the power available to you as a responder. For example, throw F-Response EE and FEMC, and RegRipper into your toolkit, and you're really expanding your capabilities.

Summit Takeaways

I recently received an email from the organizers of the SANS Forensic Summit and was asked to provide my biggest takeaways from the summit, and I wanted to post what I came up with, to see if others are seeing the same sorts of things...so, here's what I provided:

I would say that the key take-away for me is that speed is essential, but should not be sacrificed for accuracy. Incident responders need to respond quickly in order to identify, classify, and as appropriate, contain an incident. In part, this puts the onus on the organizations...even if they have consultants on retainer, it will still take time for them to arrive, and then they are slowed down even further having to get up to speed and learn the environment. Consultants can assist an organization in developing a tier 1 response, training them in triage, diagnosis, preview, containment, etc., even across an enterprise. I would also say that tools such as F-Response are paramount to this sort of activity, as well. Once the initial training is complete, the consultants can respond as tier 2 or 3 responders, assisting as necessary, but knowing that the necessary data has been collected.

It's clear that state (via PII notification laws) and federal legislation, as well as regulatory oversight (PCI, HIPAA, etc.), play a huge role in incident response. In fact, these are the primary drivers to IR at this point. Think about it...if a company was not required to notify someone when a breach of sensitive data occurred, would they?

To that end, there needs to be a much more immediate response...calling a consulting firm to come on-site to assist after the incident has occurred and been detected is a fantastic idea, but one of two things are going to happen; you're either going to continue "bleeding" data while you're waiting, or you're going to stomp all over the data and destroy the indicators (aka, "evidence") that those consultants will be looking for in order to answer the questions that need to be answered.

Okay, brief digression here...what questions will need to be answered? There are essentially three questions that need to be answered in the face of a breach of sensitive data...(a) was a system compromised, (b) did the system contain sensitive data, and (c) was that sensitive data exposed or compromised as a result of the breach? In order to determine (c), in most cases, you need to minimize "temporal proximity" (thanks, AAron, for that very Star Trek-y term!!)...that is, you need to detect the breach and collect data as close to the time of the breach as possible.

Rather than fall victim to a breach and not get the answers you need (by "you", perhaps a more appropriate way of saying it would be "your legal counsel or compliance officer"), why not get someone in ahead of time, before an incident occurs, to work with you in setting up a response plan, and training your staff in a timely, accurate response?

I've said before that tools like F-Response and Volatility (the combination of the two being referred to as Voltage) have changed the face of incident response. Having these tools available to you allow you to quickly collect and analyze memory in order to triage and categorize incidents. Too many times I've been asked to come on-site only to find out that prior to the call, the customer had already turned systems off and taken them offline. These tools will help you collect more data than every before, reduce that "temporal proximity", and at the very least have data available for tier 2 incident responders to process and analyze.

Thoughts?

PlainSight

I received word from the author today of a new open-source tool that's available called PlainSight. Eoin says that the tool is part of master's program, and at this point, the tool is somewhat proof-of-concept, but looking at the tool demos, it looks as if it has a bit of promise.

The main web page describes the tool this way:

PlainSight is a versatile computer forensics environment that allows inexperienced forensic pactitioners perform common tasks using powerful open source tools.

We have taken the best open source forensic/security tools, customised them, and combined them with an intuitive user interface to create an incredibly powerful forensic environment.

PlainSight incorporates other open-source tools (RegRipper, Volatility, etc.) to create a framework for examining disk and memory images, or local disks.

Eoin says that he plans to add the following:

- Better browser support (FF3, Opera, chrome),
- Some sort of e-mail viewer,
- Integrate in moreRegRipper plugins,
- Better support for other operating systems (currently supports Windows 98/2000/XP/Vista)

I've downloaded the ISO and would like to take a look at this as soon as I get a chance. It appears that this runs in a Linux/Knoppix environment, so perhaps some suggests might be to create a Windows version. After all, the description of the tool says it's for allowing inexperienced examiners to perform some tasks...so why not provide the capability in an environment that the examiner may be more familiar with.

Even so, at first glance, this is looks like it's the kind of thinking and effort that is needed in this community, and is definitely a step in the right direction.

Volatility 1.3 is out!

Volatility 1.3 is out! Volatile Systems (AAron, et al.) improves upon their venerable open-source memory analysis tool with this latest version, adding capabilities for extracting executables and a process's addressable memory, support for formats other than a dd-style memory dump (such as via mdd). Volatility 1.3 supports memory dumps from Windows XP SP2 & 3, and in addition, there is preliminary support for Linux memory dumps, as well.

The world of incident response is seeing changes. Incident responders have known for some time that particularly in the face of state notification laws and compliance standards from regulatory bodies, a new dimension has been added to what we do. Simple containment and eradication...get the bad guy or malware out...is no longer sufficient, as traditional first response obviates an organizations ability to answer questions about data exfiltration. Its long been known that new procedures and methodologies are needed, and in many cases, new tools. Well, folks, Volatility is one of those tools. When combined with tools like F-Response, the speed of response is maximized, allowing for greater data protection and business preservation. With F-Response increasing a responder's ability to collect data, and Volatility increasing the breadth and depth of analysis that can be performed on the memory dumps, a brave new world is opened up for incident responders!

The need for speed (without sacrificing a thorough and accurate response) is further illustrated in this SecurityFocus article, which illustrates something that incident responders and forensic analysts see all the time...AV doesn't always work the way we think it should.

Not only that, Volatility adds to a forensic analyst's toolkit, as well. The latest version has the ability to parse crash dumps, as well as hibernation files. Thanks to Moyix for all of his current and up-coming contributions, as well as folks such as Matthieu, Jesse, and all of those who put effort into the framework. Forensic analysts now have the capability to parse through and analyze additional historic remnants on a system.

Volatility requires Python, which for Windows systems is freely available from ActiveState.


Addendum: Yesterday, Matthieu updated both win32dd and the Sandman Framework. I tried out win32dd, and it worked like a champ! My first attempt resulted in some issues as a result of operator error...if you hit Ctrl-C while win32dd is running, you will get an error during every consecutive attempt to run the tool again, until you reboot. However, one of my tests as to run mdd and win32dd consecutively, and the result was files of identical size. The next step is to run Volatility 1.3 against each of them and see if there are any differences in results.