Thoughts on Analysis, Information Sharing, and Presentations

Since June of this year, I've attended a couple of conferences, not only presenting but also attending several presentations. As with any conference, presentations tend to vary with the background and experience of the author. By experience, I don't so much mean with the experience the author has in the topic they're presenting on, but more so with developing presentations specific to the audience, public speaking, etc.

One of the questions I ask myself when both creating and giving a presentation, as well as attending one, is "how is this immediately valuable and useful to me?" Often times, we attend a presentation that may have some great information, but the question then becomes, okay, but how to I use this? How do I most quickly/immediately turn this information around and use it on a case or examination?

I'm sure that a lot of other folks feel the same way, particularly after having talked about this very subject after a presentation or discussion. I've also seen this enough to add this sort of approach to my books, starting with WFA 2/e and progressing even more so into Windows Registry Forensics. With a lot of what's going on in our community, it's vitally important that analysts be able to get up from a presentation or talk, and be able to put what they've learned in a technical presentation or lab into practice. If that isn't the case...what's the point?

This is critically important when you're talking about analysis techniques, which is particularly where we need to innovate. Data is always going to be data, and it's always going to be there. There's been discussion about triaging systems, due in part to the massive increases in storage and the amount of time it takes to acquire and analyze all of that (especially acquire). So whether you're talking about browser forensics or

Innovation in Analysis
How many folks out there use RegRipper? How about using RegRipper for malware detection or analysis? Seriously.

Bear with me on this...it probably wouldn't occur to most folks to use something like RegRipper for malware detection, looking into an acquired image, or a system accessed via F-Response. However, I (and others) have used this approach time and again to our benefit.

Consider Conficker. Microsoft cites five variants. In each case, when the next variant first came on the scene, the executable file was not detected by most commercial AV tools. However, there were some consistencies across the malware family, including not just artifacts (ie, disabling system functionality) but also in the persistence mechanism.

I had the distinct honor of reviewing on of the chapters for the new book, The Malware Analyst's Cookbook, and in that chapter, the use of RegRipper was discussed. MHL and his co-authors had written several plugins for use in their work (included in the book) and to be honest, they were innovative.

Okay, so you're probably looking at this so far and thinking, yeah, but that's for malware you know about. Okay, sure, I can go with that. But here's the deal...while you may not have seen all malware, or at least a great deal of it, but what are the chances that within a large group or community of forensic and malware analysts, a LOT of malware has been seen and analyzed? Of those, their artifacts and persistence mechanisms will likely have been identified (Zeus, or ZBot falls into this category) and be worthy of a plugin.

Consider malware based on PoisonIvy. Yes, it's polymorphic, which for most of us, is going to mean that AV scanners are going to have minimal effect in detecting this stuff. However, keep in mind the four characteristics of malware...the persistence mechanism is an example of Jesse Kornblum's "rootkit paradox"; in short, most malware will want to run, but will also want to remain persistent. As it turns out, PoisonIvy isn't the only malware that uses the Registry Installed Components as the persistence mechanism.

Institutional Knowledge
How many of us use the institutional knowledge developed on previous cases/engagements on any new analysis that comes in? Most of us are going to raise our hands and say, "I do that all the time."

Now, how many of us make use of the institutional knowledge developed by other analysts, such as other team members?

Tools like RegRipper (and pretty much any tool that has some level of extensibility, including EnCase and ProDiscover) provide for sharing of institutional knowledge, but only insofar as that institutional knowledge is documented.

So What?
I'm sad to say that not all of us will ever see everything, particularly when it comes to IR and digital forensics analysis. For myself, I know that I haven't seen everything...I've seen some things several times, I've been on engagements that never where, but I haven't seen everything. Expand this to include any 5, 10, 50, or 500 analysts you want, and while it may seem that across all of us, we've seen a lot, the question then becomes...how much of that have we shared with others, or each other.

The issues the community faces are not going away. Increases in things like storage space, sophistication and proliferation of malware and compromises/intrusions, AV scanners becoming less and less of a reliable resource, to name a few. I think that most of us are familiar with these issues. So what do we do about all these?

CyberSpeak
Ovie's on a roll...don't miss the new CyberSpeak podcast! Ovie has an interview regarding triaging systems...talk about timely! ;-)

What do you need as a responder?

Many times when working with customers or talking with folks who interface with incident responders (or are incident responders), the topic of what information is pertinent to a responder comes up. When receiving calls from customers, many times the people who make the call and interface with the consultants are not themselves responders (or in some cases, even technical people), and instead represent their company from a business or legal/compliance perspective. This can often lead to misunderstandings due to differing perspectives, which ultimately leads to a lack of (accurate) information. However, this isn't restricted only to non-technical people...sometimes having the sysadmins on the call can lead to the same sort of thing, which is, again, largely due to perspective.

So...what do responders need to know when you call, or what information do your responders need access to?

Well, the first thing that an incident responder is going to need to know is your goals, or what you hope to achieve from the response activities.

In an ideal world, there are four basic sources of technical data that responders will be looking to...network traffic captures, network device log data (ie, firewalls, routers w/ ACLs, etc.), and host-based volatile (memory) and non-volatile (system image) data. There may be instances when all of this information is available, and there may be times when portions of this data is available...however, in many cases, for those unprepared for an incident, very little if any of this is available to a responder. The amount of information available to the responder is going to play a direct role in how completely the responder is going to be able to help you achieve your goals.

I once received a call from someone who needed help with a laptop. Apparently, the laptop had been infected with malware, and the immediate response was to take the laptop off of the network, wipe the hard drive, and install an updated version of the operating system. The caller wanted to know if I could tell them what the malware was, and what (if any) data had been taken. Seriously.

Some questions you're likely to be asked when you call someone to perform incident response include (but are not limited to):

- What type of incident are you experiencing? How would you characterize the incident?
- Do any of the affected systems contain "sensitive data" (per PCI, HIPAA, etc.)?
- How many systems are involved? What are their status? What are their hard drive types, configurations, and capacities (SAN/NAS, RAID 5 with a total of 100GB, mirrored, single 1TB drive, boot-from-SAN)?
- What actions have you taken already (and be honest...running an AV scan and deleting files isn't "nothing")?
- What data have you already collected, if any?
- Do you have a logical network diagram, particularly of the affected systems?

Lots of questions...do you have the answers?

Innovation

I was reading a list entry recently, looking at some of the functionality that was being added to a popular forensic analysis application, and I got to thinking...what areas of what we do (incident response and computer forensic analysis) are in need of innovation? What could we do better, through process or whatever, to do what we do better...more efficiently, more accurately, more completely?

What about adding functionality to forensic analysis applications? In the instance I was looking at, the request that had been granted was to add parsing of ASCII-based logs to the application. Is this really necessary? Is this something that needs to be added to applications that are still unstable, crash without notice or without any sort of debugging information, and currently contain far too much "functionality" so as to require a certification just to use the application (forget doing actual forensic analysis).

I'm not picking on any one application either. There's another one that I like a lot, and updates have been delayed while functionality is being added to it...functionality that is available in other tools.

What I'd like to see is a core, stable application capable of opening image files, and allowing the analyst to quickly and accurately perform keyword and grep() searches, for file content, file names, etc. From that point on, major functionality (such as parsing PST files) could be easily added as plugins, allowing the core application to remain stable.

I'm also a firm believer that too much functionality in a forensic analysis application moves that analyst further and further away from understanding the data itself. As analysts are removed from the data, their understanding of what's expected and what's unusual or suspicious lessens. One person can't be expected to know everything, but that's why we have a "community", right? Having analysts that understand how various pieces of data interact to build a more complete picture is extremely important, particularly as the sophistication of cybercrime continues to rise.

What are some areas that you feel need a little innovation? How about just shook up enough to flake off the shell of "...but that's how we've always done it"?

IR Immediate Actions

As you may remember, in December 2007 I presented at the HTCIA conference in Hong Kong, the first HTCIA conference for this chapter. It was a great conference, and a great opportunity to meet a lot of folks in this field (see the Speaker's page).

While there, I presented on Registry Analysis and Windows Memory Analysis, both of which seemed to be well received. One question came up during discussions of Windows Memory Analysis...during the presentation, I talk about different tools and techniques that are available for extracting the contents of RAM, and also talk about the difference between tools I can use as a private individual and those I can use as a consultant. At one point, someone asked me which tools and techniques I use most often as a consultant...and my response was simply "none of them". The reason for this is that in most all of the responses I've dealt with, the system (or systems) have already been turned off, rebooted, or sufficiently imposed upon by others (AV scans, running multiple tools, etc.) that even if I did, as a consultant, have an option available for collecting the contents of RAM available to me, doing so would be of little benefit to either my analysis, or any information I could provide to the customer.

Part of the issue is that as a consultant, I am extremely limited in the tools I can use to collect the contents of physical memory from a system, not so much due to the availability of a particular tool, but more so due to the license agreement associated with that tool.

However, a bigger issue is that the immediate actions performed by most first responders on-site are to take systems offline, shutdown them down, or "clean" and reboot them prior to any calls for outside assistance being made. While this may be pertinent for business preservation and continuity, it most often has a detrimental impact on any follow-on analysis that may take place.

If there is data leakage due to an intrusion (or this is suspected, or this is just a question that needs to be answered...), then the immediate reaction is (apparently) to shut the system down. This may be pertinent, particularly if there is no incident response plan in place that lets people know what they need to do, and time is required to notify and get approval for follow-on activities (such as calling consultants). This reaction appears to be fairly ingrained, and I'm not suggesting that we change it by saying DO NOT shut systems down. What I am going to suggest is that we modify those immediate actions such that pertinent information is collected from systems before they are shut down.

Wait...what? It looks like what I am saying is, go ahead and shut the systems down, or reboot them, or "clean" them...and I am. This is going to happen anyway. There's nothing I can say or do, either on this blog or as a paid consultant, to change that. In order to change that behavior, there has to be an incident response plan (CSIRP) in place that tells people what to do, and when someone shuts a system off inappropriately or incorrectly, that issue needs to be addressed. The issue is that some kind of overwhelming stimulus needs to be put in place to change this behavior, and until this happens, nothing anyone can say or do is going to change that. So...it's gonna happen anyway, and instead of changing it, let's go ahead and go with it...but throw something else in there along the way.

What can we do? Well, one thing is to put tools on systems (for a list of tools, check out my book) when they are set up, or at least at some point prior to an incident occurring. If you can't put tools and a simple batch file on systems (for whatever reason...the most common of which seems to be, "...we don't know how..."), then issue each admin/responder a CD with tools, and a thumb drive. Then make it a policy within the organization that a batch file (could be a DOS batch file, or WMI script, etc.) be run and the output saved to a thumb drive, shared drive, etc., prior to the system being taken offline.

Thoughts on Live Acquisition

Recently, I've been doing some thinking about issues surrounding live acquisitions - specifically, acquiring an image from a system that is running, as opposed to either booting the system to an alternate OS, or removing the hard drive and hooking it up to a write-blocker.

There are several methods for performing a live acquisition, but most involve running an agent or application of some kind on the system itself. You can do this using ProDiscover (install or run the PDServer agent from a CD or thumb drive), FTK Imager (run from a CD or thumb drive, and write the image files to an external drive or an already-mapped share), or with good ol' dd and netcat running from a CD. Regardless of how you choose to do this, there is an additional process running on the system...so think Heisenberg's Uncertainty Principle, but in the digital realm.

So in acquiring an image from a live system, we need to introduce a process into a system of already running processes. While our intention is to take care and disturb the 'scene' as little as possible, it is simply a fact that we're going to leave some artifacts of our activities on the system...memory is consumed by our process, as are buffers, perhaps other processes have pages written out to the pagefile, etc. However, we address this issue with thorough documentation of our procedures and methodologies.

Now, in his book Computer Evidence: Collection & Preservation, Chris Brown describes the result of a live acquisition as a "smear", as from a temporal perspective, that's what we've got. Remember, it takes time to acquire an image from a hard drive, and if the system is still live and running when you acquire that image, then there is the distinct possibility that some sectors may change after you acquire them. Rather than having a sharp, distinct snapshot in time as when you acquire an image from a hard drive that has been removed from a system, you get a "smudge" or a "smear" instead. Also, as Greg Kelly pointed out in another forum recently, some of what we would normally consider stagnant data (Registry, files, etc.) can actually be considered volatile, and subject to change during the course of our acquisition...think log files, Event Log entries, the pagefile, etc.

Now, would it be possible to minimize this effect, by limiting what's running on the system during the acquisition? I believe that the answer to this is "yes". To do this, we'd need to take a couple of things into consideration. We'd should first ask ourselves, is this necessary? Hhhhmmm...if you're imaging the system over the network, and that system is still connected to the network, what is it doing on the network while you're acquiring your image? Is it serving up web pages, processing email, etc?

Before we continue, remember, we're talking about a live acquisition here, getting an image of the hard drive, NOT collecting the contents of memory.

Okay, that being said...if you're imaging over the network, is the system still providing services (shares via the Server service, etc.) that may have a significant effect on what you walk away with? We have to consider this in the face of what effect our actions will have on the system itself when we, say, disable a service. First, we have to see what processes are running, and to do that, we need to load some software and run another process (unless you're using the ProDiscover PDServer, as the agent provides this functionality, as well). Then we have to weigh the benefits of disabling the process or service against the "costs" of the effect that our actions have on the contents of the hard drive. Is that process really processing anything? We know that it may have file handles open, etc. But is there enough of an effect on the contents of the drive that it would make a significant difference within the time it takes to acquire the image? Also, if I disable that process/service, what happens? Any log files may be closed, perhaps the operating system itself will write an entry into the System or Application Event Log, etc.

Some of the processes or services I might consider shutting down include:

• AntiVirus products - these things reach out on their own and update themselves, or run scans automatically
  
• Task Scheduler - check to see if there are any jobs scheduled to run...this can get in the way of your acquisition (also, see the "Notes from the Underground..." sidebar on pp 215-216 of my book for an interesting tidbit on hiding scheduled tasks)
• Windows Firewall - depending upon how it's configured (we'd want to check, of course) there may be some significant issues. I included a sample pfirewall.log file on the DVD with my book...I'd turned up logging on the firewall and hit it with nmap. ;-)
• Exchange/IIS - if the system is still connected to the network do you want it processing email and web pages during the acquisition? Think the same thing for the FTP and SMTP services that sometimes get installed along with IIS.

This isn't meant to be a complete list, of course, but instead it's meant to start some thinking about what we have to consider before shutting a service down, or disabling a process.

So, before doing any of this, we need to put some thought into it. What, if anything, am I going to disable or shut down? Do I even need to? In some cases, live acquistions I've done have been of systems that had already been taken off of the network (not by me) and shut down, and then rebooted so they could be acquired (acquisition via write-blocker was not possible). With no network connections, I'm not overly concerned about major changes to the system during the acquisition process...the NTP service may log a complaint with the Event Log, but little else may happen. However, this isn't always the case...we're seeing a greater need for live response, both in acquiring the contents of memory, as well as performing live acquisitions of major systems that cannot be brought down or offline. If this is the case, document it. On your acquisition worksheet or in your case notes, clearly state "these services could not be halted or disabled due to...".

This is just a first blush...getting my thoughts down, thinking through things, determining if there is even a strong enough need for something like this...perhaps a matrix of services and processes, and when its a good idea to shut them down, how to do so, etc. Is this necessary?

Also, because I know the question is going to come up...addressing this same issue in the face of acquiring the contents of physical memory is an entirely separate post. Stay tuned!

Innovation

This post is likely to be the first of several, as it's something I've been thinking about for quite a while, and it takes new form and shape every time it pops into my head. So...please bear with me...

We see all the time that cybercrime is increasing in sophistication. We see this in reports and surveys, as well as in quotes. From this, we can assume (correctly) that there is a widening gap in abilities and resources between those committing the crimes, and those investigating the crimes. This gap is created when innovation occurs on one side of the equation, and not on the other.

I guess we need to start with the question, is there a need for innovation in the field of incident response (IR), and consequently computer forensic (CF) analysis?

I know that this is going to open up a whole can of worms, because not everyone who reads this is going to interpret it the same way. Even though I get this question running around inside my brain housing group from time to time, I don't think I really have a solid grasp of the concept yet. I see things and I think to myself, "Hey, we could really use some innovation here", or as in the case of Jesse Kornblum's ssdeep, "Hey, THAT'S an innovation!!"

I know what isn't an innovation, though...hash lists are not an innovation. Not any more. Sorry. 'nuff said.

Let's look at it this way...right now, Windows systems are being investigated all the time. I'm on several public and member-only forums, so I see the questions...some of the same ones appear all the time. There are just some things that folks don't know about yet, or don't have a clear understanding of, and simply don't have the time to research it themselves. From a more general perspective, there are areas of a Windows system that are not investigated on a wide basis, simply due to lack of understanding (of what data is available and how it could affect the investigation). I firmly believe that if there were more of an understanding and more knowledge of these areas, some investigations reap significant benefits.

So, is the innovation need in the area of knowledge, communication, or both?

Vista is bringing about innovations in technology. Un- or under-documented file formats require application-specific innovations (and these include Registry entries, not just binary format).

See what I mean? It's kind of hard to put your finger on, even though it's there...just outside your direct line of vision, like trying to see someone at a distance, at night. On the one hand, cybercrime has a motivation to innovate...money. Innovations are made out of necessity. But what about other cases or issues, such as missing childern? Business innovations in technology and applications (MySpace, Xanga, IM applications, etc.) just naturally require innovations in the areas of understanding, investigations, and subsequently communications. Outside innovations in storage media have led to different (albiet, not new) means of committing information theft and fraud.

Thoughts?