Thoughts on APT

There's been a great deal of discussion lately about the advanced persistent threat, or APT, and I've seen list posts from folks adding their thoughts, or asking others to weigh in and provide any insight they may have. I see this as healthy, not only for customers, but also for the forensics community as a whole.

There are some things that are being said, quite clearly and repeatedly about this threat. For example, take a look at Wendi's post on the Mandiant blog; she presents some statistics from the M-Trends report that can give you an idea of what to look for if you suspect you've been compromised. I also think that if you view it the right way, and perhaps have a bit of context from other sources, you'll see that this upholds the Least Frequency of Occurrence (LFO) principle that Pete Silberman has described. So what this means is that responders and analysts need to look for the anomalies; not the massive spikes in activity, but the small, infrequent things that we may not notice in all the noise on a system, or infrastructure. The Mandiant folks mention this, and so do the HBGary folks...so, whether you're using LFO or MRI (thanks again, Pete!), or you're looking at digital DNA, you're looking for what is or should be standing out as anomalous and infrequent.

Okay, so...what about APT?

As I see it, there are three major groups of actors here...the good, the bad, and the ugly victim. The victims are pretty clear. The bad guys are the developers, purveyors and operators of exploits and other mechanisms (i.e., code, malware, etc...call it what you will) for malicious purposes. The good guys are LE, responders, corporate consultants, etc...those folks trying to assist the victims, most often after a data breach.

Now, a number of the good guys have been (or started) posting reports (see the Reports section at the end of this post) illustrating statistics based on the incidents they've responded to and the work they've done. Reading through these, we see a lot of information much like what Wendi included in her post. Perhaps the most important thing, in my mind, is that the numbers and information from these reports indicate that there was a cultural shift in the bad guy's realm. What I mean by that is that back "in the day", most of what we saw was malware that ran amok on networks, and folks blowing out SubSeven or NetBus to systems so that they could open and close the CD-Rom tray. No more. Systems are being targeted for either the access they provide or the data that they store and process. Malware is being modified enough so that current AV products don't detect new variants, and footprints of that malware are minimized, using mutexes so that the system is only infected once. I attended a conference in Redmond back in November, 2009, and in several of the presentations, LE stated that the bad guys are dedicated, patient, smart, well-funded, and they have an economic goal behind what they're doing.

From my perspective as a responder and analyst, as well as from reading the reports and compiled statistics, what I'm not seeing is a corresponding paradigm shift on the part of the organizations that fall victim to these intrusions and compromises. Intrusions are still going undetected; victims are being notified by external third parties weeks or months after the fact. Systems are still being compromised via SQL injection and the use of poor passwords by administrators.

One thing that really stands out in my mind is that looking at my own experience, as well as the experience of others (via reports and postings on the web), the victims are not experiencing a cultural shift that corresponds to what the bad guys have gone through. Even in the face of information that indicates that the cost of data breaches has increased, organizations continue to be breached. In all fairness, breach attempts are going to happen; however, at least one report indicates that as many as 70% of data breach victims responded to find out well after the breach from an external third party.

The point is that the bad guys have identified targets and have an economic stimulus of some kind for attaining their goals. They're dedicated and compartmentalized...someone is dedicated to discovering vulnerabilities, and often it appears to be a different party all together that employs the exploit and some new piece of malware. For the victims, we're still seeing incident prevention, detection, and response all being secondary or tertiary duties for overworked IT staff...so while the bad guys can dedicate time and resources toward getting into an organization, IF there are dedicated responders within the organization, and IF they have any recent training or experience, and IF anyone actually knows where the data resides...well, you can see my point. From the perspective of a historical military analogy, this appears to be akin to special operations forces attacking villages defended by farmers and shopkeepers.

Maybe I'm way off base here, but this whole discussion of APT seems to be showing us something that's a bit more of an expansive issue. My thinking here is that if those organizations that are storing and processing "sensitive data" (choose your definition du jour for "sensitive data") were to have a corresponding cultural paradigm shift, we might begin to see intrusions detected and responded to in a manner that would provide data and intel to law enforcement, such that there could ultimately be arrests. I know, this is easier said than done...look at the issues that have sprung up around compliance; all compliance is...really...is an attempt to mandate or legislate minimum levels of security that organizations should have already had in place. I don't want to cloud the issue (no pun intended), but my overall point here is that maybe law enforcement would be able to make arrests if they had data and intel. As a responder, too often have I arrived on-site for an incident where the customer was informed of an issue by an outside third party; no one knows definitively where critical data resides, there are no logs available, and administrators have already done "nothing", which in reality amounts to an extensive list of removing systems from the network, scanning them with AV, deleting files, and even wiping entire systems.

So we know that the bad guys are having fairly high rates of success compromising systems and infrastructures using, in some cases, well-known vulnerabilities that simply hadn't been patched. We know that in many cases, they don't need to use special privilege escalation exploits, because they get in with Administrator/root/superuser privileges. We know that in most cases, they don't upload massive sets of tools, but instead use native utilities or only one or two malware files. We know that rootkits simply don't have to be used to hide the bad guy's presence...why hide from someone who's not looking for you?

So the take away, for me, from these reports is simply that there needs to be a cultural shift on the part of those who store and process sensitive data, and it has to come from the top down. It's 2010, folks...do we still need to sell infosec to senior management? What should be the CEO's concern...that his email and IM are up and running, or that the sensitive data that his company stores and processes is secure, and his infrastructure monitored?

Reports
7Safe (UK)
Verizon
Mandiant

Addendum: There's a bit of a different perspective on APT and what it really means over at TaoSecurity (here, and commentary on the M-Trends report here). For another view or perspective on the M-Trends report, see what IntelFusion says.

One thing to keep in mind about the reports...remember that they're based on numbers compiled by the perspective groups. Each group may have a different customer base and primary line of business when it comes to what they do. What this means is that each report is going to represent a slightly different culture when it comes not only to the numbers but also what they represent.

Links

Tools
David Kovar has written a tool, in Python, to parse the NTFS $MFT, called analyzeMFT. The tool can be downloaded from this site. I've been using Mark Menz's MFTRipper to parse this data, and having other tools to do this sort of thing available can only be a good thing.

MS article on NTFS $MFT
Lance's article on Detecting Timestamp Changing Utilities

Windows 7 XP Mode
One of the interesting aspects of Windows 7, from both a usability and a digital forensics point of view is the addition of XP Mode. In short, if you have a system whose processor supports hardware virtualization (be sure to check that out!!), you can install a Windows XP SP3 virtual machine into VPC on Windows 7, and run tools that may not run (or run quite as well) on Windows 7. This sort of thing could be very useful from an analyst's perspective...with just one platform, you can run tools that don't rely on the Windows API to parse some data sources, and at the same time, you can run other tools that do require the Windows API, and even a specific version.

So, while this can be very useful, there's the question of virtualization and how it affects what the analyst needs to look for when examining a system. Diane Barrett has discussed artifacts left when someone uses Moka5 or MojoPak in presentations, and we're all aware of other virtualization tools and platforms out there...but with XP Mode, it's built into the OS shell.

The key to all this, from a digital forensics perspective, is going to be in determining where the artifacts of interest exist.

XP Mode Resources
Tony Bradley's article
LifeHacker article

AV, Symantec and the Google Thang
Symantec posted something on the Trojan.Hydraq Incident, indicating that it is associated with the Google issue that popped up recently.

Something I find concerning about their write-up is the description of the artifacts. They mention that the Trojan is a DLL and installs as a Windows service with the name "RaS[4 random characters]". Well, that's easy enough to search for across the enterprise...look for any service name that starts with "RaS". The problem is, this isn't the whole story. If the executable file is a DLL, that would indicate that it installs "under" something else, like SvcHost. This would mean that there are other artifacts; specifically, if someone finds a service with the specified name, then they should look at the Parameters subkey for the ServiceDll value...what happens if the name of the file changes from what's listed in the write-up? How about checking the SvcHost key in the Software hive?

Symantec isn't the only one who doesn't provide a great deal of useful information to folks, either. The MMPC has a write-up on rootkits, and mentions Trojan:W32/AproposMedia...here's their write-up on that one. Googling, I find that EmsiSoft, makers of the a-squared AV product, have something a bit more substantial.

SafeBoot
Didier Stevens has posted about restoring SafeMode with a .reg file, adding a bit more to his info about a virus that deletes the SafeBoot key, tricks to restore SafeBoot, and protecting the SafeBoot key from being deleted. While not an end-all, be-all security approach, it is a good idea to take a look at this and consider making it part of your system setup. After all, where would you be if you didn't have access to a bit of safety net like SafeBoot?

Safe Mode Boot Options
Safe Mode Boot options for XP (here're the options for Windows 2000)

Interesting Request
I received an interesting request in my email this morning...someone wanted to use one of my Perl scripts in part of their courseware, and was asking if it was okay to do so. I appreciate when people do that, but I didn't recognize the script: sweep.pl. I followed the link provided in the email and downloaded the script...it's a port scanner/banner grabbing script I wrote in 1998! I wouldn't call my skillz 'l33t in any sense, even now...but back then, maybe imaginative. After all, I was doing stuff back then to see if I could, and to see if I really understood the mechanics of what was going on.

Interesting Analysis

I was analyzing a Windows 2003 R2 SP2 image recently, and I ran across some interesting things that I thought might be a good idea to share.

One of the things I was doing in my analysis was creating a timeline, and saw some Event Log event records that were interesting, so I created a mini-timeline using just the Event Logs. In this case, the system had 5 .evt files...the three usual ones, and Internet Explorer.evt and WindowsPowerShell.evt. I extracted all five files from the image, and began parsing them with my own tools. At first glance, I noticed that there were Security event records dating back to early October, 2009...logins, Event Logs being cleared, etc.

As my analysis progressed, I needed to drill down on something specific, so I created a micro-timeline comprised of just the Security Event Log records, and noticed that the earliest record was from the first couple of days of December, 2009. But wait...how did my other timeline have Security events going back two months before what was apparently in the Security Event Log?

So then I ran evtrpt.pl against the Security Event Log and confirmed the number of records in the log, as well as the date range. I then proceeded to do the same thing across the other Event Logs, as well, just to be sure...and suddenly, from the WindowsPowerShell.evt log, I found 185 Security records! Ah, that's something! So I opened the file in a hex editor and viewed the ELF_LOGFILE_HEADER structure; the StartOffset and EndOffset values were the same...0x0030...indicating that both values pointed to the end of ELF_LOGFILE_HEADER structure (which is 48 bytes long), and that there were NO records in the Event Log. In addition, the CurrentRecordNumber value as set to 1, and the OldestRecordNumber was 0...again, no records. I then opened the Event Log in Event Viewer on my analysis system and was again told, no soup for you! I then went back to the hex editor and after scrolling down past a bunch of zeros, I began to see valid event records, all with "Security" as the source!

So what happened?

Well, apparently when the Event Logs were cleared, and the Event Log files themselves reallocated, at least some of the sectors contained valid event records. Also, there apparently hadn't been any Windows PowerShell events generated, so no event records were written to the log, and the Windows API (via the Event Viewer) duly reported "no records". However, the tools I've written and use do not rely on the Windows API, and instead parse the EVT files on a binary level; they search for the "magic number" for an event record, back up a DWORD to get the size, read that much information into a buffer, and then compare the initial size value to the last DWORD in the buffer. This is just the first check, but allows the analyst to locate event record structures in unstructured data, such as within "unallocated space" within the EVT file. So, not only did the 64K file contain 185 valid event records, but I also found the "smoking gun" that turned supposition (up to that point) into cold, hard fact!

A brief description of the tools (mentioned previously in this blog): evtrpt.pl is a script I use to tell me the total number of event records within a EVT file, as well as the frequency of sources/IDs, and the date range of all records in the file. I use this to get a quick view of how useful the EVT files may be, or to look for specific sources...like McLogEvent, etc.

An example of the frequency of sources and IDs looks as follows (from test data):
Security 528,2 2
Security 528,5 25
Security 538,3 1
Security 540,3 3
Security 540,8 1
Security 551 2

As you might imagine, that can be pretty useful.

I also use evtparse.pl as part of my timeline tools; I can point it at a single file, or at a directory containing EVT files, and it will parse through them, listing the output in the five-field TLN format. From here, I can create a mini-timeline using just the EVT file output, or add the events to an overall event file that also contains file system metadata info (via TSK fls.exe), etc. I also have a switch that will list only the event record numbers, in sequential order, with their corresponding time_generated times...I use this primarily to see if the system clock has been changed.

What this demonstrates is that there can be more sources of data on a Windows system than we're really aware of initially. Because of how the scripts I use were written (i.e., not using the Windows API), with minor modifications, they can also be used to find and extract records from the pagefile, memory, and unallocated space. Evtrpt.pl and an earlier version of evtparse.pl are included in the timeline_tools archive in the Win4n6 Yahoo group, and were mentioned in my second timeline analysis article in the Hakin9 magazine.

As a side note, I also found some pretty cool artifacts in unallocated space within one of the Registry hive files...in this case, the SAM file contained indications of deleted user accounts (and when they had been deleted), which corresponded to other findings in the analysis.

Analysis Stuff

Metadata
Didier has posted new versions of his PDFiD and pdf-parser tools. Didier's offerings really kind of run the gamut, don't they? Well, hey...it's all good stuff! I mean, really, other than the fact that he's updated these really great tools, what else needs to be said?

Malware
The MMPC posted some new malware descriptions recently, regarding Hamweq and Rimecud. Nice names. Signatures have been added to MRT, apparently.

An interesting aspect of Hamweq is that it apparently drops files in the Recycle Bin, and uses the Installed Components key in the Registry as a persistence mechanism. I wrote up a quick and dirty plugin for the key...entries with StubPath values that point to the Recycle Bin would be suspicious, even if they didn't indicate Hamweq was installed, specifically.

Other malware has other artifacts and persistence mechanisms. Take Win32.Nekat as an example...this one adds entries to the user's Control Panel\don't load key, effectively removing the applets from view. While not overly sophisticated (I mean, it is native functionality...), something like this would be enough to slow down most users and many admins. And yes, there is an app a plugin for that (actually, it was pretty trivial to write...one of several that I wrote yesterday).

APT
With the Google thing, there's been considerable discussion of advanced persistent threat, or APT, lately. I'm not going to jump on that bandwagon, as there are lot of folks smarter than me talking about it, and that's a good thing. Even Hogfly has talked about APT.

I get the "threat" thing, but what I'm not seeing discussed is the "advanced" part. Wendi over at Mandiant blogged about M-Trends and APT, noting some...uhm...trends such as outbound connections and mechanisms used to avoid anomaly detection. One of those mechanisms listed is "service persistence", which sounds like the malware is installed as a Windows service, a persistence mechanism. While I do think that it's a good idea to talk about this kind of thing, what I'm not seeing a lot of right now is actionable intel. Wendi and Hogfly presented some very useful information, demonstrating that all of this talk still comes down to a couple of basic questions; Have I been breached, and am I infected? How do I find out? What do I do if I am? How do I protect myself? So someone looks at both posts and uses the information there to look and see if they've been breached. If they don't find anything, does that mean they're clean? No, not at all...what it means is that you didn't find what you searched for, and that's it. Both posts presented information that someone can use to scour systems, but is that all that's really available?

I think that a very important concept to keep in mind when doing this kind of analysis is what Pete Silberman said about malware; he was absolutely correct when he described it as having the least frequency of occurrence on a system. Think about it. Malware, particularly worms, no longer want to keep infecting systems over and over again, so they use a single, unique mutex to say, "hey, I'm infectin' here!". That way, the system doesn't get so massively infected that it stops functioning; not only does that alert folks that somethings wrong, but it also deprives the attacker of the use of the system. So, you can run handle.exe from MS on a live system, and then run the output through handle.pl and see mutants listed by least frequency of occurrence.

I'm going to throw this out there...run it up the flagpole and see who salutes, as it were...but I think that the same sort of thing applies to intrusions and breaches, as well. For example, Windows systems have thousands of files, and intruders may be installing some tools to assist them in persistence and propagation, but the fact is that there are a number of native tools that are perfect for what most folks want to do. I mean, why install a tool to locate systems on the network when you can use native tools (ipconfig, netstat, nbtstat, etc.)? So intruders don't have to install or add something to the compromised system, unless the required functionality is not available in a native utility. Remember what Wendi said in her M-Trends post about using services for persistence? How many do you think are there? Do intruders install their persistence mechanisms 50 or 100 times? No...likely, they only do it once. And they can either hide as svchost.exe, pointed at an executable in another location, or beneath the legit svchost.exe, using the ServiceDll entry (and adding an entry to the appropriate SvcHost key value in the Software hive).

Timeline Analysis
To illustrate my point about least frequency of occurrence, let's talk briefly about timeline analysis. Given the minimalist nature of malware and intrusions, how can we use timeline analysis to our advantage? The approach I've been advocating allows the analyst to see multiple events, from many sources, side-by-side for comparison and analysis.

One of the things that folks ask about with respect to timeline analysis is a graphical means for representing all of the data, in order to assist the analyst. IMHO, this simply does NOT work! I have yet to find a viable means of taking in all of the data from various sources, throwing it into a timeline and hoping to use a graphical representation to pick out those anomalies which happen least often. As an example, check out Geoff Black's CEIC 2007 Timeline Analysis presentation...the fourth slide in the pack illustrates a graphical file system metadata timeline within EnCase. I agree with Geoff's assessment...it's not particularly effective.

Overall, in order to get a grip on APT, we as responders and analysts need to change our mindset. We need to understand that we can't keep looking for spikes in behavior or activity, and there is no Find All Evidence button. When you respond to or analyze a single Windows system, and consider the changes in OS, various applications that can be installed, and the infrastructure itself, what constitutes an anomaly? Honestly, I don't think that this is something a software package can tell you.

I do, however, firmly believe that training and education are the key.

Links

e-Evidence
Christina's updated the e-Evidence What's New site...check it out! There's always some good reading here...

Tools
Speaking of tools, Don's been busy...no, Don's not a tool, but he's posted a tool update over on the SecRipcord blog. In addition to the tool update, Don has a little tip he got from some guys from Mandiant on detecting if malware embedded in or infecting CHM (Windows compiled Help) files might have been run on the system.

Don also links over to IronGeek's forensically interesting spots post. It's a very interesting post, and I noticed that a lot of the Registry locations that IronGeek mentions are covered by RegRipper plugins. Also, something else to think about...IronGeek has done a great job of presenting some interesting places to look for forensic data, but I noticed that under both IE and Firefox, there's no mention of Favorites or Bookmarks, respectively.

I digress for a moment, but I really think that examining some of the areas described in my earlier post can add a great deal of context to the information retrieved via more well-known analysis. For example, one of the first items on many checklists with respect to browser analysis is to check the contents of the TypedURLs key. That's fine, but what if the user's default browser isn't IE? That's right...no entries in this key would be explained by that fact. Now, what if there are some entries in the key, and the default browser is IE? Checking the Favorites list would provide some context to what was being seen in the browser history.

Where this can also be helpful is if some sort of anti-forensic tools or techniques have been employed on the system. I actually received a question along these lines just the other day...someone interested in getting into the field asked me how analysts "deal with" anti-forensic tools, and specifically mentioned timestomp. Well, there are a number of locations on a system that contain timestamps that are not affected by timestomp or other anti-forensic tools/techniques.

It seems that I'm not the only one thinking along these lines...Kristinn has updates to log2timeline that include adding this sort of information to a timeline. Thanks to gl33da for sending me the link to Kristinn's page.

Analysis
Speaking of tools, Daniel Wesemann has a SANS ISC post on the static analysis of malicious PDFs, which is pretty easy to follow along with...and starts out using Didier's PDF tools. Very cool!

Circling back around to browser analysis yet again, I picked this one up from the sausage factory...consider the browser's session restore capabilities, something that's been around for a bit. I use Firefox (I've been using Mozilla since 0.9, from as far back as...'94?) and version 3.5 would crash often. When it did, I'd have to decide to restore sessions are just start new ones. Harry Parsonage posted a PDF on the topic...take a look.

Certs
Daniel Castro had a very interesting article in FCW regarding his thoughts on a national certification program. I think that Daniel made some very important points in his article, one being that an increase in certified individuals hasn't resulted in a corresponding decrease in vulnerabilities, incidents/breaches, and losses. I also agree with Daniel in that a national/federal certification program will result in a check-in-the-box mentality...there are plenty of examples out there to illustrate this occurring.

There are a number of excellent certification and training programs out there, and I really don't think that the issue is the programs themselves; no, it's the individual who gets the certification and what they do with it. Let's say you go away to a training program, work hard, and get the certification, and when you return back to your job, you don't use any of the newly-developed skills for 6 or more months...then what? Do you go for a skills review after an incident occurs? If your company is going to send you off to this training, shouldn't there be a purpose for it...like, this person is going to be an integral part of our security plan or CIRT? Wouldn't it then be a good idea to have a plan that starts with getting someone trained, and once they return, get them involved in the program and using those newly-learned skills?

I've seen certified responders walk up to a compromised system, open the Event Viewer, start opening event records, and use Paint to save bitmaps of the entire desktop to the system hard drive as a means of documenting their findings. Seriously. Is that the fault of the certifying organization, or whomever wrote the materials? No, of course not. The material was provided to an individual and they were certified, but what happens after that is in part up to the individual, and in part the responsibility of their management.

Rootkits
Microsoft's Malware Protection Center recently posted some observations on rootkits for 2009...pretty interesting read. However, I think that like a lot of AV service write-ups, something like this falls short of what it could be. Many times, AV service write-ups are geared toward "use our product" type of information, and sometimes not enough to be useful in an environment that may already use that product.

Working as an incident responder over the past several years, a couple of the things I've seen are that (a) a malware infection can devastate an infrastructure, (b) AV products don't protect against everything, and (c) AV services are geared toward using the product.

As an example, just about a year ago, Conficker (and later in the spring, Virut) variants would bring an infrastructure to its knees...literally. Our team would show up, and often find someone from the AV vendor already on-site or engaged. Our observations of the malware characteristics would lead us to a diagnosis, so that we could assist in identifying infected systems, which would be pulled offline and scanned with the updated virus files provided by the vendor. Often things didn't really get rolling until we stepped in because the AV vendor required a sample to analyze, and the customer was drowning.

After the incident settled down, we would have discussions with the customer to help them understand that this was a variant of previously detected malware family, and that the variant itself was sufficiently different enough to escape detection by their current and up-to-date AV product.

Anyway...enough of the soapbox, I guess...it's just too early in 2010 for that!

Browser Stuff

Many times during an examination, we'll take a look at the user's browser activity. That might include starting by getting the contents of the TypedURLs Registry key (I see that as a first step in a lot of analysis plans) and parsing an index.dat file or two; however, one thing that I rarely see in an analysis process, in case notes, or in a report is, the first thing I did was determine the default browser, as well as any other browsers that may have been installed and used.

When exploring user activity via the browser, the analyst needs to be sure to:

1. Check to see what the default browser is...yes, there's an app a RegRipper plugin for that. But keep in mind, this value can change. If the user doesn't uncheck one box in the default browser dialog that appears, they will be asked every time if they want to make the browser their default, so the question will be asked each time and they can go back and forth.

2. Determine what other browsers may be installed; don't assume that someone's only going to use Firefox. Strike that...don't assume anything. Find out. Any easy way to do this is to check the file associations for .htm(l) and .asp(x) files, as well as just see what software is installed.

3. Other places to check (these are included in RegRipper plugins, by the way...) include the user UserAssist keys (what've they been launching), the Uninstall key (for software installations), and which MSIs have been run on the system.

Many of these are quick checks...several folks who've used RegRipper have said that the tool has reduced Registry analysis from days to minutes, and thanks to the plugins, been more comprehensive than previous processes. So adding these checks to your analysis plan doesn't correlate to a significant increase in the time it takes to conduct your analysis.

One area that I rarely see discussed in browser analysis is the bookmarks file. For Firefox, the bookmarks.html file include ADD_DATE and LAST_MODIFIED entries for folders, and ADD_DATE and LAST_VISIT entries for the URLs. For IE, you'd look for the Favorites folder, which contains InternetShortcut/.url files which also include timestamps, in addition to the file MAC times themselves.

It seems to me that including this information in a timeline...should the investigation necessitate doing so...might be a source of some valuable data. For example, let's say you found an entry of interest in the user's Internet history; would it add some additional (and perhaps significant) context to the overall investigation to know that the web site was in the user's bookmarks/Favorites?

Commercial tools like ProDiscover make it very easy to populate the IE Internet History view from an image rather than just a single user. But keep in mind that it isn't IE that populates the Internet history artifacts for the user; it's the use of the WinInet APIs. What that means is that any application or tool that uses the WinInet APIs may leave similar artifacts, which is why, during some engagements, some of us have seen an Internet history for the Default User populated. In one particular instance, wget.exe was found to have been launched using System-level privileges, and the tool was found to use the WinInet APIs so we found clear artifacts of the use in the Default User's Internet history. In that particular case, the intruder used SQL injection to gain access to the MS SQL Server, and ran commands to create an FTP script file, which was then launched via ftp.exe. The script downloaded wget.exe, which the intruder verified was on the system, and then used to download additional software.

Another aspect of browser analysis (specifically for IE) is to look for Browser Helper Objects, or "BHOs". From a forensic analysis perspective, some BHOs have been known to be spyware, or worse; Symantec identified BHOs as a common loading point for malware.

This article discusses how to prevent BHOs from loading with the Explorer process, and only loading with IE.

On Firefox, Add-ons may be of interest. Here's a Symantec article that talks about BHOs (IE) and XPCOM (Mozilla).

My point is that sometimes just looking at the user's Internet browsing history may not be enough to really get a solid picture of what's going on. The existence of a particular web site that has been bookmarked or added to the user's Favorites may add valuable context to the examination. BHOs are loaded when the user starts IE, so any action taken by the BHO will be done in the user's context, and therefore will populate the user's Internet history.

So how might you use this in a real-world investigation? Well, if the user has their browser configured to delete the history when the browser is closed, or uses another tool to do so, you may find something of value in the bookmarks. Even if the history hasn't been deleted, you will be able to associate some artifacts with specific user activity.

What about the Trojan Defense? Well, with a comprehensive and thorough malware detection process, you might also include a specific check for BHOs or addons to the browser, further closing the door on that issue.

Resources
Firefox 3 Forensics
FoxAnalysis
Firefox Forensics (Machor Software - also Windows and Google Chrome Forensics)
NirSoft Browser Tools
WBF Tool

Addendum, 8 Jan
Opera Files - global history is kept in global.dat, entries have a format that looks similar to IE Favorites .url files:

Webpage Title - Something
http(s)://www.somewebsite.com/page
(possible *nix epoch timestamp)

Linkilicious in 2010

RegRipper
Paul Stutz sent me a nice email recently, telling me that he'd not only used RegRipper's rip.pl on Linux, but he'd used it to work on the NIST CFReDS hacking case. Paul posted his write-up here, and gave me the go-ahead to post the link. Check it out! What Paul does is use rip.pl and some scripting on Linux to create something of a CLI version of RegRipper, automating a good deal of what RegRipper is capable of in order to solve the challenge. This is a really good view into what someone can do with RegRipper and the tools.

SafeBoot
Didier has written a program that creates an undeletable SafeBoot key. His point is dead on...there is malware that deletes the SafeBoot key so that the user or admin cannot boot into SafeMode and recover the system. This may not be such an issue on XP (thanks to Restore Points) but it can be a pain if your IR plan includes such steps.

I have to tell you, Didier really comes out with some pretty amazing things...check out his article in the first edition of the IntoTheBoxes newsletter, on Windows 7 UserAssist keys. Also, did you know that Didier's pdfid.py tool is used by VirusTotal?

Processes
Claus also has a very good post on tools you can use to view and manage processes on Windows. A number of the tools are GUI, but some are CLI tools, and all of them could be useful during incident response. Take a look at what Claus has compiled...there are some very good options available.

Windows Firewall and Open Ports
Speaking of Claus...he posted recently about opening ports in the Windows Firewall, and that got me to thinking about the times I've examined a Windows system following an incident (intrusion??) and found that there were custom entries in the Registry that allowed traffic to or from the system.

Why CIRTs Should Fail
David Bianco's blog post on why your CIRT should fail has some excellent points, and the post is well worth the read. Take a look and let me know what you think about what he says.

In short, if a CIRT reaches what appears to be "success", there's no need for "lessons learned", right? Wrong. Ninety-nine times out of a hundred, when reviewing an incident response performed by a customer (or another analyst), there isn't a logical progression of steps, from A to B to C. Most often, it's step A, make one or two findings, and then speculate your way to Z, at which point, the book is closed on the incident. Seriously. I've been in a customer's war room when someone who hasn't been involved in the incident up to that point walks in (while the incident manager is out of the room) and announces to those in the room that the malware includes a keystroke logger. When asked how he knows that, the response is usually, "that's what hackers do...". Hey, how about we do something different...like maybe collect and analyze a sample of the malware, as well as an infected system, and see if it actually includes keystroke logging capabilities? How about if we work on fact, rather than speculation?

This is the sort of thing that can come out of lessons learned? What did we do well, and what could have gone better? Did with go with speculation because it was quicker? Could we have performed the analysis better, or could we have been better prepared?

Finally, there's a lot of talk about "best practices", and going through a lessons learned is one of those best practices for CIRTs and should be part of the CSIRP. If you're going to talk about best practices, why not follow them?

DLP
Richard Bejtlich has an excellent post that addresses thoughts on some excerpts from Randy George's Dark Side of DLP. I have to say that having been involved in a good number of response engagements that involved the (potential or real) exfiltration of sensitive (pick your appropriate definition of "sensitive", be it PCI, HIPAA, NCUA, state notification laws, etc.) data, a lot of this post rings true. First, why bother scanning for where sensitive data is stored if you don't have a policy that states what's appropriate? Really. Okay, so some IT guy runs a scan and finds sensitive data stored on a system, used by a guy in marketing. The IT guy says, "hey, you can't do that.", and the marketing guy's response is, "says who?"

I also think that Mr. George has a good point with the statement, Managing DLP on a large scale can drag your staff under like a concrete block tied to their ankles. However, I would suggest that "DLP" can be replaced with anything else that extends outside of IT, and doesn't have a policy to support it. Like incident response.

I have seen first-hand how data discovery tools have been used to great effect by organizations, just as I have seen how they've been misused. In one instance, the organization used a DLP tool to locate a very limited amount of PCI data on their network, and when they were subject to an intrusion, were able to use that information (along with other information and analysis) to demonstrate that it was extremely unlikely that the PCI data was discovered by the intruder or exposed. Thanks to picture we were able to paint for Visa, the customer only received a small fine for having the data in the first place...rather than a huge fine for the data being exposed, then having to deal with notification costs, as well as all of the costs associated with that sort of activity.

One of the biggest things I've seen when responding to incidents is that when trying to prioritize response, no one seems to know where the sensitive data is stored or processed. Knowing what data you have and use, and having the necessary policies in place to describe its storage, processing, authorized access, etc., can go a long way toward preventing exposure during an incident, as well as helping you address your response when (not if) an incident occurs.

First post of 2010

For my first post of 2010, I'm pleased to announce that the first edition of the IntoTheBoxes newsletter is out! Check it out!

I have to say that Don did a great job of putting this together! I'm not a layout editor by any stretch of the imagination, and Don (of SecRipcord and Scout Sniper fame) has done a great job of putting this all together.

I also want to thank everyone who's contributed to this issue, as well as those who's contributions will appear in subsequent issues, for their time and effort in providing an article. Our hope is that we can provide quality, hands-on, practical information that has some use or interest to as many in the field as possible.

As a close-out to 2009, Richard Bejtlich ranked WFA 2/e as one of the three best books he read in 2009! Thanks, Richard...that's high praise, indeed! That sort of establishes a tradition, as WFA 1/e was ranked #3 in 2007!

I wish everyone the best in 2010!

Lions, and tigers, and DECAF...oh, my!

Most of us are aware that DECAF is back, with a new version. Ovie had interviewed Mike and posted the podcast (I'd also posted, and others had commented), and the next day, the original DECAF was taken down. Now it's back, like Die Hard sequels, or a bad MRE...but I kid. See? Smiley face. ;-)

So, how do DECAF and tools like it affect the current state of incident response (IR) and digital forensics (DF) analysis, if at all?

In order to discuss this, let's go back to COFEE. MS released COFEE as an LE-only tool, and IMHO, that's the reason for the hoopla surrounding its release and subsequent "leak"...that it was LE-only. The fact of the matter is that while COFEE was released by MS, and includes the use of several native Windows apps and some from MS/SysInternals, it's really just a glorified batch file with some add-ons to make it's use and deployment easier...which, I think, is the key to understanding the nature of COFEE.

While COFEE was released by MS, it wasn't developed by MS in the way that Windows 7 or MSWord were...instead, it was a small group within MS that led the charge on developing and releasing COFEE to law enforcement.

Now, please understand...I am not disparaging the efforts of this group at all. In fact, I applaud their efforts in stepping forward and taking the first step to do something. I mean, there have been plenty of similar frameworks out there for folks to choose from, and some of them very good...but for some reason, COFEE was very well received. However, I will suggest that perhaps making something too easy to use isn't always the best solution. Sometimes, when it really comes down to it, it may be better...albeit not necessarily easier... to educate the user than it is to make a tool easier to use and deploy.

Given that, my own personal, overall assessment of COFEE is that it's a tool produced by folks who don't do a great deal of IR or "live response" work, for folks who don't do a great deal of IR or "live response" work. Don't make this statement out to be something that it isn't...I'm not disparaging anyone when I say this. All I'm saying is that in the corporate arena, many of us have taken the time and effort to educate ourselves on the issues and nuances of live response, whereas LE may not have had that sort of time...after all, when I'm reading or testing stuff, I'm sure most LE are diligently working to put bad guys in jail.

Then there was DECAF. I remember that there was discussion about "LE-only" this and "vulnerabilities" that. However, listening to the CyberSpeak podcast interview, there's no specific mention of what those "vulnerabilities" are. The "vulnerability" that Mike and friends found in COFEE was never specifically stated during the interview, pretty much leaving that particular issue up to speculation on the part of the community as a whole.

So where does that leave us? I'd suggest that the release of DECAFv2 really doesn't change anything at all. Here's why...

First, I don't necessarily see a vast deployment of this sort of tool. Think about it...how much encryption is being used for nefarious purposes? From what I've seen over the past 10 or so years, as well as from talking to others, the use of steganography outside of the academic or lab environment seems to be another one of those "urban legends" of digital forensics. Even when looking at some pretty sophisticated or large-scale intrusions, I (and others) haven't seen a great deal of what's generally referred to "anti-forensics". I hate to say it, but it really isn't necessary.

Second, I don't see this being deployed on servers to any great extent. I mean, really...think about it. Your company gets compromised and a great deal of money is spent to get a consulting team on-site to assist. Do you want to be the one to explain to the CEO why you installed DECAF on a system, when the efforts of the responders were hampered, or worse, the system BSoD'd?

Third, DECAF is signature-based, and does ship with some default signatures (although I have no idea what "FTK Imager Port" is from the Press Release page...I don't use any tool named that, so I guess I'm safe). Beyond that, how many users are going to go about crafting and distributing custom signatures? I mean...really. Better yet, who's going to write the DECAF signature for DECAF?

Fourth, let's say that you do encounter a system with DECAF installed...it's likely going to be an uber-admin's desktop or laptop...and there are other ways to collect data. Like pull the hard drive out and image it. Sure, you may not be able to get volatile data, but you may not need it for what you're doing, or you may have other data available to you.

Finally, I have no doubt that someone's going to come up with a DECAF detection tool. There are tools available that will detect the presence of whole disk or volume encryption on live systems, and detecting the presence of DECAF running...and doing so using a tool with random signatures (remember RootkitRevealer??) makes use of capabilities that have already been employed. In a way, this reminds me of the whole escalation stuff that happens with combat weapons. Tanks have armor, so someone makes a TOW missile. Then someone puts reactive armor on the tank. Then someone else puts a probe on the end of the TOW missile. And on. And on. DECAF comes out...someone else will ultimately produce a DECAF detection tool.

Circling back around to the beginning of the post, what we, and in particular LE, don't necessarily need is easier to use tools. Sure, there's a certain level of usability in GUI-based tools over CLI-based tools, but what's needed is education, first and foremost. After all, a knowledgeable responder is better prepared. Knowing is half the battle! or something like that...

Investigating Breaches, pt. II

As a follow-on (or one of them, anyway) to my previous post on investigating breaches, I wanted to perhaps scratch the surface a bit more (as opposed to "digging deeper") regarding the subject.

Take for instance a Windows XP laptop, which is a pretty typical corporate configuration. Let's say that some suspicious activity was seen originating from this system by network admins, when the system was connected to the corporate network. Management feels that the employee is loyal and not the issue, and they suspect that the system may have been compromised with it was connected to another network, such as the employee's home or a hotel network. As such, management would like you to examine the system and determine if it had been compromised, and if so, how and when.

Pretty easy, right? Maybe, maybe not. Like I said, often it's relatively easy to find secondary indicators of a compromise...tools that the intruder may have loaded onto the system following the initial compromise. However, it may not be so easy to find the primary indicators of the initial intrusion.

So, when starting such an investigation, what would you want to look for? Well, I'd start by getting an idea of what services and applications are installed on the system. For example, was there a web server (hey, I've seen it!) installed? FTP server? Anything used for remote access? PCAnywhere? VNC? What about applications...any P2P? Also, be sure to check the Windows firewall configuration...I've seen indicators of malware there.

Another aspect to look at is, what is the user's default browser (yes, there is RegRipper plugin for that!)? IE? Well, what files are in their cache (browser drive-bys or "browse-bys" are a big issue)? Any documents? PDF files? JavaScript? Anything that can download or carry executable code? Email? What about Outlook, and any files downloaded as Outlook attachments?

While we're on the subject of IE, index.dat files, and browsing history, don't forget about checking the Default User profile for indications of web browsing history...another quick check but hey, I've found some pretty amazing things here.

An alternative means of analysis is to mount the image and scan it for malware, being sure to see what AV was installed on the system (if any...be sure to check for MRT) and then use something else (PCTools, AVG, Avira...be sure to check licenses), looking for malware, keyloggers, etc. Other quick checks for a better view include checking the hosts file for modifications, the firewall configuration (mentioned above), running tools like wfpcheck, etc. All of this can be done rather quickly, and provide a much more comprehensive analysis than just running AV. From this, you might hope to find some secondary indicators that might at least provide a point-in-time reference, the reasoning being that a secondary indicator would occur after a primary one.

Other areas of analysis include Event Logs, Registry autostart analysis (more of a follow-on to looking into running services), and even analyzing any available crash dump logs for indications of unusual processes.

What I'd really like to do is find a forensic challenge available online that consists of analyzing a compromised Windows system, and provide a walk-through of the methods and procedures used in the analysis. While there are a number of images online for download as part of challenges, the ones I have found involve malicious activity on the system, conducted by the user...I have yet to find one that involves a system being compromised or hacked remotely.

Investigating Breaches

I recently received an email from someone who said that he wanted to learn more about "network intrusion investigations". Seeing this, I asked him to clarify, and he said that he was interested in learning what to look for when someone breaks into a system from the network.

This got me to thinking...how would one go about teaching this subject? What's a better way to see if you really understand something than to sit down and try to communicate a how-to to another person? So I started thinking about this, and it brought up another conversation I had with some other folks...actually, a series of conversations I had over about 18 months. Specifically, I had conversations about intrusion investigations with some guys who went about discovering vulnerabilities and writing exploits. My thinking was that in order for these guys to be able to report a successful exploit using a vulnerability they found, they would have to have a test system or application, and a condition to define success. I won't got into the details of the exchange...what matters here is that at one point, one of them said, "you aren't looking for artifacts of the initial intrusion...you're looking for artifacts of what the bad guy does after the exploit succeeds."

Well, I have to say, I disagreed with that at that time, but for the purposes of investigating data breaches, one of the primary things you need to determine is, was the system in question, in fact, breached? One way to answer that question is to look for indications of "suspicious" activity on the system. Was malware or some means of access or persistence installed? AV scans are a good place to start, but I'd suggest that analyzing the acquired image for indications of AV tools already installed should be the first step. Why is that? How many analysts mount an acquired image and scan it with an updated commercial AV scanning tool? Okay, you can put your hands down. Now, how many check the system or the image first to see that the tool they use wasn't the one installed on the system? See my point? If the AV scanner missed something once, what's to say it won't miss it again?

Anyway, I've already talked about malware detection, so I won't belabor the point here.

Peter Silberman of Mandiant once mentioned that malware is often "the least frequency of occurrence" on a system, and in many instance, the same may apply to an intrusion. As such, the analyst will not be looking for sudden increases in activity on a system but instead looking for very specific data points within a discrete time window. I've created several timelines in which the predominance of file system activity was the result of system or application updates, as well as AV scans run by the administrators. Often, the necessary data points or time window may be established via other means of analysis.

Overall, I don't believe that you can teach one specific means for investigating breaches, and consider that sufficient. Rather, what must be done instead is to develop an overall understanding of data sources, and what a breach "looks like", and then conduct specific analysis from there.

For example, by what means could a system be breached? Well, let's narrow the field a bit and say that we're looking at a Windows XP laptop...what possibilities are there? There are those services that may be running and listening for connections (check the firewall configuration as it may not be the default), or there may be indications of a breach as a result of user activity, such as downloads via the browser (intentional or otherwise), email, P2P applications, etc. What we may end up doing is examining the system for secondary indications of a breach (i.e., creation of persistence mechanisms, such as user accounts, etc.), and working from there to, at the very least, establish a timeline or at least an initial reference point.

Another point to remember about investigations in general is that the more data points that you have to support your findings, the better. This not only helps you build a more solid picture (and eliminate speculation) of what happened on the system (and when) but it also allows you to build that picture when some data points do not exist.

Let's look at an example...let's say that you suspect that the system you're examining may have been accessed via Terminal Services, using compromised credentials. You examine the Windows Services found in the Registry and determine that Terminal Services was set to start when the system started, and other data points to the fact that remote connections were allowed. So your next step might be to look at Security Event Log entries that would show signs of logins...but you find through Registry and Event Log analysis that auditing of login events wasn't enabled. What then? Well, you might think to look for the time that specific files had last been accessed...but if the system you're examining is a Vista system, well, by default, the updating of last access times is disabled. So what do you do?

Again...the more data points you have to support a finding, the better off you'll be in your analysis. Where one data point is good, six may be better. A knowledgeable analyst will know that while some modicum of work will be needed to establish those other five data points, much of it can be automated and made repeatable, increasing efficiency and reducing analysis time.

Perhaps a way to illustrate the overall theme of this post is to look at examples, and that's what we'll be doing in the future. In the meantime, questions and comments are always welcome.

Links and Stuff

Practicals
I love stuff that is practical...stuff you can look at, follow along on, and by the end of the hour/day, have a new skill. I received an email this morning from my Twitter account telling me that Jaime Blasco was following me...I checked out his profile and found a link to something he'd written up on analyzing malicious PDF documents, using pdfid.py from Didier Stevens. This is really good stuff, as it can help an analyst narrow down what is perhaps the most difficult question to answer...how was a system infected or compromised? Many times, it's relatively simple to tell that a system had been infected or compromised, but how (and when) that happened can often remain a mystery. In some cases, know what had infected the system can lead the analyst in investigative directions, but most often it seems to lead to speculation. Practicals like what Jaime presented demonstrate how an analyst can narrow down the infection/compromise vector, particularly since we're seeing a great deal of malware that gets on a system as a secondary or tertiary download.

Training
Speaking of practicals and training, Julia King has some great tips on budget cuts and training in a recent ComputerWorld article. Over the years, I've seen folks invest in training that really wasn't useful...immediately or otherwise...to them. This is not to say that the instruction was bad...not at all, there are some excellent training providers out there. However, I've seen people go to courses that teach them how to perform digital forensics using Linux platforms, on Linux systems...and their infrastructure is all Windows. This is simply a matter of bad corporate planning and decision making, and perhaps a lack of availability of the appropriate training when training budgets are available. Training should be specific to meet your needs, and the skills should be used immediately upon the team member's return.

Addendum: For the record, the above statement (i.e., "I've seen people go to courses that teach them how to perform digital forensics using Linux platforms, on Linux systems...") is simply an example that harkens back to an HTCIA presentation I was giving, and in the next classroom was a course similar to the one described. The "people" referred to, in this case, were attendees of the course who commented that the training was very good, but they mostly acquired and analyzed Windows systems. Nothing disparaging was said about anyone, nor about any training provider. Thanks.

IR Planning
With respect to incident response planning, here's a great quote from Julia King's article mentioned above (the quote is from a CISO):
Someday, something bad is going to happen to your company. A laptop may get stolen or data gets stolen or a virus gets inserted into your network. Before you ever get to that point...

See what he says? He's aware that it's not a matter of if an incident occurs, but when. Further the quoted CISO advocates taking steps to be prepared for when an incident does occur.

Reporting
Keith Ferrell over at DarkReading has picked up on the RAM scraper thing previously mentioned in The Register by Dan Goodin. Keith's article has a list of eight things to do to improve the situation specific to RAM scrapers, as well as concerning malware, in general. The biggest thing that's missing, however, is a change in corporate culture. Perhaps the reason why security appears so lax on these systems is because there's no real corporate direction regarding information security.

I mean, think about it. Have you ever analyzed an image acquired from a server...one of the ones, say, in a rack in the data center...and found web browser history and maybe even email files?

Tools
Speaking of servers in data centers, have you seen F-Response TACTICAL? Be sure to check out the Forensic4Cast review of TACTICAL.

Geoff Chappell has a really good site with a lot of great information available. He's done a great deal of detailed research, and has provided some of that free of charge. When I first ran across his site, I found an excellent explanation of the bootstat.dat file found on Vista and above systems, and recently completed a parser for the file. Here's an example of the output:

C:\Perl\forensics\timeline>bsparse.pl -f d:\cases\vista\bootstat.dat
Timestamp : 41995 sec
Entry Size : 64 bytes
Sev Code : 1
Version : 2
Event ID : 0x1
System Time: Fri Jun 19 11:39:41 2009 UTC

Timestamp : 41995 sec
Entry Size : 120 bytes
Sev Code : 1
Version : 2
Event ID : 0x11
Type : 0x0
Path : \Windows\system32\winload.exe

And yes, it also does TLN output:

C:\Perl\forensics\timeline>bsparse.pl -f d:\cases\vista\bootstat.dat -t -s MYSYSTEM
1245411581|BOOTSTAT|MYSYSTEM||Bootstat.dat log file init event

Okay, at this point, you're probably wondering...beyond an academic exercise in binary parsing using Perl, how is this helpful? Well, according to Geoff's site, the bootstat.dat file is the BootManager log file. Right there, that should grab you...anything that's a "log file" might be useful in forensic analysis. A log file initialization event has a time stamp associated with it (as illustrated above), which can be used to correlate additional events, such as from Windows Event Logs (.evtx files), file system metadata, etc. For example, the LastShutdownTime from the Registry would ideally be followed by a bootstat.dat log file initialization event...right?

This is another example of data that is available to a knowledgeable analyst...data that can be used to build a more complete picture of what's going on with respect the system, and the goals of the analysis. And we're not talking hours and hours of work here...we're talking about a few simple tools and a documented process. In fact, collecting data and constructing a timeline from that data can be part of an intake procedure.

Other Stuff
There's another CyberSpeak podcast up and available! Thanks to Ovie and Bret for turning out these podcasts!

Using RegRipper

Now and again, I get stories from folks who've used RegRipper or the accompanying tools (rip, ripXP) to meet their needs. When that happens, many times I'll ask if I can post their experience and as in this case, I received the "OK":

I would like to share with you the experience of a recent engagement I have done where RegRipper proved very useful. I had to analyze 4 systems and the number of user profiles in each system varied from 6 to 15. For a system I extracted the relevant hive files, renamed them to "username_NTUSER.DAT" and dumped them in one folder. What I wanted was a certain kind of user activity like typed URL, programs opened etc. by all users (on that system) in one file so that I can I can scroll through the file and get a good idea about what happened on that system i.e. what programs have been executed on the sytsem and if I find something interesting then which username does it correspond to.

First I tried using rip.exe with shell wildcard character "*" but I guess RegRipper does not support that. So I wrote a simple perl script and pointed it to the folder where all the NTUSER.DAT files were located. The script ran a particular plugin against all the hive files and then dumped the output to one file. It also added the name of the file before its output to keep track of which username the activity belongs to.

My goal is that the new version of RegRipper will obviate the need to do this sort of thing...that all you'll have to do is mount the image read-only (tools abound for this...), point RegRipper at the mounted image, and let it go.

However, in the meantime, this is one way to handle this sort of issue. No, rip.exe doesn't handle wildcards...sorry about that, but to be honest, I simply cannot keep up with everything people are going to try to do with the tools, and say ahead of time what they will and will not do. More on that later.

Writing a Perl script or a simple batch file is exactly what rip.exe was intended for! I use rip when testing plugins, and ripXP is built on that same functionality. This is also a great example of how automating a previously manual process saves time and effort, without sacrificing accuracy. In fact, accuracy and completeness are maintained, while reducing the resources it takes to perform certain tasks, making the overall analysis much more efficient.

Finally, a reminder to the community...tools like RegRipper are pretty much written in isolation. That is, I originally wrote the tools to meet my own needs, and I update them for the same purpose. Now and again, I do get submissions for improvements, and those get added, depending upon the design requirements. As one guy, I can only do so much. I've said over and over again, if there's a plugin you're interested in, send me a concise description of what you're looking for and a sample hive (emphasis added on "and" because it's not a matter of one or the other). In cases what folks have done this, I've been able to turn around the plugin fairly quickly.

Also, hives with "interesting" stuff are always a nice stocking stuffer, even all year 'round! ;-)

Addendum: I wanted to add a comment I received from David Kovar:

My primary purpose is to get a quick snapshot of how the system was used, but I'll often come back to the report to help guide me in deciding where to go with my analysis. It probably is one of the most heavily used tools in my kit.

Thanks, David!

The Trojan Defense

The malware did it...not me.

How often do we see or hear of this? I don't see this specific question very often during exams...probably because I don't work in certain environments, per se...but I do get questions peripheral to it, and I do hear the question being asked by folks who do those sorts of investigations.

More importantly, however, is another question...how does an examiner (LE or otherwise) answer that question before it gets asked? The claim that "the Trojan/worm/malware did it" is being made more and more, and it can be a challenge to address this sort of thing...but it does need to be addressed.

The fact of the matter is that there are enough data sources on a Windows system that can provide indications as to whether certain actions (downloading illicit images, collecting/exfiltrating sensitive data, etc.) were performed by malware, or intentionally by a user. These sources may vary in their location and how useful they are, depending upon the case, but by examining and correctly interpreting the preponderance of data (rather than a few selected data points), an analyst can get an overall picture of what happened.

One example is when illicit images appear on a system...did malware reach out and download those images, or did the user install P2P software and then run a specific search for those images? Or did the user install P2P software that then led to a malware infection, and then the malware downloaded the software?

All of these questions can be answered, but what it takes to do so varies from case to case, and cannot be adequately addressed in a single blog post. Suffice to say that a knowledgeable analyst needs to look at everything, and be aware of those things they do not know. By this I mean, do not dismiss the value of Registry analysis simply because you have a deadline and do not feel that you sufficiently understand Registry analysis as a whole, or how it could apply to your case. The same is true for P2P analysis.

Addressing the "Trojan Defense" involves much more than simply mounting an acquired image and running an AV scanner across it. But like I said, that's not something that can be addressed in a blog post. This isn't a sales pitch, but what's really required is training, and regular, continuous interaction with other knowledgeable analysts, so that information and experiences can be exchanged.

DF and Disclosure

The subtitle of this post should be something like, "The disclosure discussion comes to digital forensics" or "Yes, Virginia...IR software has vulnerabilities, too."

For those of you who may not be aware, it wasn't long ago that MS's COFEE was leaked. Based on some of the comments you can find online, this occurred around the beginning of the second week of November. What's the big deal? Is it that this is "LE only"? Well, now that it's been exposed and you have an idea of what's in it...are you a better person for knowing? Or are you wishing you could forget?

Also, something that a lot of folks may not realize...MS is a HUGE company. At one point, I worked for SAIC, and internally folks described the environment not as one big company, but as 400 small companies that could and did compete against each other. MS is similar...don't expect that the left hand knows what the right hand is doing all the time. Remember WOLF? There've been other efforts within MS to produce tools and documentation, but it's not as if these have been released the way Windows 7 or MS Office are released. Tools like COFEE should be considered more of a grass-roots effort, and not something you're going to see Bill or Steve on stage endorsing.

Comparing COFEE to what's out there now and has already been in use may be a bit like comparing the GI Joe accelerator suit to the real-world MOPP gear, in some ways. You'll have to excuse the analogy...we in the "community" never seem to see eye-to-eye when it comes to this sort of thing. I'm simply trying to illustrate the point that while COFEE was released by MS to LE, that doesn't mean that it's entirely bleeding edge stuff.

Then, along came DECAF, reportedly a counter-intel, anti-forensics tool meant to detect the use of COFEE, and automatically react by doing things on the system, much like an anti-virus or -spyware utility would do.

Ovie then had the opportunity to interview Mike, one of the authors of DECAF. One of the most striking things about the interview is that Mike really didn't seem to have a well thought-out rational behind the creation of DECAF. For good or for bad, MS released something to assist LE, and act as a force multiplier...as a former Marine, I totally understand that. Regardless of what you actually think of the tool itself, the idea was to give LE a capability that they did not have before; specifically, officers that are first on-scene would have a tool that would allow them to collect data (which could be used as evidence) prior to spoilation due to temporal proximity to an event (that means, get it now, before it's gone). Mike apparently had an issue with the tool, and found what he determined to be a shortcoming or "vulnerability", and produced DECAF to "exploit" that shortcoming. Interestingly enough, I've listened to the podcast twice now, and I'm at a loss...Mike never said exactly what the shortcoming or vulnerability was, other than the tool could be "fingerprinted". But isn't that true for...well...just about anything?

However, what was he really doing? He wasn't so much exploiting a software "vulnerability" as he was exploiting the training (or lack thereof) of the first responders who would use COFEE. Ovie had an excellent point...what happens if a really, really bad guy had DECAF installed, and the first responders only had COFEE? There's a possibility that the bad guy could go on hurting children due to the fact that specific evidence could not be collected. If you listen to the podcast, I really think you'll see that regardless of the example used, Ovie made a very valid point, and it really seemed that Mike maybe hadn't thought all of this through.

Mike's primary issue with COFEE throughout the interview seemed to be that it could be fingerprinted...that there were automatic means by which someone could determine that COFEE was being run on a system. Okay...but isn't that true for just about ANY software? I don't know the inner workings of DECAF, but couldn't the same thing be said for other responder tools? According to an article in The Tech Herald, it appears that COFEE's primary purpose is to automate the use of 150 (wait...150?!?) tools, some of which include tools available from the MS/SysInternals site.

How many other responders use these tools?

How many other toolkits could possibly be affected by tools such as DECAF?

Consider this...why COFEE? Why not Helix? Why not WFT? Why target a tool released by MS, and not one, say, endorsed by SANS? I'm not going to speculate on those...those are for tool authors themselves to consider.

I have to say that before I actually listened to the CyberSpeak podcast, I found out that as of Friday morning, the DECAF tool had been taken down, the web site changed, and reportedly, anyone using DECAF on a system with an Internet connection would find that the tool had phoned home and self-destructed. Kind of cool, eh...an anti-forensics tool with the ability to phone home and get updates and instructions, with no user interaction beyond turning the system on...wait, that sounds kinda like a 'bot to me...

Following right along with the CyberSpeak interview, you can watch this YouTube video, as well, that pretty much reiterates what was said on the DECAFME.ORG web page. Speaking of which, here's a quote from the page:

As a security community at large, we need to band together and start relieving some of the burden off our government by giving back.

I agree with this 100%...and I agree that two guys did have an impact. However, from the interview, Mike seems like a really smart guy, so I have to ask myself...if you see something wrong, why not fix it? If you see an issue with a tool or process, why not do something to improve it? If you find an issue with a tool, particularly one aimed at assisting LE or others with their job, why not fix it? If you have a better tool or process and want to get it out there, consider Facebook, Twitter, blogging, and even contacting guys like Ovie and Brett. If the issue is the training received by LE, remember that a lot of LE is really strapped for cash...again, contact Ovie, or your local HTCIA or InfraGuard chapter and see what you can do.

My point is, two smart guys can have an impact, but their motives will dictate the impact that they have. This wasn't an issue like what's seen in the usual "full disclosure" discussions...we're not talking about server software that's been deployed across the globe that has a severe vulnerability. We're talking about a tool that can and should be used by a discriminating, knowledgeable responder...and if you don't think that the tool is sufficient, do something to fix it...other than subverting it. Provide something better.

Thoughts?

When a tool is just a tool, pt II

Okay, this is part II for this post, because I posted an awesome rant to a thread in one of the forums, and I wanted to include that here, as well, because it kind of applied...and it's my blog, I can do what I want. ;-)

The thread can be found here, and the post I'm referring to is on the third page, in response to someone mentioning, "...don't forget that if you wind up working civil or criminal cases your tools are going to get challenged in court. Open source tools are more vulnerable to attack by the opposition than commercial tools that are standardized."

My rant, if you want to call it that, had to do with what I see as a gross misconception with respect to court cases; specifically, some commercial tools are used primarily because the analysts themselves are familiar with them, and perhaps as a result, the players in the court system have also become familiar with them. That is to say that some commercial tools are recognized within the court system, and therefore, not a great deal of additional explanation is required.

As such, it isn't the tools that are challenged in court...it's the analyst and their processes.

Also, I think another huge issue that doesn't appear to be considered when folks are making statements such as the one I quoted above is that an analyst just doesn't decide one day to walk into court, take the stand, and testify. It just doesn't happen that way.

Instead, the attorney you're working with or for (prosecution or defense) is gonna want to know your answers before he asks you questions on the stand, and the fact that you're testifying and what you're testifying about are going to be part of the discovery process...so the other side is going to have a chance to cross-examine you. As such, if there's anything that would lead the attorney you're working with to suspect that you being cross-examined would sink the case, they're not going to put you on the stand. The same is true if he or she simply doesn't feel that the results of your analysis are pertinent to their case.

With me so far? I guess what I'm saying here is that there's a heck of a lot that goes on before an analyst ever gets to the point of approaching the stand in a court of law.

Now, can we agree that an acquired image is nothing more than a stream of bits, 1s and 0s, in a file on a disk? If we can agree to that, and if the integrity of that data, that stream of bits, can be verified and validated, then why does it matter what tool I use to extract data? What does it matter if an analyst determines that an illicit image is in the image using some commercial tool's Gallery View, or by mounting the image read-only with ImDisk and viewing the image file through Windows Explorer? Regardless of the tool used, the image was there, and that doesn't change. The same is true with other data...credit card numbers, other sensitive data, etc. One tool doesn't necessarily magically make it visible where some other free and/or open source tool wouldn't be able to extract the same data.

Now, don't get me wrong...I'm not against using commercial tools. I've used them myself (and I'm seeking therapy...just kidding) when the need has arisen. But the fact of the matter is that commercial forensic applications are just like any other tool, with their own inherent strengths and weaknesses. In some cases, I've found that processes using open-source and free tools, such as timeline creation tools, have allowed me to structure data for analysis in ways not possible through the use of commercial tools. In other cases, I've found short-comings in using commercial tools, just as I've found short-comings in using open-source and free tools. That doesn't mean that commercial tools shouldn't be used...it just means that all tools should be considered just for what they are...tools.

What should matter most is the process used and documentation created by the analyst. If you thoroughly document what you've done, then why shouldn't you be able to testify about it on the stand, regardless of the tools used? I know a few analysts who've documented their work such that someone else (i.e., LE) could validate their findings via commercial tools (because that's what the LE analyst was most comfortable with) and then testify about the "findings".

So, what do you think? Are open-source tools "more vulnerable to attack"? Why does it matter if I extracted a Registry hive file from an image, and then extracted the LastWrite time from a specific key using a Perl script? Or a hex editor? Or if someone else did the same thing, but through EnCase or FTK? The fact of the matter is that if you go to that location on the disk or within the hive file, extract 64-bits, everyone who does so should arrive at the same answer...right?

Or should I just curl up in the fetal position in the corner of my office, and rock myself to sleep, chanting, "I'm a pretty girl" over and over again?