Analysis of a Win2K rootkit

I ran across this site recently (can't remember where I originally got the link) and finally got around to actually reading through it.

The first thing that jumps out at me is that the content is incorrectly titled. When talking about rootkits, particularly on Windows systems, the term most generally refers to tools that are used to hide the attacker's presence from not only the Administrator, but the operating system and native tools, as well. However, I suppose that since the attacker "had root", used two self-extracting archives (kits) to move his tools over, and "hide" from casual viewing by (a) using the attrib command, and (b) deleting some files, then technically, it is a "root kit".

Like I've said before, I really like to see things like this posted...seriously. I've heard from others that not only are such things interesting to read, but they provide a great learning opportunity, and I wholeheartedly agree.

I do have a couple of suggestions for improvement, and a question. First, the question...the author makes reference to the HKU\{SID}\Software\Microsoft\Internet Explorer\Explorer Bars\{GUID}\FilesNamedMRU key, and seems to indicate that this location can be used as an autostart location (ie, "...thus executing all of the files listed if any one of them is started.").

Does anyone have any insight on this? I'm going to try this out and see what happens, but I'd love to know how close he is on this.

Okay...suggestions...constructive stuff. I like the screen captures, but there's a lot going on in the article that would really benefit from a screen capture or two. For example, ...used an MSBlaster style exploit to open port 4444 with root privileges... Okay...but can we see that? For example, how about showing the output of fport/openports side-by-side with the output of handle.exe, showing the user context of the process? That would be cool.

Throughout the article, the author speculates as to the intentions of the attacker. I don't necessarily disagree with his assessment of the skill level of the attacker (though I'd like to see more detailed information), but I think that sometimes we can misguide ourselves when we try to guess someone's intentions.

Finally, there are some confusing statements in the article. For example,

1. The svc.bat file sets the user name of the IRC bot in win.dll (which is actually just a plain text file)...
2. ...but instead sets the machine up as a warez server via IRC. The bot installed connects to irc.efnet.com and joins the channel #XiSO...
3. Edit the C:\WINNT\system32\Setup\svchost\x.pid file to find the process id (PID) of the IRC daemon
4. This stops the IRC daemon from running
5. ...suggest the intended use for this kit is not just to run an IRC bot...

So, is it a 'bot or a daemon? Client or server? It sounds like it's a 'bot/client, based on the explanation, but the use of the term 'daemon' suggests a server or service. This is an important distinction.

The author says, "This results in 3 processes called lsass.exe, though only one is legitimate." This demonstrates a "hiding" technique that still works. Basically, the attacker started several processes named "lsass.exe", but because the executable images weren't in the system32 directory, there were no issues with WFP. Even though you won't see the full path to the executable images in the Task Manager, just having more than one copy of lsass.exe running should be a tip to even the most casual observer. Now, using 'svchost.exe' is another matter entirely...

Overall, a great job. Shout-outs/greetz to the author.

MS Office 2003 Redaction tool

I got an email this morning about a tool released by MS called the "Office 2003 Redaction Tool". This tool is an Office 2003 add-in that allows you to redact documents.

Now, many of us are aware of issues with redaction, as well as with metadata in Office documents in general. In the past, folks have posted "redacted" PDF documents to the web, only to have someone with a slow download speed find out that the redaction was...well...not a good as expected. Talk about low-tech! Some of the earliest issues I'm aware of involved someone downloading a PDF document, and then being surprised when, through a slow link, serveral words were suddenly replaced with black blocks! Hey! In a nutshell, it simply took a few more seconds for the redaction blocks to crawl through the link and make their appearance on the page. At that point, all the reader had to do was remove the blocks.

So, I downloaded and installed the redaction tool, and opened up a small document I have on my desktop. I redacted two unique words in the document, and saved the resulting document. I then opened the resulting document in a hex editor and found that in hex, the redacted words had been replaced by "7C". Wow.

This is simply an initial test. There's lots more to do. For example, it appears that the redaction tool also looks in the document properties (ie, metadata) for the same redacted words...I say "appears" because I don't know for sure yet.

Something else to consider is this...let's say that you have a Word document, and in the interests of national security, you need to redact the phrase "small happy dog". Now, let's then assume that you have the phrase "I'm really happy about my job" in the comments (metadata) for the document.

Process monitoring and malware analysis

As part of looking into ways of improving malware analysis, I got to thinking about some things. For instance, some malware may maintain encrypted strings (ie, passwords or URLs) which it then decrypts in memory, and deletes once the string has been used. Some data used by the malware process may be transient in memory. So what do you do?

Well, I had a thought...what if you could run a process for a specified period of time, dump the contents of memory (and maybe run some other tools), then resume the process for a short period of time? Using Perl, we can do this! Here's the code:


#! c:\perl\bin\perl.exe
# strap.pl
# use to launch an application (malware) and then at predefined
# intervals, suspend() the process, perform some actions (ie, dump
# contents of memory, run external programs like listdlls.exe, etc.)
# then resume the application
use strict;
use Win32::Process;

So, we're using the Win32::Process module. Now, let's set up our variables. For testing purposes, I used AIM...

my $proc;
my $app = "c:\\program files\\aim\\aim.exe";
my $args = "";
my $dir = ".";
my $exit_code;

Now, let's go ahead and create our process...

if (Win32::Process::Create($proc,$app,$args,0,NORMAL_PRIORITY_CLASS,$dir)) {
print "Process created.\n";
# Get PID
my $pid = $proc->GetProcessID();
print "Process $pid created.\n";

Now that we've created our process, let's let it run for a bit (in this case, 3 seconds), then suspend the process. What we're doing, for testing purposes is running this loop only 10 times. This can be expanded (and the Wait() time changed, as well), or an entirely different loop structure can be used.

foreach (1..10) {
$proc->Wait(3000);
$proc->Suspend();
print "Process suspended.\n";

Now that the process has been suspended, we can go ahead and do things like dump process memory (with pmdump.exe), or run tools like handle.exe, listdlls.exe, etc.

sleep(3);

Once we're done, go ahead and resume the process...

$proc->Resume();
print "Process resumed.\n";
}
}

Now, just a little error checking...

else {
print "Process creation failed: ".Win32::FormatMessage Win32::GetLastError()."\n";
exit(1);
}

So, once this script has completed, you can conduct your analysis. For memory dumps, run strings.exe against the files, or use scripts to extract email or IP addresses, etc.

Addedum: In responding to some comments this morning, I thought that something I said should be included in the text of the post itself.

What I've presented above is not meant to replace anything. Rather, it's a tool...nothing more. Like any tool, it's effectiveness depends upon how you use it. One possible scenario for the use of this tool would be where an administrator or analyst has an unknown bit of software (we won't call it malware yet b/c we don't know). When looking at the program with a hex editor (part of performing static analysis), the analyst may believe that the program is encrypted or obfuscated in some manner. Now, most admins/analysts don't have tools like IDAPro at their disposal, and don't have the necessary skill sets to use things like debuggers and disassemblers. So, moving on to dynamic analysis, they want to run the program on a "sacrificial lamb" system to see what it does. Loading up monitoring tools, the analyst may want to run the application in a more controlled fashion, dumping memory contents along the way.

The above script will let you do that. Admittedly, the script itself is raw, and in it's early stages...but the basic idea is there. The great thing about scripts like this is that they can be easily expanded. For example, let's say we replace sleep(3) the necessary command to run pmdump.exe. When the above script ends, you'll have a directory with a bunch of memory dump files. Well, at that point, the script can then:

1. Hash each of the files (using Jesse's tools)
2. Run strings.exe against each file
3. Grep() through each file looking for email or IP addresses, keywords, etc.
4. Run 'diff' between all of the files to see what changed between snapshots
5. Save the results to a flat file, a spreadsheet, or a database for analysis

To me, it's all kinda cool! Something like this would have been kind of interesting when I was using netcat to demonstrate Locard's Exchange Principle.

iPod info

I was doing my monthly check of the e-Evidence.info site this morning, and I ran across an interesting article on how iPods are used as hard drives. So I plugged an iPod into my Windows box and fired up UVCView, and pulled the serial number from the device...in this case, "0000008A0136".

From there, I went to the Registry and navigated to the HKLM\System\CurrentControlSet\Enum Registry key, then dropped down and opened up the USBStor subkey. There, I found the device ID I was looking for...Disk&Ven_Apple&Prod_iPod&Rev_1.62. Beneath this subkey, I found the instance ID that contained the serial number of the device; "0000008A0136&0". From there, I mapped the ParentIdPrefix value to the corresponding value under HKLM\System\MountedDevices and located the drive letter that the device had been mapped to; in my case, \DosDevices\G:.

So what does this all mean? If you're looking around for who's been using iPods at work, you know which key to check. If you're performing a forensic investigation, you should check under the ControlSet00x key, rather than the CurrentControlSet key.

Training Poll

A while ago, I posted about a training course based on my book. Since then, I've received emails from folks, asking about when such a course would be available near them, so I thought I'd post here and see what folks are interested in...

When I taught my Windows 2000 Incident Response course, I would generally go on-site and teach approximately 20 folks at a time, for two days. In some cases, attendees would ask me to come on-site to their location, and provide specialized content, based on the needs of their particular group. This sort of set-up (ie, having me come on-site) worked out really well for the folks who took advantage of it, because they saved a ton of money. Having me come on-site for two days to teach a dozen (most cases were closer to 20) folks the material was less expensive than sending 3 people off to other training courses (that shall remain nameless). Also, they didn't have to deal with processing travel claims for all those folks...and the material that I presented could be...and was...used immediately.

However, this kind of set up does not work for everyone. Not everyone can provide 8, 10, or more folks at one time, in one location, for two days. Some folks would be happier going off-site for three or four days. And still others would be happier doing it all online.

So, here're my questions to you, dear reader...

1. Would you be interested in a training course (2-3 days), based on my book? The material would be technical, with a lot of hands-on work, includes labs/exercises.

2. If you would be interested in such a course, what information, specifically, would you be interested in? My data hiding presentation has always been popular, but my "Windows Registry as a forensic resource" can be a bit out of reach for some folks. What type of content would you like to see, specifically? Would you be more incident response oriented?

3. What type of setting/forum would you like to see? Would you prefer to have me come on-site, or would you like to go somewhere off-site? If you're more interested in an web-based approach, can you point me toward some services?

Thanks. Feel free to post a comment (please sign it) or email me...

A/V software on web servers, revisited

The subject thread I've been following and contributing to has proved to be interesting, to say the least, and the most interesting aspects have little to do with the subject...

First off, I guess I really shouldn't be surprised how many respondants go almost immediately off-topic, without bothering to change the subject line. Bad etiquette, guys, but I guess that's to be expected.

I want to take a second or two so summarize some of the most popular responses I've seen in watching this thread, and comment on them...

There's a lot of talk about "protecting against currently unknown exploits"...and I have to ask, if it's unknown, how do you protect against it and how do you know A/V will help you? Really. One example came up of a new exploit in which the bad guy took control of the system and dropped an already-known rootkit (ie, HackerDefender, etc.) on the system...in such a case, A/V would work. Yes, and you'd be very lucky, b/c the bad guy was very stupid. If the bad guy gained such access, why would he not bother to disable or even uninstall the A/V first? Or roll back the .dat file? Or why use something that's old? If he's using a zero-day exploit, why would he then use an old rootkit?

There have been mentions of Code Red, SQL Spida, and SQL Slammer. To me, it's odd that someone would use those examples and say, "I've seen A/V come to the rescue with these viruses", when the infections could have been easily prevented in the first place with configuration settings. With Code Red, all the IIS admin had to do was disable the .ida/.idq script mappings...something not many folks used anyway. With SQL Spida, all the SQL admin had to do was put a password on the 'sa' account. Need I say it? Duh!

Another biggy is that A/V protects against the things that have been missed, the things that haven't been done. Good point, but I would suggest that perhaps the security process itself needs to revisited. If something wasn't done, why was that? Was it part of the process and the setting wasn't verified before putting the server into production? Or had it been changed? Yet again, A/V software is justified...but as a band-aid approach.

There have been respondants who are in positions where the web servers are administered by multiple folks, and perhaps even developers have admin access to the servers, for adding updates. This falls right into the same category as malware that gains SYSTEM level access...A/V isn't going to help, b/c at that point, it's GAME OVER, guys! Someone with that level of access can simply disable your A/V software.

So...am I being arrogant? I don't think so. I've managed IIS 4.0/5.0 web servers and watched the logs fill up with failed Code Red/Nimda attempts, etc. I've had NT 4.0 boxes (my own) connected to a DSL hookup with no firewall, and no A/V software...and the only time I've ever gotten a virus, worm, or rootkit on my system is when I put it there myself.

Am I saying that A/V software shouldn't be used? Not at all...I think like every tool, it has it's place. However, I do think that if a web server admin is doing their job, then it's not necessary to put A/V software on a web server. That was the original question I responded to.

Training

I recently had someone contact me, someone who had read my book and wanted to see about attending my course. Well, one way to do that would be to sign up at MegaMind.org (note: the course actually covers much more than just Window2000). Another way to do it is to have me come on-site and teach it at your facility.

This person suggested that I contact Amazon for a list of names of people who are interested in the training, but to my knowledge, Amazon doesn't maintain anything like that.

So I thought I would pose the question via the blog and see what kind of response I got...so, here it is...

If you're interested in incident response training specific to Windows systems, what sort of program do you prefer? Would you prefer traveling to another location to take the course, or having me come to your facility and teach it? How about the material? The course is specific to incident detection, verification, and resolution on live systems...but I have tailored it to the specific needs of the certain organizations (i.e., I've extended the 2-day course to 5 days, added or removed material as needed, etc.).

How about forensic analysis of Windows systems? As I've already started working on another book along those lines, I can easily see the necessary material for a detailed, hands-on course coming together.

So...what are your thoughts, preferences, comments?

A/V software on web servers

There's an interesting thread over on the SecurityFocus Focus-MS list, regarding the installation of anti-virus software on IIS 6.x web servers. I attempted to ignore it, but this morning felt the need to interject.

In a nutshell, my point has simply been that if a web server is just that...a web server...then for the most part, it's serving up content via port 80 (and possibly port 443). If the system is properly configured, why then is anti-virus software needed? If it's just a web server, and your LAN/infrastructure is properly configured and administered, what is the purpose of adding yet another software package to a system, that's going to generate yet another set of logs that will be ignored?

One post mentioned the fact that you can't possibly know all of the threats you'll face. Well, that's true...if you've got your head stuck in the sand. Remote attacks come down to two basic types...those that exploit poor configurations (ie, weak passwords, running unnecessary services, etc.), and those that exploit improper bounds checking in the software itself (ie, buffer overflows). Understanding this makes it easier to reduce the attack surface by greatly restricting the avenues available to an attacker. In fact, it is possible to reduce the attack surface to the point where you still have your necessary functionality, but pretty much the only way to compromise the system is to be the administrator sitting at the console...and at that point, no amount of anti-virus software is going to do you any good.

Another post mentioned that the web server is the public interface to the rest of the world, and anything that gets on the LAN can make it over to the web server, and you may possibly end up with an embarrassing situation (ie, defacement, malware proliferation, etc.). This may be the case...if you've got your web server installed on the same LAN segment as your employee desktops. However, the most likely avenue of attack at that point would be via NetBEUI/file sharing, and it that's enabled on your web server, then it's no longer just a web server, is it?

Take a look at some of the stuff that's hit the Internet. Remember Code Red? Well, some folks at MS set up an IIS 4.0 web server, and it was not susceptible to Code Red a full year before Code Red was launched. In fact, if everyone had simply disabled the proper script mapping the day before Code Red came out, they would not have been vulnerable.

If you're considering putting anti-virus software on a web server, I'd suggest that you look at the root cause as to why you want to do that. Perhaps a better investment would be in training of your staff (ie, support staff, as well as management), or in another product all together.

Am I saying that web servers should never have A/V installed? No, not at all...what I am saying is that before doing so, you should take a hard look at the reasons why you're doing so. Sometimes just setting some ACLs and removing unnecessary services will go a lot further than installing another software package that needs to be maintained.

As a side note, it seems to me that a lot of folks out there are of the mind, "if you're running Windows, you must have anti-virus installed." To me, this seems to be a very uneducated and misinformed position. I do not use A/V software on any of my personal systems, and have never been infected when I haven't done so intentionally. At work, there are no log entries from the corporate A/V software to indicate an infection of any kind on my workstation. Sure, if your kids are using a home system, you'd want to have anti-virus software installed, but that applies regardless of the platform.

Addendum: I was checking out Bruce Schneier's blog this morning and noticed something that rang true with this blog entry. Specifically, in one of his entries about turning off cell phones in tunnels, Bruce says, "This is as idiotic as it gets. It's a perfect example of what I call "movie plot security": imagining a particular scenario rather than focusing on the broad threats." Wow. Bruce gets lauded as a security expert when he says things like that (and I happen to think he's right). However, when I say something like, "There's no point in installing A/V software on a web server", which is pretty much along the same lines as what Bruce said, only mapped to the digital world, I end up getting emails that are better not quoted in public.

Most known threats can be protected against without the use of A/V software on web servers. New threats won't be caught because they're new, and not yet subject to scrutiny by the A/V community.

Oh, and one other thing...more than one person sent me email telling me that they "saw" A/V software protect systems from being infected with the SQLSpida worm. All I can say is, thanks guys, but you made my point for me.

Bots writing Registry entries

As I've purused some of the anti-virus sites of late, I've noticed a trend that malware...specifically, bots...are writing two particular Registry entries:

[HKCUHKLM]\System\CurrentControlSet\Control\Lsa

and

[HKCUHKLM]\Software\Microsoft\OLE

I'm seeing this with several bots...W32.Bropia, W32.MyTob, etc. Some A/V sites point out that these are variations of SD-Bot, which wrote to the keys, as well...but why? A/V companies do a great job of saying which keys get created or modified, but it's tough to figure out *why*.

What's the purpose for writing to these keys? Does it have something to do with the LSASS vulnerality in MS04-011? Is this another autostart location?

Prefetch file metadata

I exchanged emails with the anonymous poster from my previous metadata entry, and got an interesting perspective. Specifically, not enough of the folks actually performing forensic analysis of Windows XP systems are aware of the Prefetch directory and what it contains.

This reminds me of a very brief exchange I had w/ one of the virus writers from the group 29A a while back. Specifically, Benny and Ratter had written some viruses that took advantage of NTFS alternate data streams, and I asked them where they saw things going. The response I got back stated, in brief, that it was a deadend b/c everyone knows about ADSs.

Hhhhmmm...so why is it that when I talk about them at conferences, attendees sit up and say things like, "Okay...go back a sec..."??

My point is that just b/c some of us know something, we have to realize that not everyone does. Just b/c someone is, say, a forensic analyst for local, state, or even federal law enforcement, that doesn't mean that they know all of the ins and outs of Windows XP.

Keeping that in mind, .pf files within the Prefetch directory have certain metadata associated with them, specifically, the file contains several Unicode strings (view using strings.exe or BinText from FoundStone), one of which is the path to executable image. So you will see from where the executable was launched.

So...outside of strings and MAC times...what is there? Has anyone ever seen an ADS associated with a .pf file?

"Hidden" event records

I can't release the info I have yet, and I'm not trying to tease anyone...honest. However, I can say this...the method I'm using to retrieve event records from an Event Log file is very interesting. In my single-minded simplicity, I've found a way to locate "hidden" event records!

Seriously. I have a test Event Log file from a system, and using the MS API (ie, Event Viewer, psloglist.exe, and the Perl Win32::EventLog module), I can "see" 2363 total event records, running from 11239 to 13601, inclusive. However, using my Perl script, and verifying it by hand, I can also "see" event number 11238, which wasn't over written yet...the header information for the Event Log file simply tells the MS API to start with event ID 11239 (by giving not only the event ID number, but the offset where it's located within the file).

Very interesting stuff.

File Metadata

I've been working on my GMU2005 presentation regarding file metadata on Windows systems...basically, showing the types of metadata that are in and associated with various files on Windows boxen.

The stuff I've covered includes Office documents (I even include my MergeStreams demo, b/c it's way cool), PDF documents, Event Log files, and PE file headers. I also cover NTFS Alternate Data Streams and MAC times.

Am I missing anything really obvious here? My goal with this presentation is to tell the audience, "hey, guys and gals...there're all these files on Windows systems, and they're usually there by default, in many environments. There's a lot more information you can pull from them than just the fact that they exist."

I'm just trying to do a sanity check. I went back through my book to see if there's anything I really missed, and I think I've got it. Sometimes, you get to doing this stuff so often, that you stop seeing how important it is for others in the field to know it...you stop seeing the forest for the trees, so to speak.

Oh, and I emailed the guy in charge of the GMU2005 conference, and asked if I could be squeezed in at the last minute with a presentation on the Event Log file format. I specifically asked to get a slot during prime time...not at 5pm on the last day. We'll see how it goes...but I'll be putting the actual presentation together next week and getting it in the approval pipeline. That'll give me 5 presentations at a single conference. Ouch!

And yes, once the presentations have been approved for public release, I'll post them.

Where, oh, where did my little SSID go...?

Got wireless? Ever go to the Control Panel, to the Network Connections applet, open up the properties for the Wireless Network Connection, click on the Wireless Networks tab, and see a whole bunch of SSIDs listed in the "Preferred Networks" box? You probably know how they got there, and you can easily get rid of them...but have you ever wondered where they're kept on the system?

Ever imaged a Windows XP drive and wondered what wireless networks the suspect connected to?

Well, I've been digging around and I've found it. Open up RegEdit and navigate to the following key:

HKLM\Software\Microsoft\WZCSVC\Parameters\Interfaces

See a subkey that looks like a GUID there? I've got one on my system, you may have more. Well, click on the subkey and look over in the right-hand panel. If you don't see values named "ActiveSettings", "Static#0000", etc., then move on to the next GUID.

If you find one of these values, right-click it and choose "Modify". See the SSID in the binary data?

Now, my laptop is a Dell system and uses BroadCom software. If you don't see the values I mentioned in your Registry, check your client application for your wireless stuff, and let me know what you've got. I've read on the 'Net that Cisco and 3Com client apps keep the SSIDs in the Registry in plain text.

Rootkit Detection, and a prediction (of sorts)

I was over on Rootkit.com again today, reading up on some of the recent entires...if you're at all interested in Windows security, you should really consider signing up. Anyway, I was reading an article by the erstwhile Joanna Rutkowska on crossview-based rootkit detection, and was really fascinated by what I was reading. Her article on Rootkit.com discusses various issues and does a good job of outlining the "war of attrition" as the good guys develop new ways to detect rootkits, and the rootkit authors (some can be good guys, too...) develop new ways to avoid detection.

Joanna makes some interesting comments, in particular:

One may ask a simple question now: why bother to hide files at all? Isn’t the idea of “hide in the crowed” equally stealth? The answer, fortunately, is no...

and

The answer is no, because the current antivirus technology is able to find all (unhidden) executable files and then perform some kind of analysis if the given PE file looks like a potential rootkit/malware installer (for e.g. check if it uses functions like OpenProcess(), OpenSCManager(), ZwSetSystemInformation() and similar). When designing such scanner we need to remember that rootkit executable can comprises of two parts, one being an actual malware loader and the other being a (polymorphic) decoder.

I'm not sure that I entirely agree with her statement, though her reasoning is certainly sound. First off, let me just say that we all come from different places and have different opinions based on different experiences. For example, I have military training in my background, which includes the concept of Maneuver Warfare (as practiced by experts). Given that, and also given that we're seeing more and more attacks in the media that seem to take a more economic or financial focus, my thought is that we're going to see more targetted attacks.

What does this mean? Well, rather than going for mass infections, we'll likely see programs installed on fewer, but targetted machines. Am I saying that this is the death of Internet worms? Not at all...we'll have those around for a long while yet. But what I am saying is that it's very likely that the worms will be test cases...what works, how "noisy" is it, how quickly is something detected and turned over to the anti-virus vendors for analysis and signature creation? With this kind of information, the attacker can target his approach...and all without rootkit technology.

I'll give you an example. In the military, it's commonly known that during inspections, you give the inspector something to find...b/c if you don't, he won't leave until he's gone to some very dark, uncomfortable places with a microscope and a pen light. So, you give him something to find...not too significant, but enough to satisfy him so he'll...well...go away. Well, map that sort of thing over to what we've been seeing since the inception of viruses, and especially since backdoors like Back Orifice were released...when the incident occurs, it's detected b/c it has a significant and often immediately noticeable impact on systems. Well, what if the attacker decided to be really stealthy, and not give the inspector (Administrator, in this case) any cause to even look around in the first place?

Why are rootkits used? To hide the attacker's presence when the administrator or investigator comes looking. So...don't do anything to cause the investigator to look in the first place.

Where are attacks going? Think about maneuver warfare...one of the concepts is to bypass strongpoints. Marines assaulting a beach will bypass a bunker that's facing the beach, and cut it off from the rear, choking off the supply routes that keep the guys in the bunker in beans, bullets, and band-aids. The same holds true with crime...bad guys are going to attack the easy targets first...the unlocked cars and houses, the unescorted children and women, etc. Online, something that looks like it's fairly unattended/unmanaged will be attacked first. Why go after the heavily protected server, where *if* you do get in, you'll create a lot of noise in doing so (in my book, I used the example of Ethan Hawke in Mission Impossible crushing up a light bulb and spreading the shards outside the apartment in the safe house...), and someone's going to come looking.

Attacks are likely going to be targetting less well protected systems, and the attacks are likely going to have less of an impact on the systems over all. The attacks will be more subtle, and the attacker is going to take great pains to stay stealthy and hidden, by not attracting attention to the fact that he's there. Do you need rootkit technology for this? No. It's been widely seen that it doesn't take a lot of effort to remain hidden from most administrators, even if you're hiding in plain sight (no disrespect intended, guys and gals). Adding a program to a system that isn't going to be detected by anti-virus software (all that takes is something new), isn't going to create a lot of noise, and isn't going crash or overwhelm the system is all it takes.

Are you like me, and need examples and specifics? No problem. Anyone remember

Event Log file format

As a follow-up to my earlier post on the EventLogRecord structure, I wanted to mention that after no small effort, and with some assistance from someone involved with PyFlag, I was able to figure out the format of Event Log files.

Okay, that this point, you're probably thinking...so what? Well, consider this..you're analyzing a Windows system, but you're on Linux. Or you have a corrupted Event Log file, and the Event Viewer (and even psloglist.exe) can't open it. Or you're looking for event records in slack space. In any one of these instances, knowing the structure of the Event Log file would be very helpful. I've drafted a paper, and I need to see about getting it through my employee for public release. If that doesn't work, or it gets stalled, I'll see about releasing the information via another (albiet acceptable) means.

So why am I telling you this, only to say, "I can't release it yet?" Well, my thought was that if anyone has a pressing need to know something about the Event Log file format now (and I mean right now), send me an email and I'll see what I can do to answer your question(s). Otherwise, hang tight and I'll see about getting the information out.

The other reason I'm posting this is to ask about forums/magazines suitable for posting this sort of information. I've been working on articles for the Digital Investigation Journal, but it does take a while for the article to be available to the public. If there's a forum similar to the DIJ, but quicker...let me know.

Addendum: I wanted to add a couple of comments with regards to my effort in this project. First off, since I started this project, a tool called GrokEVT was released. A post hit the SF Forensics list, and I initially caught wind of it there, and since I'd posted asking about this subject, I got a couple of emails from folks pointing it out. There are also some other materials out there, but I can't provide links, b/c right now the links seem to be broken.

Anyway...GrokEVT looks like an excellent tool. It seems to do pretty much everything; extract the event records from the file, search the Registry for message files, then extract the message strings from the file. However, the documentation does state that some of these functions are "unstable". Well...it's a good start.

The one thing that the package doesn't seem to do is explain the format of the Event Log file itself. Yes, I've only looked at a small piece of the puzzle, and no, my solution isn't as comprehensive as GrokEVT or PyFlag. However, the little bit that I've done does provide the forensic analyst with the necessary information to locate event records in slack space, and extract and interpret those records. What I've also done is create a Perl script that uses several functions to retrieve event records from a file. These functions can be used to retrieve records from a corrupted or partially deleted Event Log file.

Knowledge of the Event Log file format is also useful in understanding and detecting anti-forensics techniques involving the Event Log.

My hope is that someone finds this information useful.

System Analysis

Ever notice how you go to a conference or view something online that talks about "analysis", and all you see/hear/read about is data collection?

I was at a conference a couple of weeks ago, and one of the speakers was giving a presentation on "live data analysis". The speaker did a great job talking about collecting data...but that's not analysis. Or is it?

I have to say at this point, I'm confused. I've been thinking for a while that data collection and analysis are two different things. After all, we see it all around us as two separate actions. The men and women on TV shows like "CSI" do collection...and then they perform their analysis.

At the beginning of every month, I like to drop by the E-Evidence.info site to see what wonderful new papers and presentations are posted. I love reading through some of the stuff that appears on the site. At one point, I even went back through the archives from previous months and years. There's always something interesting posted here.

Yesterday, I ran across a paper about checking Windows boxen for signs of compromise. Reading through the paper, I think it's extremely useful to the right audience, but it doesn't say anything about analysis...it's all about running tools. Some of the descriptions of tools and why they are used are pretty bare, to say the least. But like I said, it's a great paper for the right audience.

I then read through a presentation on the "live analysis" of a Linux system. Again...go here, get this tool, run it. While the presentation does present issues such as Tor networks used for anonymity and privacy (if you're interested in that kind of thing, check out VPM), it really doesn't do much to cover "analysis". Even Dana Epp's 2004 presentation includes the word "analysis" in the title, but the presentation itself only glosses over any actual analysis.

My point is that data collection is easy...it's the analysis that's the hard part, and what we need to start focusing on. The tools are there. Techniques and methodologies for collecting data abound. But I'm not saying that we shouldn't keep presenting and writing on data collection...what I am saying is that if the presentation or paper has the word "analysis" in the title, then analysis should be discussed.

Okay, I know what you're going to say..."hey, dude, chill! There are just too many possible things that someone can look for when doing analysis." And I'd agree with you. But I also think that a really cool way to do presentations is to pick something...something you've seen or done, or something that's of interest to your audience (I know, it's really hard to get information on what others are interested in...)...and go over that in detail. I've decided that for my part, I'm going to start doing more to cover the actual analysis...now that you have the data, what is it telling you...in my papers and presentations. In fact, I submitted an abstract for the DoD CyberCrime Conference for a paper/presentation that does exactly that. I don't know if the proposal has been accepted yet or not, but my intention is to walk through the analysis of a corporate case, specifically the theft of proprietary information. It should be interesting...not only in putting the presentation together, but also in the audience's reaction.

The media, and how they skew "attacks"

I read an article in the Bozeman Daily Chronicle today, about a database system housing personal information about hunters in Montana was compromised.

I have to say, I'm extremely disappointed, not only with the state of IT, but of the popular media.

Reading through the article, it's pretty clear what happened. A system owned and administered by folks in Montana was compromised, and someone tried to turn it into a warez server. In fact, the activity that appeared in the logs could have been completely automated. Many of us have seen this sort of thing before...an automated script scans for FTP servers and tries to log into the "anonymous" account. If it's able to do so, it tries to create a directory, in order to see if it has write access. If the script is successful, it either logs the IP address of the vulnerable system and moves on, or it creates the necessary directory structure and starts uploading files. Of course, this is one of many ways that this kind of activity can occur.

So what's my point? Well, if I were a hunter in Montana, I'd want to know why my personal information was on a database system that was accessible from the Internet, in a manner such that it could be attacked in this way. What service was attacked? The database is Oracle, and though that software has had it's share of vulnerabilities, I don't get the sense (again, my source being the article in question) that the database itself was attacked...but that the system it was running on was attacked through another (possibly unnecessary) service. So...why was it connected to the Internet in such a way as to be accessed by this "attacker", and what (potentially unnecessary) services/daemons were running on it and why?

Speaking of questions, the author of the article had an excellent opportunity to make a mark by asking those tough questions. I believe that legislation is getting us to the point where incidents such as this must be reported. Now what needs to happen is that knowledgeable people need to ask the tough questions...why was the system connected to the Internet in this manner? Who is responsible for the design decision? Who is responsible for the administration of the system? Once these questions start to be asked, maybe the IT folks actually making the decisions will start thinking a bit harder about what they're doing.

So, while we don't...and probably never will...have all of the information about the attack and what actually occurred, articles like this tend to spread FUD amongst the parts of the population that aren't as familiar with security (this includes a lot of IT folks) issues as some of us. I'm not saying that I'm an expert, but I do know enough to recognize FUD like this...

Reading Windows files in binmode()

For the past couple of days, I've been writing Perl scripts to parse binary data on Windows systems...I've been staring at 1s and 0s, hex, parsing strings, etc.

My first exercise was to parse PE headers...and the script works very well for legit PE files. By reviewing the information available on MSDN, and correlating that information with other sources (sorry, guys, but there are some holes in your docs!), I have been able to parse all the way down paste the data directories and into the section headers. Very cool! There's still a lot of work to do to make this script really useful, but developing it has really been very beneficial in understanding PE headers and malware.

Now, I'm working on a Perl script to parse .evt files manually, by opening the file in binmode() and parsing the byte stream. The problem I'm having is that even though the EVENTLOGRECORD structure is well documented at the MSDN site, I have not been able to find any information about the data located between offset 0 of the file, and the offset of the first record (which itself seems variable, depending on log type, operating system, etc.). Byte alignment is important, so I know that the API has some inherent method for locating the various records. However, I'm trying to read in the file, basically, a byte at a time...does anyone have any information about the .evt file header info? I'd like to figure out how to parse and make sense of this data.

While this whole thing seems like a pointless exercise, there is a method to my madness. For example, I can use scripts like this with ProDiscover 4.0 to reduce the time it takes to analyze a system. If you know what it is you're looking for, you can automate the activity, increasing efficiency, reducing mistakes, etc. So let's say I have a ProScript (ProDiscover uses a Perl module called ProScript to implement Perl as its scripting language) that I use to identify can copy files from an image. I can then automate scripts using...you guessed it...Perl in order to help me find only the suspicious things. This is referred to as "data reduction". By using the ProScript API, I can automate adding this information to my reports, as well.

More on memory dump analysis, and some other stuff

First, I'm back from the MISTI "Cracking eFraud" conference. It was great to meet a lot of the folks there, both other presenters, as well as attendees.

I attended Brian Carrier's presentation entitled "Live Forensic Analysis". Brian was kind enough to sign my copy of his book for me. I also need to get a copy of Dan Farmer's book, as well.

While I enjoyed Brian's presentation, I have to say that there wasn't a great deal of "analysis" discussed. Yes, Brian covered tools and techniques, talked about rootkits...but analysis techniques weren't really discussed. However, I don't think that a one-hour presentation was really the venue for that sort of topic. I gave a presentation earlier in the morning on looking in the Registry for specific data, and went over by a few minutes...even though I felt that I was glossing over a lot of things.

Maybe the whole idea of presenting on "analysis" really needs to be a full-day presentation, with hands-on exercises, etc.

Anyway, I wanted to blog on memory dump analysis a bit more this morning. What I'm talking about here is using tools such as dd.exe to grab the contents of physical memory; i.e., RAM. As I blogged earlier, on Windows systems, if you want to grab an "image" of physical memory, you have to generate a crash dump, in which the system halts and the contents of physical memory are written to a file. However, most of the systems we would likely see aren't set up for this...so lots of folks, including LEOs, are using dd.exe to dump the contents of physical memory.

Once you have this file, what do/can you do with it? Well, the first and most obvious thing is to run strings.exe against the file or open it up in BinText. You might also run scripts against the file, looking for specific types of strings, such as email addresses, IP addresses, etc. Such things might be useful.

Another thing you might do is to break the dump down into 4K pages and generate hashes for those pages, and then parse through the file system, doing the same thing for the files. Matches would let you know if the pages were loaded in memory (credit for that one goes to Dan Farmer).

But what about parsing kernel structures? Microsoft has many of the available structures documented in MSDN, so we know what they look like. We know, for instance, that a particular structure is so many bytes long, based on the values in the structure, and we can parse these with computer code, either in C, or in scripting languages such as Perl (using unpack()). Knowing this, we only have one thing left that we need to know...the offsets. The image file itself starts at 0 (or 0x00000000)...we need someway of finding out where the Waldo structure begins in that file, before we can start parsing it.

So...what are your thoughts? Am I off base? Am I close? Do you have have any input at all on how to determine the offsets...other than "ask Microsoft"? Are there any developers out there who can comment on this?

Upcoming conferences

Next week, I'll be presenting at the MISTI "Cracking eFraud" conference in Boston. My presentation is on using the Windows Registry as a forensic resource". I'm presenting at (get this!) 0730 on Wed. If you're there, stop by and say hi...I promise that it'll be worth your time.

In August, I'll be giving a couple of presentations at GMU2005. I'll be presenting on document metadata on Windows systems, and on tracking USB devices across Windows systems. It looks like quite a bit of time is scheduled for each of these presentations, and I'm giving them both twice...so I may sneak in MISTI presentation, as well...particularly since the USB data is incorporated in that presentation.

Once again, if you're there, stop by and say hi. At the end of my presentations, I like to give a pop quiz, with copies of my book as prizes.

Also, if you've got or know of a conference coming up where topics like this (or other topics concerning the forensic analysis of Windows systems) would be beneficial to the audience, let me know.

Memory collection and analysis follow-up

This topic is by no means closed...in fact, I think that the discussion of dealing with physical memory, memory analysis, etc., for forensics and/or malware analysis is just beginning. Given that I'm one of those folks who has spent the past couple of years knodding my head about using dd.exe to image RAM, simply because I didn't know any better, and given the number of folks I've run into who have done the same thing, it's pretty clear to me that this is an issue that needs to be addressed.

IMHO, the best way to approach this is to provide the knowledge, weigh the pros and cons, and let the user/reader make the decision on how to proceed. For example, sometimes, running tools such as the FSP would be the way to go...correlating and analyzing the results can get you a long way. However, those are user-mode tools and things might be missed...but then again, with the right combination of tools and analysis, you may be able identify those things that are missed, at least...not what the data is, but just the fact that it's missing (think of this as akin to identifying the wind...we can't see it or taste it, but we know it's there based on how it affects the environment).

In other cases, such as malware analysis (and forensics involving specific processes running on the victim system), using the debugger tools to grab the contents of process memory, and then using the same debugger tools to analyze the information retrieved, might be more desireable. You'd get some of the same information about the process as you would with the user-land tools from the FSP (i.e., loaded modules, handles, etc.), but this might be the preferred approach. Putting the tools on a CD and writing the process memory dump file to a thumb drive would be a great way to handle this.

Now, when it comes to a full-out memory dump, so far as I've been able to determine, the only real way to do this is with a crash dump. Crash dumps can be triggered manually, as mentioned in my previous post (via the MS KB article), but doing so requires advanced planning. I'd highly recommend incorporating these changes into malware analysis systems. I'd also recommend that the settings be incorporated into critical systems, as at the very least, analyzing the memory dump would make root cause analysis much easier. One of the methods MS mentions for performing troubleshooting is "send us your crash dump"...Oracle does this as well for their products. For forensic purposes, one has to take this into consideration, but I tend to believe that the benefits may outweigh the risks (i.e., overwriting exculpatory evidence when the full crash dump file is written to disk)...depending upon the situation.

I'd appreciate hearing your thoughts...considering that in the past couple of days, I've completely thrown out the idea of using dd.exe to image physical memory, and thrown out pmdump.exe for getting process memory. Now, I've got to go back and rewrite my already-published articles...

RAM, memory dumps, and debuggers...oh, my!

One of my recent posts on obtaining the contents of physical memory using userdump.exe attracted the attention of some folks as MS, and I ended up having a long conversation with Robert Hensing on the topic of obtaining and analyzing the contents of (physical) memory.

The long and short of it is this...the tools and techniques you use all depend upon what you want to do. How's that for a direct answer? No, that wasn't Robert's response...he provided much more information than that...I'm just summing it up for you. I'm going to give you a short and sweet explanation, opening this topic up for discussion.

If you want to grab the contents of physical memory, you can do so with dd.exe...keeping in mind that tools like this, as well as LiveKD, don't lock kernel memory prior to performing their dump. Therefore, what you get is a smear, of sorts, as the contents of physical memory are changing all the time, while you're dumping those contents. Also, keep in mind that dd.exe does not produce output that is compatible with MS debuggers.

Now, if you're looking to get the memory used by a process, then the way to go is to download the MS debugging tools, and run those from a thumb drive or CD. The debugging tools have a plethora of switches, but Robert was nice enough to write a VB script that's included in the tools called "adplus.vbs" that will let you easily dump the memory contents of multiple processes. The debugging tools do this by first suspending the process, and after the dump is complete, you can then either resume or kill the process. The output can then be analyzed using the debugging tools.

I have to say that this makes a lot more sense than using just pmdump.exe and strings. Robert even went so far as to point out that this can even be used when hunting for rootkits, as some rootkits will insert code into memory used by all processes...when attempting to perform analysis of the memory dump, the debugger will crash when it attempts to reference memory that doesn't exist or is hidden.

Now, what if you want to capture a snapshot of the system? Well, the only real way to do that is with a crash dump (ie, Blue Screen). And when I say "real", I'm referring to the most forensically sound method of doing this...that being said, being able to do this requires some preparation. I'd highly recommend that if you have critical systems that you're really concerned about, that you put a lot of thought into considering these options. First, take a look at KB254649 for an overview of dump file options on 2000, XP, and 2003, and then KB244139 for a feature that allows a memory dump file to be created from the keyboard. Go through the second KB article thoroughly and consider adding the recommended modifications to critical systems. I'm told that some folks out there have done this.

I'd also consider setting up testing systems with the same options, so that when you're testing malware, you can generate dumps of process memory or even crashdumps as part of your testing and then use the MS debugging tools for your analysis.

Robert added that there are debugger plugins that allow you to view and analyze very useful information, such as TCP/IP connection information, from a full crashdump, as this information is handled by the kernel.

So...the long and short of this is that some recommendations that work on Linux systems aren't exactly advisable on Windows boxen, depending upon what you want to do. Sure, you can use dd.exe to image memory, but the output isn't compatible with the MS debugging tools, and since there don't seem to be any tools available for really parsing and analyzing this information, it's of limited use.

Other useful resources include MS KB articles on debugging.

Dumping and analyzing physical memory

Well, my research into dumping and analyzing physical memory is progressing. I can't say that I'm finding a positive answer...all I can say is that the research is going well. ;-)

I got in touch with Joanna Rutkowska over at invisiblethings.org about a presentation she gave in Oct '04, and made reference to dd.exe and memory dumps (i.e., crashdumps) created by Windows tools are not compatible. This has been confirmed via other sources.

MS has a tool called userdump.exe (1, 2) that you can use to collect process memory, but it requires that you run a setup program that installs a kernel-mode driver, so it has to be done ahead of time.

An alternative to this kind of crashdump analysis and debugging is LiveKD.

Memory Dump Analysis

One of the things I'm seeing, or should I say, have been seeing for a while, is a move away from the purist approach to forensics, in that actual practitioners are moving away from the thinking that the process starts by shutting off power to the system. I've corresponded with folks using the FSP, and other similar toolkits, be they homebrew, or the WFT.

Besides collecting volatile data, one of the things that's talked about is imaging memory, or collecting a memory dump. For the most part, this has been talked about, but when I've asked the question (as I did at HTCIA2004), "what would you do with it?", most people respond with a blank stare. Then someone way, way in the back (or it sounds like they're way, way in the back...) says, "run strings on it." Okay, that's fine...but what then? How do you associate anything that you find in memory with the case you're working on?

Well, as a start, Mariusz Burdac, over at seccure.net, has released a white paper entitled, "Digital forensics of the physical memory". Now, some of the grammar may throw you off a bit, but keep in mind that this is a start in the right direction. The table of contents of the paper are pretty impressive, addressing some of the issues faced in performing analysis of memory dumps. Over all, I think that the paper is a really good contribution, and a definite step in the right direction. The collection and analysis of information (re: evidence) from live systems is going to become even more important as time goes on.

However, one thing that really threw me was the fact that the author started the paper off by using the FU Rootkit and SQL Slammer worm as examples to justify performing memory dump analysis. The issue I have with this is that the author then goes into analyzing memory from Linux systems...the FU Rootkit and SQL Slammer worm affect Windows systems. The author makes no mention of Windows systems other than to say that the analysis of memory dumps "can be done".

In order to collect the contents of physical memory from a Windows system, you should look into the Forensic Acquisition Utilities from George Garner. The web site includes examples of how to use dd.exe to obtain an image of PhysicalMemory.

One needs to keep in mind, however, that while the system is running, pages are swapped out of memory and into the pagefile (ie, pagefile.sys). Therefore, if you were able to identify the area of memory used by a process, it would be a fragment of the total amount of memory (RAM + pagefile) used by the process. If you're interested in a particular process, I'd recommend using pmdump, instead.

MS does have some documentation regarding using debugger tools to analyze memory dumps. According to MS documentation:

A complete memory dump file contains the entire contents of physical memory when the Stop error occurred. The file size is equal to the amount of physical memory installed plus 1 MB. When a Stop error occurs, the operating system saves a complete memory dump file to a file named systemroot\Memory.dmp and creates a small memory dump file in the systemroot\Minidump folder.

So, maybe that gets us on our way a bit. This is used to analyze Stop errors, but perhaps it can also be used to analyze memory dumps, as well.

It looks like more research is required, as well as some testing. As I'm digging into this, I'd appreciate hearing from folks with regards to what they did, what worked, what didn't work, etc.

Shooting oneself in the foot...

I haven't found any really good malware analysis postings lately, but higB (*secureme blog) came to my rescue and posted about a recent, and personal, incident.

In a nutshell, he infected himself with a Trojan, and then went about figuring out what it did. Reading through it, I see that he did a lot of things right.

One of his comments in particular seemed interesting to me: "system.exe looked normal to me." I'm sure this is the case a lot of times, to a lot of admins. I'm on XP Home right now, and don't see "system.exe", though I do see "System" and "System Idle Process" (via Task Manager). Even using tlist.exe, I don't see anything called "system.exe".

Take a look at his post...what would you have done differently? What things would you have done that higB didn't do? What do you think of his tools and techniques for analyzing the file?

Some help needed with PE headers

The day before yesterday, I started digging into PE headers while looking at some malcode. One of the tools I've been using is PEView, and another is FileAlyzer. Both tools have proven extremely useful in viewing the PE Headers, as well as other information about the file, breaking things down a little more beyond a simple hex editor.

Here's my question, though. Just prior to the PE headers (ie, before the "PE\0\0") is the MS-DOS stub program, and according to everything I've been able to find on the topic, this is put in place by the linker. Evidently, this is a holdover dating back to MS-DOS 2.0, and for some reason is still in use today. This is the section of code that, when you open a PE file in a hex editor, you see something like "This program cannot be run in DOS mode." I've seen variations of this...which leads to the question. Between the notification about running in DOS mode and the PE header, you'll often times see lots of binary data. Sometimes (as with netsh.exe on XP Pro, for example), you'll see the letters spelling out "Rich".

Does anyone know what this is?

I'm fairly sure that it's the contents of the stub program added by the linker, but I'd like to get confirmation on that, and perhaps even see if it's possible to tie a PE file to a particular linker or development environment.

Here's an example of how I'm thinking that this could be used...let's say you've got a case where you're pretty sure that someone developed a program. You've got a copy of the executable (worm, whatever...) and you think that the suspect created it. You may or may not find bits and pieces of code in slack space. But let's say you find a development environment, such as Cygwin or Borland or MS Visual C++. Would it be possible to tie the stub program added by the linker to the development environment, by comparing the MS-DOS stub program in the PE file to the version of "winstub.exe" (or whatever the default is) on the suspect's machine?

Bonus Question: The "magic number" for a Windows executable is "0x5A4D" (or "MZ"). What is the significance of "MZ", and were did it come from?

Schneier on Attack Trends

Bruce Schneier posted an interesting blog entry the other day on attack trends seen by Counterpane's monitoring service. The post seems to be excerpted from his essay, which is an interesting read. His blog entry was also /.'d, along with some supporting information.

Some of the interesting trends that Bruce talks about include such things as "hacking" moving from a notoriety-based, hobbyist activity to out-and-out economic crime. Examples of this include extortion, as well as the rental/sale of botnets.

The data Counterpane collected supports what others have been seeing. Worms and other malcode used to be written as proof of concept, and some of it even got released into the wild. Now, some malware authors are writing code for demonstration, but providing private versions of the code, with greater capabilities, to those willing to pay for it. It seems that folks are learning from history...while writing something that's annoying can be fun and you can get your 15 minutes of fame amongst your friends, one wrong step and you could end up in jail (just ask this guy). So why not take a targetted approach to your attack, remain quiet and patient, and collect information/data for later use? Some malware now has built-in rootkit capabilities in order to hide activity (Trojan.Blubber, Trojan.Drivus, Backdoor.Ryejet).

Another trend Bruce mentions is the increased sophistication in malware. There's evidence that shows worms becoming more intelligent in their reconnaissance and propogation techniques. The Win32.spybot.KEG worm, for example, includes multiple capabilities, in that it performs scans for specific vulnerabilities, can communicate it's findings over IRC, includes a backdoor, get the contents of the clipboard, grab images from a web cam, etc.

Rather than looking at these as separate trends, consider them together. Attacks are coming quicker, and the attacks and malware are becoming more sophisticated. Malware is getting onto the network via some exposed gateway or rogue (ie, forgotten) system, and scanning for specific vulnerabilities. These tools are becoming less "noisy" (ie, looking for specific things, rather than taking a shotgun approach), moving quicker, and include the necessary capabilities to hide from all but the most sophisticated investigator.

Combine this with the continuing trend of IT as a rapidly growing industry (ie, more and more people moving into IT everyday) - which means that every day, there are new/green/un- or under-trained administrators - and you've got a pretty interesting scenario.

The scary part is that the growth of cybercrime combined with the growth of (excuse me for saying this) "security-challenged" administrators and IT managers opens up the investigative arena for explosive expansion. What does this lead to? An old friend of mine recently told me about an issue he had with a computer system, where he had to determine whether certain documents were on the system. He took the Windows XP system to a "forensic expert" who was really just an expert MAC user...who also never found the documents. Also, my friend gave the "forensic expert" explicit instructions to NOT connect the system to a network under any circumstances...and when he went by the expert's office, he found the system connected to the expert's network via RJ-45/Cat-5 cable, and the ethernet activity lights on the system blinking furiously.

The point of all this is that the attackers continue to be lightyears ahead of the victims. The need for training and education in order to (a) recognize that an incident has occurred or is occurring, and (b) do something about it is paramount.

MS Security Document

I ran across an interesting document today at the MS Download Center entitled, "The Security Monitoring and Attack Detection Planning Guide" (in PDF).

So far, all I've given it is a quick glance, but it looks like it has some fairly good information in it. For example, chapter 2 discusses tools for correlating security events. But there's a big "uh-oh" in there, too. The document mentions the Event Comb MT tool used for correlating Security Event Log entries (and ONLY Security Event Log entries) from across machines...but then goes on to state that Event ID 12294 (account lockout threshold exceed on the default Administrator account) is reported to the System Event Log. Doh!

For the most part, it looks like the document really addresses a lot of the common sense things that MS has been pushing for years...things like taking a look at who has Admin privileges in your organization (and why they have it), taking a system-wide approach to design (rather than a band-aid, patch it up approach), etc.

Overall, it does look like a good resource, if for no other reason than for providing Appendix A, "Exclude Unnecessary Events". This is one of those sections that made me go "hhhhmmmm"...if an event is "typical behaviour" and deemed "unnecessary", why was it included at all? Well, at least MS has provided some kind of an explanation of various events, so rather than knocking them, I'll thank them.

Case studies, and T&E again

I was going through the E-Evidence.info site again this morning...the site is updated with new stuff each month...and saw the Internal Investigations Case Study presentation by Curtis Rose. This is a very informative read, even for technical weenies such as myself who really love the "down in the weeds" stuff.

Something really jumped out at me on slide number 10, though. When I say "jumped out", I mean deja vu, because I know I've been here before. The specific statements surround an internal investigation conducted by a sysadmin (and please don't think I'm using this as an opportunity to bust on sysadmins, because I'm not...not this time, anyway):

The investigative memorandum generated by the system administrator was biased and clearly written to substantiate the suspect was responsible

Really? Go figure. I've been here before, where a sysadmin reports on an incident in such a way as to support his original hypothesis...the one he developed shortly after receiving the first pager alert. At 2am.

A basis for much of the document was information from connection logs, which the memorandum indicated were manipulated

I can't begin to tell you how many times I've seen this, particularly in public lists. In the same post, someone who has "conducted" an "investigation" will state the source of evidence as being authoritative, but also suspect.

People, you can't have it both ways. Is it just me?

One final bullet that I'll comment on is:

What limited analysis was conducted was performed directly on the victim systems

Again, I can't tell you how many times I've seen or heard of this..."Task Manager didn't show any unusual processes."

"So that one process that looks like 'svchost', but is really called 'scvhost'...that one isn't 'unusual' to you?"

So what's with the rant? It's a need for education, folks! Education of whom? Well...get ready for this one...of IT Managers, from the C-level down. If your organization isn't hiring the right people, and the right number of people, to staff your IT department, they're doing themselves a disservice. Of course, some places may choose to do this as a sort of self-imposed governor (like one of those things they used to put on U-Haul truck engines so they wouldn't go over 65 mph, no matter how hard you pushed on the gas pedal).

When I say, "the right people", I'm referring to folks who don't necessarily look at their day job as just that. It seems sometimes that the job market is tight...so having someone who doesn't even try to keep up on things, even on their own time, doesn't make a great deal of sense when there're lots of people out there who do, and would want that job.

But you can't rely simply on self-education and -training. Hiring the right numbers of people will allow for things like taking time off for training and continuing education. One of the approaches I found to be very effective was to come on-site to provide my training. This way, admins were out of the office and engaged in training, but they weren't completely out of pocket. In fact, in at least one case, an Exchange admin used the new skills he'd learned to solve an issue over lunch during the second day of training. I've even recommended splitting the training into "port-starboard"...instead of sending everyone off-site for two days, I'd come on-site for four days and teach the course. The first two days, I'd train half of the staff, and then train the other half during the second two days.

My point is that there are a variety of options available for training and education...it just depends on where you choose to look, and how badly you want it.

Finally, for the guys and gals who are in those positions where continuing education and advancement in the IT field is non-existant...can I recommend Monster.com?

IR Tools

I ran across something this morning over on Mike Howard's blog about the use of netsh in troubleshooting the XP SP 2 firewall. The blog entry points to an MS KB article entitled, "Troubleshooting Windows Firewall settings in Windows XP Service Pack 2". I'm always on the lookout for good tools to use, and this one looks great for getting some pretty good information from XP systems, particularly concerning the firewall.

It may not be abundantly clear what I'm talking about if you read the above KB article, so start by taking a look at netsh.exe on an XP box by typing "netsh /?" at the command prompt. There are several options available, but if you're interested in just collecting information about the XP firewall, type "netsh firewall show /?".

Training, time, and job responsibilities

As sort of a crossover from my last post, I thought I'd blog about some of the responses I received directly to my email inbox...

The biggest thing I'm seeing, from the responses as well as my own personal/professional experience, is that security takes time...and time is usually something in short supply. IT managers (and by this I mean all the way up to C-level folks) don't count on the fact that staying abreast of security issues takes time. Even if you're in an all-MS shop, with only Cisco routing equipment (as I've been), staying up on the latest viruses, effects of patches (and the systems they apply to), etc., all take time. Usually what ends up happening is that either (a) "security" is an ambiguous assignment to an already-overtasked and under-trained admin, or (b) the security guy/gal is seen sitting around staring at their monitor, so they're asked to help out with everything from helpdesk to router installations.

Now and again, I get perturbed at the content I see in posts to public forums. One of my pet peeves is the admin who posts to one of the SecurityFocus lists, and respondants ask questions for clarification...yet through the life of the thread, the original poster (OP) never responds. In the few instances where I've been able to track the OP down via direct email, 99.99% of the time, the reason for the disappearance is that something else more important came up (ie, Shiny Object Syndrome).

Another pet peeve is the OP who will ask a question that could have been answered, or perhaps simply better phrased, had the OP done some research of their own prior to posting. Sometimes a simple Google search is sufficient, other times simply putting together their own test would have answered their question.

So, what do these peeves of mine have to do with the subject at hand? Well, the first has to do with time...in many cases, it seems that admins are turning to public forums for their answers, but don't want to give out too much information about their networks or the situation they're dealing with. In many cases, troubleshooters/respondants need simple things such as the operating system/application name and version...which the OP may not feel comfortable giving out. However, the real issue is the fact that the OP is posting in the first place...they obviously have an issue they're dealing with but don't have the time to learn basic troubleshooting skills, troubleshoot the problem themselves, or get on the phone with techsupport for the application in question.

The second peeve has to do with education and training, at least indirectly. Well, now that I think about it, they both do. In a lot of cases, when I've asked people why they haven't done their own research, the ones that don't feel like they're being bullied (and they're not) will tell me that they don't have the time. There's no harm in asking questions, but sometimes questions can be answered or better phrased if you reason things through, do a little basic research, and even try something for yourself.

Taking training and education a step further, I've often wondered why there are so many people who lurk on public lists, some who post, and so very few who publish anything. When I say publish, I'm not necessarily talking about writing a book or getting an article published...what I'm talking about is doing some testing, documenting the methodology so that it can be duplicated and verified, and then writing up your results. I think that if there were more of this, the computer forensics community itself would be better served, as a whole. However, when I've asked about this, I've been told that the "requirements" are too rigorous...it takes too much time, and many people actually write so poorly, that they don't want to have to go through the headache of constant editing (and the sense of rejection they may feel when someone corrects their spelling and/or grammar).

You know what? You don't have to be a PhD to get something published. Actually, it's pretty easy...of course, I'm saying that as someone who's already done that (and tried to help others do the same). Not only is the act of getting something published educational in and of itself, but just going through the process of discovery teaches us a lot. I set up a testing methodology before where I've had to go back and redo everything, because after I was done I realized that there was something else I could have done. And then once we put our methodology and findings out there, we're all better for it.

[rant off]

The need for training

I ran across something interesting this morning...it's not new, but it's the first time I've seen it. I was checking out what's new over on the E-Evidence site and somehow made it to an article that quoted Kate Seigfried about a study she'd conducted. The article said that cyberforensics is a discipline still in it's infancy.

Here's an interesting quote from the article:
In academia, Purdue University’s Center for Education and Research in Information Assurance and Security recently produced a study on the state of the computer forensics’ science. The study found forensic investigative procedures at present were still constructed in an informal manner that could impede the effectiveness or integrity of the investigation. Unfortunately, the study pointed out informal nature of the procedures could prevent verification of the evidence collected and might diminish the value of the evidence in legal proceedings.

Forensic investigative procedures are still constructed in an informal manner? What? The article isn't explicit enough to really say a whole lot, but I know that several law enforcement agencies will document their procedures, which other agencies will use as the model.

The article goes on to say that Eugene Spafford sees two key questions:

1. How do we formalize the process of cyber forensic evidence gathering and analysis using appropriate and rigorous scientific method.

Evidence gathering is the easy part. For the most part, there are formalized processes out there for imaging drives. There are issues that need to be addressed, such as terabyte storage capacities (after all, where are the golden eggs kept these days but in jinormous databases??), RAID, etc., but these can be overcome.

Now, finding evidence is a different matter...that involves search and analysis techniques that haven't been formalized. Why is that? Well, I have a couple of thoughts on that, but would like to hear from you with regards to your thoughts on the matter.

2. How do we augment information systems so as to produce better audit and evidentiary trails while at the same time not exposing them to additional compromise.

I'm not sure, but it would seem to me that making use of the inherent capabilities of the system would be a good start. What I find odd is that there are so many "hardening guides" out there for Windows systems, and we still see these systems being compromised. When you talk to admins, they don't seem to have the knowledge themselves, and some say that there are just too many guides out there - which one or ones are "authoritative"? Point them to the NSA guides (after all, who's more "authoritative" than the NSA??) and many of them will blindly install the settings, and then wonder why they can't do anything.

I think what he's referring to is to design and build systems (remember the Orange Book of the Rainbow Series??) with more robust auditing built in...don't make it something the admin has to add or configure separately, because it won't happen.

[rant]
On a side note, it still mystifies me why MS would produce a "network operating system" that has NO inherent capability to get audit logs (i.e., Event Logs) off of systems. Even with the old NT-style domains, BDCs wouldn't automatically send their logs to the PDC or a designated server...you had to install separate software. How is that a "network" os?
[/rant]

But I digress...

I started looking and found the study referred to in the article (from 2003) entitled, "The Future of Computer Forensics: A needs analysis survey".


This study, conducted by Marcus Rogers and Ms. Seigfried, provides some interesting information that I would think is still true today, almost two years later. Their survey found that training, education, and certification is the top issue mentioned by the respondants, while lack of funding was the least reported issue.


Training, education, and certification? Lack of formalization? Well, they're probably right. I've been to conferences before where one presenter will have "???" in an area of their presentation (the specific example in mind involved NTFS ADSs, OLE documents, and where file summary information is kept...), while another presenter at the same conference had detailed information and even a demonstration that answered the question. The first presenter was a LEO, the second was a private citizen.


A couple of years ago, I was talking to a guy who provided computer forensics training to LEOs. He asked me if NTFS ADSs could be transferred over the network. I told him via file sharing, yes...but not via other protocols, such as FTP and HTTP. He bet me that they could, so we set up a demonstration. Turns out I was right...I knew the answer because I'd already done the research.


My point is that there are folks out there doing reproduceable, verifiable research...but it doesn't seem to be getting out there, even if it's presented at conferences, written into papers, articles, and books. Why is that?

Data Reduction, revisited

I thought I'd take a moment to revisit the topic of data reduction.

What steps are you using to perform data reduction? What are you doing to sort the wheat from the chaff, as it were?

Some of the data reduction steps I'm aware of include:

• Hash sets - look for known good or known bad files
• File signature analysis - look for files whose header information doesn't match up nicely with the file extension
• File version info - parse binary files for file version info, and flag those that don't have any
• Keyword searches - depending on the case, look for files/sectors containing certain key words

Hash sets can be used to sift through those hundreds or thousands of operating system files...the ones that we know are good, and therefore we're not interested in them. You can also use hash sets to look for known bad files, as well.

What else are folks doing?

Remember the KISS principle

How often do you see someone post to a list with a question that they could have answered themselves had they bothered to test it out?

I ran across two recently...one that didn't have much of anything to do with Windows forensics at all, but applies more to human nature...

The first was a question about files changed when a system is recovered. You know, for some reason, you can't boot a system, so you pop the CD in and reinstall the operating system...in doing so, what files are altered in the process. I suggested to the original poster (OP) that he try running a 'simple' test to find out, and the response I got was the tests weren't all that simple. His reasoning was that you'd have to check every version of Windows.

Basically, it sounded to me as he was arguing himself completely out of discussion. After all, who out there knows ahead of time when they're going to have to recover their system, and runs a integrity checking tool ahead of time?

My suggestion to him was to keep the scope of the issue small...pick an exemplar system such as XP Home or Pro, and define your problem and methodology, in such a way that the testing process you use is repeatable. For example, say that all you have available is Windows XP Pro. Note the patch level (ie, service pack, any additional patches/hotfixes)...you can do with with psinfo or WMI. Run an integrity checking tool on the files in the system32 directory...use something like FCIV or md5deep (from Jesse Kornblum). Then 'recover' the system and re-run the integrity checking tool.

You may also want to get file versioning information from the binary files in the system32 directory, as well.

Correct me if I'm wrong, but to me, asking the question "are files changed when you recover a system, and if so, which ones" really doesn't do a lot to progress the community, particularly when no one's going to do any testing.

The second post had to do with PDA forensics...the OP asked if anyone had experience using dd for PDA forensics. It was kind of an odd question, as he also stated that his employer was just about to buy (or had just purchased) the Paraben product. The kicker to the post was the statement that the OP made about not finding anything on Google. A quick search turned up info at E-Evidence.info, as well as

I've got a question about a Registry value...

The other day, on one of the lists I am subscribed to, someone raised an interesting point. This issue addresses the following Registry key:

HKLM\System\ControlSet00x\Enum\IDE

Beneath this key are subkeys that are specific to IDE drives on the system. Similar to the USB storage device key, this key has device instance ID subkeys, and beneath each of those are unique instance ID subkeys. Each of these unique ID subkeys has a Registry value called "UINumber".

Now, the issue that was brought up was this...when Windows XP is installed on a system, it looks for other drives that have operating systems installed and assigns the UINumber value accordingly. Therefore, if the UINumber value for a drive is other than 0, that should indicate that XP was installed on a system with other operating systems...right?

The poster stated that he had done testing that demonstrated this assumption. What I'm looking for is any documentation regarding how the value is set. Yes, I've done some exhaustive searches on Google and at the MS site, and haven't found anything that addresses how the UINumber value is set for hard drives.

Anyone got anything?

Seltzer on Rootkits

I received a link to Larry Seltzer's new article on rootkits this morning. It's dated 20 April, so why do I mention it?

While the article comes across as a "too-little-too-late" rehash, I do think that it is important to keep these things in the mind and eye of the public. However, I think that it's important to do so with some responsibility. The article starts down that road by mentioning that, oh, yeah, by the way...for a rootkit to take hold, it first has to get on your system. Yeah, well...ok, so most normal users may not really be aware of that.

The article also mentions the Strider Ghostbuster tool from MS...but makes no mention of the fact that this tool really isn't available. Note that the article clearly states that the tool "...works by listing..."...rather than "will work" or "should work". Either way, the article easily misleads the reader.

Is there a cause for fear? Yeah, sure...without a doubt. But that fear should be tempered with knowledge. The fear should not be so much that it causes fear and paralysis...with knowledge, that fear should be akin to that nagging feeling you get when you're leaving your house in the morning. Did I remember to turn off the stove? Did I turn off the water in my sink? Did I remember to wear pants?