Friday, December 17, 2004

Am I wrong on this?

I have to wonder I wrong to espouse my opinion with regards to incident response activities, particularly on Windows systems?

The frustration I run into is something I see both online and at work. Here's the admin in charge of a Windows system (could be a dual-hatted Linux admin, or an MCSE) finds that something suspicious is going on. He does some cursory checks, such as a port scan, perhaps, and with exception of a few vague statements ("I didn't see anything suspicious in the Task Manager"), provides no further information. At that point, he now wants to know if the box has been "hacked" (I cringe at the use of that word).

My point about this is, why can't people simply do the few simple things to gather information before making a decision, either to blow away to box, or post online? So the question I come to, is, is it wrong for me to espouse a view that an academic approach needs to be taken to incident response? When I say "academic" approach, what I'm referring to is collecting and analyzing facts. It's simple, people...really. Cause and effect. I can't get into my car...not because it's been "hacked" (shudder), but because I'm using the wrong key. Therefore, when your Windows 2003 server suddenly becomes a warez server, one thing to do would be to start checking your logs. If you've got IIS installed, check the web and FTP logs. That's a good start.

As a side note, I just want to say that I once worked at a large telecommunications company, and while investigating just such an incident, I requested the IIS 5.0 web logs from the Windows 2000 server...and received a zipped archive containing three .evt files. Uh...guys...IIS does not log to the Event Log.

Finally, on that topic...why do so many admins seem to think that the only thing Windows systems are good for is to be rebuilt? The default behaviour seems to be, "I found something I don't recognize, so I'm going to blow the system away and rebuild it." And what good does that do, particularly if not long after you go live, the system is re-infected or re-compromised? No root cause analysis (RCA) is done, so how do you know what caused the system to start acting up? And remember folks, not all issues are fixed with a patch...I've handled a great many warez incidents in which the FTP server was set up to allow the anonymous user to write to the drive. As yet, I haven't seen a patch for that.

Another thing I see is folks talking about malware or processes being "hidden from Windows". "Hidden" how? Most malware to "hides" from the administrator by not using names such as "badsoftwarehere.exe" and "nastywarez.exe"...oddly enough, this is still highly effective. Forget rootkits...for the vast majority of exploits and "hiding", nothing that sophisticated is necessary. I've done analysis on systems where I've looked over the admin's shoulder at the Task Manager, with him pointing and saying, "See? See? Nothing!", and I can clearly see two spyware processes. Ugh. I know I've got superpowers, but that's not one of them. The processes aren't "hidden", you're simply incapable of seeing them...there's a difference.

Am I wrong to ask people to collect facts, and be specific?


Anonymous said...

Nope, you're not wrong at all. Frustrates me as well to see blatant disregard for the quest of knowledge. =)

Joe said...

I realize this is an old post, but this is a topic I feel is still really relevant today. System Administration is something that usually comes along with preconceived notions by most... IT / Security elites. Do you review logs, lock down your servers, have activity baselines, etc.
While all of these tasks are important and should be standard.... I’ve been in the field for some time, and have found that to be rare for these things to occur. Unfortunately most Admins I have come across excel in running just the systems they work with, in small to medium size companies, the list of hardware isn’t very varied, or if it is varied, its because the equipment is 5 years old and cheap. The Admin might not know that IIS logs are not in the event viewer, but does know the printer on the executive floor backwards and forwards.
What I am getting at, is I have always seen the trend for most System Admins to be good at keeping things running, and the burden of forensics (even simple steps) has been left to the security department, or a contractor/professional. This is something I have struggled with in my career, as its hard to find a company that will pay you for being security minded, instead of a "Yes" man / just keep it up and running. For example not too long ago, a company server had been compromised (non production website defacement) and I was told to stop analyzing the root cause because the receptionist outlook is crashing. While Im not trying to down play an end users impairment, the security concern isn’t there in most places UNTIL it affects the bottom line.
Its a real shame too, because when a system/device/server is compromised, its usually the admin that takes the heat.
For the reasons mentioned (sorry for the book of a post) its always refreshing to me when I see security guides posted for administrators, who for instance might not be familiar with analyzing malware with a hex editor, or able to understand creating customized IDS signatures.