Tuesday, July 14, 2009

SANS Summit Question

During the recent SANS Forensic Summit, there were a number of questions handed to Rob during the panels that went unanswered due to time constraints. After the IR panel, I took a look at some of the yellow index cards with questions and grabbed one in particular to answer. The question I found was (no author info was available):

What is the worst thing an IR team internally will do?

As I kept thinking about this, I kept coming back to two answers...either do nothing at all, or do too much of the wrong thing. Now, I'm assuming with my response that an incident has been detected, the IR team has been called into action, and some kind of response process has been initiated. At that point, you very likely have something that occurred to indicate that there is an incident, so you may very well have sensitive data being actively exfiltrated from the environment, so doing nothing at all could be extremely detrimental/harmful to the organization.

On the other hand, grabbing systems, running AV scans, deleting files, even wiping systems and reinstalling them can also be harmful to the organization. In a typical breach situation, the questions that need to be answered are:

1. Was the system compromised?
2. Did the system contain some kind of sensitive data?
3. Did "yes" to #1 lead to the exposure/exfiltration of #2?

If you destroy the indicators or "evidence" of what occurred, and cannot therefore determine the answers to these questions, where does that leave you? If you're just re-installing everything with no idea of the attack/infiltration vector, how do you protect yourself in the future? Heck, how do you even know that you fixed the issue, particularly if you have no idea how long the bad guy or their malware have been on the systems? You may be re-installing the malware itself!

So then I thought to myself, why do IR teams (both internal and external) do this sort of thing? In my experience, a lot of times it has to do with assumptions that are made...very often, incorrect assumptions. Due to lack of knowledge, skills, tools, and/or training, IR teams are very often under the gun to provide answers to management, or just get things working again. If you don't know what to look for, it's simply easier to make assumptions without any hard data and proceed on from there. I mean, really...if you destroy the data, who's going to be able to question you? I'm not saying that this is being done maliciously...what I am saying is that I have seen both internal IR teams as well as consultants make some very unfounded SWAG statements about an incident with no data to back them up, and proceed on from there. Very often, I'll just stand there, shaking my head, as they charge off into the sunset.

So, what's myanswer? The question was about IR teams, and not specifically management...so I'd have to say that, IMHO, the worst thing an internal IR team can do is NOT take it upon themselves to develop their own knowledge and skill sets. "I don't know what to do because management won't send me to training" should be an indicator that you've got the wrong people on your team...both for internal teams as well as consulting companies that provide response services. Not everything is going to be included in a class, and having to sit in a classroom to learn something means that the team member (or members) are unavailable for that time. More importantly, if you do get the opportunity to attend a class, but are unable to process the information and use it in your environment, that basically means that you got a nice paid vacation...and to be honest, I'd much rather have one of those some place other than in a classroom!

In summary, the worst thing an IR team can do is not learn from their mistakes, and not take it upon themselves to expand their skill sets and improve their processes.

Thoughts?

12 comments:

Unknown said...

In IR, field experience is worth so much more than lab/class experience. I started taking a person out with me on simple cases to build up their knowledge set and experience levels. Now I've got them acquiring servers, rebuilding raids and doing full blown investigations. It's very cool to have someone that wants to expand their skillset.

I want to expand on your first part of the summary. The worst thing an IR team can do is not learn from their own mistakes or help the victim organization learn from theirs.

H. Carvey said...

Great comment.

Many times after an engagement, someone may be a bit overwhelmed by all of the opportunities out there for developing their knowledge and skill sets. The best place to start, however, is with your last engagement...what could have gone better? What could you as an individual or as a team done better?

Dave Hull said...

Good thoughts. Every team should include a "lessons learned" phase in their incident response methodology. It doesn't have to be formal, but it should be on the list of procedures for incident response.

Another frequent mistake I've seen is the incident responder who comes into a situation and alienates (or worse) the people closest to the problem, be they sys admins, DBAs or developers.

As incident responders, it's essential to have good rapport with those folks because you're going to need their help.

Whatever happened to those remaining Summit questions?

Btw, loving WFA 2E.

H. Carvey said...

Whatever happened to those remaining Summit questions?

That's a good question for Rob Lee. He encouraged me to take the one I took...so I guess the answer to your question would be, "You left them there."? ;-)

e0n said...

Yeah, the thing that irks me the most is when an analyst says that they can’t do their job because they did not get any training. True that training is very important; however as a professional you need to rely on yourself. There are more than enough resources, books, blogs etc. to learn from, i.e. WFA 2/e :P. I always enjoy working with people that take it upon themselves to learn the skills they need on their free time/down time or on the job. In this field, I feel it is required. Also the more you learn on your own and show worth to management, the more they want to send you to training. Just my two cents.

Dave Hull said...

Hm. Maybe Rob was trying to tell me something... ;)

Bugbear said...

Great Post!

I brought this very subject up to management about three weeks ago. I am currently putting together the company's IR Plan and Procedures from scratch and am dealing with new MA laws surrounding PII. The days of running an AV and walking away are over.

Also reading WFA 2E - great book, keep up the good work!

Bugbear said...

Great Post!

I brought this very subject up to management about three weeks ago. I am currently putting together the company's IR Plan and Procedures from scratch and am dealing with new MA laws surrounding PII. The days of running an AV and walking away are over.

Also reading WFA 2E - great book, keep up the good work!

Anonymous said...

Couldn't agree more with the comment about no training not being an excuse for not being able to perform IR duties.

There's all sorts of free tools and resources available for learning IR, all that's required is that you go looking for them and put in the effort to use them. There really is no excuse.

ddewildt said...

I think another important aspect is to decide who is running the incident. I have been involved in incidents in the past where there was no clear person or team running the incident, and rather people haphazardly running off and doing their own thing. It gets very hard to track and can ultimately prove detrimental to the reponse activity and potentially destroys evidence that is needed down the track.

I'd second the comments about good rapport with the people running the system - they are going to know the most about it and you as a responder can not be as effective without their help. It gets a bit hard when its their fault your in the situation though :)

H. Carvey said...

ddewildt,

...decide who is running the incident...

I completely agree that this is important, but from my perspective, this is a function of management (ie, ensuring proper leadership and processes/procedures are in place...), and the question was specifically about teams...which is why I didn't address it.

But like I said, you're right, and you do bring up an excellent point. One of the biggest things I've seen as a result of a lack of leadership in IR has been the rampant spreading of incorrect information, either because the necessary response activities weren't performed correctly (or at all), or simply because it's easier and more fun to speculate about something ("...the malware has a keystroke logger and rootkit in it...") than it is to actually do the work to get the data.

Anonymous said...

Not learning from your mistakes and reviewing (and documenting) what could have been done smarter with your peers is IMHO a big mistake. Every assignment is a huge opportunity to learn and improve. In this field you never stop learning. Regards, James