Terry Cutler, CTO of the Canadian firm Digital Locksmiths, had an interesting article published in SecurityWeek recently. The article is titled, "You've been hacked. Now what?", and provides a fictional...albeit realistic...description of what happens when an incident has been identified. A lot of what is described in the article appears to have been pulled from either experience (IR is not listed as an available service on the company web site) or from "best practices". For example, in the article, the assumption appears to be made that if a compromise occurs, corporate cell phones must be assumed to have been compromised (with respect to calls...email wasn't mentioned).
The article talks about not disconnecting systems, which in many cases is counter to what most victims of a compromise want to do right away. However, I completely agree with this...unfortunately, the article doesn't expand beyond that statement to say what you should do.
Now, what I do NOT agree with is the statement in the article that you should "get help from an ethical hacker". First off, given the modern usage of the term "hacker", the phrase "ethical hacker" is an oxymoron...like "jumbo shrimp". While I do agree that some of the folks performing "ethical hacking" are good at getting into your network (as stated in the article, "Ethical hackers are experts at breaking into your system the same way a hacker will."), I don't agree that this necessarily makes them experts at protecting networks, or more importantly, scoping the incident and determining where the attack came from.
In the years that I have been an incident responder, the one thing that consistently makes me a cringe is when I hear someone say, "...if I were the hacker, this is what I would have done." Folks, where that thinking takes you can be irrelevant, or worse, can send your responders chasing way down rabbit holes. Think CSI, and go where the evidence takes you. I've seen instances where the intruder had no idea what organization he'd compromised and simply meandered about, leaving copious and prolific artifacts of his activity on all systems he touched. I've also seen SQL injection attacks where, once in, the intruder was very focused in what they were looking for. Sometimes, it's not so much about the corporate assets as it is loading keystroke loggers on user systems in order to harvest online banking credentials.
What you should be doing is collecting data and following the evidence, using the information you've collected to make educated, reasoned determinations as to where the intruder is going and what they are doing. Do not make the assumption that you can intuit the attackers intentions...you may never know what these are, and you may chase down rabbit holes that lead to nowhere. Instead, focus on what the data is telling you. Is the intruder going after the database server? Were they successful?
The best way to go about establishing an organic capability for this sort of work (at least, for tier 1 and/or 2 response) is to establish a relationship with a trusted adviser, someone who has experience in incident response and digital forensics, and can guide you through the steps to building that organic capability for immediate response.
At this point, you're probably wondering what I mean by "organic", and why "immediate response" is something that seems so necessary. Well, consider what happens during a "normal" incident response; the "victim" organization gets notified of the incident (usually by an external third party), someone is contacted about providing response services, contract negotiations occur, and then at some time in the future, responders arrive and start to learn about your infrastructure so that they can begin collecting data.
The way this should be occurring is that data collection begins immediately, with incident identification as the trigger...if this doesn't happen, critical data is lost and unrecoverable. The only way to do this is to have someone onsite trained in how to perform the data collection.
A lot of local IT staff look at consultants as the "experts" in data collection, and very often don't realize that before collecting data, those "experts" ask a LOT of questions. Most often, the consultants called onsite to provide IR capabilities are, while knowledgeable, not experts at networking, and they are definitely not experts in YOUR infrastructure and environment. |
I'm not even talking about getting to prosecution at this point...all I'm talking about is that data that is necessary to determine what happened, what data may have been compromised is quickly decaying, and if steps are not taken to immediately collect and preserve this data, there very likely will be a significant detrimental impact on the organization. Now, the only reason that this isn't being done now is because onsite IT staff don't have the training. So, work with that trusted adviser and develop a process and a means for collecting the necessary data, and documenting it all.
Going back to the SecurityWeek article, I completely agree...don't disconnect the system as your first act. Instead, have the necessary tools in place and your folks trained in what to do...for example, collect the contents of physical memory first, and then do what you need to do. This may be to disconnect the system from the network (leaving it powered on), or making an emergency modification to a switch or firewall rule in order to isolate the system in another manner. If the system is boot-from-SAN, you may also want to (for example) have a means in place for acquiring an image of the system before shutting it down. Regardless of what needs to be done, be sure that you have a documented process for doing it, one that allows for pertinent data, as well as business processes, to be preserved.
Ever wondered, during an incident, what kind of person (or people) you're working against? This eWeek article indicates that the impression that hackers are isolated, socially-inept "lone wolf" types is incorrect; in fact, according to the article, "hackers" are very social, sharing exploits, techniques and even providing tutorials. Given this, is it any wonder why folks on the other side of the fence are constantly promoting sharing? The bad guys do it because it makes sense, and makes them better...so why aren't we doing more of it?
Definitely agree with this blog post. I never understood why companies that perform pentests and other security assessments get hired to do IR & forensics works. Most of the time I have seen this in action, the consultant doing the IR has 0 experience in it and only know whats to look for from what their pentests generate (web logs, login history, etc), very high-level things, but wouldn't know what to do if given a registry hive or even just an offline disk image without a bunch of tools to do the work for them.
ReplyDeleteI also cringe whenever I hear the term 'ethical hacker'. Coming from management, it usually means the company/its decision makers still have some unease over hiring "hackers" -- meaning they don't understand security very well. I cringe just the same when a person touts themselves as an "ethical hacker" as it usually indicates their only experience with security is clicking buttons in a GUI instead of security having been a passion/hobby, and they went corporate to make a living. Obviously people in the latter group produce much better results and are more valuable.
Several years ago, at the first or second SANS Forensic Summit, I was on a panel where we were discussing IR. One of the questions was, "What should a customer be looking for in their responders?", and one of my comments was to ask if response is what the responders do. I know, sounds kind of weird...but how many responders do response full time? When you take your car in to get worked on, do you want the guy working on it to have just come off of a 2 month contract as a carpenter?
ReplyDeleteI do firmly believe that, for the right people, experience in other areas (vuln assessments, pentesting, etc.) can provide valuable insight, but in my experience, it ends up providing tunnel-vision and myopia, instead.
I still think "ethical hacking" is an oxymoron, of sorts, and that anyone who wants to gain access to your assets is going to do things that you'd look at as perhaps "unethical".
Excellent post Harlan. Every time I read a post from yourself I learn something new and get exposure to a new tool. Fiddler just helped me identify the .exe that was downloaded to a PC that became infected. I could not locate it until I used Fiddler and saw the binary in the traffic so easily with Fiddler. Very cool.
ReplyDeleteRegards,
Brian P.
Brian,
ReplyDeleteThanks. Not sure where the connection to Fiddler came in, but cool.
What did you find out? What was downloaded, and where did it go?