Carbon Black

I was recently afforded an opportunity to download and install the Carbon Black (Cb) standalone server; for testing purposes, I installed it on a Windows 7 host system, and got it up and running right away. 

If you're not aware, Cb is a small sensor that you can load on a Windows system, and it monitors executables being launched.   When a new process is started, information about that process is monitored by the sensor, and sent to the server for correlation and presentation.  The server can be maintained by the Kyrus guys, or it can be maintained within your infrastructure.  If you choose to maintain the server yourself, that's fine...it just means that you're going to need to have someone work with the tool, get familiar with it, and monitor it.

Getting Cb set up is easy.  The process of generating a license and getting the necessary sensor from the guys at Kyrus was simple, straightforward, and very quick.  From there, I rolled out the first sensor to a Windows XP VM that I set up as a guest on the Windows 7 host system via VMPlayer.  Shortly after installing the sensor on the target system, I began seeing events populating the dashboard, just like what I saw in the demo I attended in July.  I even began doing things on the system that one might expect to see being done during the recon phase of an incident, such as running some native tools to start mapping the network just a bit.  My "infrastructure" is a bit limited, but I got to see the Cb functionality first hand.

It's clear that a great deal of thought and effort has gone into the creating Cb, as well as structuring the interface, which is accessible via a browser.  I found that the interface and functionality is far more intuitive that other, similar tools I've seen, and in very short order, I was able to narrow down the information I wanted, based on my experience as an incident responder.

That's another thing I like about Cb...I can use the experience I've developed as an incident responder to get timely answers, and I don't have to learn someone else's framework or methodology.  For example, I look at the dashboard and see a "suspicious" process...in this case, I'd run "ipconfig /all" on the XP VM system...and all I have to do is click on an icon to see the parent process (in this case, cmd.exe).  I could also see loaded modules and files that had been modified.  I could even get a copy of the new executable that was 'seen' by the sensor.  Future enhancements to the sensor include monitoring of network connections and Registry keys.

All of this is very reminiscent of the demo...consider a "normal" incident involving a browser drive-by or some phishing attack against an employee.  Having Cb installed before an incident is the key, as it would allow a responder to very quickly navigate through the interface and once a suspicious process is located (based on a time hack, process name, or in the near future, network connection...), the responder can quickly identify the parent process or any child processes.  So why is this so special?  Well, most times for me, as a responder, I don't get called until after an incident occurs...which means processes have long since run and exited, and there may even have been efforts to clean up (ie, files deleted, etc.).  However, with Cb, there would still be a history of process execution, copies of the EXEs, etc.  In the case of July's demo, there were three stages to the attack, the first two of which launched and exited once their work was complete.  Further, of the three EXEs, AV detection of each was spotty, at best.

Having something like Cb deployed in your environment before an incident occurs is really the best way to approach not just using tools like this, but incident response in general.  In addition, Cb has a range of other uses...just ask the Kyrus guys for their case studies and stories, particularly the one about the CIO who used Cb to save thousands of dollars on Office licenses, based on actual, hard data (Deming, anyone?).

Cb is also a valuable tool to deploy during an incident...it's like sending out a bunch of lightweight little LP/OPs (listening or observation posts) to cast a wide net for data collection and correlation.  Under normal conditions, responders need to start taking systems down or performing live acquisitions, and then start analyzing those acquisitions, along with log files, network captures, etc., in order to begin scoping the incident and identify other systems that need to be acquired and analyzed.  Deploying Cb could (depending upon the type of incident) provide more information in a quicker manner, and require fewer resources (ie, fewer analysts/responders to send on-site, pay for travel, lodging, etc.), and reduce the overall impact on your infrastructure.  Deploying Cb in conjunction with F-Response would REALLY up your game to a whole new level.

Carbon Black is a tool that should be in your arsenal, as it changes the dynamics of incident response, in favor of the targets and responders, and takes away a lot of the advantages currently enjoyed by intruders.  If you have an infrastructure of any size, you should be calling the Kyrus guys.  It's not just large infrastructures that are being targeted...don't believe me?  Spend a rainy Sunday reading through Brian Krebs' blog, and then, if you can stop crying, look me straight in the eye and tell me sincerely that you're below the bad guy's radar.

In my experience as a responder, when a call came in, we'd start collecting information, not only about the incident, but we'd also have to get information from the customer that we'd use to populate the contract, so we'd have a couple of things going on in parallel.  Even so, it would still take us some time to get on-site (6 hrs, sometimes out to 3 or more days...), and then we'd have to start collecting data.  Even though we'd asked for network device and firewall logs to be collected, in some cases, it was only as a result of the incident itself that the customer found out that, "hey, wait a minute, what do you mean we aren't logging on the firewall (or web server)?"  I've had only one instance where I showed up and some data had actually been collected...in every other instance, when an incident occurred, the customer was completely unprepared and we didn't start collecting data until someone got on-site. 

Think about that, as well as any other incidents you may have encountered, for just a moment.  Now, imagine what it would be like if the customer who called you already had a contract and a relationship in place, and you'd helped them install Cb.  With the contract already set up, that's one thing you don't have to deal with...and with Cb rolled out, data is already being collected.  So, while they're on the phone, you can begin to assist them, or you could VPN into their infrastructure and access the Cb server yourself.  If a team needs to be deployed to assist, then you're already collecting information (if you have F-Response rolled out or the local IT staff has the training, memory and images may also be collected) even before the responders are able to get airline tickets!

Cb is like Burger King for IR...have it your way!

Thoughts on IR

Those of us in the security community like to share ideas through analogy; I'm sure that's to convey technical issues in an understandable (to others) manner.  As a former military member, there are a number models that I use and refer to in analogies, particularly when communicating to other former military members.

Along those lines, something struck me the other day...the Internet is very much like the ocean, and organizations connected to the Internet are like ships on the ocean.  For the most part, ships (and submarines) are designed to do well on the ocean, within their limitations.  However, like the Internet, there are risks involved and potentially negative events (attacks, etc.) can originate from anywhere (above, or on or below the surface of the ocean, and from any direction) at any time.  There are a lot of events that occur on the Internet all the time, and many have no effect at all on organizations, both large and small.  Some only affect smaller organizations, while larger organizations are not affected at all by these events.

However, the Internet is not the only origination point for negative events.  Devastating events can also originate internally within a ship, just as they can with an organization or company.  As such, internal and external threats are well understood by the captain, and that understanding is subsequently conveyed to the crew.  So, not only do ships have things like radar, sonar and manned watches to protect them from external threats, but there are internal monitors, as well...gauges to monitor pressure and flow rates in pipes, etc.  There are also crew members who monitor these gauges, and keep track of the state of various functions aboard the ship at all times.  Minor events can be detected and addressed early, before they become major incidents, and more significant events are detected and the appropriate individuals warned 

Another thing to consider is this...the risks of operating on the ocean are well understood, and that understanding has guided the construction of the ships themselves.  US Navy ships have, among other things, watertight compartments that can be sealed, preventing fire or flooding (the two primary threats to most ships at sea) from expanding.  For an example of this, consider the sinking of the RMS Titanic (read the first paragraph of the Collision section) versus the bombing of the USS Cole - even with a hole in the hull at the waterline, the Cole did not sink.

Even though all of these risks are understood and planned for, Navy ships still have damage control (DC) teams.  These a members of the crew with regular jobs on the ship, but they are also trained to respond effectively when an incident occurs.  That's right...most naval personnel get some training or familiarity with damage control and what it takes, but there are individuals specifically designated with DC duties.  The leaders and members of the DC teams are identified by name, and they all have specific responsibilities, and they also understand each other's responsibilities...not to critique what the other team members do, but to understand where each team member fits in the response process, and to be able to take over their role, if necessary (due to injury, etc.).  These teams have designated, pre-staged equipment and conduct regular training drills, with the idea being that a missile or torpedo strike against a ship, or even a fire breaking out in the galley, doesn't necessarily wait for the most opportune moment for the crew...as such, the DC team must be able to respond under the worst of conditions.

The purpose of the DC team is to control the situation and minimize the effect of the incident on the health and operation of the ship and its crew.  The Executive Officer (second-in-command) of the ship is usually the person responsible to the captain for the training of the DC team, while the Engineering Officer is usually designated as the damage control officer (ref).

So, where does this model fit in with today's organizations?  Do the risks of operating interconnected IT equipment appear to be understood?  Who within the organization is the DC team leader?  Who are the members of the DC team and how often do they drill?  Perhaps more importantly, what type of monitoring is in place?  Where are the gauges and who monitors them? Sure, these may be questions asked by some guy with a blog, but they're also asked by those responsible for assessing regulatory compliance, be it the PCI DSS (para. 12.9 specifies the requirement for an IR team), HIPAA, FISMA, NCUA, etc.  Further, state notification laws (what are we up to...46 states with notification laws at this point?) such as California's SB-1386 have an implication of a response capability; after all, how would questions be answered without someone getting answers?

Thoughts?  Does the DC team model fit?

Proactive IR

There are a couple of things that are true about security, in general, and IR specifically.  One is that security seems to be difficult for some folks to understand (it's not generally part of our culture), so those of us in the industry tend to use a lot of analogies in an attempt to describe things to others that aren't in our area of expertise.  Sometimes this works, sometimes it doesn't.

Another thing that's true is that the current model for IR doesn't work  For consulting companies, it's hard to keep a staff of trained, dedicated, experienced responders available and on the bench, because if they sit unused they get pulled off into other areas (because those guys "do security stuff") and like many areas of information security (web app assessments, pen testing, malware RE, etc.) the hard-core technical skills are perishable.  Most companies that need such skills simply don't keep these sorts of folks around, as they look to consulting companies to provide this service.

Why doesn't this work?  Think about it this way...who calls emergency incident responders?  Well, those who need emergency incident response, of course.  Many of us who work (or have worked) as incident responders know all too well what happens...the responders show up, often well after the incident actually occurred, and have to first develop an understanding of not just what happened (as opposed to what the customer thinks may have happened), but also "get the lay of the land"; that is, understand what the network infrastructure "looks like", what logs may be available, etc.  All of this takes time, and that time means that (a) the incident isn't "responded to" right away, and (b) the clock keeps ticking as far as billing is concerned.  Ultimately, what's determined with respect to the customer's needs really varies; in fact, the questions that the customer had (i.e, "what data left our network?") may not be answered at all.

So, if it doesn't work, what do we do about this?  Well, the first thing is that a cultural shift is needed.  Now, follow me here...all companies that provide a service or product (which is pretty much every one of them) have business processes in place, right?  There's sales, customer validation, provisioning and fulfillment, and billing and collections...right?  Companies have processes in place (documented or otherwise) for providing their product or service to customers, and then getting paid.  Companies also have processes in place for hiring and paying employees...because without employees to provide those products or services, where would you be?

Ever since I started in information security, one of the things I've seen across the board is that most companies do not have information security as a business process.  Companies will process, store and manage all manner of sensitive data...PCI, PHI, PII, critical intellectual property, manufacturing processes and plans, etc...and not have processes for protecting that data, or responding to incidents involving the possible exposure or modification of that data.

Okay, how about those analogies?  Like many, I consider my family to be critical, so I have smoke alarms in my home, fire extinguishers, we have basic first aid materials, etc.  So, essentially, we measures in place to prevent certain incidents, detect others, and we've taken steps to ensure that we can respond appropriately to protect those items we've deemed "critical".

Here's another analogy...when I went to my undergraduate education, we were required to take boxing.  If you're standing in a class and see everyone in line getting punched in the face because they don't keep their gloves up, what do you do?  Do you stand there and convince yourself that you're not going to get punched in the face?  When you do get punched in the face because  you didn't keep your gloves up, do you blame the other guy ("hey, dude! WTF?!?!") or do you accept responsibility for getting punched in the face?  Or, do you see what's happening, realize that it's inevitable, listen to what you're being told, and develop a culture of security and get your gloves up?  The thing about getting punched in the face is no matter what you say or do afterward, the fact remains...you got punched in the face.

Here's another IRL example...I recently ran across this WaPo article that describes how farms in Illinois are pre-staging critical infrastructure information in an easily accessible location for emergency responders; the intention is to "prevent or reduce property damage, injuries and even deaths" in the event of an incident.  Variations of the program have reportedly been rolled out in other states, and seem to be effective.  What I find interesting about the program is that in Illinois, aerial maps are taken to each farm, and the farmers (those who established, designed, and maintain the infrastructure) assist in identifying structures, etc.  This isn't a "here's $40K, write us a CSIRP"...instead, the farmer has to take some ownership in the process, but I guess they do that because a 1 hour or one afternoon interview can mean the difference between minor damage and loosing everything.

Sound familiar?

As a responder, I'm aware of various legislation and regulatory bodies that have mandated the need for incident response capabilities...Visa PCI, NCUA, etc.  States have laws for notification in the case of PII breaches, which indirectly require an IR capability.  Right now, who's better able to respond to a breach...local IT staff who know and work in the infrastructure every day (and just need a little bit of training in incident response and containment) or someone who will arrive on-site in anywhere between 6 and 72 hours, and will still need to develop an understanding of your infrastructure?

If the local IT staff knew how to respond appropriately, and was able to contain the incident and collect the necessary data (because they had the training and tools, and processes for doing so), analysis performed by that trusted third party adviser could begin much sooner, reducing response time and overall cost.  If the local IT staff (under the leadership of a C-level executive, like the farmer) were to take steps to prepare for the incident...identify and correct shortfalls in the infrastructure, determine where configuration changes to systems or the addition of monitoring would assist in preventing and detecting incidents, determine where critical data resides/transits, develops a plan for response, etc...just as is mandated in compliance requirements, then the entire game would change.  Incidents would be detected by the internal staff closer to when they actually occur...rather than months later, by an external third party.  Incident response would begin much quicker, and containment and scoping would follow suit.

Let's say you have a database containing 650K records (PII, PCI, PHI, whatever).  According to most compliance requirements, if you cannot explicitly determine which records were exposed, you have to report on ALL of them.  Think of the cost associated with that...direct costs of reporting and notification, followed by indirect costs of cleanup, fines, lawsuits, etc.  Now, compare that to the cost of doing something like having your DBA write a stored procedure (includes authorization and logging) for accessing the data, rather than simply allowing direct access to the data.

Being ready for an incident is going to take work, but it's going to be less costly in the long run when (not if) an incident occurs.

What are some things you can do to prepare?  Identify logging sources, and if necessary, modify them appropriately (add Process Tracking to your Windows Event Logs, increase logs size, set up a means for centralized log collection, etc.).  Develop and maintain accurate network maps, and know where your critical data is located.  The problem with hiring someone to do this for you is that you don't have any ownership; when the job's done, you have a map that is an accurate snapshot, but how accurate is it 6 months later?  Making incident detection and tier 1 response (i.e., scoping, data collection) a business process, with the help of a trusted adviser, is going to be quicker, easier and far less costly in the long run, and those advisers will be there when you need the tier 3 analysis completed.

What about looking at things like Carbon Black?  Cb has a number of uses besides just IR, and can help you solve a number of other problems.  However, with respect to IR, it can not only tell you what was run and when, but it can keep a copy of it for you...so when it comes to determining the capabilities of the malware downloaded to your system, you already have a copy available; call that trusted adviser and have them analyze it for you.

Remember the first Mission: Impossible movie?  After his team was wiped out, Ethan made it back to the safe house and as he reached the top of the stairwell, took the light bulb out of the socket and crushed it in his jacket, then spread the shards on the floor as he backed toward his room.  What this does is provide a free detection mechanism...anyone approaching the room isn't going to know that the shards are their until they step on them and alert Ethan to their presence; incident detection.

So what are you going to do?  Wait until an incident happens, or worse, wait until someone told you that an incident happened, and then call someone for help?  You'll have to find someone, sign contracts, get them on-site, and then help them understand your infrastructure so that they can respond effectively.  When they're first there, you're not going to trust them (they're new, after all) and you're not going to speak their language.  In most cases, you're not going to know the answer to their questions...do we even have firewall logs?  What about DHCP...do we log that?  What will happen is that you will continue to hemorrhage data throughout this process.

The other option is to have detection mechanisms and a response plan in place and tested, and have  a trusted adviser that you can call for assistance.  Your local IT staff needs to be trained to perform the initial response, scoping and assessment, and even containment.  While the IT director is on the phone with that trusted adviser, designated individuals are collecting and preserving data...because they know where it is and how to get it.  The questions that the trusted adviser (or any other consulting firm) would ask are being answered before the call is being made, not afterward ("Uh...we had no idea that you'd ask that...").  That way, you don't loose the whole farm, and if you do get punched in the face, you're not knocked out.

By the way...one final note.  This doesn't apply solely to large companies.  Small business are loosing money hand over fist and some are even going out of business...you just don't hear about it as much.  These same things can be done inexpensively and effectively, and need to be done.  The difference is, do you get it done, even if you have to have a payment plan, or do you sit by and wait for an incident to put you out of business and lay off your employees?

Thoughts on Analysis, Information Sharing, and Presentations

Since June of this year, I've attended a couple of conferences, not only presenting but also attending several presentations. As with any conference, presentations tend to vary with the background and experience of the author. By experience, I don't so much mean with the experience the author has in the topic they're presenting on, but more so with developing presentations specific to the audience, public speaking, etc.

One of the questions I ask myself when both creating and giving a presentation, as well as attending one, is "how is this immediately valuable and useful to me?" Often times, we attend a presentation that may have some great information, but the question then becomes, okay, but how to I use this? How do I most quickly/immediately turn this information around and use it on a case or examination?

I'm sure that a lot of other folks feel the same way, particularly after having talked about this very subject after a presentation or discussion. I've also seen this enough to add this sort of approach to my books, starting with WFA 2/e and progressing even more so into Windows Registry Forensics. With a lot of what's going on in our community, it's vitally important that analysts be able to get up from a presentation or talk, and be able to put what they've learned in a technical presentation or lab into practice. If that isn't the case...what's the point?

This is critically important when you're talking about analysis techniques, which is particularly where we need to innovate. Data is always going to be data, and it's always going to be there. There's been discussion about triaging systems, due in part to the massive increases in storage and the amount of time it takes to acquire and analyze all of that (especially acquire). So whether you're talking about browser forensics or

Innovation in Analysis
How many folks out there use RegRipper? How about using RegRipper for malware detection or analysis? Seriously.

Bear with me on this...it probably wouldn't occur to most folks to use something like RegRipper for malware detection, looking into an acquired image, or a system accessed via F-Response. However, I (and others) have used this approach time and again to our benefit.

Consider Conficker. Microsoft cites five variants. In each case, when the next variant first came on the scene, the executable file was not detected by most commercial AV tools. However, there were some consistencies across the malware family, including not just artifacts (ie, disabling system functionality) but also in the persistence mechanism.

I had the distinct honor of reviewing on of the chapters for the new book, The Malware Analyst's Cookbook, and in that chapter, the use of RegRipper was discussed. MHL and his co-authors had written several plugins for use in their work (included in the book) and to be honest, they were innovative.

Okay, so you're probably looking at this so far and thinking, yeah, but that's for malware you know about. Okay, sure, I can go with that. But here's the deal...while you may not have seen all malware, or at least a great deal of it, but what are the chances that within a large group or community of forensic and malware analysts, a LOT of malware has been seen and analyzed? Of those, their artifacts and persistence mechanisms will likely have been identified (Zeus, or ZBot falls into this category) and be worthy of a plugin.

Consider malware based on PoisonIvy. Yes, it's polymorphic, which for most of us, is going to mean that AV scanners are going to have minimal effect in detecting this stuff. However, keep in mind the four characteristics of malware...the persistence mechanism is an example of Jesse Kornblum's "rootkit paradox"; in short, most malware will want to run, but will also want to remain persistent. As it turns out, PoisonIvy isn't the only malware that uses the Registry Installed Components as the persistence mechanism.

Institutional Knowledge
How many of us use the institutional knowledge developed on previous cases/engagements on any new analysis that comes in? Most of us are going to raise our hands and say, "I do that all the time."

Now, how many of us make use of the institutional knowledge developed by other analysts, such as other team members?

Tools like RegRipper (and pretty much any tool that has some level of extensibility, including EnCase and ProDiscover) provide for sharing of institutional knowledge, but only insofar as that institutional knowledge is documented.

So What?
I'm sad to say that not all of us will ever see everything, particularly when it comes to IR and digital forensics analysis. For myself, I know that I haven't seen everything...I've seen some things several times, I've been on engagements that never where, but I haven't seen everything. Expand this to include any 5, 10, 50, or 500 analysts you want, and while it may seem that across all of us, we've seen a lot, the question then becomes...how much of that have we shared with others, or each other.

The issues the community faces are not going away. Increases in things like storage space, sophistication and proliferation of malware and compromises/intrusions, AV scanners becoming less and less of a reliable resource, to name a few. I think that most of us are familiar with these issues. So what do we do about all these?

CyberSpeak
Ovie's on a roll...don't miss the new CyberSpeak podcast! Ovie has an interview regarding triaging systems...talk about timely! ;-)

Temporal Proximity

A couple of years ago, I heard someone...a really smart someone...talk about "temporal proximity". I know that sounds Star Trek-y, but the reference was to initiating response activities as close to an incident as possible.

Several of the industry reports indicate that the majority of incidents that the vendors have responded to have been the result of third-party notification to the victim. That alone indicates a number of things, the lack of temporal proximity (perhaps a better description would be "temporal dispersion") being one of them.

Why is this so important? Well, a lot of good...no, strike that...a lot of critical information can exist in memory, making it very volatile. Processes complete, network connections terminate. The longer you wait, the more this happens...and the system is more likely to be rebooted.

Some intruders get on systems, run tools, and them leave with data. When they do so, they may delete their toolkits. I've seen batch files that include the "del" command for just that purpose. Well, the more temporal dispersion you have from the incident, the less likely you are to recover the deleted files...in case you haven't heard, Windows (especially XP) has it's own built-in antiforensics measures.

Okay, you're probably wondering what I'm talking about...so I'll tell you. For Windows systems, even if you don't interact with the system, stuff still happens, particularly with XP. Just let an XP system sit for a couple of days, and you'll see. Restore Points are created every 24 hours, and if the disk space available for the RPs is getting short, others will be deleted. A limited defrag is run every three days. And this is just for an XP system that sits there with no network connectivity and no one interacting with it. Now, add to that things like Windows software and application updates...you know, the stuff that just kind of happens automatically with a network connected system. Even with minimal auditing enabled, stuff still gets logged to the Event Log...more so on Vista and Windows 7, simply because there are so many more logs.

Now, add to the mix that no one within your infrastructure is aware of an incident (intruder, malware, etc.), and systems remain up, functioning, operational and in use. I've been on engagements where we collected data from a system and then three days later collected the same data...and you'd swear that they were two different systems. Prefetch files had been deleted, deleted files had been overwritten by OS and application updates, applications and tools being run, etc.

In order to achieve temporal proximity, you need a couple of things. First, visibility...if you don't have visibility into your infrastructure, how will you know when something occurs? You can't really expect to know when something goes wrong or changes if you're not monitoring, right?

Second, you need a plan. What's you're IR plan? Acquire memory and disk, and then take the system offline? Or panic and not do anything at all until someone who has no idea what's going on makes a decision? I can't tell you the number of times I've responded and found out that the incident had been detected a month prior, and the infected/compromised system had been left up the entire time.

Me: "You know the intruder has been siphoning data off of this system for the past month, right?"

Them: "We didn't know what to do."

This happens more than you'd care to know, and not just to one vertical...not just PCI, but to many, many types of victims.

One final note...Marines in training learn what are referred to as "immediate actions". These are simple tasks that you use to clear a jammed weapon. They're simple when you're on the range, on a bright, sunny day after a good night's sleep. You can ask a range coach if you're doing it right. But we're trained on this over and over because you never need it in those conditions...when you're going to need that reaction to be programmed is during an assault, at 2:30am, after you've gone without sleep for two or more days and maybe haven't eaten in as long. And it's raining. And it's cold.

Are your IT assets critical to your business? If I were to back up a truck and take all of your computers...all desktops, laptops, servers, etc...how would that affect your business? It would disappear, wouldn't it? Well, if IT assets are so critical to your business, why not protect them? The bad guys aren't coming into your organization and walking out with boxes full of papers...they're coming into your network and stealing data that way. And they're successful because in many cases, they have greater visibility into your infrastructure than you do.

Lessons in IR

Something in the news out of Tulsa, OK, this morning really provided an excellent lesson in IR.

Basically, the story goes that someone saw what they thought might be one of the deadliest spiders on the planet, panicked, and killed it. An expert in spiders asked to see the body of the spider, but it wasn't available...it had been destroyed.

How many times has this happened to you as a responder?

Caller: "Help! We were hit with the deadliest Windows worm known to man!"

You: "Okay, calm down. How do you know?"

Caller: "We received an alert on our AV console!"

You: "Okay, good. What did it say?"

Caller: "We don't know."

You: "Uhm...okay. Have you isolated any infected systems or preserved a sample of the malware?"

This is where things just kind of go downhill. But the news article is a great example of how things go wrong on a daily basis in IR...

Malware for Incident Responders - Examples

I thought that now would be a good time to take a step back and look at the malware characteristics that have been presented, and see how they can be used to understand malware so that such incidents can be prevented, detected, and better responded to, not only by first responders, but also by forensic analysts. During many malware incidents, you'll hear things like, "we cleaned the systems, but they keep getting infected", or "we don't know how the malware is infecting systems." My hope is that the characteristics will provide a framework that can be used to answer these, and other questions.

Initial Infection Vector
One thing we're always curious about is how the malware originally got on a system or into an infrastructure. Many times we see malware on a system and find out that it makes its way through the network like a worm, exploiting a vulnerability or a specific set of vulnerabilities. However, many times, these vulnerabilities are obviated at the perimeter, either by the fact that the ports are not accessible via the firewall, or the network is NAT'd behind the firewall, or something similar. So the question remains, how did the malware originally make its way into the network? Where is patient zero?

Initial infection vectors (IIVs) can come from a variety of sources. For example, a user opens an email attachment and either the worm kicks off or a downloader gets installed, which then reaches out and grabs the worm. Other IIVs can include the web browser, USB thumb drives, etc. Heck, even digital cameras can be the IIV for malware!

Propagation Mechanism(s)
Propagation mechanisms are how the malware "gets around" and infects other systems once it's in your infrastructure. Some malware...worms, in particular...like to propagate by exploiting vulnerabilities on the network. Some worms use just one exploit, and others actually use multiple exploits, going with the one that works. This may also be the initial infection vector, or it may be how the malware propagates once it's on your network.

Very often, malware will exploit operational business functionality in order to propagate. From the exploit example, worms will not only exploit vulnerabilities, but also exploit the operational business functionality of not patching systems in a timely manner. A lot of malware currently exploits the operational business functionality of having all users run with Administrator privileges. The release of Conficker has seen a move to exploit file sharing; by placing a copy of the malware at the root of a share, and adding an autorun.inf file, the malware exploits the operational business functionality of file servers (as well as the fact that MS dropped the ball with respect to the NoDriveTypeAutorun Registry setting).

Persistence Mechanism(s)
Persistence mechanisms are those artifacts that allow malware to survive a reboot. Jesse Kornblum's "Rootkit Paradox" paper points out that rootkits (a form of malware) wants to remain hidden from view but at the same time, it wants to run; extending that paradox just a bit, something similar can be said for most malware...it wants to run, but in most cases it also wants to remain persistent across reboots. As such, there are a finite number of ways that malware (any software, really) can do this on Windows systems. Most of us are familiar with some of the persistence mechanisms in the Registry, in particular the ubiquitous "Run" key.

Another popular means of ensuring persistence is to load the malware as a Windows Service, or in the case of kernel-mode rootkits, as a device driver (*.sys file). This can be further extended by not only loading the malware as a service, but loading it as a DLL under the Svchost service. Doing so has the effect of loading the malware and hiding it in plain sight; by that, I mean that if a responder lists the running processes, either through Task Manager or some other tool, the malware won't be readily visible, as its running as part of another service.

A variant of this persistence mechanism that I've personally seen in the field is to install two services...one that's the actual malware, and another that checks to make sure that the malware is running after the system has been rebooted. I assisted a customer with a situation like this where they had identified one persistence mechanism, "cleaned" the system by removing the key and the files, and then rebooted...only to have the malware return. Identifying the other service allowed us to finally clean the systems.

Yet another persistence mechanism is to use Scheduled Tasks, also referred to as "AT jobs". Conficker is a recent example of malware that uses this persistence mechanism...the responder users an AV vendor's tool to remove the malware file itself, as well as any Registry keys, but shortly after the malware is removed, it's back again!

Artifacts
Artifacts differ from persistence mechanisms in that they are both essentially artifacts, but persistence mechanisms allow the malware to survive reboots and remain...well...persistent. Many times, malware will leave artifacts or "footprints" of its presence, either by design or simply by its interaction with the system. For example, some malware modifies the hosts file; this isn't a persistence mechanism, but it is a modification that the malware is designed to do. Some malware does this so that processes on the system that need to make network connections...AV updates, for example...don't succefully resolve the domains that they intend to for updates, etc.

Other artifacts can include disabled or even deleted security applications and services, as well as modifications to the Windows Firewall to register the malware as an exception to the firewall rules, allowing it to communicate to the Internet.

Artifacts are not limited to host systems (ie, Registry entries, files created or modified, log file entries, etc.); artifacts can also be found in network traffic, or in log files on other systems or devices. For example, one of the big things I've seen in a number of malware write-ups is malware's "phone home" capability, reaching out either to an IRC server or requesting a URL so that the infected systems are logged in the web server logs. Many AV companies are listing the various URLs, and malware authors have been randomizing the domain selection, making it a full time job to develop and maintain blacklists on firewalls and web proxies. However, the point is that while these URLs or IP addresses do change, they are transient artifacts that are found in the network connections on the system, as well as in network traffic.

The examples I've provided are not a complete list by any means. However, they are meant to illustrate a framework which can be used to understand and address malware from several perspectives, including initial triage of an incident, on-site or first response to a known or as-yet-unidentified malware incident, or forensic analysis following an incident. Once the framework is in-place, there are a number of resources that can be used to fill in the gaps, if you will: vendor-provided analysis, static/dynamic analysis of the malware itself, or analysis of captured network traffic and log data.

Malware for Incident Responders - FakeXPA

I've mentioned here before that many times it's very difficult for experienced incident responders to really get their heads wrapped around a malware issue, such as an infection in a corporate infrastructure. As difficult as it is for folks who do this kind of work, imagine how difficult it can be for the IT staffers managing that environment and attempting to recover from such an infection. I recently spent some time working with someone who'd been hit with a variant of a 9 year old dropper. Yes, that's right...9 years old. The secondary download was about 2 years old, and their AV product was able to detect these baddies...well, as long as the AV product installed was up-to-date and enabled. No extra.dat files were required.

Again, as I've mentioned before, one of the issues faced by responders (IT staff, as well as guys like me) is a lack of information from AV companies...most do not provide necessary information to perform detection, eradication and recovery effectively, simply because they're not in the business to do so. A commercial AV company's goal is to get you to use their product, so their response is to attempt to get a copy of the variant, develop a definition or signature for it, and see if that signature works for you...all while you just want it gone!

As a side note, one of the things that software vendors, in general, don't realize is this...by now, most folks are aware of the fact that things won't always be perfect. All products will have flaws, and most consumers know that. So the key at this point is, when someone does have a problem, how do you respond? Customers are happiest with the first person that feels their pain and really helps them.

Okay, enough of that. I thought that after our last example, it might be a good idea to take a look at some of the malware that comes out, and take a look at it from the perspective of an incident responder (and subsequently, a forensic analyst). An example I ran across today was W32/FakeXPA. Let's take a look at this from perspective of the framework we've already put together...

First off, W32/FakeXPA is a "rogue" security product, which infects a system and makes the user believe that they have some sort of issue (or no issues at all) on their system. Other similar malware includes Trojan:W32/AntivirusXP. Fortunately, Trojan:W32/FakeXPA was added to MRT in December, 2008...so there's some protection there.

Initial Infection Vector
No specific infection vector is identified. MS says that "various methods" are used to infect systems, so I'm sure that responders should look to either email attachments or a secondary download based on an initial dropper via email or the web browser. However, this also does not rule out other infected media (thumb drives, CDs) as the transport mechanism.

Artifacts
MS's write-up on W32/FakeXPA states that among other things, this malware may add an "XP Antivirus" value to the HKCU\Software key. The most recent information available from the MMPC also states that a recent variant modifies the hosts file.

Other artifacts may include an entry in the user's Start Menu folder (ie, "%UserProfile%\Start Menu\XP Antivirus XXXX")

Some of the different variants have different artifacts. For example, from MS's write-up, the 2010 variant installs as an BHO, as well as adding a subdirectory and files to the "C:\Documents and Settings\All Users\Application Data" directory.

While the "under the hood" artifacts of an infection vary, the commonality is that the user will see the fake antivirus GUI giving alerts to various threats detected on the system. Most peoples reaction will be, "yes, get rid of all of it!!", but responders need to look to whether or not the alerts are coming from the legit installed AV product or not. See the MS write-ups for screenshots of some of the variants.

Propagation Mechanism
This particular bit of malware doesn't appear to propogate once it's on a system. This is classified as a Trojan, not a worm.

Persistence Mechanism
According to the MS write-up, W32/FakeXPA writes a file to the %ProgramFiles% or "%ProgramFiles%\XP Antivirus" directory and creates an entry in the user's (HKCU hive) Run key. This appears to be the common persistence mechanism across the variants (the 2010 variant also includes a BHO), and provides responders with a great way to check individual machines, systems within a domain, as well as acquired images for indications of an infection.

Interestingly, the latest MMPC write-up ends with a listing of SHA-1 hashes for the latest variant. Come on, guys...not only do many products rely on MD5 hashes (EnCase, Gargoyle, etc.) but using static hashes in the face of malware that varies just does not make sense! Why not use fuzzy hashes?

Resources
VirusTotal page for one variant of FakeXPA (see how other AV vendors are detecting it)
ThreatExpert (2009 variant, A360 variant)
Another rogue "security product"

Final Note
Look at some of the differences between the various write-ups...for example, compare the write-up on the ThreatExpert 2009 variant to the MMPC Security Portal write-up for the same variant. Both identify some of the files and Registry keys left by the malware, but ThreatExpert identifies the packer, which means another means of detection may include Scout Sniper, from the illustrious Don Weber!

HBGary: FastDump and Responder

Thanks to Rich Cummings, I was recently able to take a look at HBGary products that they offer with respect to physical memory collection and analysis; specifically, FastDump Pro and Responder Professional.

First, the FastDump product is pretty cool. The free version of the tool allows you to dump the contents of physical memory from pre-Windows 2003 SP 1 systems (XP, Windows 2003 w/ no Service Pack). Now, a lot of folks are going to look at FastDump Pro and wonder why it's available for a fee; well a close look at the write up for the FastDump Pro should very quickly make anyone realize that the tool is definitely worth what they're charging; FDPro is not encumbered by the 4GB limit, works up to Windows 2008 (Windows 7 Ultimate Beta shouldn't be a problem, either), and it handles both 32- and 64-bit versions of Windows. That's A LOT packed into a $100 executable! FDPro also has the capability to incorporate collection from the pagefile, as well; however, in the limited testing I've done so far, analysis tools other than Responder won't necessarily "understand" the .hpak format.

Before we look at the Responder product, I'll have to upfront about my testing...my focus was incident response, and I really didn't intend to fully exploit Responder's malware analysis capabilities. So, essentially, while I had access to an evaluation version of the Responder Pro product, I was really using what amounted to the capabilities in the Field Edition. However, one of the things I've really been pushing with respect to incident response is speed...when an incident occurs, information collection and analysis needs to start as soon as possible, and tools like FastDump Pro and F-Response give you that speed in collection; Responder gives you speed in analysis for a range of Windows operating systems through a common interface.

So I started off by creating a case in Responder and loading the first memory dump/snapshot from the DFRWS 2005 Memory Challenge. Now, the snapshot can be a raw memory dump, collected via dd.exe (no longer available), F-Response + {enter a tool here}, FastDump, FastDump Pro, etc. Responder will identify the operating system of the memory dump and extract a good deal of information, making it available to the responder via the user interface (UI). So, once the memory dump has been collected, it just takes a couple of mouse clicks to get to the point where the responder is actually looking at the contents of the memory dump, viewing things such as the active process list, network connections, etc.

When I first looked at the Responder product a bit ago, as an incident responder, one of the issues I had as being able to quickly and easily find what I was looking for...in particular, the command line used to launch each of the processes in the active process list. Well, not only is this now available in the current version of the product, but you can also drag the columns in the UI to a more suitable location. For example, I dragged the column for the process command line over to line up the process name, PID, parent PID, and command line so that I could see everything together and quickly run through the entries.

You can also view the open network sockets from the memory dump in a very netstat-like format. An option that the Responder product provides is the ability to export the data you're viewing in a variety of formats (Note: the export functionality was disabled in the evaluation version). This allows you to use either screen scrapes of the Responder UI or exports of the data for reporting, or you export the data you've got and use tools similar to Gleeda's vol2html.pl to modify the format a bit.

Now, one of the options when importing a snapshot is to "Extract and Analyze All Suspicious Binaries"; this allows for a modicum of analysis to occur while importing the snapshot. What is "suspicious" is defined by rules visible in a text file, which means that as you become more familiar with the tool, you can comment out some of the rules, uncomment some, or add your own.

With Responder, you can also view the open handles and network sockets for a specific process, view, analyze, or save a copy of a binary (exe or DLL/module), run strings against a binary, etc. There is a great deal of capability in this tool, and there's no way I'm even beginning to scratch the surface. From an IR perspective, tools like this provide the first responder with a means of getting answers quickly, while at the same time being able to "answer new questions later". This is an extremely powerful capability...imagine quickly triaging an incident and being able to narrow down from your 500 possible systems the 12 or so that may be "in scope". Consider the cost savings. And when you do acquire physical memory, you've also got a copy of the malware (if there is any) in an unencrypted, un-obfuscated state.

Admittedly, Responder doesn't give you the same granularity, deep-dive capabilities, and flexibility of Volatility, but it does allow you to import memory snapshots from a range of Windows versions and puts the tools in your hands to quickly get the answers you need; that in itself is a huge plus! Again, I did not really dig into the full spectrum of capabilities of FastDump Pro and Responder, so if you're interested in really exploiting HBGary's capabilities for doing malware analysis, you should definitely consider giving them a call.

Characteristics of Effective Incident Response

The Need
There is a need for effective incident response, now more than ever. However, the key to incident response is incident preparedness. Responding without being prepared to respond correctly is what turns an incident into a major data breach and a major embarrassment.

Addendum: If you don't think incidents are going to happen, or that they aren't going to happen to you, take a look at this SecurityFix post from Brian Krebs. To be clear, the very first sentence says "reported"...which perhaps indicates a subset of what was actually detected.

There are three primary characteristics of effective incident response, and knowing and understanding these lead to being better prepared to respond to incidents:

1. Completeness of Data
When I am called to respond to an incident, there are four sources of data I will generally look for, to some degree, regardless of the type of incident:

- Network traffic captures
- Logs from network devices (firewalls, routers, etc.)
- Host-based volatile data
- Host-based non-volatile data

Now, you may not need data from every source for every incident, and the value of the data from these sources may vary depending upon the incident. In all the time I've been doing IR, in various capacities, I can't recall a single time when I've had all four sources of data available...in fact, in most cases, I'm lucky to get one, and it's a miracle to have two.

2. Accuracy of Data
How accurate the data is in many cases can depend upon what it is you're actually trying to do. For example, capturing portions of volatile memory using a batch file and third party tools (ie, tlist.exe, tcpvcon.exe, etc.) to get the list of active processes, network connections, etc., may be accurate enough for your needs. However, in other instances, collecting the full contents of physical memory may be the only means of achieving accuracy (and completeness) of data.

3. Temporal proximity
This is a term I borrowed from Aaron Walters (of Volatility fame), not because it's cool and Star Trek-y sounding, but because it makes perfect sense. The sooner you begin responding to an incident, the more complete and accurate data you're going to get, and your overall response is going to be more effective and lead to better remediation, etc.

Since this field is wrought with analogies, let's try this one...as a homeowner, do you have smoke detectors in your home? How about fire extinguishers? I do. How about insurance? When you called for your insurance, did the insurance company ask how far you are from the nearest hydrant? How about from the nearest fire department? Did you get a home inspection prior to purchasing your home? Is it up to code with respect to exits, etc? Do you know what to do in case of a grease fire in your kitchen?

So, your home is full of valuables, in particular, your family. If your home were to catch fire, what would you do? First off, how would you know? Then, once you knew, what would you do? Would you just wait until the house burned down to call the fire department?

Now, map your home to your network infrastructure, and a fire to a data breach. Where are your valuables? Do you have a detection mechanism? Are you able to respond immediately and correctly to a grease fire?

This example shows, in part, that temporal proximity plays an important role in incident response, but it also highlights the need for approaching it the right way, which may include training. So what does this mean for the coming year? Well, it should mean that training will be more important than ever. Think about it. If a small fire starts in your house, would you (a) wait for an hour, then call the fire department, (b) wait for the house to burn down completely, or (c) begin putting the fire out yourself immediately? With respect to computer security incidents, who is in a better position to respond immediately...the sysadmin who is currently logged into the console, or an responder such as myself who is 24-48 hrs away from being on-site? Not only is the sysadmin there, but she more than likely knows the systems and the architecture very well; an external responder such as myself is going to have to get up to speed on your architecture, and even then won't have all of the little nuances.

This is all fine and good, but as Lon Solomon is fond of saying, "so what?" The fact is that many organizations simply don't put any effort or resources into their response plan because they feel that there's no requirement to do so.

External Forces
I'm not going to make a prediction for the future, but the primary drivers for incident response in 2009 (and beyond) will continue to be external forces; specifically, legislative and regulatory compliance requirements. Why is that? Because they have to be. After all, most of us think that if someone is making a business out of storing and/or processing our sensitive (PII, PHI, PCI) data, then of course they'll do everything they can to protect and secure that data. I mean, why wouldn't they, right? After all, if a buddy wants you to hold $50 for him, or the ring he's going to present his bride at their wedding, then you're going to do everything you can to protect that data, right? For many of us, this is just what we think of as common sense, but that's sadly not the case with your sensitive data. Look here, or here. And things only start to change when some external forces come into play, and those forces are strong enough to act as the stimulus to cause that change...those external forces being legal (think CA SB-1386, etc.) or regulatory compliance (think HIPAA, Visa PCI, etc.) requirements.

Changing the Face of IR

Major corporations handle our sensitive data...referred to as PII, PHI, PCI. Sometimes, they don't do such a great job of securing and protecting that data. The Dept of Veterans Affairs. TJX. World Bank. IMF. The list goes on. Data breaches happen...there's no question about that. If they didn't, guys and gals like me would be out of jobs.

Historically, the stimulus for change with respect to infosec (particularly with respect to IR) has been external to organizations. An attack or major breach stimulates some change, but its not long lived...once the panic wears off, executive management fails to see ROI from the resources that were suddenly (re: knee-jerk reaction) invested. According to Richard Bejtlich and others, there is no ROI from security...not directly anyway.

Legislation and regulation...with consequences...has a more lasting effect. Visa's PCI defines "compliance", which while not being what most of would consider "security", is at least a step in right direction. There are other regulatory/oversight bodies that provide their own guidelines...NCUA, HIPAA, etc. Section 748.2 of the NCUA Regulations provides guidance on "response programs". The PCI DSS (paragraph 12.9) provides compliance standards for an incident response plan.

Every organization with employees has a payroll process. Why? Well, without it, employees wouldn't get paid, and we all know that the CEO has to get paid, right? And oh, yeah...if you don't pay your employees, they don't come to work...you know where this one is heading. Many organizations have disaster recovery and business continuity plans, backup systems, etc. But why do some organizations not have computer security incident response plans, even when some regulatory body tells them that they need to have one?

Regulatory body definitions of "sensitive data" aside, what about corporations that loose intellectual property (IP)? Did you read this 12 Nov article in USAToday (additional commentary on the story at TaoSecurity)? Many organizations subsist primarily on their IP...remember Ira Winkler's Corporate Espionage?

The Verizon Business Security group put together some interesting statistics in their 2008 Data Breach Investigations Report; for example:

83% of attacks were not "highly difficult" (re: low-hanging fruit)
85% of attacks were opportunistic (re: "hey, look...someone left their keys in their car...")
87% of attacks could have been avoided with reasonable security controls
66% of breaches involved data the victim did not know was on the system

Perhaps one of the most interesting and revealing statistics was that reportedly 75% of breaches were not discovered by the victim. What this means is that data from within those organizations network infrastructure was compromised and exposed, and the breached organization had no idea until someone told them.

So, if its not enough that some regulatory oversight body requires you to have an incident response plan, how about the inevitability of an incident occurring? What's it gonna take for organizations to plan for an incident occurring, rather than reacting (poorly) after one has occurred? Oddly enough, any change in this regard with have nothing whatsoever to do with the victim "doing the right thing", and has everything to do with legislation and regulatory oversight.

IR Preparedness

As a responder, I often hear two reasons for folks calling for help after a breach has occurred (aside from the rather obvious one, that is) - the caller either wants to be sure that they're performing "due diligence", or they're interested in bringing in a disinterested third party to have a look at things.

Every now and then when I get a chance to sit back and reflect, something strikes me. Most often those things were right there in front of me all along, and I just never noticed them. Things like these reasons that people give me.

Take for example the reason of "due diligence". It occurs to me that if someone were really interested in performing due diligence, they would've called me before the incident occurred, to ensure that they were prepared to handle an incident. Closing the barn door and shutting the stall doors after the horse has left is not "due diligence".

What about the "disinterested third party"? If you call a consulting company to come help you contain and recover from an incident, the fact that you're paying them obviates the "disinterested" part of the description, doesn't it?

Many times, folks at an organization looking for assistance after an incident has been detected will say that they want an outside firm to handle the examination in case the issue ends up going to court. But what's the real difference between the victim organization performing the investigation and an outside firm doing so? The outside firm most often has a documented process, or at the very least documents what they did in their report.

The primary reason that consulting companies are called when an incident occurs is that the organization was simply unprepared for an incident. Regardless of legislative or regulatory requirements, they're simply unprepared.

This is not to say that when an incident occurs, you should not call someone for assistance. Rather, I'm reiterating my response from the SANS Forensic Summit panel when Rob Lee asked the panel members about response time to a call...my response was that the best time to call someone for help is before an incident occurs. At the very least, you can protect your organization by demonstrating due diligence.

Costs of not being prepared
Fines - legislative/regulatory oversight often comes with some kind of fine
Notification - writing and mailing letters is going to cost time and money
Charges - 20,000 Visa credit card numbers are exposed...who's going to pay for the cost of reissuing all of those credit cards? You get three guesses, and none of the answers are "Visa" or "the card holder". And two of your guesses don't count.
Suits - Civil suits have been brought against organizations as a result of a breach; just defending yourself is expensive

Something very important to remember about regulatory requirements is that in many cases, unless you're able to definitively identify the data that was exposed, you may have to notify on ALL data that could potentially have been exposed. So, if the database containing 6 million records was compromised, and you were not prepared for an incident, and you think that only 23,000 records were exposed...but you don't know for sure...guess how many people you're going to have to notify? You get three guesses, and the first two don't count.

Benefits of Incident Preparedness
Compliance - with legislative and regulatory requirements
Lower overall cost - the upfront cost of doing nothing is...nothing; in the long run, however, costs mount.
Confidence - from your Board of Directors and the consumer (b/c you're demonstrating "due diligence")

As long as we use IT assets to conduct business, and as long a people are part of the process, there will always be a need for incident response. Incidents are always going to occur, without question. The difference today (and tomorrow) is if you're going to be prepared for an incident...or not.

What do you need as a responder?

Many times when working with customers or talking with folks who interface with incident responders (or are incident responders), the topic of what information is pertinent to a responder comes up. When receiving calls from customers, many times the people who make the call and interface with the consultants are not themselves responders (or in some cases, even technical people), and instead represent their company from a business or legal/compliance perspective. This can often lead to misunderstandings due to differing perspectives, which ultimately leads to a lack of (accurate) information. However, this isn't restricted only to non-technical people...sometimes having the sysadmins on the call can lead to the same sort of thing, which is, again, largely due to perspective.

So...what do responders need to know when you call, or what information do your responders need access to?

Well, the first thing that an incident responder is going to need to know is your goals, or what you hope to achieve from the response activities.

In an ideal world, there are four basic sources of technical data that responders will be looking to...network traffic captures, network device log data (ie, firewalls, routers w/ ACLs, etc.), and host-based volatile (memory) and non-volatile (system image) data. There may be instances when all of this information is available, and there may be times when portions of this data is available...however, in many cases, for those unprepared for an incident, very little if any of this is available to a responder. The amount of information available to the responder is going to play a direct role in how completely the responder is going to be able to help you achieve your goals.

I once received a call from someone who needed help with a laptop. Apparently, the laptop had been infected with malware, and the immediate response was to take the laptop off of the network, wipe the hard drive, and install an updated version of the operating system. The caller wanted to know if I could tell them what the malware was, and what (if any) data had been taken. Seriously.

Some questions you're likely to be asked when you call someone to perform incident response include (but are not limited to):

- What type of incident are you experiencing? How would you characterize the incident?
- Do any of the affected systems contain "sensitive data" (per PCI, HIPAA, etc.)?
- How many systems are involved? What are their status? What are their hard drive types, configurations, and capacities (SAN/NAS, RAID 5 with a total of 100GB, mirrored, single 1TB drive, boot-from-SAN)?
- What actions have you taken already (and be honest...running an AV scan and deleting files isn't "nothing")?
- What data have you already collected, if any?
- Do you have a logical network diagram, particularly of the affected systems?

Lots of questions...do you have the answers?

Summit Takeaways

I recently received an email from the organizers of the SANS Forensic Summit and was asked to provide my biggest takeaways from the summit, and I wanted to post what I came up with, to see if others are seeing the same sorts of things...so, here's what I provided:

I would say that the key take-away for me is that speed is essential, but should not be sacrificed for accuracy. Incident responders need to respond quickly in order to identify, classify, and as appropriate, contain an incident. In part, this puts the onus on the organizations...even if they have consultants on retainer, it will still take time for them to arrive, and then they are slowed down even further having to get up to speed and learn the environment. Consultants can assist an organization in developing a tier 1 response, training them in triage, diagnosis, preview, containment, etc., even across an enterprise. I would also say that tools such as F-Response are paramount to this sort of activity, as well. Once the initial training is complete, the consultants can respond as tier 2 or 3 responders, assisting as necessary, but knowing that the necessary data has been collected.

It's clear that state (via PII notification laws) and federal legislation, as well as regulatory oversight (PCI, HIPAA, etc.), play a huge role in incident response. In fact, these are the primary drivers to IR at this point. Think about it...if a company was not required to notify someone when a breach of sensitive data occurred, would they?

To that end, there needs to be a much more immediate response...calling a consulting firm to come on-site to assist after the incident has occurred and been detected is a fantastic idea, but one of two things are going to happen; you're either going to continue "bleeding" data while you're waiting, or you're going to stomp all over the data and destroy the indicators (aka, "evidence") that those consultants will be looking for in order to answer the questions that need to be answered.

Okay, brief digression here...what questions will need to be answered? There are essentially three questions that need to be answered in the face of a breach of sensitive data...(a) was a system compromised, (b) did the system contain sensitive data, and (c) was that sensitive data exposed or compromised as a result of the breach? In order to determine (c), in most cases, you need to minimize "temporal proximity" (thanks, AAron, for that very Star Trek-y term!!)...that is, you need to detect the breach and collect data as close to the time of the breach as possible.

Rather than fall victim to a breach and not get the answers you need (by "you", perhaps a more appropriate way of saying it would be "your legal counsel or compliance officer"), why not get someone in ahead of time, before an incident occurs, to work with you in setting up a response plan, and training your staff in a timely, accurate response?

I've said before that tools like F-Response and Volatility (the combination of the two being referred to as Voltage) have changed the face of incident response. Having these tools available to you allow you to quickly collect and analyze memory in order to triage and categorize incidents. Too many times I've been asked to come on-site only to find out that prior to the call, the customer had already turned systems off and taken them offline. These tools will help you collect more data than every before, reduce that "temporal proximity", and at the very least have data available for tier 2 incident responders to process and analyze.

Thoughts?

The Need for Speed

The recent Best Western issue illustrates an important point, which was mentioned in many of the posted articles on this issue...

Compliance != Security

In the face of compromises or any other potential/verified breach, a quick response is essential. You don't know if you have sensitive data (PCI, PHI, PII, etc.) leaving your network, and your first, most immediate and natural reaction (i.e., disconnecting systems) will likely expose you to more risk than the incident itself. Wait...what? Well, here's the deal, kids...if a system has sensitive data on it, and was subject to a compromise (intrusion, malware infection, etc.), and you cannot explicitly prove that the sensitive data was not compromised, you may (depending upon the legal or regulatory requirements for the data) be required to notify, regardless.

So...better to know than to not know...right?

What you need to do is quickly collect the following items:

1. Pertinent network (i.e., firewall, etc.) logs
2. Network packet capture(s)
3. Full or partial contents of physical memory
4. An image acquired from the affected system

Of course, this assumes a great deal...that you have a CSIRP in place, that you've already identified systems that store/process sensitive data, and that you've got the tools and training to collect any of this data. Some of it is fairly trivial...collecting firewall logs may simply be as easy as a call to your NOC. Collecting packet captures is as simple as having a Windows laptop with Wireshark installed, or Linux laptop. Tools for dumping the contents of RAM have recently taken a surge forward, and acquiring an image from a live system (can't shut that server down??) is as straightforward as getting a copy of FTK Imager.

Remember to DOCUMENT everything you do! The rule of thumb is, if you didn't document it, you didn't do it.

What other tools are available? In the case of Best Western, as well as any other organization with remote systems (located in distant data centers or storefronts), something like F-Response may prove to be extremely valuable! If you're not sure about F-Response and don't believe the testimonials, give the Beta Program a try. With the Enterprise Edition of F-Response already deployed (or simply pushed out remotely as needed), getting the data you need is amazingly straightforward!

So why do all this? Why go through all this trouble? Because you will likely have to answer the question, was sensitive data leaving my network? The fact of the matter is that you're not going to be able to answer that question with nothing more than a hard drive image, and the single biggest impediment to doing the right thing (as opposed to something) in a case like this is time...when you don't have the tools, training or support from executive management, the only reaction left is to unplug systems and hope for the best.

Unfortunately, where will that leave you? It'll leave you having to answer the question, why weren't you prepared? Would rather have to face that question, or actually be prepared?

If you want to learn what it takes to be prepared, come on by the SANS Forensic Summit and learn about this subject from the guys and gals who do it for a living!

Resources
CSO Online - Data Breach Notification Laws, State by State
SC Magazine - Data Breach Blog

The Demented Musings of an Incident Responder

I respond, therefore I am. H. Carvey, 2008

I thought I'd jot down some of the things I see from time to time in the field...

In some network infrastructures, there's no need to use rootkits or other obfuscation methods.

In fact, these attempts to hide an intruder's activity may actually get the intruder noticed...incorrectly programmed or implemented rootkits lead to BSODs, which get you noticed. Look at some of the SQL injection attacks...not the ones you saw in the news, but the ones you didn't see...well, okay...if you didn't see them, then you can't look at them. I get it. Anyway, the other SQL injection attacks would punch completely through into the interior network infrastructure, and the intruder would be on systems with privileges above Administrator. From there, they'd punch out of the infrastructure to get their tools...via TFTP, FTP (the ftp.exe client on Windows systems is CLI and allows you to use scripts of commands), their own wget.exe, etc. From there, the intruder would use their tools to extend their reach...create user accounts, etc...and many times go unnoticed.

However...while we are on the subject of rootkits...I was reading a post over on the Volatility Tumblr blog, and came across this interesting bit of analysis of the Storm bot from TippingPoint's Digital Vaccine Labs (DVL). Apparently, the rootkit capabilities of this malware were "copied" (their word, not mine) from existing code. The point of this is that some issues with rootkits are being solved by using existing, proven code. Symantec has a post on the Storm bot, showing that it doesn't just blow up on a system.

Many times, an organization's initial response exposes them to greater risk than the incident itself, simply by making it impossible to answer the necessary questions.

Perhaps more so than anything else, state notification laws (CA SB-1386, AB-1298, etc.) and compliance standards set forth by regulatory bodies ('nuff said!) are changing the face of incident response. What I mean by that is that rather than just cleaning systems (running AV, deleting files and accounts, or simply wiping and reinstalling the system) is no longer an option, because now we need to know if (a) there was "sensitive data" on the system, and then (b) if the malware or compromise led to the exposure of that data.

What this leads us to is that the folks closest to the systems...helpdesk, IT admins, etc...need to be trained in proper response techniques and activities. They need to have the knowledge and the tools in place in order to react quickly...like an EMT. For example, rather than pulling a system offline and wiping it, obtain a physical memory dump, disconnect the system from the network, obtain an image, and then wipe the system. Properly control, preserve, and maintain custody of the data so that forensic analysts can do their thing. This process is somewhat over-simplified, I know, but it's something that can be used as a basis for getting those important questions answered. Add to that network traffic captures and any available device logs, and you've pretty much got most of what incident responders such as myself are looking for when we are asked to respond.

What's in your clipboard

When I've written about performing incident response data collection (here and here), I've mentioned retrieving any available data from the clipboard. Others have mentioned the same thing. I've mentioned it as a way of collecting as complete a set of information as possible...what might appear to be the work of a Trojan may, in fact, be the doing of the user themselves. In the past, while working for a telecomm company, we found that a user was attempting to access routers using a GUI telnet app, and had copied the password he was using to the clipboard so that he could easily paste it into the GUI.

Just today, I saw this blog entry that identified malware that actually overwrites the contents of the clipboard, so that if the user pastes a URL into the address bar of the browser, the malicious one will be pasted instead.

So, just a thought...maybe it's about time for me to add entry back into my IR script. You can obtain a copy of pclip.exe here.