Friday, January 29, 2010

Thoughts on APT

There's been a great deal of discussion lately about the advanced persistent threat, or APT, and I've seen list posts from folks adding their thoughts, or asking others to weigh in and provide any insight they may have. I see this as healthy, not only for customers, but also for the forensics community as a whole.

There are some things that are being said, quite clearly and repeatedly about this threat. For example, take a look at Wendi's post on the Mandiant blog; she presents some statistics from the M-Trends report that can give you an idea of what to look for if you suspect you've been compromised. I also think that if you view it the right way, and perhaps have a bit of context from other sources, you'll see that this upholds the Least Frequency of Occurrence (LFO) principle that Pete Silberman has described. So what this means is that responders and analysts need to look for the anomalies; not the massive spikes in activity, but the small, infrequent things that we may not notice in all the noise on a system, or infrastructure. The Mandiant folks mention this, and so do the HBGary, whether you're using LFO or MRI (thanks again, Pete!), or you're looking at digital DNA, you're looking for what is or should be standing out as anomalous and infrequent.

Okay, so...what about APT?

As I see it, there are three major groups of actors here...the good, the bad, and the ugly victim. The victims are pretty clear. The bad guys are the developers, purveyors and operators of exploits and other mechanisms (i.e., code, malware, it what you will) for malicious purposes. The good guys are LE, responders, corporate consultants, etc...those folks trying to assist the victims, most often after a data breach.

Now, a number of the good guys have been (or started) posting reports (see the Reports section at the end of this post) illustrating statistics based on the incidents they've responded to and the work they've done. Reading through these, we see a lot of information much like what Wendi included in her post. Perhaps the most important thing, in my mind, is that the numbers and information from these reports indicate that there was a cultural shift in the bad guy's realm. What I mean by that is that back "in the day", most of what we saw was malware that ran amok on networks, and folks blowing out SubSeven or NetBus to systems so that they could open and close the CD-Rom tray. No more. Systems are being targeted for either the access they provide or the data that they store and process. Malware is being modified enough so that current AV products don't detect new variants, and footprints of that malware are minimized, using mutexes so that the system is only infected once. I attended a conference in Redmond back in November, 2009, and in several of the presentations, LE stated that the bad guys are dedicated, patient, smart, well-funded, and they have an economic goal behind what they're doing.

From my perspective as a responder and analyst, as well as from reading the reports and compiled statistics, what I'm not seeing is a corresponding paradigm shift on the part of the organizations that fall victim to these intrusions and compromises. Intrusions are still going undetected; victims are being notified by external third parties weeks or months after the fact. Systems are still being compromised via SQL injection and the use of poor passwords by administrators.

One thing that really stands out in my mind is that looking at my own experience, as well as the experience of others (via reports and postings on the web), the victims are not experiencing a cultural shift that corresponds to what the bad guys have gone through. Even in the face of information that indicates that the cost of data breaches has increased, organizations continue to be breached. In all fairness, breach attempts are going to happen; however, at least one report indicates that as many as 70% of data breach victims responded to find out well after the breach from an external third party.

The point is that the bad guys have identified targets and have an economic stimulus of some kind for attaining their goals. They're dedicated and compartmentalized...someone is dedicated to discovering vulnerabilities, and often it appears to be a different party all together that employs the exploit and some new piece of malware. For the victims, we're still seeing incident prevention, detection, and response all being secondary or tertiary duties for overworked IT while the bad guys can dedicate time and resources toward getting into an organization, IF there are dedicated responders within the organization, and IF they have any recent training or experience, and IF anyone actually knows where the data resides...well, you can see my point. From the perspective of a historical military analogy, this appears to be akin to special operations forces attacking villages defended by farmers and shopkeepers.

Maybe I'm way off base here, but this whole discussion of APT seems to be showing us something that's a bit more of an expansive issue. My thinking here is that if those organizations that are storing and processing "sensitive data" (choose your definition du jour for "sensitive data") were to have a corresponding cultural paradigm shift, we might begin to see intrusions detected and responded to in a manner that would provide data and intel to law enforcement, such that there could ultimately be arrests. I know, this is easier said than done...look at the issues that have sprung up around compliance; all compliance an attempt to mandate or legislate minimum levels of security that organizations should have already had in place. I don't want to cloud the issue (no pun intended), but my overall point here is that maybe law enforcement would be able to make arrests if they had data and intel. As a responder, too often have I arrived on-site for an incident where the customer was informed of an issue by an outside third party; no one knows definitively where critical data resides, there are no logs available, and administrators have already done "nothing", which in reality amounts to an extensive list of removing systems from the network, scanning them with AV, deleting files, and even wiping entire systems.

So we know that the bad guys are having fairly high rates of success compromising systems and infrastructures using, in some cases, well-known vulnerabilities that simply hadn't been patched. We know that in many cases, they don't need to use special privilege escalation exploits, because they get in with Administrator/root/superuser privileges. We know that in most cases, they don't upload massive sets of tools, but instead use native utilities or only one or two malware files. We know that rootkits simply don't have to be used to hide the bad guy's presence...why hide from someone who's not looking for you?

So the take away, for me, from these reports is simply that there needs to be a cultural shift on the part of those who store and process sensitive data, and it has to come from the top down. It's 2010, we still need to sell infosec to senior management? What should be the CEO's concern...that his email and IM are up and running, or that the sensitive data that his company stores and processes is secure, and his infrastructure monitored?

7Safe (UK)

Addendum: There's a bit of a different perspective on APT and what it really means over at TaoSecurity (here, and commentary on the M-Trends report here). For another view or perspective on the M-Trends report, see what IntelFusion says.

One thing to keep in mind about the reports...remember that they're based on numbers compiled by the perspective groups. Each group may have a different customer base and primary line of business when it comes to what they do. What this means is that each report is going to represent a slightly different culture when it comes not only to the numbers but also what they represent.


Anonymous said...

Harlan, this is a great write up on one of the many issues we face as IR's and Examiners. I feel like a salmon swimming upstream almost, no, every day. Please keep up the great work. Your work and analysis assist me greatly as Im a one man show in a huge company ans welcome all the assistance I can get.

Paul Bobby said...

The corporations that are part of the Defense Industrial base have executed this cultural/paradigm shift - we began this process in '07.

Anonymous said...

When do you think we will start to see the benefits of that cultural shift, Paul? I think we need staff to concentrate on "offensive" security in order to be able to "defend" from these attacks.

Kai Axford said...

Great write-up on this topic! This is the next big issue as the threat landscape continues to evolve. More and more we are seeing the topic of information/economic warfare becoming a major concern. The recent Aurora attack was an obvious example. Look forward to more coverage and your insight.


jbmoore said...

China has our industry. Now the battle for knowledge shifts from what we can teach them and help them build to what information we know and covet. Why is anyone surprised by this shift to a higher level of knowledge acquisition? Why are people surprised that stealth and persistence are valued in these attacks? None of this should be a surprise to anyone. And why does it take a year for these disclosures to make the news? That's a year for the bad guy's to stay ahead of the curve and our "leaders" to be out of the loop policy wise.

Benjamin Wright said...

Yes, these threats demand the urgent attention of CEOs. Thinking about this from the CEO perspective, I argue that the public communications response to a security incident is becoming just as important as the technical (or even legal) response. Thoughtful public messages are part of an effective security program. Observe how much public noise Google made two weeks ago regarding the attack on it. Whether you agree with what Google said or not, it's clear that public relations is central to the way Google handled this attack. The PR response came from the top of the company. -Ben

H. Carvey said...


Thanks for you comments.

I'd like to reiterate my point, and that is that if the bad guys are compartmentalizing skill sets and employing specialized skills with a dedicated, economic goal in mind, wouldn't it then be prudent for those organizations that could fall victim to do something similar?

To illustrate my point, let's look at a data breach engagement...bad guy gets in via SQL injection, puts a couple of unique tools on a system or two, and then uses native tools to locate and extract "sensitive data" (IP, manufacturing or bid plans, PCI/PII/PHI data, etc.).

Weeks later, someone on the outside finds out about this and let's the victim know. They call for help, and that responder has to spend hours convincing the IT staff that based on the web server logs, there is, in fact, a database somewhere within the infrastructure that provides dynamic content to the web server. In short order, the IT staff becomes overwhelmed and/or disinterested and leaves the responder to figure out what happened.

In the course of the examination, the responder determines that not only had there been probes and recon preceding the actual SQL injection attack, but once inside, the intruder moved around system to system with relative ease due to a shared domain admin account with a weak password (i.e., was easily guessed). In short, the entire incident was so rife with policy and 'best practice' violations that the responder found indications of several other incidents that had gone completely unnoticed.

From my experience conducting PCI forensic assessments, this is a pretty typical scenario. Very rarely did I arrive on-site to find anyone who knew where the "sensitive data" was stored/processed.

Again, my point is that organizations storing/processing sensitive data of any kind can no longer sit back and remain oblivious to data being exfiltrated from their systems.

paulbain said...

Harlan Carvey wrote:
It's 2010, folks . . . do we still need to sell [the need for] infosec [information security] to senior management? What should be the CEO's concern . . . that his email and IM are up and running, or that the sensitive data that his company stores and processes is secure, and his infrastructure monitored?

IMO, a useful term would be "Stalag 13 security," by which I mean "a false sense of security that arises from either stupidity or ignorance." The majority of my employers (the ones that I have had in recent years) suffered from Stalag 13 security. IMO, Stalag 13 security explains many decisions regarding information security that are made by CEO's, CIO's, and CFO's. IMO, few of those clowns understand info sec. They have no clue.

-- Paul D. Bain

Anonymous said...

Note to Paul Bobby: Paul, *some* defense contractors have had a paradigm shift, and some are now doing incredible work in this area - I applaud those who are. Unfortunately, many more are still blind to the problem and fail to understand the ramifications of the threat. I have seen some of these organizations receive the "third-party" notification Harlan refers to, only to dismiss the issue ("we blocked traffic to that external domain, and the activity stopped" or "we couldn't find the computer you told us about" [so we ignored the fact that something Really Bad is on our network]). Having been in IT security for 10+ years, and dealing with APT for close to 7 now, the overall state of corporate security is still lagging far behind the methods and capabilities of the threat.

Harlan: Thank you for your plug for information sharing with law enforcement. Many of the companies who are successfully battling APT are doing so in part through partnership with LE. LE can provide briefings or, in some cases, more detailed information about the threat, and can assist should you have an incident. Information provided to LE allows them to do their job - follow up on these attacks, investigate the perpetrators, and work to stop this activity. It can be a mutually beneficial relationship, and it *does* work to address the problem. I encourage people to reach out to their local LE (through local infosec groups like ISSA, or partnerships such as Infragard) for additional options.

Christa M. Miller said...

We've talked about this before -- corporations believe they have an economic DISincentive when it comes to IR. They believe they are still taking the chance that it will not happen to them, that bad PR is worse for their bottom line than quietly dealing with a breach afterward.

As you point out, though, with the frequency of incidents increasing, companies can't ignore the rising chances that they'll be compromised. Therefore they need careful cost-benefit projections comparing action vs. inaction.

Benjamin is spot on above that PR is a big part of these issues. Unfortunately many companies don't have adequate crisis communications plans in place at all, much less those dealing with security breaches. PR needs to be worked into the above cost-benefit analysis, because as he points out, it can end up offsetting a lot of the cost if done correctly.