Thoughts on APT

There's been a great deal of discussion lately about the advanced persistent threat, or APT, and I've seen list posts from folks adding their thoughts, or asking others to weigh in and provide any insight they may have. I see this as healthy, not only for customers, but also for the forensics community as a whole.

There are some things that are being said, quite clearly and repeatedly about this threat. For example, take a look at Wendi's post on the Mandiant blog; she presents some statistics from the M-Trends report that can give you an idea of what to look for if you suspect you've been compromised. I also think that if you view it the right way, and perhaps have a bit of context from other sources, you'll see that this upholds the Least Frequency of Occurrence (LFO) principle that Pete Silberman has described. So what this means is that responders and analysts need to look for the anomalies; not the massive spikes in activity, but the small, infrequent things that we may not notice in all the noise on a system, or infrastructure. The Mandiant folks mention this, and so do the HBGary folks...so, whether you're using LFO or MRI (thanks again, Pete!), or you're looking at digital DNA, you're looking for what is or should be standing out as anomalous and infrequent.

Okay, so...what about APT?

As I see it, there are three major groups of actors here...the good, the bad, and the ugly victim. The victims are pretty clear. The bad guys are the developers, purveyors and operators of exploits and other mechanisms (i.e., code, malware, etc...call it what you will) for malicious purposes. The good guys are LE, responders, corporate consultants, etc...those folks trying to assist the victims, most often after a data breach.

Now, a number of the good guys have been (or started) posting reports (see the Reports section at the end of this post) illustrating statistics based on the incidents they've responded to and the work they've done. Reading through these, we see a lot of information much like what Wendi included in her post. Perhaps the most important thing, in my mind, is that the numbers and information from these reports indicate that there was a cultural shift in the bad guy's realm. What I mean by that is that back "in the day", most of what we saw was malware that ran amok on networks, and folks blowing out SubSeven or NetBus to systems so that they could open and close the CD-Rom tray. No more. Systems are being targeted for either the access they provide or the data that they store and process. Malware is being modified enough so that current AV products don't detect new variants, and footprints of that malware are minimized, using mutexes so that the system is only infected once. I attended a conference in Redmond back in November, 2009, and in several of the presentations, LE stated that the bad guys are dedicated, patient, smart, well-funded, and they have an economic goal behind what they're doing.

From my perspective as a responder and analyst, as well as from reading the reports and compiled statistics, what I'm not seeing is a corresponding paradigm shift on the part of the organizations that fall victim to these intrusions and compromises. Intrusions are still going undetected; victims are being notified by external third parties weeks or months after the fact. Systems are still being compromised via SQL injection and the use of poor passwords by administrators.

One thing that really stands out in my mind is that looking at my own experience, as well as the experience of others (via reports and postings on the web), the victims are not experiencing a cultural shift that corresponds to what the bad guys have gone through. Even in the face of information that indicates that the cost of data breaches has increased, organizations continue to be breached. In all fairness, breach attempts are going to happen; however, at least one report indicates that as many as 70% of data breach victims responded to find out well after the breach from an external third party.

The point is that the bad guys have identified targets and have an economic stimulus of some kind for attaining their goals. They're dedicated and compartmentalized...someone is dedicated to discovering vulnerabilities, and often it appears to be a different party all together that employs the exploit and some new piece of malware. For the victims, we're still seeing incident prevention, detection, and response all being secondary or tertiary duties for overworked IT staff...so while the bad guys can dedicate time and resources toward getting into an organization, IF there are dedicated responders within the organization, and IF they have any recent training or experience, and IF anyone actually knows where the data resides...well, you can see my point. From the perspective of a historical military analogy, this appears to be akin to special operations forces attacking villages defended by farmers and shopkeepers.

Maybe I'm way off base here, but this whole discussion of APT seems to be showing us something that's a bit more of an expansive issue. My thinking here is that if those organizations that are storing and processing "sensitive data" (choose your definition du jour for "sensitive data") were to have a corresponding cultural paradigm shift, we might begin to see intrusions detected and responded to in a manner that would provide data and intel to law enforcement, such that there could ultimately be arrests. I know, this is easier said than done...look at the issues that have sprung up around compliance; all compliance is...really...is an attempt to mandate or legislate minimum levels of security that organizations should have already had in place. I don't want to cloud the issue (no pun intended), but my overall point here is that maybe law enforcement would be able to make arrests if they had data and intel. As a responder, too often have I arrived on-site for an incident where the customer was informed of an issue by an outside third party; no one knows definitively where critical data resides, there are no logs available, and administrators have already done "nothing", which in reality amounts to an extensive list of removing systems from the network, scanning them with AV, deleting files, and even wiping entire systems.

So we know that the bad guys are having fairly high rates of success compromising systems and infrastructures using, in some cases, well-known vulnerabilities that simply hadn't been patched. We know that in many cases, they don't need to use special privilege escalation exploits, because they get in with Administrator/root/superuser privileges. We know that in most cases, they don't upload massive sets of tools, but instead use native utilities or only one or two malware files. We know that rootkits simply don't have to be used to hide the bad guy's presence...why hide from someone who's not looking for you?

So the take away, for me, from these reports is simply that there needs to be a cultural shift on the part of those who store and process sensitive data, and it has to come from the top down. It's 2010, folks...do we still need to sell infosec to senior management? What should be the CEO's concern...that his email and IM are up and running, or that the sensitive data that his company stores and processes is secure, and his infrastructure monitored?

Reports
7Safe (UK)
Verizon
Mandiant

Addendum: There's a bit of a different perspective on APT and what it really means over at TaoSecurity (here, and commentary on the M-Trends report here). For another view or perspective on the M-Trends report, see what IntelFusion says.

One thing to keep in mind about the reports...remember that they're based on numbers compiled by the perspective groups. Each group may have a different customer base and primary line of business when it comes to what they do. What this means is that each report is going to represent a slightly different culture when it comes not only to the numbers but also what they represent.

Analysis Stuff

Metadata
Didier has posted new versions of his PDFiD and pdf-parser tools. Didier's offerings really kind of run the gamut, don't they? Well, hey...it's all good stuff! I mean, really, other than the fact that he's updated these really great tools, what else needs to be said?

Malware
The MMPC posted some new malware descriptions recently, regarding Hamweq and Rimecud. Nice names. Signatures have been added to MRT, apparently.

An interesting aspect of Hamweq is that it apparently drops files in the Recycle Bin, and uses the Installed Components key in the Registry as a persistence mechanism. I wrote up a quick and dirty plugin for the key...entries with StubPath values that point to the Recycle Bin would be suspicious, even if they didn't indicate Hamweq was installed, specifically.

Other malware has other artifacts and persistence mechanisms. Take Win32.Nekat as an example...this one adds entries to the user's Control Panel\don't load key, effectively removing the applets from view. While not overly sophisticated (I mean, it is native functionality...), something like this would be enough to slow down most users and many admins. And yes, there is an app a plugin for that (actually, it was pretty trivial to write...one of several that I wrote yesterday).

APT
With the Google thing, there's been considerable discussion of advanced persistent threat, or APT, lately. I'm not going to jump on that bandwagon, as there are lot of folks smarter than me talking about it, and that's a good thing. Even Hogfly has talked about APT.

I get the "threat" thing, but what I'm not seeing discussed is the "advanced" part. Wendi over at Mandiant blogged about M-Trends and APT, noting some...uhm...trends such as outbound connections and mechanisms used to avoid anomaly detection. One of those mechanisms listed is "service persistence", which sounds like the malware is installed as a Windows service, a persistence mechanism. While I do think that it's a good idea to talk about this kind of thing, what I'm not seeing a lot of right now is actionable intel. Wendi and Hogfly presented some very useful information, demonstrating that all of this talk still comes down to a couple of basic questions; Have I been breached, and am I infected? How do I find out? What do I do if I am? How do I protect myself? So someone looks at both posts and uses the information there to look and see if they've been breached. If they don't find anything, does that mean they're clean? No, not at all...what it means is that you didn't find what you searched for, and that's it. Both posts presented information that someone can use to scour systems, but is that all that's really available?

I think that a very important concept to keep in mind when doing this kind of analysis is what Pete Silberman said about malware; he was absolutely correct when he described it as having the least frequency of occurrence on a system. Think about it. Malware, particularly worms, no longer want to keep infecting systems over and over again, so they use a single, unique mutex to say, "hey, I'm infectin' here!". That way, the system doesn't get so massively infected that it stops functioning; not only does that alert folks that somethings wrong, but it also deprives the attacker of the use of the system. So, you can run handle.exe from MS on a live system, and then run the output through handle.pl and see mutants listed by least frequency of occurrence.

I'm going to throw this out there...run it up the flagpole and see who salutes, as it were...but I think that the same sort of thing applies to intrusions and breaches, as well. For example, Windows systems have thousands of files, and intruders may be installing some tools to assist them in persistence and propagation, but the fact is that there are a number of native tools that are perfect for what most folks want to do. I mean, why install a tool to locate systems on the network when you can use native tools (ipconfig, netstat, nbtstat, etc.)? So intruders don't have to install or add something to the compromised system, unless the required functionality is not available in a native utility. Remember what Wendi said in her M-Trends post about using services for persistence? How many do you think are there? Do intruders install their persistence mechanisms 50 or 100 times? No...likely, they only do it once. And they can either hide as svchost.exe, pointed at an executable in another location, or beneath the legit svchost.exe, using the ServiceDll entry (and adding an entry to the appropriate SvcHost key value in the Software hive).

Timeline Analysis
To illustrate my point about least frequency of occurrence, let's talk briefly about timeline analysis. Given the minimalist nature of malware and intrusions, how can we use timeline analysis to our advantage? The approach I've been advocating allows the analyst to see multiple events, from many sources, side-by-side for comparison and analysis.

One of the things that folks ask about with respect to timeline analysis is a graphical means for representing all of the data, in order to assist the analyst. IMHO, this simply does NOT work! I have yet to find a viable means of taking in all of the data from various sources, throwing it into a timeline and hoping to use a graphical representation to pick out those anomalies which happen least often. As an example, check out Geoff Black's CEIC 2007 Timeline Analysis presentation...the fourth slide in the pack illustrates a graphical file system metadata timeline within EnCase. I agree with Geoff's assessment...it's not particularly effective.

Overall, in order to get a grip on APT, we as responders and analysts need to change our mindset. We need to understand that we can't keep looking for spikes in behavior or activity, and there is no Find All Evidence button. When you respond to or analyze a single Windows system, and consider the changes in OS, various applications that can be installed, and the infrastructure itself, what constitutes an anomaly? Honestly, I don't think that this is something a software package can tell you.

I do, however, firmly believe that training and education are the key.