Rods and Cones, and EDR "blindness"

I ran across an interesting post recently regarding blinding EDR on Windows systems, which describes four general techniques for avoiding EDR monitoring. Looking at the techniques, I've seen several of these techniques in use on actual, real world incidents. For example, while I was with the Crowdstrike Overwatch team, we observed a threat actor reach out to determine systems with Falcon installed; of the fifteen systems queried, we knew from our records that only four were covered. We lost visibility because the threat actor moved to one of the other eleven systems. I've also seen threat actors "disappear from view" when they've used the Powershell console rather than cmd.exe, or when the threat actor has shell-based/RDP access to systems and uses a GUI-based tool. EDR telemetry includes process creation information, so we might "see" a GUI-based tool being launched but after that, no new processes are created based on whatever options the threat actor chose, or buttons they pushed, so without some other visibility (file system, Registry, network telemetry) we'd have no visibility into what they were doing.

I know this sounds pretty simplistic to some, but I really don't get the impression that it's commonly understood throughout the industry, and not just with customers.

I've previously discussed EDR bypass in this blog. Anyone who's been involved in a SOC or SOC-like role that involves EDR, including IR engagements where EDR is deployed, has seen many of the same conditions, such as incomplete deployment or roll-out of capabilities. A while back, for some products, EDR monitoring did not natively include the ability to block processes or isolate systems; this was a separate line item, if the capability was available. We see customers purchase the capability, and then complain when it didn't work...only to find out that it hadn't been deployed. I've seen infrastructures with 30,000 or 50,000 endpoints, and EDR deployed to only 200 or fewer systems.

The take-away here is that when it comes to EDR, finding a blind spot isn't really the profound challenge it's made out to be, particularly if you understand not only the technology but also the business aspect from the customer's point of view.

All that being said, what does this have to do with "rods and cones"? Okay, just bear with me for a moment. When I was an instructor in the military, we had an exercise for student Lts where we'd take them out at night and introduce them to the nuances of operating at night. Interestingly, I learned a great deal from this exercise, because when I went through the same school several years previously, we didn't have this exercise...I assume we were simply expected to develop an understanding ourselves. Anyway, the point is that the construction of the human eye means that the cones clustered around the center of the eye provide excellent acuity during daylight hours, but at night/low light levels, we don't see something when we're looking directly at it. Rather, for that object to come into "view" (relatively speaking), we need to look slightly to one side or the other so that the object "appears" to us via the rods of the eye.

What does this have to do with EDR? Well, assuming that everything else is in place (complete deployment, monitoring, etc.), if we are aware of blind spots, we need to ensure that we have "nearby" visibility. For example, in addition to process creation events, can we also monitor file system, Registry, and network events? Anyone who's dealt with this level of telemetry knows that it's a great deal of data to process, so if you're not collecting, filtering, and monitoring these events directly through EDR, do you have some other means or capability of getting this information if you need to do so? Can you retrieve/get access to the USN change journal and/or the MFT, to the Registry (on a live system or via triage retrieval), and/or to the Windows Event Logs? The fact is that while EDR does provide considerable visibility (and in a timely manner), it doesn't 'see' everything. As such, when a threat actor attempts to bypass EDR, it's likely that they're going to be visible or leave tracks through some other means which we can access via another data source.

An analogy I like to use at this point is that when walking by a pond, if someone throws a rock into the pond, we don't actually have to see them throw the rock, nor do we have see that a rock broke the surface of the pond, to understand what's happened. We can hear the splash and see ripples against the shore, and know that something happened. When it comes to monitoring endpoints, the same is true...we don't always have to 'see' an action to know that something happened, particularly something that requires a closer look or further investigation. Many times, we can observe or alert on the effect of that action, the impact of the action within the environment, and understand that additional investigation is required.

An additional thought regarding process creation is that process completion is a "blind spot" within some EDR products. SysMon has a process termination event (event ID 5), but most EDR tools only include process creation telemetry, and additional access and analysis are needed to validate whether the process executed completely, or if something occurred that prevented the process from completing normally. For example, many SOC analysts have likely seen threat actors use the Mp-Preference Powershell module to impact Windows Defender, and made statements in tickets and incident diaries to that effect, but what happens if Windows Defender is not enabled, and some other security control is used instead? Well, the command will have no effect; using the Mp-Preference module to, say, set exclusions in Defender will not result in Registry entries or the corresponding Windows Event Log records if Defender is disabled.

So, keeping in mind that for something bad to happen on a system, something has to happen, the take-away here is that EDR by-passes or blind spots are something to be understood, not feared. Many times, we can get that "good harvest from low-hanging fruit" (quote attributed to David Kleinatland), but sometimes we need a bit more. When we employ (or deploy) one tool, we need to understand both it's strengths and shortcomings, and come up with a plan to address any gaps. When considering EDR, we have to understand the impact of coverage (or lack thereof), both on the endpoint as well as across the infrastructure; we have to understand what it can and cannot 'see', how to properly configure it, and how to use it most effectively to "find bad". Sometimes, this may mean that we'll get a signal that something may be amiss, rather than something that clearly jumps out to us as "this is BAD!!!" Sometimes, we may rely on the impact of the action, rather than directly identifying the action itself.