Friday, December 21, 2012

Are You Being Served?

Larry Daniel recently posted to his Ex Forensis blog regarding a very interesting topic, regarding "The Perils of Using the Local Computer Shop for Computer Forensics".  I've thought about this before...when I was on the ISS ERS (and later the IBM ISS ERS) team, on more than one occasion we'd arrive on-site to work with another team, or to take over after someone else had already done some work.  In a couple of instances, I worked with other teams that, while technically skilled, were not full-time DFIR folks.  Larry's post got me to thinking about who is being asked to perform DFIR work, and the overall effect that it has on the industry.

There's a question that I ask myself sometimes, particularly when working on I doing all I can to provide the best possible product to my customers?  As best I can, I work closely with the customer to establish the goals of the exam, to determine parameters of what they are most interested in.  I do this, because like most analysts, I can spend weeks finding all manner of "interesting" stuff, but my primary interest lies in locating artifacts that pertain to what the customer's interested in, so that I can provide them with what they need  in order to make the decisions that they need to make.  As much as I can, I try to find multiple artifacts to clearly support my findings, and I avoid blanket statements and speculation, as much as I can.

Also, something that I do after every exam is take a look at what I did and what I needed to do, and ask myself if there's a way I could do it better (faster, more comprehensive and complete, etc.) the next time.

Let's take a step away from DFIR work for a moment.  Like many, I make use of other's services.  I own a vehicle, which requires regular upkeep and preventative maintenance.  Sometimes, if all I need is an oil change, I'll go to one of the commercial in-and-out places, because I've looked into the service that they provide, what it entails, and that's all I need at the moment.  However, when it comes to other, perhaps more specialized maintenance...brake work, inspections recommended by the manufacturer, as well as inspections of a trailer I own...I'm going to go with someone I know and trust to do the work correctly.  Another thing I like about working with folks like this is that we tend to develop a relationship where, if during the course of their work, they find something else that requires my attention, they'll let me know, inform me about the issue, and let me make the decision.  After all, they're the experts.

Years ago...1992, in fact...I owned an Isuzu Rodeo.  I'd take it to one of the drive-in places to get the oil changed on a Saturday morning.  The first time I took it to one place, I got an extra charge on my bill for a 4-wheel drive vehicle.  Hold on, I said!  Why are you adding a charge for a 4-wheel drive vehicle, when the vehicle is clearly 2-wheel drive?  The manager apologized, and gave me a discount on my next oil change.  However, a couple of months later, I came back to the same shop with the same vehicle and went through the same thing all over again.  Needless to say, had I relied on the "expertise" of the mechanics, I'd have paid more than I needed to, several times over.  I never went back to that shop again, and from that point on, I made sure to check everything on the list of services performed before paying the bill.

Like many, I own a home, and there are a number of reasons for me to seek services...HVAC, as well as other specialists (particularly as a result of Super Storm Sandy).  I tend to follow the same sort of path with my home that I do with my vehicles...small stuff that I can do myself, I do.  Larger stuff that requires more specialized work, I want to bring in someone I know and trust.  I'm a computer nerd...I'm not an expert in automobile design, nor am I an expert in home design and maintenance.  I can write code to parse Registry data and shell items, but I am not an expert in building codes.

So, the question I have for you, reader, is do you know that you're getting quality work?  To Larry's point, who are you hiring to perform the work? 

At the first SANS Forensic Summit, I was on a panel with a number of the big names in DFIR, several of whom are SANS instructors.  One of the questions that was asked was, "what qualities do you look for in someone you're looking to hire to do DFIR work?"  At the time, my response was simply, "what did they do last week?"  My point was, are you going to hire someone to do DFIR work, if last week they'd done a PCI assessment and the week prior to that, they'd performed a pen test?  Or would you be more likely to hire someone who does DFIR work all the time? I stand by that response, but would add other qualifications to it.  For example, how "tied in" are the examiners?  Do they simply rely on the training they received at the beginning of their careers, or do they continually progress in their knowledge and education?  Do they seek professional improvement and continuing education?  More importantly, do they use it?  Maybe the big question is not so much that the examiners do these things, but do their managers require that the examiners do these things, and make them part of performance evaluations?

Are you being served?

Addendum:  Why does any of this matter?  So what?  Well, something to consider is, what will a CEO be reporting to the board, as well as to the SEC?  Will the report state, "nothing found", or worse, will the report be speculation of a "browser drive-by"?  In my experience, most regulatory organizations want to know the root cause of an issue (such as a compromise or data leakage)...they don't want a laundry list of what the issue could have been.

In addition, consider the costs associated with PCI (or any other sensitive information) data theft; if an organization is compromised, and they hire the local computer repair shop to perform the "investigation", what happens when PCI data is discovered to be involved, or potentially involved?  Well, you have to go pay for the investigation all over again, only this time it's after someone else has come in an "investigated", and this is going to have a potentially negative effect on the final report.  I think plumbers have a special fee for helping folks who have already tried to "fix" something themselves.  ;-)

Look at the services that you currently have in your business.  Benefits management.  Management of a retirement plan.  Payroll.  Do you go out every month and select the lowest bidder to provide these services?  Why treat the information security posture of the your organization this way?


Cheeky4n6Monkey said...

How can we know we are doing/getting quality work?
As a (naive?) newbie, I would think having a common standard of training would help.

Clients can get a basic indicator of their potential investigator's skillset/ability to learn.
Investigators will have a structured method for improvement.

But in reality ...
- multiple jurisdictions of the world
- different areas of investigation (eg crime vs e-discovery)
- the changing nature of technology
will probably mean we will be stuck with a variety of organisations and no singular certifying body.
And if clients can't find a definitive indicator of "expertise", they will probably default to the lowest bidder aka the computer shop.

Anyway, just my 2 cents ... :)

H. Carvey said...


Thanks for the comment.

...we will be stuck with a variety of organisations and no singular certifying body.

I don't know...I think that the CDFS may be making some in-roads in this direction.

...if clients can't find a definitive indicator of "expertise", they will probably default to the lowest bidder...

That's really no different than what's happening now.

At a recent conference, I asked a room full of people, who was performing shellbag analysis...two people raised their hands, and one of the stated that they hadn't actually done that type of analysis since they had attended SANS training. I then asked who in the room were interested, during exams, in determining a user's actions...and about 80-90% of the hands went up.

So, my question would be, why isn't everyone doing shellbag analysis, or at least asking about it? Are they not doing it because it was not part of the insert vendor name training they attended? Or, are they not doing it because it's not something that their management comes back and asks them about after they've submitted a report? it not something they do because someone else didn't put it on the checklist?

Going back to your comment (which I tend to agree with...), how is the consumer of DFIR services going to make a determination as to the quality of services they are rendered? My first thought was to look at the reporting...what does the analyst say that they do or did in the report? However, I'm not sure how much they would be able to discern from that report.

I don't's an interesting question.

Unknown said...


The challenge of picking a qualified analyst really transcends DFIR. It could be rephrased to state how do I find a quality attorney, CPA, or anyone who offers a professional service.

It's a very good question you ask, by the way, because in my experience, there is no simple formula.

Let me give you the example of selecting an corporate attorney for a firm I worked for. We did our research and ultimately picked a "big name" AMLAW 100 firm whose client list was a who's who of Silicon Valley hot start-ups. The Partner who represented us did nothing but corporate work for start-ups (which we needed) and went to Stanford Law. So we were very comfortable with out pick. Top firm, the Partner has impeccable credentials...we would be well served, right? Wrong.

We weren't well served. The quality of the work and advice we received was consistently below our expectations, and certainly not to the level that earned them such a lofty reputation.

The problems was, the partner was overextended. He had taken on too many clients and he had more work than he could handle. And since the firm I worked for wasn't the next Facebook, our work got pushed to his Senior Associate...then to the Junior Associate, and then to the Paralegal.

So we thought we were doing the right thing by picking based on the firms reputation and the attorney's credentials, but none of that matters if he doesn't do the work. Bait and switch. Pay for the Stanford grad but the work gets done by a paralegal flunky.

So we switched firms. We continued to look at the firm's reputation but we also got guarantees on who would do the work and we asked very direct questions about how much work the Partner did vs. the Associates. That made a huge difference.

So my advice to anyone looking for an DFIR analyst/consultant would be the same. Pay attention to credetials and industry involvement, but if you're a small operation, think twice about going to a big firm where you could be lost in the shuffle. A smaller firm that can dedicate time and resources to you could just be the better option.