Kicking off the Information Security D-List Interviews is Paul “PaulDotCom” Asadoorian. Paul gave me the opportunity to pick his mind via email and IM over the past week.

Tell me a little about yourself.

I live in Rhode Island, and have all my life, where I have always been a computer geek and lots of other things. I started programming when I was 7 years old on the Apple IIe computer. I’m somewhat of an over achiever and earned the nick name “Salad Shooter” shortly after I founded “PaulDotCom”. My first real job in the industry was an intern for a small software company. I did some programming, but my primary job was “Backup Boy”, or “BUB” for short. As the “BUB” I had to go around to all of the development systems and perform backups. Keep in mind this was well before USB thumb drives, and involved magnetic tape drives, pliers, screw drivers, and DoS like commands on a PoS specific operating system (IBM 4690). I am thankful for all that I learned and appreciate things more getting my start at the bottom (I mean it doesn’t get any more bottom than being called “BUB”).

Currently I am the “Product Evangelist” for Tenable Network Security. Its my job to use the products in real-world environments and tell people about the features and use-cases. For PaulDotCom, I produce and host the weekly “PaulDotCom Security Weekly” podcast, which now includes both audio and video. I also participate in our security consulting work, performing penetration tests and web application assessments.”

How did you get interested in information security?

I was working as system/network administrator for a small company, you know the right of position where you have to know something about everything (UNIX/Linux, Windows, Networking, Printers, phone switches, etc…). I started to grow tired of Windows and was beginning to work with Linux (I installed Red Hat 5.2 from floppy disks). I appreciated the control, but knew that it came with great responsibility when it came to security. My friend’s computer had gotten hacked, so it got me curious about security. I started to secure computers in the office with mixed success, some would be fine after system hardening, and some would not function as well. This proved very challenging and really started to become a focus in my career. I left that job and took a full time position as a UNIX systems administrator, primarily Solaris. After working there from some time two things happened that put me on the track for security: 1) The firewall admin got sick and I took over firewall maintenance for several Checkpoint firewalls 2) We had to undergo a security audit and I was tasked with hardening our 20+ UNIX systems. I’ve never looked back and made security the focus in my career ever since!

What is your educational background (e.g. formal schooling, certifications, self-learning, etc.) and did it add value to your information security career?

Certainly, I have a Bachelors of Science in Information Systems, with a strong focus on business (I graduated from Bryant College). I’d say that my computer courses helped me to fill in the gaps of everything that I learned on my own and on my job. I also earned two SANS level certifications (before there were silver and gold), which was very rewarding and really helped my career. I believe that certifications are valuable, especially when you are starting out. For me, it helped me learn and apply so much to my career, even more than it was a resume builder. Certifications are what you make of them, and I did my best to make the most of them by studying hard, creating a lab at home

What did you want to be when you grew up? Would you rather be doing that?

I so wanted to be a baseball player when I was growing up. I don’t think that would be the best career for me now, especially seeing as I wasn’t always very good at it! If I had to choose a new career it might be as a martial arts instructor, fishing guide, or furniture maker.

What projects (if any) are you working on right now?

Well, there is the usual stuff I have going on at PaulDotCom. We are working hard to expand and grow in the areas of Internet radio and Internet TV. My other research project that I will be embarking on soon has to do with embedded systems. I did a lot of research on the embedded side, and even some presentations on the security aspects. I’ve let it rest for too long and want to get back into it. I don’t want to give it all away, but the goal is to raise awareness on how widespread and dangerous embedded security problems are today, and how its only going to get worse, not better.

It looks like you’re invovled in a lot of projects. How do you balance family time with project time?

I think the answer you will hear from many of us is balance with work and family life is a constant struggle. I’d say that sometimes I do it really well, and other times I fail miserably at it. I always try to learn from each experience and try to get better at time management as time goes on. I think I’m getting better, my family may have other thoughts 😉

How do you guys come up with “themes” for the podcast?

At this point it really happens very naturally. I think in the beginning we really had to struggle to come up with content, such as stories for discussions and technical segments. We did a lot of listener feedback in the beginning as well. Since then we have made the technical segments a formal part of the show, and try to do at least one per episode. A technical segment is a “How-To”, including audio and a wiki page entry, explaining to the audience how to do something technology or security related. We draw on our experiences, so its usually whatever we were working that week. The Wiki has also been a tremendous success, as we create a wiki page for each show and document the tech segments, interview questions, and stories for discussion each week. I can’t imagine trying to produce a show without the Wiki
technology! Overall, we try to focus on whats happening in information security, but without being the “accident jumpers” of security media. For example, we like to have on guests that are maybe not so well known, but are the true “rock stars” of the industry.

What is your favorite security conference (and why)?

Just like beer and cigars, its tough to have just one. I really like Shmoocon and Defcon, they are fun in their own unique ways, so I always make sure myself and the entire PaulDotCom team make it out to both events. I meet so many cool people and share new ideas at those two conferences especially.

What do you like to do when you’re not “doing security”?

I of course enjoy spending time with my family (they call me “Clark Griswald” around this time of year). My non-tech relaxing hobbies include fishing from my small freshwater boat, and smoking cigars (preferably fishing while smoking cigars). I also practice martial arts, including Kung Fu and Tai Chi.

What area of information security would you say is your strongest? What about your weakest?

I’d say I’m the strongest in penetration testing and network security, and a splash of embedded systems knowledge. My weakest area is probably deep, in-depth system forensics, I know enough to get by, but its not my area of specialty.

What advice can you give to people who want to get into the information security field?

Wow, that’s a loaded question! I’ve been asked this question many times, and discussed the topic on the PaulDotCom show a few times as well. I finally made a blog post which goes through in detail how I recommend people get their start in information security.

How can people get a hold of you (e.g. blog, twitter, etc.)

My blog is http://pauldotcom.com, but its not just me who blogs, it’s the entire PaulDotCom team. I am also on Twitter as @pauldotcom where I can be found announcing various PaulDotCom things, talking about cigars, information security, and keeping others in check 😉

After speaking with Harlan Carvey on several online communities we both frequent he agreed to be interviewed on Windows forensics and his new book:

How did you get into forensics?

I started in the commercial infosec arena as a consultant doing vulnerability assessments and pen tests. At one point, I started working for a company, and a forensics guy needed some assistance. With something of a security background and a clearance, as well as some technical knowledge, I helped out and began to see the other side of the coin. I began to see the early stages of understanding that Locard’s Exchange Principle applied to the digital world just as well as the physical world.

From there, I had opportunities to not only talk to and ask questions of folks performing “forensic investigations”, but I started performing my own incident response, and looking for ways to do my job better. From there, I grew into the forensics field.

What training would you recommend to those interested in entering the forensics field?

Right now, it’s hard to say much about “training”. Much of what’s needed is specialized…however, I believe that a great deal of what’s needed can be taught to anyone who is interested in learning. Training itself does not suffice…I believe that you have to have an interest in the field. Hand-in-hand with the training is some kind of standard that needs to be met…too much of the forensics field is considered an “art” at this point.

I would say, however, that a great way to get started is to have a broad base of knowledge and understanding of network computing. You don’t have to be an expert in writing TCP/IP-based applications, but you do need to have a fundamental understanding of network communications (peer-to-peer, client-server, etc.) as well as how operating systems and applications work.

What other books have you written?

My first book, “Windows Forensics and Incident Recovery”, from AWL/Pearson Ed was published in July 2004.

What makes a good forensic examiner?

Curiosity and a willingness to learn. No one of us knows everything. As a community, we know much more than any one person. A willingness to put aside your assumptions and fears, and a willingness to share your thoughts and knowledge with others, I believe, is paramount.

What should an organization look for when hiring a consultant or organization to provide forensic services?

I think that the primary focus should be on the focus of the consultant or organization. How important is it to the consultant or organization to understand and meet your business needs? Do they come in and tell you that they’re going to have to take your production systems down during prime operating hours, or do they look for other ways to meet or even augment your business needs? Are they simply interested in meeting your needs with purely technical solutions that don’t take your business needs into consideration? Also, how realistic and honest with you are they when it comes to taskings?

It’s also a good idea to see if you can talk to other clients and see what kind of track record the consultant or organizations brings with them.

Do you recommend any training in particular or additional books?

There are a number of books out there that are very good for learning; I’d start with “The Cuckoo’s Egg”, by Clifford Stoll. Eoghan Casey’s “Crime Scene Investigation” books, particularly the second edition, are very good, as is Chris Brown’s “Computer Evidence: Collection and Preservation”, Brian Carrier’s “File System Forensic Analysis”, and “Forensic Discovery” by Farmer and Venema. But it’s not about one book, or one set of books. A broad base of knowledge and information is extremely useful…so including “TCP/IP Illustrated” by Stevens, “Google Hacking” by Johnny Long and even “Corporate Espionage” by Ira Winkler will all have their benefits.

I see many people in the security field with past military experience on their resumes. If someone were looking to get into security, by way of military service, what branch and trade would you nudge him or her towards?

First off, I’d nudge them toward the military, in general. Military experience exposes young people to new things and forces them to deal with things they may never have encountered before. Ultimately, these become “war stories”…we hear them all the time; good bosses, bad bosses, experiences, etc. Having life experience is important, as is encountering a variety of different experiences, as security (in general) puts us in a position where we have to work in the face of potentially adversarial conditions. In such conditions, you have to remain calm and professional.

For specific branches, I’ve always been impressed with the emphasis the Air Force places on off-duty education when it comes to promotions. However, most experiences are what you make of them, so working in computer networking in any capacity, in any branch, is a great place to start. Learning to solve problems and adapt to situations is important in consulting.

What about those who start in civilian life?

Pretty much the same thing, but without the military component, obviously. Working in networking, starting out as a junior helpdesk or sysadmin is a great way to get started.

What is your opinion on technical training and certifications as a method for preparing people for forensic work?

Technical training and certifications have a good basis in getting people prepared, but in my perspective, they are only good if they are actually used and evaluated. In the Marine Corps, every Marine receives training in the care, feeding, use and deployment of their weapon…the M-16 service rifle. Even in peace time, Marines further receive annual refresher training. In the civilian world, I am often called on-site to assist in incidents, and see certificates on the walls of the cubicles…people are sent away for training, receive a certification, and then not only do not use it, but are not evaluated on the use of the knowledge or training by their managers.

If there is one person in our industry whom you would like to meet, who would it be and why?

I don’t like to name drop, but I’ve had the great fortune of meeting many people in the community, however briefly. The one person that comes to mind that I haven’t met, and would love to sit down with him, have a beer, and really pick his brain about incident response is Troy Larson. And not because his name is well-known, but because he’s encountered challenges that many of us may never see. To me, he’s the “yeti” of the security industry…he’s apparently found in the Pacific Northwest, there have been sightings, and we may see footprints or glimpses of him now and again, but I have yet to actually meet anyone who’s talked to him. 😉

What do you think is the biggest obstacle to having management “buy in” to an incident handling process?

After working in the commercial information security arena of over 10 years, and having been involved in physical and communications security in the military, I really don’t know what that obstacle is, or what accounts for a lack of buy-in. A very visceral part of me feels that it’s a fear of loosing control. However, I think that that fear, in a lot of ways, comes from an inherent lack of management or leadership ability in a lot of ways. There are a lot of managers who have a great deal of confidence in their ability to perform their tasks, and to hire and lead others, and manage resources to accomplish those tasks. I guess in some ways, to many managers, security seems like a wild west show, with a lot going on and some of it seeming contradictory, particularly if your only window into the security world is through trade journals and the media.

In a lot of ways, I see security (in general) marginalized, perhaps for any number of reasons. Unfortunately, our society is turning to legislation, fines, and even jail time to change this corporate culture, largely due to the fact that many of those in a position to make decisions about security aren’t making the necessary culture changes (and subsequent decisions) themselves.

In the late ’90s, the concern was, how do I sell security to my manager or my customer? In today’s day and age, we shouldn’t have to be doing this any longer. Online incidents have gone from joy-riding on the Information Superhighway to cybercrime with an economic basis and motive, and we need to keep up accordingly, not only in the sense of incident preparation, prevention and detection, but also in response. Security incidents resulting in an economic gain for one side and a loss for another is no longer a matter of if, but when.

What is the one security oriented product/service/project that is missing today?

I don’t think that there’s really one product or service…it’s more of a mindset or component of our corporate culture that’s missing. There are a number of products out there that can be used to meet the needs of particular environment or infrastructure, but regardless of how many products or services you throw at a problem, until the owners…the sysadmins, IT directors, and most importantly, senior management…change their mindset and culture, none of that is really going to matter.

For example, imagine an infrastructure where systems are put into service with little to no configuration management or control, and even if the systems and/or applications generate logs, no one is monitoring them. If that’s the case, what good is it going to do to purchase log aggregation tools, or other tools or products that generate logs?

Why write a book on Windows Forensics?

Because there wasn’t one, at least not one like the one I wrote. I’d done some research into particular areas of forensics on Windows systems, and started presenting at conferences and writing little snippets into online forums. Over time, questions (and in some cases, misquotes) started coming. I found that others were interested in the same thing. I wanted a way to document the information, and keep it available…there was no way I was going to be able to memorize everything…even I would need a reference to refresh my memory about research I’d done or something I’d found.

Simply posting on the web…on site, forum, or blog…didn’t seem to be enough; people were still asking the questions…so why not write a book? I was fortunate enough to find a publisher (Syngress, now owned by Elsevier) who was not only willing, but excited to publish it.

What programming and/or scripting languages do you recommend to assist in Windows forensic analysis? What about Unix/Linux forensic analysis?

Perl. Sometimes batch files on Windows systems are enough to get the job done, but for a greater level of granularity of control, I tend to go with Perl.

Have you ever purchased hard drives on eBay to see what you could glean from them?

No, but I have purchased them from a small shop near the local university. I’m afraid that if I do that anymore, I’ll regret what I end up finding. After all, when doing research into metadata in MS Word documents, I Googled for those types of documents from .mil and .gov sites…you’d be surprised what’s out there.

What type of environment would you recommend a person construct to help them learn forensic analysis?

Acquiring an image of a system is easy…I firmly believe that anyone who is interested in learning can be taught how to acquire images of systems. The truly hard part to teach is how to analyze a system, because it is a “system” in the truest sense of the word. You need to understand how artifacts on a system are created and modified. Once you do, then the absence of an artifact where you expect to see one is in itself an artifact.

In order to do that, it isn’t really all that difficult to set up an environment to teach or to learn forensic analysis. A couple of systems on a simple network are all that’s really needed. From there, there are a number of freeware tools available that allow just about anyone to set up a forensic analysis learning lab. For example, using tools from MS’s SysInternals site, you can monitor the Registry and file system on live systems for changes in real-time, as they occur, during different events. Keeping Locard’s Exchange Principle in mind, this will show us what artifacts we can expect to see as different events occur.

Acquiring an image of a system is really fairly trivial, but the analysis of that system can take time. Using freeware tools such as FTK Imager and dd.exe, you can acquire images of Windows test systems, even those running in virtualized environments. Then, using ProDiscover Basic from Technology Pathways, you can create a .vmdk file for the image, and then use VDK (or the GUI interface VDKWin) to mount the image on a Windows system as a read-only file system, and then using Perl, you can perform deep analysis of both the Registry and file system. Using ACLs and Jesse Kornblum’s md5deep, you can ensure the integrity of the image file(s) during this process.

Have you ever run into a situation where you’ve been asked to perform forensic analysis or recovery of data on an OS which you are unfamiliar with? How did you make out?

Yes, I have. In such cases, I do as much research as I can before hand, and ask a lot of questions along the way. I may not ask those questions of or in front of the customer, but I will ask someone. One of my goals is to develop professional networks, getting to know people that I can go to, if not for an answer, then a pointer to the right direction. No one person can know everything, and by sharing information, we’re all much smarter.

Any plans for a book on a new topic?

I’ve been approached with a couple of ideas. Some of those ideas have included things that are already out there. Others are new. What I’m most interested in, however, is meeting the needs of the community.

What is the one Windows forensic tool that you can’t live without, and why? What about Unix/Linux?

Perl. Perl, as a scripting language, gives me the power and flexibility to extract, correlate, and display data so that it can be more easily understood and analyzed.

Do you have to have an intimate knowledge of the operating system you are performing analysis on or are there some specific areas that people can focus on to obtain the maximum information?

IMHO, specialization is becoming more and more of a requirement, due to the sophistication of the incidents that are being detected.

In general, I feel that a good broad base of knowledge is required for any investigator, but more and more, the age of “Nintendo forensics” is drawing to a close. Incident investigations are becoming less about finding a couple of images or movies on a system and then closing the case. Questions are being asked that cannot be answered using the DOS-era standard of response; ie, power off the system, remove and acquire the hard drive. Keyword searches do not find Registry entries that are binary data types, or ROT-13 “encrypted”.

For example, one of the best sources of information regarding autostart locations within the Registry is anti-virus sites, where the vendors provide write-ups of the malware that they’ve analyzed. This is due to the fact that malware authors are finding these autostart locations and using them in their malware, and the vendor may not even be aware of them!

Now, I do not believe that everyone needs to have intimate knowledge of an operating system and/or applications. However, I do believe that there needs to be a community-based approach to the research; otherwise, if the research is supported solely by an individual or an organization, the results of that research are going to remain close-hold and out of the hands of folks who need it.