Building a Successful Security Team

Building a good security team can be difficult for an organization. There are many obstacles out there, even for those organizations that have unlimited recruiting resources. This chapter will discuss the "do's and don't's" of building a strong security empire. Such knowledge is essential for managers, staff, and consultants looking to develop security within an environment.

Determining Whether a Security Staff Is Even Required

The first question we need to answer is whether or not an organization even needs a formal security staff. The security initiatives of some successful organizations have been run solely by an IT director and a good network engineer. A dedicated and specialized security team can be difficult to hire and retain, and some organizations look for months without finding suitable security engineers. A good security team can also be quite costly depending on the level of expertise desired. In general, "You get what you pay for," so it is important to be realistic when budgeting for this type of employee.

As a guide to determine what sort of staffing is required for security within an environment, take the factors listed in Table 9.1 into consideration. Add the corresponding point values to the score for each box that applies to your environment. This should give you a general idea of how to look at security staffing needs.

Table 9.1. Calculating Security Requirements ^[a]

Question	0 points	1 point	2 points	3 points	5 points
How many workstations does the organization operate?	<75	>75	>200	>500	>1,000
How many servers (Web, email, firewall, etc.) does the organization operate?	<5	>5	>15	>30	>40
Does the organization have a full-time, dedicated Internet connection?	No	Yes
Does the organization allow for external parties (customers, partners, vendors) to connect to any of the servers via the Internet or a WAN?	No		Yes
Does the organization allow for remote dial-in access, wireless communications, or VPNs?	No	Yes
Does the organization host sensitive information, or are servers absolutely critical for the continuity of business?	No	Yes
Does the organization have to conform to legal regulations, or contracts that address security issues?	No	Yes
Total Points

Example: If you have 200 computers (2 points) and an Internet connection (1 point), your score would be 3.

Of course, there are many other factors to weigh in the matter that cannot be represented in a chart or formula. After calculating these primary factors, let Table 9.2 act as a guide to assist in your staffing considerations.

Insight into the Security Engineer Market

Good security engineers have been historically very hard to come by. Oftentimes demanding salaries 20 50% higher than the average networking engineer, security engineers seem to find no shortage of employment opportunities, even in times of economic strife.

Table 9.2. Interpreting the Results

Score	Consideration
0 1	Such a low score would indicate that there is probably no need to have a full- or part-time security engineer. If a major security decision is required, using a temporary consultant may be the best choice. As always, the IT staff should still be security-minded and follow the rules of security. No organization of any size should ever neglect the maintenance of security within their environment.
2 3	This score would indicate the need for an individual or group to be directly tasked with security responsibilities. There is still probably no need for a full-time security engineer, but security must be organized and someone should be in charge of managing and reporting on security measures. Spending a few hours a week with dedicated security focus is recommended.
4 5	This score would indicate that the organization should invest in at least a part-time security engineer, perhaps an individual who has security tasks as half of his/her daily chores. This individual should be formally trained in security with a few years of experience.
6 8	This score suggests that a full-time security engineer should be employed by the company. This individual should be formally trained and have several years of experience.
9+	A score this high suggests the need for multiple security engineers, or a security team. Quite often, the successful scaling approach includes having a single security expert and one or more mentors who perform the daily security chores while developing security skills.

At the beginning of this book, we looked at several factors that separate security from other practices in IT. In short, security is global, high-pressure, extremely dynamic, and is a relatively new practice compared to system and network engineering. The combination of these factors means a shortage of good security engineers. Many would-be security professionals shy away from the more intense factors that surround the occupation, while others simply dislike the human aspects (considering that their primary goal in becoming an engineer was to remove themselves from the human element anyway).

What does all of this mean to us? Simply that security is a different field than others we have had to deal with in the past. When we hire a security engineer, we must hire someone who knows technology, is highly creative, and is capable of interacting with other humans, a difficult combination of skills. Organizations that are able to quickly hire engineers without a good compensation package are either extremely lucky, or are not receiving the quality they need.

What Is a Security Professional?

Security professionals come in all shapes and sizes and can fit just about anywhere in an organization chart. Some high-level security-minded professionals have little use for deep and dirty technical knowledge, while the security-minded firewall-jockey has little need to know the ins and outs of performing a financial risk and impact analysis. There is, however, a middle area that a normal "good security engineer" will fall into, a central base of skills that most non-specialized security professionals should possess.

Overview of Skills and Knowledge

Almost every component that makes up an IT infrastructure has some relationship to security. This includes, but is certainly not limited to, all servers, desktops, applications, operating systems, networking equipment, communication protocols, and physical security. Thus, an individual's security skill set must often span many different technologies. An effective security engineer is going to have to know or learn, to some degree, about most of the technologies deployed within the environment.

The Security Mind (Rules and Virtues)

Before we talk about the nitty-gritty security fundamentals, we must first concern ourselves with the most important aspect of a good security professional: It is important to look beyond the specific technologies an individual knows and ascertain his/her security knowledge and ability to grasp the concept of security as a whole. Just about anyone can configure a firewall, but most people could not begin to tell you how to use it effectively.

This is a very important concept to remember since it is quite common to be deceived by technical know-how. Many people who are in the security market, or who desire to become security professionals, are simply interested in the technology of security. During an interview, it is quite easy to be impressed when someone knows how many bits a network overflow attack should consist of, but such knowledge does not warrant placing someone in charge of information security. Detailed technical knowledge is an extremely useful tool in security, but it is useless unless it is guided by an understanding of the virtues and rules of security.

Operating System Skills

One of the key functions of a security professional is to protect systems from being compromised. Knowledge of how to protect a server or workstation is essential to the average security professional's career. To secure a computer, it is useful to have a strong knowledge of the underlying operating system. A good security professional should have an understanding of the following operating systems (of course, this will vary depending on the environment):

• Windows Microsoft Windows is a fairly simple operating system to understand. The way in which security is implemented and managed, however, can get very complicated on the Windows NT, 2000, and XP platforms. Many of the security features within the operating system are hidden from sight to avoid confusing end-users. While pretty much anyone can quickly figure out how a Windows-based server operates, it is recommended that all security professionals spend some time gaining knowledge of how the underlying security architecture works. This is true for both individual systems and enterprise distributions.

• UNIX (especially Solaris) More complex than Windows for the average user, knowledge of UNIX is almost a requirement for security professionals in a UNIX-based organization. The speed and robustness of UNIX has made it a standard for high-end applications, and its ability to be "tweaked" has made it a favorite for security programmers and hackers alike. Unlike desktop applications, security applications have a tendency to appear first in UNIX, then in Windows. The underlying infrastructure of UNIX has many more security "details" to worry about and it can be much more complicated to implement security. Ultimately, however, UNIX can arguably achieve more security than Windows and, as such, is essential to the security professional's skill set.

• Linux One of the biggest pushes for the Linux operating system has been from the security world. As the leader in publicly accepted Open Source operating systems, Linux has the potential to be secured tighter than most other operating systems. Many high-end applications have been designed to run on Linux-based servers, and they are becoming more common each year. Though Linux is very similar to other forms of UNIX, it does have its own set of unique security components. Knowledge of Linux is a valuable tool for any security professional.

Networking

All security professionals should have a good knowledge of how modern networks function. Many aspects of security and hacking rely on the use and manipulation of networks, networking devices, and networked systems. TCP/IP is just about the only protocol to discuss as related to security, since most other protocols are becoming less and less common to find. Security professionals should understand how networks function and how devices communicate over a LAN, WAN, and over the Internet. A strong knowledge of all the common communication protocols like Telnet, SMTP, File Transfer Protocol (FTP), HTTP, DNS, and ICMP is certainly a plus.

Security Technologies

Security technologies, like firewalls and IDSs, are often what organizations consider to be the primary skills of a security professional. While there are other, more important factors, a security professional should indeed know a good deal about firewalls, network intrusion detection, host intrusion detection, system security (operating system hardening), VPNs, and general encryption. It is also important to know a bit about various security-related topics like packet filtering, proxying, zoning, structures for logging, and reporting. The individual should also be well-versed in the art of penetration and vulnerability testing (as discussed below).

Hacking Technologies

A security professional must have at least some knowledge of how hackers work and operate. Though it is a bad idea to hire anyone who enjoys hacking in his or her spare time (discussed later), it is important for the individual to know the concepts, tools, and technologies of hackers. This should include penetration testing, vulnerability scanning, sniffing, and malicious coding. Knowledge of hacker exploit tools, such as viruses, worms, back doors, and Trojan horses is essential.

It is important for a security professional to be able to recognize an attack when it occurs, whether it is on a network, in a system, or from someone calling randomly through an organization asking for passwords. Most defenses are designed to report interesting activities, but we must still be able to recognize an attack from small pieces of information.

Programming

Though certainly not a requirement, some programming knowledge is extremely useful in security (especially when dealing with Linux, Solaris, or other forms of UNIX-based security). Working knowledge of Perl or some other common scripting language is a big asset to anyone in security.

Written Policies

Policies and procedures (discussed later) are the only hope any organization has for maintaining long-term security efforts. Being able to recognize where policies need to exist and writing them down or doctoring up someone else's policies are vital skills for anyone practicing security; more so is knowing where and when a policy needs to be created, and being willing to do the work. Be wary of anyone continually suggesting new security measures and never talking about a policy or procedure.

About Hiring Hackers

One glorified concept that certainly justifies a paragraph or two in this book is the idea of hiring hackers for the security staff. The temptation is very prevalent to do so, and has been further inspired by movies, the media, and the simple logic that a hacker should know better than anyone how to keep systems from being "hacked."

I have been interviewing security engineers for private organizations and consulting firms for many years now. One of the key elements of my security interview process is to inquire as to whether or not the interviewee was ever a hacker. The question, of course, is usually phrased somewhat cleverly, so as to inspire a tale of some great hacking feat. If an interviewee receives a checkmark in my interview box marked "Potential Hacker," the interview notes are trashed and we move to the next candidate.

In this book, I discuss the difficulty of dealing with employees who have hacker tendencies, and point out how difficult it can be to find and stop such individuals. Take this concept up a few thousand notches when considering hiring a "potential hacker."

There are, of course, a few exceptions, but I emphasize the word "few." Being in charge of, or on a team that is in charge of, information security is somewhat of a power trip. It inspires great pride in most, even if it is just being in charge of security for the local drugstore. For an engineer sitting at a desk with numerous icons of all the latest hacker tools strewn across his or her desktop, the temptation to use them is incredible. For the person in charge of information security, there is often nothing but conscience preventing him or her from completely taking over the IT infrastructure. Now, imagine giving such power to a hacker.

We must, to some degree, inherently trust our security engineers. It could almost be included in the job description: "Must be trusted with the keys to the kingdom." Finding someone we can trust, who can go through the daily turmoil and political battles of work, and whom we could even potentially fire without having a destructive worm released through the infrastructure, is very difficult indeed. If we start to look at hackers to fill these positions, we are asking for a nightmare.

Story of an Employed Hacker

To quickly share an experience, there was one exception that was made to the "no hiring a hacker rule." A very young and extremely bright individual asking for a moderate salary and with the gleam of potential in his eye was hired as a security consultant. It was known that this individual had been a hacker, but he seemed to be on a straight path and it was too great an opportunity to pass up. Employment lasted approximately eight months with this individual, who would occasionally hint at the hacker life he lived when at home.

The sheer potential of an intelligent hacker employed within the company was enough to cause sleepless nights for those in charge of him. Every time this individual would ask for a raise, every time he made a formal complaint, and every time he asked for special privileges, it was in the minds of all the managers that, "We really don't want to upset this guy." Even when it came time to terminate the individual (for unrelated reasons), the company had to spend countless hours checking and double-checking the systems accessible to this individual for a potential threat. Throughout the entire history of his employment and termination, the individual never made a single (known) attempt to hack the employing company. But even years after, the possibility exists that this individual could have left a hidden remote control application or time bomb within the company or its clients.

Moral of the story: If an organization does decide to hire someone who is potentially a hacker, it is very important to think through the whole lifecycle of employment, all the problems, headaches, and worries it could cause, and only then determine if it is worth it.

Training Security Personnel from Within

Training employees within the organization is highly recommended for many reasons. Total cost of ownership of the employee is normally much lower when he or she is trained; it keeps his/her skills sharp and keeps him/her satisfied in the job; and, training is often the only chance an organization has of acquiring a good security engineer.

Some forms of security training are heavily priced. Taking advantage of the lack of security engineers, training organizations and technology vendors have seen fit to add a nice 50 100% to the bill for a security class. Sadly, the quality of such training is usually no higher than normal technical classes. Paying 50% more for training does not increase the potential of creating a good security engineer. This being said, I highly recommend the idea of training internal staff on security, but offer the following suggestions:

1. Choose very carefully whom you send to training. You want to pick employees who will be attentive and who will retain and apply what they have learned. Also, security classes are very similar to hacking classes, and you don't want to send a disgruntled employee to learn "Hacking 101."

2. Normally, it is more valuable to train in classes that are more global than ones that are product-specific. Unless required, don't take the "Brand-X Firewall" class when you would get more from the "Firewalls for Everyone" class and reading the Brand-X firewall manual.

3. Security conferences are usually a great way to get a good mix of technology, theory, and design. I highly recommend attending conferences like "SysAdmin, Audit, Network, Security" (SANS), where there are numerous short sessions, which can be followed up with independent study.

4. Complement the training class with independent study. Security is a very large field and it is extremely dynamic. A class is valuable for introductions and for some subjects that require hands-on experience. But reading books, magazines, and online articles can provide more information than a class.

Hacker Training

After reading the previous section concerning the problems with "hiring a hacker," one may get the sense that an organization should have nothing to do with hacker knowledge. This is not the case, as knowledge of hacking is essential for anyone to be able to secure an environment. It is highly encouraged that security professionals spend a good amount of time reading through hacker Web sites, catching up on the latest "How to Hack a…." articles, and even downloading and experimenting with hacker tools. All of this, however, should be done within the following guidelines:

1. Company policy should dictate that anyone visiting hacker Web sites, downloading hacker tools, or participating in any way within the hacker community must first get approval from a manager or executive. A hacker tool acquired without permission should be considered a security violation.

2. Whenever a hacker Web site is visited, a file is downloaded, or a tool is experimented with, it should be done on a system and network completely separate from the main company network. Doing this through a separate Internet connection, through a modem, or from a separate leg on the firewall is highly recommended if possible.

3. For anyone using a security tool, including common sniffers, it is recommended that written permission from the CIO is obtained and the letter is then safely locked away. There have been incidents where confidential information was exposed to a security professional without intention, and that individual was terminated or prosecuted for the act.

Interviewing a Security Professional

Conducting security interviews is rarely simple. This does not mean, however, that we have to leave the hiring process up to "luck," or trial and error. I have included the following key points to consider when looking to hire a security professional:

Assess the Ability to Think Out of the Box

Good security requires an individual to think of things that may seem a bit far out, or perhaps deal in abstract logic. When all of a sudden an email system begins forwarding all messages through another server before reaching their destination, the individual has to be able to recognize that something may be wrong and then figure out why anyone would want to do that. A useful tactic to use in an interview is to catch the candidates off guard and force them to think on their feet. Out of the box questions like "Why would you attack a candy maching to the Internet" or "In what ways is an online transaction more secure than actualy going to the store?" Such questions are asked when the actual answer doesn't matter as much as the thought process followed.

Assess the Ability to Follow a Process

Practicing security often involves following logical processes to derive a conclusion about a particular situation. It is very common for a security engineer to be told, "I think someone broke into our system," and then asked to perform an assessment. The process followed does not have to be long or complicated, but it absolutely must be methodical and repeatable. Thus, in an interview, it can be very helpful to give someone a scenario with little information and ask how the candidate would handle it. For example, tell the individual that someone comes to him/her in a panic saying that the email server has been hacked and is now not working. Then ask how he/she would go about handling the situation.

Asking questions like these, we are looking for some clear and logical way of getting to the bottom of the situation. Our ideal candidate will follow a simple process that informs management, deals with the immediate situation, and then performs a logical analysis while documenting every step. A key phrase to listen for is: "Follow the corporate policy and procedure in the matter." Also, consideration should be taken to preserve evidence, but at the same time, the immediate unavailable system must be taken care of. Primarily, we are looking to see if an individual is even aware of such considerations, and secondarily if he/she is able to incorporate them into a procedure.

Use a Group to Ask Technical Questions

Sometimes, it can be hard to ask direct security questions when there are no security experts within the organization. Though by no means should we expect a candidate to be an expert in every field of technology, he/she should be able to answer questions about operating systems, networking, and a few other key technologies. If there is no one in the environment comfortable asking tough security questions, round out the interview with questions from other practices. Ensure the candidate knows the fundamentals of each area and then ask him/her to expound on the security implications of each. If the candidate is good, everyone in the room may learn something. In this scenario, it is important to take good notes to verify any complicated security information later.

Certifications and Other Tall Tales

Certifications are a great concept, and it would be wonderful if they actually worked. To be certified in something should reflect an experienced understanding of the subject. It should measure knowledge that has been practiced, that can be applied practically, and that will be retained for a long period of time. If this was the case, then an employer would be able to look at a list of acronyms next to a person's name and feel confident that the individual is proficient in those subjects.

Sadly, the vast majority of security certifications in today's market simply indicate that an individual was able to memorize a series of facts before writing them all down on paper. The candidate could shortly after forget the information, usually because he/she never really understood it in the first place. This is a hard lesson learned through several years of hiring and terminating certified technology professionals.

I myself have several certifications to my name, as do the majority of my colleagues. Such paperwork is quite useful to have since most organizations do not have the expertise to proper assess security skills and must therefore refer to the symbols next to a candidate's name. Sadly, I know of no good security professionals who place any value on the processes they went through to acquire their certifications.

Over the years, the majority of professionals I have hired have had no security certifications at all. After some thought and study in the matter, I realized that to accurately certify someone as knowledgeable in a subject would be very difficult and would not allow for profit to be made. Since having people certify and recertify at $100, $300, and $500 a shot is incredible revenue for these training organizations, the idealistic goal of a having a truly useful certification is very hard to achieve. A true, unbiased certification cannot be used to grow an organization or make a profit.

Using a Screening Company

Screening companies are becoming more common for organizations interested in hiring security engineers. A screening company will ask a candidate security questions on behalf of an organization and then provide a score that indicates the candidate's level of comprehension. These organizations are good time-savers, but they have many drawbacks. The individuals performing the interviews are normally just technical enough to ask the questions and understand the answers. Since security requires more creativity than fact-retaining, a good security interview can't be confined to stock questions. If, however, the local staff consists of no security experts, these organizations may provide good value.