Chapter 16: Addressing Staffing and Training Issues | Hardening Network Infrastructure. Bulletproof Your Systems Before You Are Hacked.

The requirement to harden one s network infrastructure is a very broad and very deep requirement. Expertise is required on a multitude of products and technologies. This creates a problem because it is nearly impossible to find someone with the breadth and depth of knowledge that an organization requires. It is simply too much information for one person to know.

You can address this need in one of two ways. The first method is to bring in the expertise that you need through hiring additional personnel or using contractors/ consultants . The second method is to train your people for the relevant skills that your environment requires. In this chapter, we are going to look at how to address those staffing and training issues to ensure that you have the necessary skills to succeed at hardening your network infrastructure.

Staffing Issues

If you want to succeed at hardening your network infrastructure, you must have a competent and qualified staff that can properly plan, implement, and maintain your environment. In small environments, staffing may not be an issue because you can have a single person fulfilling multiple roles, and the requirements for those roles are generally much simpler. In larger and more complex environments, however, there arises a need for multiple people to address the various roles and responsibilities. This is due to the fact that people have a finite amount of time they can spend working on things. If there is more that needs to be done than they have time for, either tasks will get overlooked or you will need to augment your staff to address any issues that are not being handled. Here are the three predominate schools of thought regarding how to augment your staff:

Increase staff headcount. This involves hiring new employees to be members of the IT organization.
Utilize contractors. This involves bringing in outside expertise to address your staffing needs on a temporary or long- term contract basis.
Outsource. This involves bringing in an external company to handle some or all of the IT responsibilities.

Increasing Staff Headcount

Adding headcount is simply increasing the number of staff employees by recruiting and hiring new ones. One of the benefits of increasing your staff headcount is that these workers become members of the company and, in theory at least, have more loyalty to the company than the other choices. A drawback of adding to your staff, however, is the increased overhead of having to provide for employee benefits, such as health care and retirement.

Utilizing Contractors

Utilizing contractors to augment your staff is a very viable option. You can use contractors temporarily to help when you have a lot going on, and you can cut back when things slow down again. One of the big benefits of utilizing contractors is that you have the ability to pick and choose a specific expertise much more readily than is typical with staff employees. Many contractors are able to gain a specialization more easily than a staff employee, who tends to need to be an expert at many things. Another benefit is that the company does not have the overhead of providing benefits to the contractor, as compared to a staff employee. A drawback, however, is that the contractor is not an employee of the company and therefore you don t know where this person s loyalties lie ”with their employer or with their customer. Another drawback is that not all contractors live up to expectations, though you can often address this issue by having a good screening process before you hire a contractor.

Outsourcing

Outsourcing is one of the more controversial methods of augmenting your staff. It involves using an external source for some or all of your IT staffing needs. This is typically done in one of two fashions . The first method is for the company to hire an external company that handles all IT responsibilities. The benefit to this is that it allows the company to focus its resources on corporate objectives without needing to concern itself with IT objectives. The outsourcing firm can handle those responsibilities. The drawback, however, is that the company is no longer in control of IT, which can make it very difficult to accomplish technology goals, especially if those goals are not defined in the outsourcing contract.

The second method is to outsource certain parts of the IT organization, typically the helpdesk and related functions. Although this has the same benefit as the prior method, the drawbacks include the previously stated reason as well as the fact that the outsourced personnel reside offsite or even in a foreign location. Outsourcing also has the potential for political backlash caused by user /employee resentment if it s not handled properly.

Recruitment and Retention

Recruitment and retention are two of the most difficult staffing issues you will have to face. First, you have to identify a method to find good people for your organization. Then you have to try to keep them so that you do not lose that investment in their skills and knowledge. You need to make wise choices in both regards if you want to have an effective staff that can properly manage and maintain your environment.

Recruitment

Recruiting good people for your organization is the first step in being successful at hardening your network infrastructure. If you can t find the right people to do the jobs that need to be done, you will not be able to begin, much less succeed.

You can use many methods of recruitment to identify good candidates for your organization. In some cases, the education of the candidate may be important. The proper candidate must have a college degree, preferably in a discipline that complements the goals and objectives of the position they will fill. In other cases, experience may be what matters. You need someone who has been there and done that, and can identify the pitfalls and prevent your organization from falling into them. And in yet other cases, certification may be the key. You need someone who carries the credentials of a vendor or organization that identify this individual as being skilled in the products or technologies you require.

One aspect of recruitment to not overlook is the possibility of moving someone into the position or promoting people internally. Although the candidate might not be technically as proficient as someone else, it is a great morale booster and an excellent reward for employee loyalty. It sends the message that the company is willing to help those employees who have done right by the company.

Each method has its pros and cons. In seeking candidates with a college degree, you can identify folks who have gone through a formal education process and, in the case of business majors, can bring a good amount of business acumen into a field that struggles sometimes to align technology and business objectives. At the same time, by restricting yourself to only recruiting candidates with degrees, you might overlook someone who has the experience you need ”and experience can be more valuable than any degree in some cases. In seeking candidates with certification, you can identify folks who have demonstrated at least a basic degree of competence in a specific product or technology. However, certification is a double-edged sword. It is so easy today to simply braindump the required information to obtain certification that often it is hard to really know how competent someone might be based on their certification.

Keeping the Bar High for Certifications

Some certifications have taken steps to ensure, as best as can be done, that candidates can back up the certification with the relevant skills. Some certifications, such as the Cisco Certified Internetwork Expert (CCIE), rely on practical application labs to ensure that candidates can demonstrate the required degree of knowledge. Other certifications, such as the International Information Systems Security Certification Consortium (ISC) Certified Information Systems Security Professional (CISSP), rely on peer recommendation and verifiable experience to ensure that candidates can demonstrate the required degree of knowledge.

This leaves us with what I believe is the most important aspect of any prospective candidate ”experience. Although having a degree or certification certainly has its value, experience is really king, especially for your mid- and senior-level staff. You need to have people who have done what you are trying to do or accomplish, or have enough industry experience that they can identify the proper methods of hardening your network infrastructure if you want to be successful.

The Technical Interview One of the most overlooked steps of identifying a solid candidate is the technical interview. Many times people look at technical interviews from the perspective of I don t have time to spend an hour or an afternoon with someone. However, if this person is someone you may be trusting your enterprise to, I would ask, how could you not have the time to spend to find the right candidate?

The Lost Art of the Technical Interview

In all the positions I have interviewed for ( roughly 30), I have only had two technical interviews that I considered tough interviews. The first was with a software development company, where I spent over four hours interviewing with eight different people. The second was with a consulting company, where I spent an entire afternoon interviewing with four different people, ranging from pure technical interviews to one interview regarding business decisions, politics, interpersonal skills, and how to handle difficult customers ”and that was after three phone interviews. Good technical interviews are invaluable in identifying the best candidates.

In designing a good technical interview, you should approach the subject from multiple angles. On one hand, you want to ask directed questions, such as What commands would you run to enable OSPF routing on a Cisco router? These allow you to more readily gauge the breadth and depth of a candidate s knowledge for those times when you truly need an expert who knows a product inside and out. These very specific questions, however, are not the end-all-be-all of interviewing. In fact, they are probably not nearly as important as some other types of questions you could ask, such as scenario-driven and troubleshooting questions. These are perhaps the most valuable of all questions because they allow you to get a feeling for how the candidate solves problems. They also demonstrate the candidate s logical thought process. As part of my technical interviewing process, I also like to ask a troubleshooting scenario question. Although the goal is to see if the candidate can determine the proper solution, the most important objective is to allow me to see how the candidate deals with unknown or uncertain situations. It is easy to demonstrate something that you know, but it is far more difficult to figure out something that you don t know ”and that is a skill I have always prized in candidates because so much of the time technical folks are fixing things that they might not fully know or understand. Finally, give the candidate an opportunity to detail something that they have done. This grants them the comfort factor of being able to describe something they are comfortable with, and it allows you to see how well they can articulate solutions ”a valuable skill for building executive summaries and gaining management support.

Background Checks In addition to identifying the type of candidate that will properly fill a position and performing a solid technical interview, you should also check the background of a potential candidate. At a minimum, you should verify the references of any prospective candidate. Again, in my personal experience, my references have only been checked about 25 percent of the time. Although references are not a guarantee of quality, they at least provide some measure of validation of a candidate s credentials and employment history.

Limitations of Employer References

Most large companies now only verify employment because they are very afraid of lawsuits due to revealing any info about former employees. Consequently, many companies do not allow reference checks. In fact, my last employer was one of these companies. Be aware of this limitation when you check references, especially references from former employers .

Some environments require strict background checks. This is particularly true in regard to banking and federal contracts. Indeed, when you consider that many network administrators, helpdesk personnel, and security professionals will be privy to sensitive corporate information, it is a worthwhile investment to verify the criminal and credit history of your candidates. Practically speaking, someone who has administrator-level access to your network effectively has access to all the data that flows across it.

A couple of methods are available for performing a background check. With the Internet, one of the easiest ways to find out about someone is to simply google them. You can do this by going to www.google.com and entering the person s name in the search field. Although there is no guarantee that something will turn up, it is a good free method of checking someone out before you pursue more costly alternatives.

If your company subscribes to LexisNexis (www.lexisnexis.com), you can use it to research just about anything, including people. LexisNexis maintains an online legal database of virtually every person and company in the world. The only real drawback of LexisNexis is that it can be a costly solution.

Also, several online background-check firms can perform the background check for you and provide you with the results. Here are some of the better known firms:

US Search (www.ussearch.com)
Employment Screening Resources (www.esrcheck.com)
American Background (http://www.americanbackground.com)

Security Clearances In conjunction with performing background checks, it may be necessary to obtain security clearances for your employees. This is especially true if you are going to be trying to obtain any government contracts. You need to be aware of the three common levels of security clearance:

Level 1 (Top Secret) Top-secret clearance requires candidates to profile their lives for the previous ten years and requires a periodic reinvestigation every five years . The cost of qualifying for top-secret clearance can cost tens of thousands of dollars and take up to a year to process.
Level 2 (Secret) Secret clearance requires candidates to profile their lives for the previous five years at the same level of detail as top-secret clearance. Secret clearance also requires a periodic reinvestigation every ten years. The cost of qualifying for secret clearance can cost thousands of dollars and take up to a year to process.
Level 3 (Confidential Clearance) Confidential clearance requires a much lower level of detail than secret and top-secret clearance. This is the lowest level of clearance and must be reinvestigated every 15 years.

For government contracts/employees, all security clearances are investigated by the Defense Security Service (DSS). You can find more information about the DSS at http://www.dss.mil. The process for obtaining a security clearance involves submitting an electronic personnel security questionnaire (EPSQ), which as a Microsoft Word document is 31 pages in length and involves providing information regarding personal information, citizenship, locations lived, education history, employment history, personal references, spouse and family members information and citizenship, military history, records and history of foreign dealings, medical records, police records, drug activity, alcohol activity, investigation records, financial records, civil court records, and personal and professional association memberships. Upon the completion of this form, the DSS will then investigate the background of the candidate. Clearly this is a very time-consuming process.

Bonded and Insured In addition to all the background checks and security clearances that may be required for a candidate, once you hire someone, it may be necessary to obtain bonded insurance for the employee. This is especially true for contractors and consultants. The reason for obtaining bonded insurance is to protect your company and your employee from liability in the event that something they do causes the company to lose money.

Bonding and Background Checks

Bonding companies will not just bond anyone . For the cost of bonding, you can often get a little bit of a background investigation as well, provided by the bonding company.

Retention

Now that you have hired a good candidate, the next step is keeping them. Even in a tough job market, people can and will find other opportunities. Money is always a potential issue. People simply have to be paid a competitive wage. In addition to that, however, here are some other things you can do to help keep good employees:

Provide a casual work environment. Although there are times when it is appropriate to dress in a coat and tie, the ability to come to work dressed casually cannot be overlooked as an excellent quality-of-life measure for retaining good people.
Provide a technically challenging environment. By and large, people are in this industry because they enjoy the challenge. Although this does not mean you need to be a bleeding-edge technical company, it behooves you to maintain a technically challenging environment by constantly looking at new things that can be done. Otherwise, your employees may get bored and move elsewhere.
Provide flexible hours. Flexible work hours come in many forms. In some cases, it means allowing an employee to come in early or late to avoid the hassle of rush hour. In other cases, it means allowing the employee the opportunity to telecommute or work from home from time to time. Depending on the position to be filled, some jobs really don t need the person to be physically present to effectively perform their duties .
Provide the proper tools. One of the most frustrating things on the job is being asked to do something without having the proper tools (hardware or software). Although sometimes it is not feasible to provide everything an employee wants, when at all possible, you should provide them with the tools they need to do their jobs in an efficient fashion.
Provide a good career path . You should provide a solid career path that allows the employee to grow with the company. One of the best things you can do is provide a technical career path in addition to a management career path, allowing your employees to decide the direction they believe best suits them.
Provide positive feedback. Although not limited to technical staff, if employees only ever hear from their managers when something is broken or to complain about the quality of work being delivered, it makes for low morale and employees who will monitor the classifieds hoping something better will come along. Be sure to take notice when projects are completed early and come in under budget.

Individual Roles and Responsibilities

Before you can determine which candidate is the most appropriate for a given position, you need to identify the roles and responsibilities required in your organization. This allows you to identify exactly what it is you are looking for in a candidate. Although I will break each role down individually, you may be able to have multiple roles handled by a single person, provided the workload is not too much; likewise, you might need to have multiple people handling the responsibilities of high-workload roles:

Chief Security Officer (CSO) This role is responsible for managing all security personnel and projects in an organization.
WAN Administrator This role is responsible for implementing, managing, and maintaining all WAN equipment, protocols, and technologies.
LAN Administrator This role is responsible for implementing, managing, and maintaining all LAN equipment, protocols, and technologies.
Firewall Administrator This role is responsible for implementing, managing, and maintaining all firewall hardware, software, and technologies.
VPN Administrator This role is responsible for implementing, managing, and maintaining all VPN hardware, software, and technologies.
Public Key Infrastructure (PKI) Administrator This role is responsible for implementing, managing, and maintaining all PKI hardware, software, and technologies.
Intrusion Detection/Prevention Administrator This role is responsible for implementing, managing, and maintaining all intrusion detection/prevention system hardware, software, and technologies.
Internal Security Auditor This role is responsible for testing and auditing the security policies, procedures, and posture of an organization.
Change Controller This role is responsible for managing all aspects of the change-control process. Your change controller is one role that should not share responsibilities with any other positions. This will eliminate any potential conflicts of interest with regard to network changes that need to occur.
Network Architect This role is responsible for designing all aspects of the organization s network infrastructure.
Virus Software Administrator This role is responsible for implementing, managing, maintaining, and updating all virus software in the organization.
Capacity Planner This role is responsible for monitoring bandwidth usage and network performance and uses that information to project the sizing of circuits and determine the necessary bandwidth capacity required to effectively pass the required levels of data.
Security Tester This role often works in concert with the Internal Security Auditor and is responsible for performing penetration and vulnerability tests to validate the actual security level of the network.
Incident Response Team Leader This role is primarily responsible for managing and coordinating incident-response activities.

Organization/ Group Roles and Responsibilities

As with the individual roles and responsibilities in your organization, it is helpful to identify the organizational/group roles and responsibilities, which allows you to group individuals in easily managed teams :

WAN Management Team This group consists of WAN Administrators and WAN Architects and is responsible for all aspects of WAN design, planning, implementation, administration, and maintenance.
LAN Management Team This group consists of LAN Administrators and LAN Architects and is responsible for all aspects of LAN design, planning, implementation, administration, and maintenance.
Security Team This group consists of all Firewall Administrators, VPN Administrators, PKI Infrastructure Administrators, and IDS/IPS Administrators and is responsible for all security-specific devices, software, and technologies.
Virus Software Management Team This group consists of all Virus Administrators and is responsible for all aspects of virus-protection software, from the desktop to e-mail servers to gateway virus protection.
Change Management Group This group consists of individuals from every other information technology group as well as representatives from the respective line of business (LOB) groups, as required. This group is responsible for approving and managing all changes that occur in the environment and preventing implementation conflicts.
Audit/Vulnerability Assessment/Penetration Testing Group This group consists of all security auditors and penetration testers and is responsible for testing the environment to ensure that it complies with all policies and procedures. In addition, this group should actively test the security posture of the organization by periodically attempting to compromise security on the network. This group should not contain members of any of the groups they will be testing, to avoid any potential conflicts of interest.
Incident Response Team This group consists of individuals who are responsible for handling security incidents. The incident response team is detailed in more depth in Chapter 17.

Knowledge Management

Knowledge management is one of the most crucial staffing issues you will need to address. There was a time when an individual who knew everything about the network or about their responsibilities, even if no one else did, was considered a tremendous benefit to a company. That is no longer the case. An individual who knows how something works when no one else does is no longer a benefit. Instead, they are a liability ”and a large liability at that. In today s environment, people have to share knowledge with each other. Being the only person who knows how something works should not be allowed to be considered job security. In fact, you need to make it clear that being in such a position makes for an even more insecure job. The reason for this is simple: the more that an individual knows without sharing that knowledge with their peers, the greater the difficulty you ll have trying to replace that person if they leave the company. You are better off to have that happen sooner than later.

This does not mean you necessarily need to let folks know they will be fired for not sharing information. Instead, you need to build a culture and environment that encourages and rewards the sharing of information, in addition to letting them know that it is not acceptable to withhold information. You can make this happen in a number of ways:

Have regular team meetings. You should have regular team meetings to ensure that, at the very least, everyone is aware of what everyone else is doing, even if they might not necessarily know how to perform those tasks.
Require documentation of all changes. You should require as a component of your change-control process that all changes must have the appropriate documentation created or updated prior to these changes being approved. If the network diagrams have not been updated, for example, the change should be rejected.
Encourage mentoring. Encourage your senior personnel to mentor and teach your junior personnel. This has the twofold benefit of helping your senior personnel develop career skills that will benefit them in the long run as well as helping your junior personnel develop skills and expertise by learning from your best and brightest.
Have regular chalk-talk sessions. A chalk-talk session is simply an informal training session. These should not take up more than an hour or so and are a great way to share concepts and show employees what each other is doing. For example, you might have a chalk-talk session that demonstrates how the router ACLs are configured and what traffic is being permitted and denied .
Provide a central data repository. Provide a central, secured location such as an SSL-protected website for your employees to share data, information, whitepapers, and technical notes as well as all the policies, procedures, and documentation related to the network. This provides an easy-to-locate and easy-to-navigate method for people to find information related to how the network is designed and functioning.
Cross-train people. Not only will individuals benefit from knowing more, and thus becoming more flexible in their career both in and outside the company, but the company will benefit from the increased number of people who know different areas of the network.
Move people around. Letting someone stay in the same job for years is not good for them or good for business. As technologies change, they may find themselves obsolete. Also, moving people around merges knowledge and skills. This does not mean that people are arbitrarily moved into areas they know nothing about; it simply means that they have opportunities to work in other areas. Moving people around is also a good security measure, because it is harder for someone to defraud the company, and fraud is more easily discovered .