One of the most difficult aspects of hardening your network infrastructure is managing the changes that are going to occur on your network. Regardless of how well you plan, you are going to have to make changes to your network to address security updates, technology updates, and even acquisitions and employee moves. All these changes can undermine ”or improve ”the security on your network.
In this chapter, we are going to look at how you can manage the changes that will inevitably occur on your network while ensuring that you maintain your security posture at an acceptable level. We are also going to look at how you can implement a patch management strategy to address not only security patches and updates, but also general patches and updates in your network.
Your change control process has two fundamental areas. The first is planning for changes. This is often called the change planning process . This is the step where you want to identify the risks involved in the change and build the change planning requirements to ensure that the change is successfully implemented. Before any changes are made, you should take a look at the key tasks required to ensure the success of the potential changes. The second step is managing the change. This area is often called the change management process . This is where you need to define the process that approves and schedules the changes to ensure that there is adequate notification and minimal user impact as a result of the change.
The most effective way to implement a successful change control policy is to define a process flow that identifies the key steps and requirements of the change control process, as shown in Figure 14-1.
We will refer to this flowchart as we discuss how to plan and manage changes in the next sections. First, however, we need to look at who manages the changes in our environment.
Before you can begin effecting any changes in your environment, you need to define who is responsible for managing those changes. Defining the change management team is another one of those political endeavors you will need to undertake in your enterprise. You need to identify all the people who will be involved in accepting or denying change management requests . Your change management team does not need to be made up exclusively of technical experts. In fact, it shouldn t be made up exclusively of technical experts. This is because the goal of the change management team is not to investigate the technical accuracy of the changes being proposed. To achieve this, your change control team should be made up of a mix of people from all your major business units and organizations so that they can provide the necessary business impact information to properly judge the change request. At the same time, you need to try to keep your change control group as small as possible; otherwise , it becomes very difficult to coordinate all the members . If there are too many people, it can be difficult ”if not impossible ”to have effective change control meetings.
Change control can be one of the most difficult hardening steps you can implement on your network. This is due in large part to the fact that change control often interferes with the ability of people to do what they want, when they want to do it. No one likes to be told no, and yet change control is there to do exactly that in some circumstances.
At the same time, the value of a good change control system cannot be underestimated. Change control is a critical component in ensuring that your network does not spin out of control. We have all seen networks that are in some degree of shambles. Did the administrators plan for the network to be an insecure , unmanageable mess? Of course they didn't, but much like that old Talking Heads song Once in a Lifetime, one day they wake up, look at their network, and think to themselves , How did I get here? Because changes happened on the network, and because no one evaluated and managed these changes to make sure they knew and understood the impact, the network eventually became something very different from the network they thought they had.
Change control isn't all good, however. As critical as it is to managing your network, if change control isn't implemented with wisdom and planning, it can become such an intolerable burden on your network that it interferes with your ability to manage your network properly. We have all heard ”and many of us have been a part of ”change control horror stories, where making even the simplest of changes requires so much effort that you start to question whether the change is worth the effort required. In these cases, change control ceases being something that helps you harden your network and becomes something that ultimately contributes to the greater likelihood of a security incident occurring.
The final aspect of change control to consider is the human element. People don't like change. Many folks are threatened by change because they have become comfortable with how things work, and they are afraid that when a change is made it will upset that comfort factor. In some cases, this is a valid concern, but in others it is not. In an ideal world, we wouldn't need to make any changes ”but we don't live in an ideal world. In our world, new technologies and security threats continue to come out, and we must react accordingly .
The goal of the change management team is to review each change to ensure that all associated documentation is complete, to ensure that the appropriate risk levels are defined, and then to investigate and evaluate the impact of the change on the business and business requirements before approving or disapproving the requested change. If there are any conflicts or questions about the request, the change control team should handle resolving the conflicts and questions so the change can proceed. The change management team is also responsible for scheduling and communicating the change to all affected parties.
One of the difficulties of involving the line of business (LOB) on the change management team is that the LOB representative may be inundated with changes that do not affect their LOB and therefore they don't care about them. You have a couple of ways in which you can approach this problem.
One thing you can do is to involve the LOB representative only in the change control meetings that directly affect them. The other thing you can do is to remove the LOB representative from the change management team; however, you must ensure if you do this that you still engage the LOB in discussions and ensure that they sign off on the risks and changes.
You will need to determine which method is the most effective to use in your environment.
A specialized role within your change control team is the change controller. Your change controller is generally a member of your IT team who acts as a project manager for all the change process details. Some organizations will attempt to have the change controller role shared with other roles within IT, but if you are in a large environment, this really needs to be a dedicated person to keep track of and manage the status of all potential requests in your organization. If the volume of changes is large enough, you may need to go a step further and divide this role up among several controllers, with responsibilities divided by geographic region. The change controller should be responsible for the following elements of the change control process:
Accepting and reviewing all change requests to ensure completeness and accuracy. Your change controller needs be a central point of contact to properly manage all this information.
Managing and running all change control meetings. (I recommend weekly or biweekly meetings.)
Presenting the completed change requests to the change review board for approval or disapproval.
Preventing any scheduling conflicts by maintaining the calendar and schedule of all changes as well as business- related scheduling data (such as holidays, company all-hands meetings, and so on).
Publishing all meeting notes and helping to communicate all changes to the relevant implementation teams and user groups, possibly distributing a calendar of changes to LOB leads.
Ensuring that only authorized changes are implemented and that all changes are implemented in a timely fashion in accordance with business requirements and technology standards.
Providing after-action reports to ensure that the changes were indeed successful and had no negative impact on any business processes.
Providing all required metrics and data requested by management regarding the status of the change control team. This can include the volume of changes, average turnaround time, number of changes accepted and rejected, number of backouts, number of changes that generated new incidents, number of changes that did not produce the appropriate results, number of emergency changes, and the degree of client satisfaction.
The change planning process is where you do all the legwork before the change is implemented in your mainstream environment. I ve said it before, but it bears repeating: Proper prior planning prevents poor performance. If you do not plan your changes before you make them, you will rapidly lose control of your environment. To help ensure that you plan your changes appropriately, you should make sure you perform the following six steps:
Recognize the need for a change.
Identify the change that is needed.
Define the scope of the change.
Perform a risk analysis.
Test the change.
Plan the change.
The first step of your change control process is to identify the need for a change. Within a security context, this can be done by following the principles laid out in Chapter 13. Keep informed of new security exploits and, by extension, the potential need for changes in your environment by strictly adhering to a daily checklist, as follows :
Monitor the BugTraq and NTBugTraq mailing lists.
Monitor vendor mailing lists.
Check www.cert.org for any new advisories.
Check your virus software vendor s website for any new advisories by subscribing to vendor mailing lists.
In many cases, there isn t just one way of addressing an issue. There might be multiple solutions that will achieve the same result. During this phase, you need to identify the change that makes the most sense for your environment. Make sure you know and understand all the alternatives so that you can select the most effective solution. At the same time, be prepared to return to this phase to try the alternatives in the event that the original change doesn t work.
The next step is to identify what the intent and purpose of the change is. You need to define what systems and users are affected by the change, including not only the end users but the administrators who are responsible for implementing the change. When you define the change, you should make sure you include a complete technical definition of the change and the reason for the change. During this phase, you should answer the following questions:
What is the nature of the change? Consider what the change is going to address and how the change is going to address it.
What systems are impacted by the change? Consider not only the direct systems you may be changing but also incidental systems that could be affected. For example, if you are making a change on your external firewall, the ability for devices on the DMZ to communicate with systems on the internal network could be impacted.
Who is affected by the change? Define the users who will be affected by the change. This is primarily so that you know who needs to be notified of the potential change to make sure you take into account their schedule. For example, if you need to make a change that will impact your accounts receivable department, and it is the end of the month, this might not be a good time for this particular change.
Once you have identified the potential need for a change, you need to perform a risk assessment. There are actually two parts to the risk assessment that you need to perform.
The first risk assessment is actually defining the level of risk associated with the threat the change seeks to mitigate. This is an important step because the level of risk a change mitigates is critical in identifying how quickly a change needs to be made. Low-risk changes can likely be made at a much more leisurely pace, perhaps being addressed at the regularly scheduled change control meetings and on the regularly scheduled days identified for making changes. Medium-risk changes may or may not be able to wait for a predefined change schedule. High-risk changes might require immediate action that occurs outside the traditional change control mechanism.
The second risk assessment is defining the level of risk associated with making the change in question. The objective here is to understand what the potential negative impacts of making the change are. On the surface, it is easy to look at a security issue and say, We must make this change. Practically speaking, however, we have to make sure that the changes we make will not cause even more problems than they solve. Examining the risk involved in the change will allow you to judge the worst-case scenario to ensure that you have looked at all the pros and cons associated with the change before you actually make the change. This will also assist you in identifying the level of testing and validation to undertake before making the change. You can use the following five general risk levels to assign the risk of making the change:
High potential impact to a large number of users or business-critical resources due to the introduction of new products, technologies, or features The change will involve expected network downtime.
High potential impact to a large number of users of business-critical resources due to a change in the existing environment This change could involve increased network traffic or involve enterprise-wide changes (such as backbone router changes, routing protocol changes, or access list changes). There may be some network downtime involved in the change.
Medium potential impact to a smaller number of users because of a nonstandard change The change may involve new or changed products, technologies, and features. The change may also involve some network downtime.
Low potential impact involving a minimum amount of users This change will generally involve bringing new locations and services online that are not yet used by the general user community. This level can also be used to define higher-level changes that have been tested in your production environment.
No service or user impact This level generally defines making standard configuration changes, such as changes to passwords, banners, descriptions, SNMP community strings, and so on, that present no expected network downtime.
Although this chapter focuses on changes for security-related reasons, these concepts also apply to any other type of change you would make in your environment. For example, changes due to increased functionality, failed hardware, end-of-life hardware, business acquisitions, and so on, should all go through a change control process and risk assessment before you make them.
The next phase of the change planning process is to test the change that will be made. Testing the change can be made in a lab or with controlled users in your production environment. For example, if you need to make a change to your routers, you might test the change first on a router that is not heavily used, that does not control critical business processes, or that is only used by a small group of users and work with those users to make sure that they can still do what they need to after the change has been implemented.
One of the things you can do to help identify how much testing you should perform on a change is to identify the types of changes and then define the level of testing expected for each change. For example, the following changes should be tested in a test environment if at all possible:
System image changes
Updates to critical resources
Routing configuration changes
Other changes might be OK to test in a controlled production environment, depending on the circumstances. Here are some examples:
Access list changes
Virus signature updates
Content filtering updates
Device configuration changes
The most important aspect of testing the change is to make sure your test, as near as possible, mimics the environment and the method of the change you will make. For example, if your change is updating an access list on a router and you test that change while physically consoled into the router, you may find that when you make that change in production on a remote router, the access list blocks your Telnet access, something that you would not have discovered in your test.
Another testing gotcha you should be aware of is that you test the change as all users who will be impacted by the change. In testing, make sure you test as every type of user who could be affected by the update. If you only test an update as an administrator, you can almost bet on your regular users having a different experience. I have lost count of how many changes I ve tested by rolling the changes to the IT staff first, only to discover that once they were rolled out to the general user population, their access rights did not allow for the changes to occur.
This brings up an important point: Be careful to make sure when testing with your IT staff that you do not do things that may prevent your IT staff from working. In fact, I recommend that you try to identify users across your environment who are willing to test changes that need to be made. These people are easy to identify. They are your users who are geeks like you and me. They like technology and like being involved in technology, and using them to help you test changes is a good way to give them their technology fix in a more productive manner than them deciding on their own to help you in your environment.
If the test is successful, you are ready to plan the change for the production environment and submit the change control request. If the test was not successful, however, it is time to return to the change identification phase (step 2) and attempt a different change, or make changes to the current change in an attempt to achieve success.
This step is where you bring all the prep work you have performed together to build a change plan that will subsequently be submitted to your change control management team prior to rolling out on your network. Your overall goals during the change planning step are to ensure that the following points have been addressed:
Make sure that all resources required for the change have been identified and put in place. Answer the question, who needs to perform the change? This will help ensure that everyone required for the change is aware of the change.
Ensure that a clear goal has been defined for the change. Answer the question, why are we making this change? This will help ensure that everyone understands why the change is being made.
Ensure that the change adheres to all your organization s standards and procedures. Answer the question, are we following the security policy and any relevant procedures that have been defined? If no procedure has been defined, this is also an excellent opportunity to write one. This helps ensure that any changes will conform to your current architecture or engineering design guidelines or constraints.
Create a backout procedure. Answer the question, how do we undo the change if it doesn t work, or if it doesn t go well? This will help ensure that if things go wrong or don t work out the way you expected, you have a structured plan for reversing the changes.
Define any necessary escalation paths. Answer the question, who do we contact if we run into any problems? This helps ensure that you know who to talk to in the event that you run into a problem. This could include internal or external technical support resources as well as any key line of business (LOB) representatives.
Define the affected users and any downtimes required for notification purposes. You should have identified the users during the scope process. At this point, you should take that information and make sure you have a notification process for informing those users of the change. This will help ensure that your users are not caught by surprise by the change. If the impacted users are a critical LOB, you probably want signoff from the LOB on a change before implementing it.
Update all network documentation. Make sure that all relevant network documentation and diagrams are updated and ready to be submitted at the time you make the change control request. This is so important because, as you know, if you don t make sure your documentation is updated at the time of the change, it will quickly be forgotten. In fact, any change control request that does not contain this information should be rejected. This helps ensure that your network documentation is always an accurate reflection of your network.
When you plan the change for your network, you also need to generate the change control request you will be submitting. Your change control request should contain the following information:
Name and contact information, including phone number, e-mail address, pager, cell phone, and department of the person making the request.
Date the request is being submitted.
Target date for implementing the change.
The change control number. This number should be generated by a central authority, such as the change control project manager or the change controller.
Helpdesk tracking number (if applicable ).
Description and security bulletin information of the vulnerability the change addresses (if applicable).
Type of change being requested. Software (OS, application, service pack), hardware (processor, memory, disk), network (router, switch, WAN, LAN) and security are good general types you can use.
Description of the change. Be as informative as possible here, in particular, making sure that you explain the positive impact of making the change and the negative impact of not making the change.
Risk level of the change, including any potential service interruptions and the number of users impacted by the change.
System name, IP address, and location for all affected devices.
User group contacts (if applicable and available).
Whether the change has been tested.
How the change was tested.
Completed test plan, including the exact procedure for making the change. This should include any code or configuration examples as well as information regarding how the change will be deployed, whether a reboot is required, and whether the change can be uninstalled . Include planned timelines and milestones (for example, bring up and test T3 links by 1 A.M .).
Backout plan. Include timelines for backout and what the cutoff time is, after which there is no backout. For example, if the change window is 1 A.M . to 4 A.M ., and backout will take two hours, your cutoff time is 2 A.M . If it doesn t work by then, you must decide to back out or get signoff and plow forward.
Whether this change will be migrated to other locations.
Any prerequisites or assumptions required for the successful implementation of the change.
When you generate your change control request, you need to approach it from the perspective of there is no such thing as too much information. Make sure you include any of the following components as a part of your change control request: current and updated topology and configuration, physical rack layouts, hardware and hardware modules, software versions, software configuration, cabling requirements, network topology maps, port assignments and addressing, device naming and labeling, name resolution updates and changes, circuit identifiers and assignments, network management update requirements, out-of- band management requirements, solution security information, and exact change procedures.
Whereas the change planning process was about preparing for a change, the change management process defines the approval and scheduling of the change to ensure that the correct level of user notification and impact occurs. During the change management process, you want to focus on undertaking steps directly related to the actual rollout and implementation of the change in your environment. There are six phases of the change management process:
Review the change control request.
Communicate the change.
Implement the change.
Review the change.
The first step of the change management process is to review the change control request to determine whether the change should be approved or rejected. During the review process, you should compare the change against documented standards and best practices to ensure that the change complies with your business and security policies. You also need verify that all related documentation that will need to be updated as a result of the change has been updated and submitted as a part of the change control request. If this does not happen, the change should be rejected due to lack of documentation. You should also verify with the affected LOB that the change is acceptable to them and that the change schedule does not conflict with any other changes or previously scheduled tasks. For example, if the change affects the accounting group and is scheduled for the last weekend of the month when accounting is frequently working overtime to complete end-of-month transactions, the change should be rejected or at the very least the schedule should be adjusted.
If the change is rejected, the process returns to the change identification phase. This does not necessarily mean that a new change needs to be identified. For example, if it is merely a scheduling conflict, the same change can be kept, with all the related testing and preparation, and the scheduled date and times are merely changed when the request is resubmitted.
Once a change has been agreed upon, the next phase is to communicate this change to all users in your organization. Communication is one of the most effective means to ensuring the success of your changes, and the success of IT in general. If you doubt me, sit down with some of your users and ask them what the one thing IT does that annoys them to no end, and most will tell you, They don t tell us what is going on.
Because the nature of the changes you are making can be so complex, you should build a matrix to help define who will be affected by the change and any downtime that might result from the change. This is also important because different groups may require different degrees of information related to the change. For example, if you are upgrading your corporate DHCP servers, your users probably only need to know that they will not be able to connect to any network resources during the change. Your support staff probably needs to know exactly what change you are making that is causing the users to not be able to connect to any network resources during the change.
At a minimum, your communications process should address the following points:
Set realistic expectations. An old mentor of mine used to tell me, under-promise and over-deliver. You should set the appropriate expectations not only of what to expect during the change, but what to expect after the change.
Identify support resources. Identify who is responsible for making the change and who should be contacted for any questions or problems related to the change.
Communicate operational requirements. Make sure you define what applications and systems are being changed or affected by the change.
Inform your users. Make sure every person who could be affected by the change is notified about the change. When in doubt, notify them anyway. Make sure you choose a notification mechanism that is not going to be affected by the change. For example, if you are going to be making a change on the e-mail server, you might not want to use it as your primary notification mechanism.
One method of communicating changes to your user community is to use some sort of simplified impact rating for informing users of changes. For example, you could designate that green changes mean if everything goes properly, they will never know anything has changed. Orange changes mean they will see something different, but it shouldn't impact the way they work. Red changes mean they will work differently due to the change. This lets users not get so many notices they simply start ignoring them. It also illustrates the changes (that is, red changes) the users need to pay attention to.
Now that the change has been communicated, it is time to acquire the resources necessary to implement the change. These include not only hardware and software resources, but people resources as well in the form of the implementation team. The implementation team is responsible for having the technical expertise and skills necessary to make the change. In many cases, this will be the people who submitted the change request in the first place. The implementation team should be involved in the planning phase to contribute in the design and development of all aspects of the change that is being requested. The implementation team is primarily responsible for ensuring the change then goes off without a hitch while adhering to any and all organizational standards, policies, and requirements. To assist in this, your implementation team needs to be able to answer the following questions:
How and how thoroughly are they going to test the change?
How will the test and change be rolled out?
How long will testing and rollout last?
How can the change be removed?
What is the backout criteria?
The final requirement of the implementation team is to make sure that all results and deviations from the submitted change control request are documented and submitted to the change controller when the change has been completed.
The person or group that performs changes to the network should not be the person or group that approves those changes. There should be a separation of duties ”to remove the possibility of someone wanting to do something they shouldn't and then approving themselves. In some environments, there could be legal regulatory requirements to keep these entities separate.
After all the preparation work has been completed, the request has been submitted, the change control team has approved the change, the implementation team has been identified, and the users have been notified, it is time to actually make the change. During the change implementation phase, the implementation team should be monitoring and verifying that all the steps defined in the implementation plan are being followed. During this time, the implementation team should also be routinely and regularly testing the change to ensure that all systems that were supposed to be affected by the change have indeed been affected. Some of the testing and verification steps that can be performed to validate the change are as follows:
Connectivity testing through the use of extended pings and performance monitoring tools For example, you can run the command ping “t < ipaddress > and it will continue sending out ping requests until you disable it by pressing CTRL-C .
Traceroutes to verify network routing functions For example, you can run the tracert command on Windows or traceroute on Unix or Cisco IOS “based devices. You can also run show ip route on your routers to output the active routing information on your routers.
End-user desktop and application testing For example, if you changed a router that exists between the users and their e-mail server, have the users attempt to send and receive e-mail to make sure everything is still working.
File transfers or traffic-generation methods to verify network performance For example, you can run programs such as UDPFlood or Blast from Foundstone (www.foundstone.com) to generate traffic.
Many of these traffic-generation tools are detected as denial of service programs in modern virus-protection software. You may need to add exclusions on the systems that you want to run these programs on to prevent the virus-protection software from detecting and deleting the traffic-generation software.
Bit error rate testor (BERT) testing for new circuits You will likely need your provider to run this test, or you can use a BERT tester such as the Harris TS-350 Test set.
BERT testing interrupts network services, so make sure you schedule the time to run the BERT test to ensure that the network outage is acceptable.
Interface statistics monitoring For example, you can run the show interface command and look for values that are not what was expected:
local-rtr#show interface e0 Ethernet0 is up, line protocol is up Hardware is Lance, address is 0050.736b.fec3 (bia 0050.736b.fec3) Internet address is 192.168.173.99/27 MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Queueing strategy: fifo Output queue 0/40, 0 drops; input queue 1/75, 0 drops 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 22743 packets input, 3233863 bytes, 0 no buffer Received 21345 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 30152 packets output, 2835588 bytes, 0 underruns(0/0/0) 0 output errors, 0 collisions, 17 interface resets 0 babbles, 0 late collision, 1 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out
Log file verification For example, if you implemented a change to block traffic on a certain port, you can check your system logs to see if the traffic is actually being blocked. This can be done by reviewing syslog or even the console logging capabilities through the use of the terminal monitor command on your IOS-based routers.
Viewing system configurations For example, you can compare your previous configuration to the new configuration to verify that the changes have been made.
Enterprise management station availability and verification For example, you can verify that the network device is still being monitored by your enterprise management station and is performing as expected.
Once you have performed the change, the next phase is verifying that the results were successful and that all network documentation and enterprise management applications have been updated to reflect the change. You should also back up the configuration from all relevant devices to reflect any new configurations. Some specific items to verify are listed here:
Visio or other physical and logical network diagrams.
Network policies and procedures.
Name resolution (DNS and so on) works against the device loopback address and adheres to corporate naming standards.
Name resolution (DNS and so on) works against the device interface addresses and adheres to corporate naming standards.
Removal of any old DNS entries or enterprise management objects from the network.
Standard enterprise management (SNMP, RMON, and so on) on the device.
Fault management tools are updated according to the change.
Inventory management tools are updated.
All network management scripts, databases, and spreadsheets are updated.
IP address number plans and assignments are updated.
Out-of-band management access maps, phone numbers , and documentation are updated.
VLAN numbering plans are updated.
Software code and hardware device matrixes are updated.
Protocol and packet filters and ACL documentation are updated.
Routing protocol standards and configurations are updated.
DHCP server reservations are updated.
In addition, you should keep all the change documentation in the event that you experience a problem that may be related to the change, or have to make a change in the future that the current documentation could assist in the planning and implementation of.
The final phase of the change management process is to perform an after-action report on the change. The goal of the after-action report is first to ensure that the change met the goals and objectives used for the justification for the change. If it does not, you need to ask and answer the question, why? You also need to review whether all tasks required by the Update All Network Documentation and Enterprise Management Applications phase were completed to the satisfaction of the change management team. Finally, you should review any and all problems encountered ” especially if you had to roll back the change ”to see if there is anything that could have been done to improve the process. This is helpful in developing a lessons learned document that will help make sure future changes go more smoothly.
As you have read through this chapter, you might find yourself thinking, OK, this sounds great in the perfect world where I am in control of everything, but in the real world, this process seems like it is overbearing. In fact, a common response to implementing proper change control is often, Whoa, that seems like way too much. The good news is that it doesn t have to be that way. If you want to be successful in implementing a change control process in your organization, you need to consider the following:
You absolutely must have upper management support. If you do not have it, you will fail.
Your change control process must be flexible.
Change control is a learned habit.
A reality of all businesses is that everyone has their territory. The network guys are in charge of the routers, the server guys are in charge of the servers, the desktop guys are in charge of the client PCs, and so on. Each group takes ownership of their territory. To the network guys, these are their routers. It doesn t matter that they are really the company s routers; the network guys don t usually see it that way. This creates a built-in problem when we talk about change control because change control has the potential to work across multiple groups territories , acting in many ways as a central control. The psychological issue then becomes having an outside group that has the appearance and, in some cases, the effect of telling other people what to do with what they perceive as their resources. This is a recipe for disaster that has to be addressed with two things.
First, you have to have tact in dealing with these groups. They don t want to feel like they are giving up control, so your job is to convince them that they are not doing that. I had a mentor who used to call it finding your champion. You should identify individuals in all the groups you are going to have to work with who want to work with you. Then, instead of you fighting the battles to get things done, you can empower that person to do it. Why? Because they are already a part of the group. They can say the same thing you would and have it be accepted much more readily than if you were to say it. They become your champions , fighting the battles that you can t win.
Second, you have to have strong upper management support. When push comes to shove and when tact and diplomacy have failed, you must have upper management support. They can step up to the plate and say, We are doing it this way because change control said so. You want to use this support as little as possible, though, because it is going to leave a bitter taste in some folk s mouths. Choose your battles wisely, and, most important, you should ask yourself, Can I make a compromise here instead of having to force the change? If you can, then compromise.
Another step in making your change control process successful is to implement a flexible change control process. The strongest process is one that is flexible when it needs to be, yet rigid when required. In addition to being open to compromise where possible, you have to understand that not all changes are the same, and therefore not all changes require the same change control flow. There are generally three types of changes you will need to make on your network:
Immediate, critical, or high-risk changes
Urgent, important, or medium-risk changes
Nonurgent, routine, or low-risk changes
The type of change should dictate how you engage the change control process. Your change control process should define a regular and routine time to meet and discuss the changes that need to be made ”perhaps every week or every two weeks. In addition, you should have a regularly scheduled day for most changes to occur. For example, you could say that all changes must occur on Thursdays. The benefit of this is that it allows you to make changes part of your regular activities so that they become less of an intrusion and more of a normal occurrence in your environment. It also allows your various groups to get into a routine for how they approach making changes so that they include the change control process as part of their planning, and not as an afterthought.
However, not all changes can wait for a regularly scheduled meeting. This is why you must also build a process that can account for immediate or urgent changes in addition to your nonurgent and routine changes. You need to ensure that your change control process has a mechanism defined for meeting and planning changes in a much quicker fashion than your routine and regularly scheduled process. This is commonly referred to as emergency change management .
Emergency change management does not abandon the concept of change control; it merely expedites it. For example, let s say a new virus is released in the wild that you are going to be highly susceptible to. You do not have time to wait for your regularly scheduled meeting and then until the next Thursday to roll out new virus signatures. You need to make a change now. To this end, you must identify the type of situations that warrant an emergency change and who is authorized to make those changes. This creates a limited change control process flow, as illustrated in Figure 14-2. Many of these steps are simply lightweight counterparts to the full change control process we previously defined.
Remember that change control is a learned habit. You are going to have to educate your users ”but more important, your IT staff ”of the requirements and responsibilities of change control. Do not assume just because you have technical gods on your staff that they understand what is necessary for change control. Set their expectations and responsibilities so that they know what is expected and what they need to do to meet those expectations. Finally, show them the value in adhering to change control. The knee-jerk reaction is going to be, This is just making my job more difficult. You need to make sure it s understood that although it may be more difficult in planning and implementing the actual changes, managing the changes appropriately will make their day-to-day jobs as well as troubleshooting much easier.