There is one absolute in our world: there will always be change. How we manage change determines the success or failure of our endeavors. There is a German proverb that states, To change and to change for the better are two different things. If changes aren t managed properly, they can often have negative consequences, even if the changes were meant to improve things and are completely well intentioned. This is especially true in the IT world, where there is a delicate balance between a properly run system and a chaotic one.
You now know how to update your systems via the mechanisms provided by the distributions, but that is only half the battle. The reasons to implement a change management system are numerous and directly relate to the security and availability of a system. When your hardening steps have been completed, all the improvements you made could be wiped out by a single improperly planned change. Patching can be a destructive process if not managed properly or sensibly. As with any operating system, patches can help repair your systems and decrease the vulnerabilities, but they can also create denial of service or downtime if not properly tested and managed. Let s look at a real-life scenario that demonstrates the points of change management and testing of patches before deployment.
Jane, a webmaster for all internal and external web sites at her company, recently completed a hardening project of her internal web site (no access from the WAN) where all the steps in this book were taken. Joe, a junior system administrator, sees that there is a new vulnerability in the Apache packages on his favorite security site and decides to patch all the servers immediately to prevent someone from exploiting this vulnerability. He doesn t realize that there are some applications depending on the specific version of Apache and its feature set on the system that Jane administers. On Friday, Joe installs the patches on all systems without testing and leaves for the day, not to return until Monday. Jane gets a call on the weekend that the internal web site is down and the weekend staff needs access. Jane logs in via VPN and tries to figure out why the web applications won t work. After many hours of troubleshooting, she determines that the version of Apache is different from the one she needs for the applications. After more time spent reverting to the old version of Apache, she is finally able to get the web server up and running.
This scenario could have been completely avoided by using simple patch management and change management procedures. If Joe had contacted Jane, or at a minimum documented the change, many hours of downtime and problem resolution would have been averted. If the change had gone through a change control board, the problem may have never occurred at all. Even though the patch was needed, the urgency level was not as high since this was only an internal server with no connections to the outside. One interesting thing about this whole scenario is that there was a change management process in place, but it wasn t followed due to some underlying problems. Change management is the solution to this dilemma.
Many companies have realized that change management is crucial for their business to run effectively. Companies often institute monolithic change management processes with multiple layers of control and overly complicated procedures. The processes are usually followed closely at first, but over time degrade to the point where the process isn t followed or is followed with the minimum requirements. If your organization has a change management process that isn t working, determine the cause of the failure. You should keep the following in mind when creating your change management program:
Is it overly complicated?
Does it impede normal operations in a manner that isn t consistent with the level of change (such as too much documentation for minor changes)?
Is there management support and enforcement of the policy?
Is the company culture resistant to change management?
Is the change management policy properly documented in a way that users can understand?
These five questions may have surprising answers, and can often point out common causal factors in the failure of a change management process. When creating a change management process, you don t need to start with a large scope that includes all processes within your organization. You can start with a small log of all changes that occur to a server kept offline and progress from there. Creating a change management process is beyond the scope of this book, but a simple process can be implemented using the policy creation guidelines discussed in Chapter 16. Knowing you need a change management process is the first step in the creation of the process.
One other thing to note is the monitoring of your change management process, because the best laid plans can be thwarted if there is no supervision. An example of this is a company that had implemented a change management process that was thought to be successful in managing change. The change management process had been put in place to document every change that occurred on the system, regardless of the type of change, because of significant system outages or disruptions of service caused by improperly managed change. In fact, the manager in charge of the business segment commented that the systems ran better on the weekends when no one was around to tweak or otherwise modify them. The process had been in place for almost a year, but there were still problems, because it turned out that the users were not following the change management policy.
Some things to consider when creating a change management process are to ensure that patches and updates are planned and tested on a nonproduction server. Users and management must be informed of changes. Unplanned outages due to patches can create a hostile or apprehensive environment. Another important consideration is to ensure that the changes align to business goals. In our example of the Apache web server, the patching of the system didn t align to business goals, because the changes caused problems and were not critical enough to justify the change and expenditure of resources. These are only some of the things you should consider when creating a change management process.
After installing all your patches, you need to monitor them to ensure there are no unintended consequences. Patches may not immediately cause problems, but problems may show up days later when the patch was forgotten. You could minimize the impact of changes by using a test environment that mirrors your production environment, but realistically , most companies don t have the funding to run a mirror image of their production environments, so monitoring is crucial.
Patching is a critical but necessary evolution in the maintenance and security of modern operating systems. Keep your patching as timely as feasible and use your established change management and monitoring procedures to ensure maximum availability.