Maintaining Windows Server 2003

Maintaining Windows Server 2003 systems isn't an easy task for administrators. They must find time in their firefighting efforts to focus and plan for maintenance on the server systems. When maintenance tasks are commonplace in an environment, they can alleviate many of the common firefighting tasks.

The processes and procedures for maintaining Windows Server 2003 systems can be separated based on the appropriate time to maintain a particular aspect of Windows Server 2003. Some maintenance procedures require daily attention, whereas others may require only yearly checkups. The maintenance processes and procedures that an organization follows depend strictly on the organization; however, the categories described in the following sections and their corresponding procedures are best practices for organizations of all sizes and varying IT infrastructures.

Note

These tasks are in addition to those examined earlier in this chapter.

Daily Maintenance

Certain maintenance procedures require more attention than others. The procedures that require the most attention are categorized into the daily procedures. Therefore, it is recommended that an administrator take on these procedures each day to ensure system reliability, availability, performance, and security. These procedures are examined in the following three sections.

Checking Overall Server Functionality

Although checking the overall server health and functionality may seem redundant or elementary, this procedure is critical to keeping the system environment and users working productively.

Some questions that should be addressed during the checking and verification process are the following:

• Can users access data on file servers?

• Are printers printing properly? Are there long queues for certain printers?

• Is there an exceptionally long wait to log on (that is, longer than normal)?

• Can users access messaging systems?

• Can users access external resources?

Verifying That Backups Are Successful

To provide a secure and fault-tolerant organization, it is imperative that a successful backup to tape be performed each night. In the event of a server failure, the administrator may be required to perform a restoration from tape. Without a backup each night, the IT organization will be forced to rely on rebuilding the server without the data. Therefore, the administrator should always back up servers so that the IT organization can restore them with minimum downtime in the event of a disaster. Because of the importance of the tape backups, the first priority of the administrator each day needs to be verifying and maintaining the backup sets.

If disaster ever strikes, the administrators want to be confident that a system or entire site can be recovered as quickly as possible. Successful backup mechanisms are imperative to the recovery operation; recoveries are only as good as the most recent backups.

Although Windows Server 2003's backup program does not offer alerting mechanisms for bringing attention to unsuccessful backups, many third-party programs do. In addition, many of these third-party backup programs can send emails or pages if backups are successful or unsuccessful.

Monitoring the Event Viewer

The Event Viewer, shown in Figure 22.12, is used to check the System, Security, Application, and other logs on a local or remote system. These logs are an invaluable source of information regarding the system. The following event logs are present for Windows Server 2003 systems:

Figure 22.12. The Event Viewer utility.

• Security log The Security log captures all security-related events that are being audited on a system. Auditing is turned on by default to record success and failure of security events.

• Application log Specific application information is stored in the Application log. This information includes services and any applications that are running on the server.

• System log Windows Server 2003specific information is stored in the System log.

Domain controllers also have these additional logs:

• File Replication Service Any events relating to the File Replication Service are captured in this log.

• Directory Service Events regarding Active Directory, such as connection problems with a global catalog server or replication problems, are recorded here.

• DNS Server Anything having to do with the DNS service is cataloged in the DNS Server log.

All Event Viewer events are categorized either as informational, warning, or error. Logs show events of the types shown in Figure 22.13.

Figure 22.13. Event types.

Note

Checking these logs often will help your understanding of them. There are some events that constantly appear but aren't significant. Events will begin to look familiar, so you will notice when something is new or amiss in your event logs.

Some best practices for monitoring event logs include

• Understanding the events that are being reported

• Setting up a database for archived event logs

• Archiving event logs frequently

To simplify monitoring hundreds or thousands of generated events each day, the administrator should use the filtering mechanism provided in the Event Viewer. Although warnings and errors should take priority, the informational events should be reviewed to track what was happening before the problem occurred. After the administrator reviews the informational events, she can filter out the informational events and view only the warnings and errors.

To filter events, do the following:

1.	Start the Event Viewer by choosing Start, Programs, Administrative Tools.
2.	Select the log from which you want to filter events.
3.	Right-click the log and select View, Filter.
4.	In the log properties window, as shown in Figure 22.13, select the types of events to filter.
5.	Optionally, select the time frame in which the events occurred. Click OK when you're done.

Some warnings and errors are normal because of bandwidth constraints or other environmental issues. The more you monitor the logs, the more familiar you will become with the messages and therefore will be able to spot a problem before it affects the user community.

Tip

You may need to increase the size of the log files in the Event Viewer to accommodate an increase in logging activity.

Weekly Maintenance

Maintenance procedures that require slightly less attention than daily checking are categorized in a weekly routine and are examined in the following sections.

Checking Disk Space

Disk space is a precious commodity. Although the disk capacity of a Windows Server 2003 system can be virtually endless, the amount of free space on all drives should be checked daily. Serious problems can occur if there isn't enough disk space.

One of the most common disk space problems occurs on data drives where end users save and modify information. Other volumes such as the system drive and partitions with logging data can also quickly fill up.

As mentioned earlier, lack of free disk space can cause a multitude of problems including, but not limited to, the following:

• Application failures

• System crashes

• Unsuccessful backup jobs

• Service failures

• The inability to audit

• Degradation in performance

To prevent these problems from occurring, administrators should keep the amount of free space to at least 25%.

Caution

If you need to free disk space, you should move or delete files and folders with caution. System files are automatically protected by Windows Server 2003, but data is not.

Verifying Hardware

Hardware components supported by Windows Server 2003 are reliable, but this doesn't mean that they'll always run continuously without failure. Hardware availability is measured in terms of mean time between failures (MTBF) and mean time to repair (MTTR). This includes downtime for both planned and unplanned events. These measurements provided by the manufacturer are good guidelines to follow; however, mechanical parts are bound to fail at one time or another. As a result, hardware should be monitored weekly to ensure efficient operation.

Hardware can be monitored in many different ways. For example, server systems may have internal checks and logging functionality to warn against possible failure, Windows Server 2003's System Monitor may bring light to a hardware failure, and a physical hardware check can help to determine whether the system is about to experience a problem with the hardware.

If a failure has occurred or is about to occur, having an inventory of spare hardware can significantly improve the chances and timing of recoverability. Checking system hardware on a weekly basis provides the opportunity to correct the issue before it becomes a problem.

Checking Archive Event Logs

The three event logs on all servers and the three extra logs on a DC can be archived manually, or a script can be written to automate the task. You should archive the event logs to a central location for ease of management and retrieval.

The specific amount of time to keep archived log files varies on a per-organization basis. For example, banks or other high-security organizations may be required to keep event logs up to a few years. As a best practice, organizations should keep event logs for at least three months.

The following script, named Logarchive.vbs, can retrieve event logs and store them in a central location. The process may take a long time (up to a few hours) depending on the size of the log files as well as how many servers you're pulling from. Avoid running this script over slow WAN connections so that bandwidth is conserved.

Set WS = CreateObject("Wscript.Shell") Set FSO = CreateObject("Scripting.FileSystemObject") DateString = CurrentDate() ServerName = "HOFS01" Purge = True on error resume next StartTime = Now Output "---------------------------------" OutPut "Started at:  " + CStr(Now) Output "" Set System = GetObject("winmgmts:{(Backup,Security)}\\" + ServerName + "\root\CIMV2") If Err.Number = 0 Then     Set colLogs = System.ExecQuery("select * from Win32_NTEventLogFile",,&H30) For Each refLog In colLogs             LogName = ServerName+ "_" + LogFileName(refLog.LogFileName) +  "_" + DateString     If FSO.FileExists("C:\Logs\" + LogName + ".evt") Then  FSO.DeleteFile("C:\Logs\" + LogName + ".evt")     If Purge Then           RetVal = reflog.ClearEventlog("C:\Logs\" + LogName + ".evt")           Else               RetVal = reflog.BackupEventlog("C:\Logs\" + LogName + ".evt")           End If           If RetVal = 0 Then               Output vbTab + "Log was archived in .evt format: " + LogName +  ".evt"               If Purge Then Output vbTab + "All events were cleared from  the log"           Else                 Output vbTab + "Error while archiving in .evt format."           End If     Next Else     Output vbTab + "Failed connect to the server" End If Set colLogs = Nothing Set refLogs = Nothing Set System = Nothing Output "----------------------------------------" OutPut "Finished at:  " + CStr(Now) Output "" Output "" Set WS = Nothing FullLog.Close Set FullLog = Nothing Set FSO = Nothing Function CurrentDate     Today = Date     If Month(Today) < 10 Then         CurrentDate = "0" + CStr(Month(Today))     Else         CurrentDate = CStr(Month(Today))     End If     If Day(Today) < 10 Then         CurrentDate = CurrentDate + "0" + CStr(Day(Today))     Else         CurrentDate = CurrentDate + CStr(Day(Today))     End If     CurrentDate = CurrentDate + CStr(Year(Today))     If Hour(Time) < 10 Then         CurrentDate = CurrentDate + "0" + CStr(Hour(Time))     Else         CurrentDate = CurrentDate + CStr(Hour(Time))     End If End Function Function LogFileName(LogName)     Select Case LogName         Case "Application"             LogFileName = "app"         Case "Directory Service"             LogFileName = "dir"         Case "DNS Server"             LogFileName = "dns"         Case "File Replication Service"             LogFileName = "rep"         Case "Security"             LogFileName = "sec"         Case "System"             LogFileName = "sys"     End Select End Function Sub Output(Text)     wscript.echo text     FullLog.writeline text End Sub 

Another file, logarchive.ini, is required when using logarchive.vbs. This file, shown next, contains a list of servers and the following archiving modes:

• T means purge after archiving.

• F means archive only.

  servername,T servername,F servername,F 

To use logarchive.vbs, do the following:

1.	Verify that the logarchivelog.vbs and logarchive.ini files are in a pathed directory.
2.	Right-click the logarchive.ini file and type the list of servers on which you want to archive event logs.
3.	Choose Start, Run and type cmd to open a command prompt.
4.	At the command prompt, type cscript logarchive.vbs.

The command in step 4 archives all the event logs for the servers that were specified in the logarchive.ini file. The log files are stored in the directory specified in the script. Logs will be labeled in the following format:

servername_logname_date.log

For example, sfdc01_sec_02202003.log is the name for the SFDC01 server's Security log, archived on February 20, 2004.

It is recommended that you label log files in the following manner:

• _sec_ Security log

• _app_ Application log

• _sys_ System log

• _rep_ File Replication log

• _dns_ DNS Server log

• _dir_ Directory Service log

Note

logarchive.vbs does not purge the event logs.

Running Disk Defragmenter

Whenever files are created, deleted, or modified, Windows Server 2003 assigns a group of clusters depending on the size of the file. As file size requirements fluctuate over time, so does the number of groups of clusters assigned to the file. Even though this process is efficient when using NTFS, the files and volumes become fragmented because the file doesn't reside in a contiguous location on the disk.

As fragmentation levels increase, as illustrated in Figure 22.14, disk access slows. The system must take additional resources and time to find all the cluster groups in order to use the file. To minimize the amount of fragmentation and give performance a boost, the administrator should use Disk Defragmenter to defragment all volumes. As mentioned earlier in the chapter, Disk Defragmenter is a built-in utility that can analyze and defragment volume fragmentation. Fragmentation negatively affects performance because files aren't efficiently read from disk.

Figure 22.14. Disk Defragmenter tool.

To use Disk Defragmenter, do the following:

1.	Start Disk Defragmenter by choosing Start, Programs, Accessories, System Tools.
2.	Select the drive to either analyze or defragment.
3.	Click either the Analyze or Defragment button to begin the defragmentation process. Note that simply selecting the Defragment button will automatically analyze the drive before proceeding with the defragmentation process.

Tip

Disk Defragmenter included in Windows Server 2003 is a scaled-down version of Diskeeper. Using a third-party defragmentation product, such as Diskeeper, enables you to schedule defragmentation during nonpeak hours. Many other features of third-party products, such as defragmenting servers remotely, would greatly benefit a company.

Running the Domain Controller Diagnostic Utility

The Domain Controller Diagnostic (DCDIAG) utility provided in the Windows Server 2003 Support Tools is used to analyze the state of a domain controller (DC). It runs a series of tests, analyzes the state of the DC, and verifies different areas of the system, such as

• Connectivity

• Replication

• Topology integrity

• Security descriptors

• Netlogon rights

• Intersite health

• Roles

• Trust verification

DCDIAG should be run on each DC on a weekly basis or as problems arise. DCDIAG's syntax is as follows:

[View full width]
dcdiag.exe /s:<Domain Controller> [/u:<Domain>\<Username> /p:*|<Password>|""] [/h: {parameter}] [/q:{parameter}] [/v:{parameter}] [/n:<Naming Context>] [/f:<Log>] [/ ferr:<Errlog>][/skip:<Test>] [/test:<Test>] 

Parameters for this utility are

• /h Display this help screen.

• /s Use <Domain Controller> as the home server. This is ignored for DCPromo and RegisterInDns tests, which can only be run locally.

• /n Use <Naming Context> as the naming context to test. Domains may be specified in NetBIOS, DNS, or distinguished name (DN) format.

• /u Use domain\username credentials for binding with a password. Must also use the /p option.

• /p Use <Password> as the password. Must also use the /u option.

• /a Test all the servers in this site.

• /e Test all the servers in the entire enterprise. This parameter overrides the /a parameter.

• /q Quiet; print only error messages.

• /v Verbose; print extended information.

• /i Ignore; ignore superfluous error messages.

• /fix Fix; make safe repairs.

• /f Redirect all output to a file <Log>; /ferr will redirect error output separately.

• /ferr:<ErrLog> Redirect fatal error output to a separate file <ErrLog>.

• /c Comprehensive; run all tests, including nondefault tests but excluding DCPromo and RegisterInDNS. Can use with /skip.

• /skip:<Test> Skip the named test. Do not use in a command with /test.

• /test:<Test> Test only the specified test. Required tests will still be run. Do not use with the /skip parameter.

Valid tests that can be run include the following:

• Connectivity Tests whether DCs are DNS registered, pingable, and have LDAP/RPC connectivity.

• Replications Checks for timely replication between domain controllers.

• Topology Checks that the generated topology is fully connected for all DCs.

• CutoffServers Checks for servers that won't receive replications because their partners are down.

• NCSecDesc Checks that the security descriptors on the naming context heads have appropriate permissions for replication.

• NetLogons Checks that the appropriate logon privileges allow replication to proceed.

• Advertising Checks whether each DC is advertising itself and whether it is advertising itself as having the capabilities of a DC.

• KnowsOfRoleHolders Checks whether the DC thinks it knows the role holders of the five FSMO roles.

• Intersite Checks for failures that would prevent or temporarily hold up intersite replication.

• FsmoCheck Checks that global roles are known, can be located, and are responding.

• RidManager Checks to see whether RID master is accessible and whether it contains the proper information.

• MachineAccount Checks to see whether the machine account has the proper information. Use the /RecreateMachineAccount parameter to attempt a repair if the local machine account is missing. Use /FixMachineAccount if the machine's account flags are incorrect.

• Services Checks to see whether DC services are running on a system.

• OutboundSecureChannels Verifies that secure channels exist from all the DCs in the domain to the domains specified by /testdomain. The /nositerestriction parameter will prevent the test from being limited to the DCs in the site.

• ObjectsReplicated Checks that machine account and DSA objects have replicated. You can use /objectdn:<dn> with /n:<nc> to specify an additional object to check.

• frssysvol Checks that the file replication system (FRS) SYSVOL is ready.

• kccevent Checks that the Knowledge Consistency Checker is completing without errors.

• systemlog Checks that the system is running without errors.

• DCPromo Tests the existing DNS infrastructure for promotion to the domain controller.

• RegisterInDNS Tests whether this domain controller can register the Domain Controller Locator DNS records. These records must be present in DNS for other computers to locate this domain controller for the <Active_Directory_Domain_DNS_Name> domain. Reports whether any modifications to the existing DNS infrastructure are required. Requires the /DnsDomain:<Active_Directory_Domain_DNS_Name> argument.

• CheckSDRefDom Checks that all application directory partitions have appropriate security descriptor reference domains.

Note

Topology, CutoffServers, and OutboundSecureChannels are not run by default.

Monthly Maintenance

It is recommended that you perform the tasks examined in the following sections on a monthly basis.

Maintaining File System Integrity

CHKDSK scans for file system integrity and can check for lost clusters, cross-linked files, and more. If Windows Server 2003 senses a problem, it will run CHKDSK automatically at startup.

Administrators can maintain FAT, FAT32, and NTFS file system integrity by running CHKDSK once a month. To run CHKDSK, do the following:

1.	At the command prompt, change to the partition that you want to check.
2.	Type CHKDSK without any parameters to check only for file system errors.
3.	If any errors are found, run the CHKDSK utility with the /f parameter to attempt to correct the errors found.

Testing the UPS

An uninterruptible power supply (UPS) can be used to protect the system or group of systems from power failures (such as spikes and surges) and keep the system running long enough after a power outage so that an administrator can gracefully shut down the system. It is recommended that an administrator follow the UPS guidelines provided by the manufacturer at least once a month. Also, monthly scheduled battery tests should be performed.

Validating Backups

Once a month, an administrator should validate backups by restoring the backups to a server located in a lab environment. This is in addition to verifying that backups were successful from log files or the backup program's management interface. A restore gives the administrator the opportunity to verify the backups and to practice the restore procedures that would be used when recovering the server during a real disaster. In addition, this procedure tests the state of the backup media to ensure that they are in working order and builds administrator confidence for recovering from a true disaster.

Updating Automated System Recovery Sets

Automated System Recovery (ASR) is a recovery tool that should be implemented in all Windows Server 2003 environments. It backs up the system state data, system services, and all volumes containing Windows Server 2003 system components. ASR, shown in Figure 22.15, replaces the Emergency Repair Disks (ERDs) used to recover systems in earlier versions of Windows.

Figure 22.15. The ASR utility.

After building a server and any time a major system change occurs, the ASR sets (that is, the backup and floppy disk) should be updated. Another best practice is to update ASR sets at least once a month. This keeps content in the ASR sets consistent with the current state of the system. Otherwise, valuable system configuration information may be lost if a system experiences a problem or failure.

To create an ASR set, do the following:

1.	Open Windows Server 2003's Backup utility by choosing Start, Programs, Accessories, System Tools.
2.	Click Advanced Mode from the first screen in the Backup or Restore Wizard.
3.	Click the Automated System Recovery Wizard button.
4.	Click Next in the Automated System Recovery Preparation Wizard window.
5.	Select the backup destination and then click Next to continue.
6.	Click Finish when you're done.

Note

This process might take a while to complete, so be patient. Depending on the performance of the system being used and the amount of information to be transferred, this process could take several minutes to a few hours to complete.

Updating Documentation

An integral part of managing and maintaining any IT environment is to document the network infrastructure and procedures. The following are just a few of the documents you should consider having on hand:

• Server build guides

• Disaster recovery guides and procedures

• Checklists

• Configuration settings

• Change configuration logs

• Historical performance data

• Special user rights assignments

• Special application settings

As systems and services are built and procedures are ascertained, document these facts to reduce learning curves, administration, and maintenance.

It is not only important to adequately document the IT environment, but it's often even more important to keep those documents up to date. Otherwise, documents can quickly become outdated as the environment, processes, and procedures change as the business changes.

Quarterly Maintenance

As the name implies, quarterly maintenance is performed four times a year. Areas to maintain and manage on a quarterly basis are typically fairly self-sufficient and self-sustaining. Infrequent maintenance is required to keep the system healthy. This doesn't mean, however, that the tasks are simple or that they aren't as critical as those tasks that require more frequent maintenance.

Checking Storage Limits

Storage capacity on all volumes should be checked to ensure that all volumes have ample free space. Keep approximately 25% free space on all volumes.

Running low or completely out of disk space creates unnecessary risk for any system. Services can fail, applications can stop responding, and systems can even crash if there isn't plenty of disk space.

Changing Administrator Passwords

Administrator passwords should, at a minimum, be changed every quarter (90 days). Changing these passwords strengthens security measures so that systems can't easily be compromised. In addition to changing passwords, other password requirements such as password age, history, length, and strength should be reviewed.

Maintaining the AD Database

AD is the heart of the Windows Server 2003 environment. Objects such as users, groups, OUs, and more can be added, modified, or deleted from the AD database. This interaction with the database can cause fragmentation. Windows Server 2003 performs online defragmentation nightly to reclaim space in the AD database; however, the database size doesn't shrink unless offline defragmentation is performed. Figure 22.16 shows the differences in fragmented versus defragmented AD databases.

Figure 22.16. Fragmented versus defragmented AD databases.

NTDSUTIL is the tool for maintaining AD databases. It is used to defragment the AD database, but it also performs other routines such as cleaning up metadata left behind by abandoned domain controllers and managing Flexible Single Master Operations (FSMO).

Note

Compacting the AD database requires the DC to be rebooted.

To use NTDSUTIL to defragment the AD database, do the following:

1.	Restart the DC.
2.	When the initial screen appears, press the F8 key.
3.	From the Windows Advanced Options menu, select Directory Services Restore Mode (Windows domain controllers only).
4.	In the next screen, select the Windows Server 2003 operating system being used.
5.	Log on to the Windows Server 2003 system.
6.	Click OK when the informational message appears.
7.	At a command prompt, type NtdSUTIL files.
8.	At the File Maintenance prompt, type compact to %s, where %s identifies an empty target directory. This invokes Esentutl.exe to compact the existing database and write to the specified directory. The compaction process is illustrated in Figure 22.17. Figure 22.17. Using the NTDSUTIL.
9.	If compaction was successful, copy the new ntds.dit file to %systemroot%\NTDS and delete the old log files found in %systemroot%\NTDS.
10.	Type quit twice to exit the utility.
11.	Restart the computer.

Other uses for the NTDSUTIL include, but aren't limited to, the following:

• Info Analyzes and reports the free space, reads the Registry, and then reports the sizes of the database and log files.

• Integrity Performs an integrity check on the database, which detects any kind of low-level database corruption. This can take a long time to process if the AD database is large. It's important to note that you should always run Recover prior to running an integrity check.

• Recover Attempts to perform a soft recovery of the database. This task scans the log files and ensures all committed transactions therein are also reflected in the data file.

MMC 3.0

MMC 3.0 is a natural evolution in the Microsoft Management Console line. It integrates seamlessly with the new WinRM toolset to provide powerful tools for system management. MMC 3.0 offers a similar look and feel with some additional features. The first is the Action Pane. It appears on the ride side of the frame and lists all actions that are possible for the selected item. This is a much appreciated feature and decreases down time looking for functions. There is also a new Add or Remove Snap-in Wizard that gives options for MMC 3.0 snap-ins and allows nested display. Finally, this new version adds improved error correction for faults in snap-ins that can cause the MMC to fail.

MMC 3.0 works in conjunction with .Net 2.0 and allows C# developers much freedom in creating their own snap-ins for Microsoft, and even for custom applications. This piece finally closes the loop between developers and the operational staff who support an application once it moves to the production stage.

Microsoft plans a roadmap for migrating current snap-ins to the MMC 3.0 structure and adding additional functionality to current tools.