Recovering from Memory Leaks and Deadlocks

Web applications have some vulnerabilities where over time their performance can significantly diminish. These vulnerabilities fall into two categories: memory leaks and deadlocks. Both can severely affect the performance of a Web application, even to the point where it comes to a grinding halt. And the .NET CLR offers some substantial help in these areas.

Memory Leaks

Wait a minute, though; .NET applications don't have memory leaks ”right? Yes, in theory that is correct. Unfortunately, though, some developer oversights can cause memory to leak. Take, for instance, an application variable in which user information is stored. What if information is added but never removed? What if the application variable holds more and more information as time goes by? The answer is that the memory the application consumes will go up, and the available free memory on the server will go down. And when available memory goes down, so does performance.

When the memory consumption gets so great that performance is poor, an application restart is necessary to get things back on track. ASP.NET applications can be configured so that they will restart when memory conditions reach a certain point. Actually, the Apache Web server has been doing this for years. It's great that Internet Information Server (IIS) enables ASP.NET applications to do this because all my Unix friends can't give me a hard time about this anymore. For years I've had to endure their jabs when I mention having to restart my Web server.

Setting a threshold for application restart couldn't be easier. All that's required is a simple addition to the Web.Config file as follows :

 <processmodel memorylimit="75" /> 

This line causes the application to restart when available memory falls below 75%. You can set this percentage to anything that you feel is best for your application. Then, when the available memory falls below the threshold, a new instance of your application is spun up. This happens while the old instance is still satisfying requests. With both applications running, all new requests go to the new instance, and requests made to the old application are drained. Eventually, all requests to the old application are satisfied and no more requests are made to that application. At this point, the old application is released and all memory it was holding is released. This way, any new instance of an application will not be noticed by users.

Deadlocks

There are many cases where an ASP.NET application must synchronize its use of resources. There might be a shared resource such as an Application variable that should be accessed by only a single thread, and simultaneous access could, for whatever reason, prove to be detrimental to the application's performance. In these cases, developers usually use the Lock() method to restrict the execution of a piece of code to a single thread and thereby ensure that only a single instance of code is accessing a resource. The Unlock() method then releases the locking mechanism so that other threads can execute the code.

There is an inherent performance hit for using these mechanisms. For starters, a certain amount of overhead is spent using Lock() and Unlock() . And what is probably even worse is that some threads have to wait until the lock is removed before continuing. In some cases, this significantly reduces an application's performance. But this can't always be avoided, and judicious use of Lock() and Unlock() is okay.

However, one of the problems with anything that holds up the execution of a thread, is that code which is interwoven with other code that uses locking mechanisms can sometimes result in a deadlock. This can happen when one thread locks a piece of code and then goes on to execute another piece of code. But another thread might be executing the second piece of code and have it locked, and then might need to execute the first piece of code, which is already locked. In cases such as this, each piece of code is waiting on the other piece of code to become unlocked. They can't proceed because each locked thread is waiting on the other. This a classic deadlock and brings a Web application to a grinding halt.

ASP.NET applications can be restarted in cases of deadlocks. A threshold can be set in the Web.Config file where the application restarts if enough requests build up in the queue. When a deadlock occurs, any request that gets hung up waiting for the locked code stays in the queue. For this reason, the request queue is a good indicator of a deadlock condition. Of course, you should be careful not to set the threshold so low that high traffic conditions will exceed the threshold. Reasonable values can be obtained using the performance monitor before the Web.Config file is edited.

The following entry restarts the application when there are 500 requests in the queue:

 <processmodel requestqueuelimit="500" /> 

The same seamless method of starting a new instance of the application and gracefully moving requests from the old application to the new happens in the case of deadlock restarts just as it does for memory restarts.

The processmodel attributes can be seen in Table 21.1. They determine the behavior of the CLR with regards to your application.

Table 21.1. The processmodel Attributes

Attribute	Option	Description
enable		Specifies whether the process model is enabled.
	true	Indicates that the process model is enabled.
	false	Indicates that the process model is not enabled.
timeout		Specifies the number of minutes until ASP.NET launches a new worker process to take the place of the current one. The default is infinite.
idleTimeout		Specifies the number of minutes of inactivity until ASP.NET automatically shuts down the worker process. The default is infinite.
shutdownTimeout		Specifies the number of minutes allowed for the worker process to shut itself down. When the timeout expires , ASP.NET shuts down the worker process. The time is expressed in hr:min:sec format, so 0:00:05 is 5 seconds. The default is 5 seconds.
requestLimit		Specifies the number of requests allowed before ASP.NET automatically launches a new worker process to take the place of the current one. The default is infinite.
requestQueueLimit		Specifies the number of requests allowed in the queue before ASP.NET begins returning "503 - Server Too Busy" errors to new requests. The default is 5000.
memoryLimit		Specifies the maximum allowed memory size , as a percentage of total system memory that the worker process can consume before ASP.NET launches a new process and reassigns existing requests. The default is 40%.
cpuMask		Specifies which processors on a multiprocessor server are eligible to run ASP.NET processes. The cpuMask value specifies a bit pattern that indicates the CPUs eligible to run ASP.NET threads. For example, the cpuMask value 13 represents the bit pattern 1101. On a computer with four CPUs, this indicates that ASP.NET processes can be scheduled on CPUs 0, 2, and 3, but not on CPU 1. ASP.NET launches one worker process for each eligible CPU. If the webGarden attribute (see following) is set to false, the cpuMask limits worker processes to the number of eligible CPUs. (The maximum allowed number of worker processes is equal to the number of CPUs). By default all CPUs are enabled and ASP.NET launches one process for each CPU.
webGarden		Controls CPU affinity (the habit of threads to be serviced by a preferred processor) when used in conjunction with the cpuMask attribute. (A multiprocessor Web server is called a Web garden.)
	true	Indicates that CPU usage is scheduled by the Windows operating system. The default is true.
	false	Indicates that the cpuMask attribute is used to specify which CPUs are eligible to run ASP.NET processes.
userName		If present, this attribute runs the worker process with a different Windows identity than the default process. By default, the process runs as the IIS identity (System). If valid credentials are presented in the username and password attributes, the process is run with the given account. The other special value is "Machine," which, when used in conjunction with autogenerate as the password, runs the process as an unprivileged ASP.NET service account. The default value is "System."
password		If present (and in conjunction with a user name ), this attribute causes the worker process to run with the configured Windows identity. The default is "autogenerate."

Scheduled Restarts

Waiting on poor memory conditions and deadlocks is one way to manage application restarts. You also can schedule restarts based on either elapsed time or requests. The following entry in the Web.Config restarts the application after 50 hours of operation:

 <processmodel timeout="50" /> 

The following entry in the Web.Config restarts the application after 75,000 requests:

 <processmodel requestlimit="75" /> 

It is important to note a few things regarding the application restarts. First, the ASP.NET files don't have to be recompiled at the next request. The compiled binaries are saved to disk and then reloaded in the new application. The second important point is that session state can be stored in the ASPState Windows NT service, or in SQL server. The following line in the Web.Config causes the session state to be stored in the ASPState service:

 <configuration>    <sessionstate inproc="false" server="localhost" />  </configuration> 

If the service is not started, use the following command line to start it:

 net start aspstate 

Session state can be stored on a remote machine also. The following addition to the Web.Config file causes the session state to be stored in the ASPState service on a machine at 101.102.103.104:

 <configuration>    <sessionstate inproc="false" server="101.102.103.104" />  </configuration>