Entity Bean Locking and Deadlock Detection

The following sections provide information on what entity bean locking is and how entity beans are accessed and locked within JBoss. They also describe the problems you may encounter as you use entity beans within your system and how to combat these issues. Deadlocking is formally defined and examined. Finally, you'll learn how to finetune your system in terms of entity bean locking.

Why JBoss Needs Locking

Locking involves protecting the integrity of your data. Sometimes you need to be sure that only one user can update critical data at a time. Sometimes, access to sensitive objects in a system needs to be serialized so that data is not corrupted by concurrent reads and writes. Databases traditionally provide this sort of functionality with transactional scopes and table- and row-locking facilities.

Using entity beans is a great way to provide an object-oriented interface to relational data. Beyond that, entity beans can improve performance by taking the load off the database through caching and delaying updates until absolutely needed in order to maximize database efficiency. But with caching, data integrity is a problem, so some form of application server-level locking is needed for entity beans to provide the transaction isolation properties that are common with traditional databases.

The Entity Bean Life Cycle

With the default configuration of JBoss there is only one active instance of a given entity bean in memory at one time. This applies for every cache configuration and every type of commit option. The life cycle for this instance is different for every commit option, though:

For commit option A, this instance is cached and used between transactions.
For commit option B, this instance is cached and used between transactions, but it is marked as dirty at the end of a transaction. This means that at the start of a new transaction, ejbLoad must be called.
For commit option C, this instance is marked as dirty, released from the cache, and marked for passivation at the end of a transaction.
For commit option D, a background refresh thread periodically calls ejbLoad on stale beans within the cache. Otherwise, this option works in the same way as A.

When a bean is marked for passivation, the bean is placed in a passivation queue. Each entity bean container has a passivation thread that periodically passivates beans that have been placed in the passivation queue. A bean is pulled out of the passivation queue and reused if the application requests access to a bean with the same primary key.

On an exception or transaction rollback, the entity bean instance is thrown out of the cache entirely. It is not put into the passivation queue and is not reused by an instance pool. Except for the passivation queue, there is no entity bean instance pooling.

Default Locking Behavior

Entity bean locking is totally decoupled from the entity bean instance. The logic for locking is totally isolated and managed in a separate lock object. Because there is only one allowed instance of a given entity bean active at one time, JBoss employs two types of locks to ensure data integrity and to conform to the EJB spec:

Method lock The method lock ensures that only one thread of execution at a time can invoke on a given entity bean. This is required by the EJB spec.
Transaction lock A transaction lock ensures that only one transaction at a time has access to a given entity bean. This ensures the ACID properties of transactions at the application server level. Because, by default, there is only one active instance of any given entity bean at one time, JBoss must protect this instance from dirty reads and dirty writes. So the default entity bean locking behavior locks an entity bean within a transaction until it completes. This means that if any method at all is invoked on an entity bean within a transaction, no other transaction can have access to this bean until the holding transaction commits or is rolled back.

Pluggable Interceptors and Locking Policy

You have seen in this chapter that the basic entity bean life cycle and behavior are defined by the container configuration defined in the standardjboss.xml descriptor. The following is the container-interceptors definition for the Standard CMP 2.x EntityBean configuration:

[View full width]
 <container-interceptors>     <interceptor>org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.LogInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.SecurityInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.TxInterceptorCMT</interceptor>     <interceptor>org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>     <interceptor metricsEnabled="true">org.jboss.ejb.plugins.MetricsInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.EntityCreationInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.EntityLockInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.EntityInstanceInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.EntityReentranceInterceptor</interceptor>     <interceptor>org.jboss.resource.connectionmanager.CachedConnectionInterceptor< /interceptor>     <interceptor>org.jboss.ejb.plugins.EntitySynchronizationInterceptor</interceptor>     <interceptor>org.jboss.ejb.plugins.cmp.jdbc.JDBCRelationInterceptor</interceptor> </container-interceptors>

The interceptors shown here define most of the behaviors of the entity bean. The following are the interceptors that are relevant to this section:

EntityLockInterceptor This interceptor's role is to schedule any locks that must be acquired before the invocation is allowed to proceed. This interceptor is very lightweight and delegates all locking behavior to a pluggable locking policy.
EntityInstanceInterceptor The job of this interceptor is to find the entity bean within the cache or create a new one. This interceptor also ensures that there is only one active instance of a bean in memory at one time.
EntitySynchronizationInterceptor The role of this interceptor is to synchronize the state of the cache with the underlying storage. It does this with the ejbLoad and ejbStore semantics of the EJB specification. In the presence of a transaction, this is triggered by transaction demarcation. It registers a callback with the underlying transaction monitor through the JTA interfaces. If there is no transaction, the policy is to store the state upon returning from invocation. The synchronization polices A, B, and C of the specification are taken care of here, as is the JBoss-specific commit option D.

Deadlocking

This section describes what deadlocking means, how you can detect it within an application, and how you can resolve deadlocks. A deadlock can occur when two or more threads have locks on shared resources. Figure 5.8 illustrates a simple deadlock scenario. Here, Thread 1 has the lock for Bean A, and Thread 2 has the lock for Bean B. At a later time, THRead 1 tries to lock Bean B and blocks because Thread 2 has it. Likewise, as Thread 2 TRies to lock Bean A, it also blocks because Thread 1 has the lock. At this point, both threads are deadlocked, waiting for access to the resource already locked by the other thread.

Figure 5.8. A deadlock definition example.

The default locking policy of JBoss is to lock an entity bean when an invocation occurs in the context of a transaction until the transaction completes. Therefore, it is very easy to encounter deadlock if you have long-running transactions that access many entity beans or if you are not careful about ordering the access to them. You can use various techniques and advanced configurations to avoid deadlocking problems. They are discussed later in this chapter.

Deadlock Detection

Fortunately, JBoss is able to perform deadlock detection. JBoss holds a global internal graph of waiting transactions and what transactions they are blocking on. Whenever a thread determines that it cannot acquire an entity bean lock, it figures out what transaction currently holds the lock on the bean and adds itself to the blocked transaction graph. Table 5.1 shows an example of what the graph might look like.

Table 5.1. An Example of a Blocked Transaction Graph
Blocking Transaction	Transaction That Holds Needed Lock
Tx1	Tx2
Tx3	Tx4
Tx4	Tx1

Before the thread actually blocks, it tries to detect whether there is a deadlock problem. It does this by traversing the block transaction graph. As it traverses the graph, it keeps track of what transactions are blocked. If it sees a blocked node more than once in the graph, then it knows there is deadlock and throws an ApplicationDeadlockException. This exception causes a transaction rollback, which in turn causes all locks that transaction holds to be released.

Catching `ApplicationDeadlockException`

Because JBoss can detect application deadlock, you should write an application so that it can retry a transaction if the invocation fails because of the ApplicationDeadlockException. Unfortunately, this exception can be deeply embedded within a RemoteException, so you have to search for it in your catch block. Here's an example:

 try {     // ... } catch (RemoteException ex) {     Throwable cause = null;     RemoteException rex = ex;     while (rex.detail != null) {         cause = rex.detail;         if (cause instanceof ApplicationDeadlockException) {                 // ... We have deadlock, force a retry of the transaction.             break;         }         if (cause instanceof RemoteException) {             rex = (RemoteException)cause;         }     } }

Viewing Lock Information

The EntityLockMonitor MBean service allows you to view basic locking statistics as well as print out the state of the transaction locking graph. To enable this monitor, you uncomment its configuration in the conf/jboss-service.xml file:

 <mbean code="org.jboss.monitor.EntityLockMonitor"        name="jboss.monitor:name=EntityLockMonitor"/>

EntityLockMonitor has no configurable attributes. It does have the following read-only attributes:

MedianWaitTime The median value of all times threads had to wait to acquire a lock.
AverageContenders The ratio of the total number of contentions to the sum of all threads that had to wait for a lock.
TotalContentions The total number of threads that had to wait to acquire the transaction lock. This happens when a thread attempts to acquire a lock that is associated with another transaction
MaxContenders The maximum number of threads that were waiting to acquire the transaction lock.

EntityLockMonitor also has the following operations:

clearMonitor This operation resets the lock monitor state by zeroing all counters.
printLockMonitor This operation prints out a table of all EJB locks that lists the ejbName of the bean, the total time spent waiting for the lock, the count of times the lock was waited on, and the number of transactions that timed out waiting for the lock.

Advanced Configurations and Optimizations

The default locking behavior of entity beans can cause deadlock. Because access to an entity bean locks the bean into the transaction, this can also present a huge performance/throughput problem for an application. The following sections walk through various techniques and configurations that you can use to optimize performance and reduce the possibility of deadlock.

Short-Lived Transactions

You should make your transactions as short-lived and fine-grained as possible. The shorter a transaction, the less likelihood that it will have concurrent access collisions and the more the application throughput will go up.

Ordered Access

Ordering the access to entity beans can help lessen the likelihood of deadlock. This means making sure that the entity beans in a system are always accessed in exactly the same order. In most cases, user applications are just too complicated to use this approach, and more advanced configurations are needed.

Read-only Beans

Entity beans can be marked as read-only. When a bean is marked as read-only, it never takes part in a transaction. This means that it is never transactionally locked. Using commit option D with this option is sometimes very useful when a read-only bean's data is sometimes updated by an external source.

To mark a bean as read-only, you use the read-only flag in the jboss.xml deployment descriptor, as shown in Listing 5.11.

Listing 5.11. Marking an Entity Bean as Read-only by Using `jboss.xml`

 <jboss>     <enterprise-beans>         <entity>             <ejb-name>MyEntityBean</ejb-name>             <jndi-name>MyEntityHomeRemote</jndi-name>             <read-only>True</read-only>         </entity>     </enterprise-beans> </jboss>

Explicitly Defining Read-only Methods

Now that you understand the default locking behavior of entity beans, you're probably wondering, "Why lock the bean if it's not modifying the data?" JBoss allows you to define what methods on an entity bean are read-only so that it will not lock the bean within the transaction if only these types of methods are called. You can define these read-only methods within a jboss.xml deployment descriptor. Wildcards are allowed for method names. Listing 5.12 shows an example of declaring all getter methods and the anotherReadOnlyMethod as read-only.

Listing 5.12. Defining Entity Bean Methods as Read-only

 <jboss>     <enterprise-beans>         <entity>             <ejb-name>nextgen.EnterpriseEntity</ejb-name>             <jndi-name>nextgen.EnterpriseEntity</jndi-name>             <method-attributes>                 <method>                     <method-name>get*</method-name>                     <read-only>true</read-only>                 </method>                 <method>                     <method-name>anotherReadOnlyMethod</method-name>                     <read-only>true</read-only>                 </method>             </method-attributes>         </entity>     </enterprise-beans> </jboss>

The Instance per Transaction Policy

The Instance per Transaction policy is an advanced configuration that can totally wipe away deadlock and throughput problems caused by JBoss's default locking policy. The default entity bean locking policy is to allow only one active instance of a bean. The Instance per Transaction policy breaks this requirement by allocating a new instance of a bean per transaction and dropping that instance at the end of the transaction. Because each transaction has its own copy of the bean, there is no need for transaction-based locking.

This option does sound great, but it has some drawbacks right now. First, the transactional isolation behavior of this option is equivalent to READ_COMMITTED. This can create repeatable reads when they are not desired. In other words, a transaction could have a copy of a stale bean. Second, this configuration option currently requires commit option B or C, which can be a performance drain because an ejbLoad must happen at the beginning of the transaction. But, if an application currently requires commit option B or C anyway, then this is the way to go. The JBoss developers are currently exploring ways to allow commit option A as well (which would allow the use of caching for this option).

JBoss has container configurations named Instance Per Transaction CMP 2.x EntityBean and Instance Per Transaction BMP EntityBean defined in the standardjboss.xml file that implements this locking policy. To use this configuration, you just have to reference the name of the container configuration to use with your bean in the jboss.xml deployment descriptor, as shown in Listing 5.13.

Listing 5.13. An Example of Using the Instance per Transaction Policy

 <jboss>     <enterprise-beans>         <entity>             <ejb-name>MyCMP2Bean</ejb-name>             <jndi-name>MyCMP2</jndi-name>             <configuration-name>                 Instance Per Transaction CMP 2.x EntityBean             </configuration-name>         </entity>         <entity>             <ejb-name>MyBMPBean</ejb-name>             <jndi-name>MyBMP</jndi-name>             <configuration-name>                 Instance Per Transaction BMP EntityBean             </configuration-name>         </entity>     </enterprise-beans> </jboss>

Running Within a Cluster

Currently, there is no distributed locking capability for entity beans within a cluster. This functionality has been delegated to the database and must be supported by the application developer. For clustered entity beans, it is suggested to use commit option B or C in combination with a row-locking mechanism. For CMP, there is a row-locking configuration option. This option uses a SQL select for update when the bean is loaded from the database. With commit option B or C, this implements a transactional lock that can be used across the cluster. For BMP, you must explicitly implement the select for update invocation within the BMP's ejbLoad method.

Troubleshooting

The following sections describe some common locking problems and their solutions.

Locking Behavior Not Working

Many JBoss users observe that locking does not seem to be working and see concurrent access to their beansand thus dirty reads. Here are some common solutions for this:

If you have custom container-configurations, make sure you have updated these configurations.
Make absolutely sure that you have implemented equals and hashCode correctly from custom/complex primary key classes.
Make absolutely sure that your custom/complex primary key classes serialize correctly. One common mistake is to assume that member variable initializations will be executed when a primary key is unmarshaled.

IllegalStateException

An IllegalStateException with the message "removing bean lock and it has tx set!" usually means that you have not implemented equals and/or hashCode correctly for your custom/complex primary key class; it may also mean that your primary key class is not implemented correctly for serialization.

Hangs and Transaction Timeouts

One long outstanding bug of JBoss is that on a transaction timeout, that transaction is only marked for a rollback and not actually rolled back. This responsibility is delegated to the invocation thread. This can cause major problems if the invocation thread hangs indefinitely because things like entity bean locks will never be released. The solution to this problem is not a good one: You really just need to avoid doing anything within a transaction that could hang indefinitely. Common mistakes are making connections across the Internet and running a web crawler within a transaction.