10.3 Email for System Integration

I l @ ve RuBoard

Most developers are familiar with controlling remote applications via email. Almost anyone who has ever subscribed to a mailing list has emailed a subscription request, a status change, or an unsubscribe to [e-mail] , and received a response indicating that action was taken. More recently, emails to customer service addresses at major companies are read first by software, which attempts to match them to a set of prewritten FAQs, and then are passed to a human only if no relevant FAQ can be found. Even then, the human customer service employees might handle the request via a web interface that serves up each incoming message, allows them to enter a response, and dispatches the result back to the customer.

10.3.1 Consider Email as an Enterprise Bridge

Email can provide an inexpensive, easy-to-implement linkage between multiple applications.

Most email exchanges expect that a human will be on at least one side of the conversion. In a J2EE setting, this is still often true, but not always. The BugZilla open source bug-tracking system, for instance, implements synchronization between installation by sending emails containing XML representations of the data being synchronized. Each instance periodically checks an email address for messages from the other system and uses the contents to update its own database. This implementation allows synchronization between bug databases running on different networks, across firewalls and VPNs, and without system availability concerns because the messages are queued.

This strategy is most effective on internal networks that are already secured, or when the security requirements of the application are minimal. However, when coupled with the security practices discussed later in this chapter, this approach can be used safely and effectively in a wide variety of contexts.

When implementing systems that receive email, consider existing options before creating your own. The James Mail Server, from the Apache Software Foundation, is a pure Java SMTP, IMAP, and POP mail server. James is built around the JavaMail APIs and introduces the concept of a "maillet," which is somewhat like a Java servlet but processes incoming email instead of incoming HTTP requests . If you're considering building advanced mail-handling support into an application, James is worth considering, as it provides mail transport and storage services along with a framework for processing incoming messages.

10.3.2 Retrieve Incoming Mail Efficiently

Inefficient access to email message stores can result in performance degradation.

If you're using messaging middleware, J2EE provides a convenient way to receive incoming messages via message-driven EJBs. The J2EE container will deliver messages to EJBs as they arrive from the external messaging server. While some message queue vendors might have "email bridge" support built into their products, there is no specific J2EE mechanism for receiving messages.

In most cases, including whenever a POP3 message store is involved, you have to explicitly check for mail, and J2EE applications have a few methods for doing so. The first is to check in response to some user activity, either in an EJB session bean or a Java servlet. In most cases, this is not a terribly useful thing to do. Users will not respond well to a three- or four-second (or potentially much longer) wait while the system takes care of housekeeping that is not related to the task at hand.

Instead of holding users hostage to the mail component, most applications deal with mail retrieval by launching one or more background worker threads. This generally has to be done at the application's controller layer, usually by launching a thread from a servlet's initialization method or when the servlet context is being initialized . Remember that EJBs are explicitly forbidden from creating or maintaining threads of any kind. For more information, see Section 10.3.7 later in this chapter.

JavaMail supports a listener framework for inboxes, which is discussed in the next section.

A final option involves a standalone application running in its own Java Virtual Machine (JVM), periodically waking up to check mail and then transferring the contents of the incoming messages to another system for processing (such as connecting to an EJB session bean or a simple object access protocol [SOAP] service). You'll see how to quickly ship JavaMail messages off to SOAP-based web services later in this chapter.

10.3.3 Use Inbox Listeners Carefully

JavaMail allows you to assign a listener to a message store, assuming that the server supports adding new messages while a user is connected. IMAP does, but POP3 does not. To be notified when a new message arrives, create a new class implementing MessageCountListener and associate it with a Folder object. The messagesAdded( ) method will be called whenever messages are added to the folder.

In most real-life situations this isn't a great way to go. It requires keeping the connection to the server open, which consumes network resources. The mail server might decide to time you out, or a network problem could knock out the connection. Some servers place a limit on the amount of time one user can be connected. So if you take this approach, implement the ConnectionListener interface and associate it with the folder and message store objects. This will allow you to recreate the connection and folder and assign a new listener.

10.3.4 Choose an Effective Delivery Mechanism

Because JavaMail doesn't provide any provision for having mail delivered to an application, there is an architectural decision to be made regarding how incoming messages will be retrieved. In practice, this comes down to deciding whether you want to retrieve messages from an IMAP mailbox or a POP3 mailbox. In some environments this decision will be made for you, but when possible, it's helpful to evaluate the two options.

POP3 has the advantage of simplicity. It's almost impossible to find a mail server that doesn't support POP, and implementations are very mature. The protocol itself is older and is geared toward listing, retrieving, and deleting messages from a single, undifferentiated pool. For simple process flows, such as when one mailbox is dedicated to a particular application and the application deletes messages after retrieving them, POP is generally the best choice.

POP falls short when dealing with complex message stores. It has no support for folders, no query system, and minimal support for downloading parts of messages. This is where IMAP comes in. Like POP, IMAP allows clients to connect to remote message stores. However, while POP assumes that the client will download messages and store them locally, IMAP assumes that messages will be stored on the server and provides support for multiple mailbox folders to aid in organization. More advanced IMAP implementations can provide a degree of write access to the contents of the message store, as well as server-side queries to find particular messages.

IMAP has been around long enough that it is almost as ubiquitous as POP, and it can offer some real advantages when building email-enabled applications. Mail for multiple applications (or multiple types of messages for a single application) can be directed to IMAP folders via rules on the server, allowing the JavaMail application to read particular folders rather than scan every message. Folders can also be used to store messages after they've been dealt with by an application.

10.3.5 Beware Email in Transactions

Every interface between two applications creates a potential transactional nightmare. Introducing email collection into an application introduces the need to maintain an external resource, i.e., the incoming mailbox. Needless to say, the transaction support in J2EE was not intended to handle this kind of activity. If an EJB session bean uses JavaMail to send email or delete a message from a mailbox, rolling the transaction back will not unsend or restore the messages.

Commercial enterprise messaging systems sometimes have features that allow the sender of a message to control the message after it has been sent. Standard email does not have this capability, so once a message has left the flow of execution it cannot be recalled.

When sending messages, try to hold off until the very end of a transaction, and try to design your transactions so that user input, particularly if provided via email that might arrive at an indefinite time in the future, comes at a break between transactions. If the transaction must send an email to a live user early in the process, it should send the user another message if the transaction is rolled back.

10.3.6 Cut and Run

The best practice for incorporating incoming email into your applications is to get the message out of the email layer as quickly as possible and into a transaction that can be controlled more tightly. The mailbox should never be used as a receptacle for data related to an ongoing process. Instead, the mail should be read out as quickly as possible and loaded into another queue. A database is an obvious choice here.

When retrieving messages, log in, do your business, and log out as quickly as possible. Staying connected and periodically polling for new messages consumes network resources, which is reason enough to try and avoid doing this, but this also makes it much more difficult to write code that will properly handle dropped connections to the mail server.

Connecting to most well-written servers will obtain a lock on the mailbox. If there is any possibility at all that more than one thread (or more than one application) will access a mailbox at the same time, each thread or application should be aware of the possibility of conflict, and should fail gracefully when this happens. For the same reason, applications should be sure to close connections as soon as possible to free up access to the resource. If this can be avoided, so much the better.

Needless to say, information about the state of a mailbox should be considered out-of-date as soon as the connection to the mailbox is closed. This is particularly true of the message IDs provided by the message store, which are not constant or unique beyond the context of the current connection.

10.3.7 Use an Email Management Framework

Single applications should have as few points of contact with the external email layer as possible. This allows the centralization of logic to deal with concurrency issues ”only one thread ever accesses the mailbox. To help you out with this, I present a basic framework for handling incoming messages. The framework is based on the Strategy design pattern, which allows you to plug in different kinds of processing behavior. ^[4] Having this framework will allow you to easily extend your applications to handle different kinds of messages by creating new instances of a monitoring thread pointed at different mailboxes and passing in handler objects that know how to deal with particular message types.

^[4] For more information on this and other design patterns for software engineering, and on design patterns themselves , check out Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides (Addison-Wesley).

Start by declaring an interface that can be implemented by classes that will be able to handle incoming messages. I will keep it simple for this example, so you will define just one kind of interface, a BlockingMailHandler , which handles a particular message and returns when processing for that message has completed. (See Example 10-4.)

Example 10-4. BlockingMailHandler

 public interface BlockingMailHandler  {   public boolean handleMessage(javax.mail.Message message); } 

A more complete implementation would result in a BlockingMailHandler from a MailHandler , and probably define an AsynchronousMailHandler as well.

To handle incoming messages, you create a class that implements the BlockingMailHandler interface and does something interesting in the handleMessage( ) method. The handler implementation is responsible for any thread synchronization issues. If the ultimate target of an incoming message (or the information it contains) is a Java object, that object generally shouldn't implement the BlockingMailHandler interface directly. Instead, create a new class that acts as an intermediary for the target object. If the target is an EJB, you don't have any choice in the matter ”you must implement a wrapper class.

Now, you'll define a MailMonitor thread class, which takes a BlockingMailHandler and feeds it incoming messages (see Example 10-5). You can construct a new MailMonitor when initializing a servlet context, or launch it from a standalone application. Each MailMonitor thread will wake up every 30 seconds and retrieve all the messages in a mailbox. Each message is processed by the handler, which returns true or false to indicate whether the message should be deleted.

Example 10-5. MailMonitor

 import javax.mail.*; import javax.mail.internet.*; import java.io.*;  . . .  public class MailMonitor extends Thread {   boolean interrupted = false;   BlockingMailHandler handler = null;      public MailMonitor(BlockingMailHandler mh) {     handler = mh;   }    . . .    public void stopRunning(  ) {     interrupted = true;   }      public void run(  ) {     Session session = Session.getDefaultInstance(System.getProperties(  ));     Store store = null;     try {       store = session.getStore("imap");     } catch (NoSuchProviderException nspe) {       nspe.printStackTrace(  );       return;     }      . . .      while(!interrupted) {       try {         if (!store.isConnected(  )) // Should always be true           store.connect("mail.invantage.com", -1, "carl", "carl@mit");                    Folder folder = store.getDefaultFolder(  );         folder = folder.getFolder("INBOX");         if (folder =  = null) {           System.out.println("Unable to open INBOX");           return;         }          . . .          // Try to open read/write. Open read-only if that fails.         try {           folder.open(Folder.READ_WRITE);         } catch (MessagingException ex) {           folder.open(Folder.READ_ONLY);         }          . . .          int totalMessages = folder.getMessageCount(  );             try {           Message messages[  ] = null;           messages = folder.getMessages(1, totalMessages);            . . .            for (int i = 0, n = messages.length; i < n; i++) {             boolean mbDelete = handler.handleMessage(messages[i]);                    // Delete the message.             if (mbDelete) {               messages[i].setFlag(Flags.Flag.DELETED, true);             }           } // End for         } catch (MessagingException e) {         System.out.println("Unable to get Messages");         }          . . .          // Close the folder and message store.         folder.close(true); // Purge deleted messages         store.close(  );          . . .        } catch (MessagingException e) {         System.out.println(e.toString(  ));         return;       }               try {         this.sleep(30 * 1000);       } catch (InterruptedException e) {         System.out.println("Exiting");         return;       }     }   } }  . . . 

Enhancements to this framework for a production environment are left as an exercise. Beyond additional configuration options, it would be helpful to support multiple handlers so that one mailbox can receive more than one type of message (remember, it's unsafe to have more than one application routinely accessing the same mailbox). A more complete implementation is available on this book's web site (http://www.oreilly.com/catalog/javaebp.

One major concern when implementing a mail monitor in a production system is what do to about messages that aren't recognized. Your implementation deletes only messages that are properly handled. You could set the standard that a handler should delete a message if it doesn't recognize it, but that solution is unnecessarily draconian, particularly if you want to implement the multiple handlers feature. Another option is to implement a handler that deletes unknown messages, but then that handler must be kept up to date on the format for all valid messages.

Ultimately, there's no substitute for human review. With IMAP, you can move messages to other folders after processing, where they can be reviewed by software or by users. A similar approach can be used to handle messages that were not recognized by the system .

10.3.8 Incorporate Security

Remember how, back in the introduction, we went over some areas where Internet email had a few problems? There really isn't much that we can do about lack of support for message routing and quality-of-service guarantees , short of switching over to a heavier-grade messaging system. And for most application-to-application communications problems, that's exactly what you should do.

You can do something about the security problem, however. Most of the intercomponent communications we've discussed in this chapter have been via XML. The W3C has defined standards for using digital signatures both to sign and encrypt XML documents. Provided that your digital signatures are managed in a secure manner, signing documents allows you to prevent other systems or malicious users from providing XML input for your email-enabled applications. Encrypting the XML will prevent its contents from being revealed in transit, even if the message has to leave your private network and cross over onto the Internet.

Signing and encrypting an XML file produces another XML file, which can be attached to a mail message just like any other file. There is no standard Java API for using digital signatures with XML, but several freely available Java tools support the standard, including the XML Security package from the Apache XML Project (http://xml.apache.org).

When a human is one end of an email conduit, the S/MIME standard allows digitally signed and encrypted emails that can be managed by an S/MIME-compliant mail reader. JavaMail does not natively support S/MIME, but several commercial toolkits are available that enable you to add S/MIME support to your applications fairly easily.

10.3.9 Use Secure Protocols

Many mail servers now support Secure Sockets Layer (SSL) encryption for particular protocols (including SMTP, POP, and IMAP). If your server supports SSL, your JavaMail application can use it as well, reducing the likelihood of third parties eavesdropping on your communications. To do this, you need to create a JavaMail Session object that uses the Java Secure Socket Extension (JSSE) system, which is included in Java Development Kit (JDK) 1.4 or is available as an add-on for earlier Java 2 implementations.

To use JSSE with a particular protocol, you need to install the JSSE provider according to the instructions included with the JSSE installation. You can then create an SSL-aware session by setting the following three properties within the Session objects.

Property	Value
mail. protocol socketFactory.class	javax.net.ssl.SSLSocketFactory
mail. protocol socketFactory.fallback	true if the connection should resort to non-SSL if the server does not support SSL; false otherwise
mail. protocol socketFactory.port	The remote port for the new socket factory

You will also have to set the default port for the protocol if it is not the standard port for the protocol.

If the remote server uses a server certificate not supported by the JSSE implementation, you will need to add the remote certificate to your local set of trusted root certificates. See the JSSE documentation for more information.

10.3.10 Use JAF for Bridging Content

Most JavaMail programmers have built up a passing familiarity with the JavaBeans Activation Framework (JAF). JAF was originally intended as a content-neutral mechanism for handling MIME-formatted data. For several years , JavaMail was the only API that used JAF explicitly. This changed with the introduction of SAAJ, a Java API for handling SOAP messages with attachments. SAAJ provides a standard interface for creating, sending, reading, and receiving messages based on SOAP Version 1.1 and the SOAP with Attachments extension. SOAP messages contain XML content and optional MIME-formatted file attachments, and are usually transported from one system to another via HTTP. For more on SOAP and Java, check out Dave Chappel and Tyler Jewell's Java Web Services (O'Reilly).

SAAJ also uses the JAF to handle file attachments. This makes it easy to write email-based frontends for SOAP services, or to transform SOAP messages, complete with attachments, into email. In fact, SMTP can even be used as the transport layer for SOAP, in lieu of HTTP.

For example, imagine a document cataloging system that has been provided with a web services interface. Clients can add documents to the catalog by providing a document ID and the document itself. The web service implements this by accepting a SOAP message containing the ID and header. You can add email support to this application by providing an email address users can send documents to. Your software will read the incoming messages and convert them to SOAP calls.

Example 10-6 extends your MailMonitor component by defining a handler that converts incoming email into SOAP messages. The SOAP call is kept simple: the handler just pulls the email subject and inserts it as a document ID.

Example 10-6. SOAPTransferMailHandler.java

 import javax.activation.*; import javax.xml.soap.*; import javax.xml.messaging.*; import javax.mail.*;  . . .  public class SOAPTransferMailHandler implements BlockingMailHandler {   URLEndpoint soapEndpoint = null;    . . .    SOAPTransferMailHandler(String epURL) {     soapEndpoint = new URLEndpoint(epURL);   }      public boolean handleMessage(javax.mail.Message message) {    . . .      try {       SOAPConnection con =          SOAPConnectionFactory.newInstance(  ).createConnection(  );                SOAPMessage soapMsg = MessageFactory.newInstance(  ).createMessage(  );        . . .        // Populate a basic SOAP header.       SOAPEnvelope envelope = soapMsg.getSOAPPart(  ).getEnvelope(  );       Name hName = envelope.createName("header", "ws",          "http://www.company.com/xmlns/soapservice");       SOAPHeader header = envelope.getHeader(  );       SOAPHeaderElement headerElement = header.addHeaderElement(hName);       SOAPElement docIdElem = headerElement.addChildElement("title", "ws");       docIdElem.addTextNode(message.getSubject(  ));        . . .        Object content = message.getContent(  );        . . .        if (content instanceof Multipart) {         Multipart multipart = (Multipart)content;          . . .      for (int i=0; i < multipart.getCount(  ); i++) {           Part part = multipart.getBodyPart(i);           if(Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition(  ))) {             DataHandler attachmentContent = part.getDataHandler(  );             AttachmentPart ap =                       soapMsg.createAttachmentPart(attachmentContent);             soapMsg.addAttachmentPart(ap);           }               }       }          con.call(soapMsg, soapEndpoint);          } catch (Exception e) {       e.printStackTrace(  );       return false;     }          return true;    } } 

More advanced systems might pull the XML SOAP body from the email message body. This approach can also be used to bridge web services through firewalls where email services are available.

I l @ ve RuBoard