Providing IMAP Access to Mailboxes

IMAP is a powerful and full-featured protocol for mail management. But its many features are both a blessing and a curse. IMAP is a great protocol to use: it provides all the necessary tools to store, organize, and mail on a central server. But IMAP's complexity makes it an intimidating protocol to implement: the base IMAP protocol spec in RFC 2060 runs to more than 80 dense pages. IMAP also puts a lot of burden on the mail server. IMAP servers have to support message parsing (to allow the client to download only selected parts of a message), message searching (using a special query language), and the ability to reference messages using two completely different sets of numeric identifiers (message sequence numbers and unique IDs).

This complexity has historically caused all but the most dedicated developers to steer clear of IMAP, settling for the simpler but less capable POP3. So the fact that twisted.mail includes IMAP support is a great opportunity for developers. It makes it possible for you to write your own custom IMAP server, but without having to deal with all the nasty details of the IMAP protocol.

8.4.1. How Do I Do That?

To make an IMAP server, write classes to implement the IAccount, IMailbox, IMessage, and IMessagePart interfaces defined in twisted.mail.imap4. Then set up a realm that makes your IAccount implementation available as an avatar. Wrap the realm in a Portal, and set up a Factory that will pass the Portal to an imap4.IMAP4Server Protocol. Example 8-4 demonstrates a complete IMAP server.

Example 8-4. imapserver.py

from twisted.mail import imap4, maildir

from twisted.internet import reactor, defer, protocol

from twisted.cred import portal, checkers, credentials

from twisted.cred import error as credError

from twisted.python import filepath

from zope.interface import implements

import time, os, random, pickle



MAILBOXDELIMITER = "."



class IMAPUserAccount(object):

 implements(imap4.IAccount)



 def __init__(self, userDir):

 self.dir = userDir

 self.mailboxCache = {}

 # make sure Inbox exists

 inbox = self._getMailbox("Inbox", create=True)



 def listMailboxes(self, ref, wildcard):

 for box in os.listdir(self.dir):

 yield box, self._getMailbox(box)



 def select(self, path, rw=True):

 "return an object implementing IMailbox for the given path"

 return self._getMailbox(path)



 def _getMailbox(self, path, create=False):

 """

 Helper function to get a mailbox object at the given

 path, optionally creating it if it doesn't already exist.

 """

 # According to the IMAP spec, Inbox is case-insensitive

 pathParts = path.split(MAILBOXDELIMITER)

 if pathParts[0].lower() == 'inbox': pathParts[0] = 'Inbox'

 path = MAILBOXDELIMITER.join(pathParts)



 if not self.mailboxCache.has_key(path):

 fullPath = os.path.join(self.dir, path)

 if not os.path.exists(fullPath):

 if create:

 maildir.initializeMaildir(fullPath)

 else:

 raise KeyError, "No such mailbox"

 self.mailboxCache[path] = IMAPMailbox(fullPath)

 return self.mailboxCache[path]



 def create(self, path):

 "create a mailbox at path and return it"

 self._getMailbox(path, create=True)



 def delete(self, path):

 "delete the mailbox at path"

 raise imap4.MailboxException("Permission denied.")



 def rename(self, oldname, newname):

 "rename a mailbox"

 oldPath = os.path.join(self.dir, oldname)

 newPath = os.path.join(self.dir, newname)

 os.rename(oldPath, newPath)



 def isSubscribed(self, path):

 "return a true value if user is subscribed to the mailbox"

 return self._getMailbox(path).metadata.get('subscribed', False)



 def subscribe(self, path):

 "mark a mailbox as subscribed"

 box = self._getMailbox(path)

 box.metadata['subscribed'] = True

 box.saveMetadata()

 return True



 def unsubscribe(self, path):

 "mark a mailbox as unsubscribed"

 box = self._getMailbox(path)

 box.metadata['subscribed'] = False

 box.saveMetadata()

 return True



class ExtendedMaildir(maildir.MaildirMailbox):

 """

 Extends maildir.MaildirMailbox to expose more

 of the underlying filename data

 """

 def __iter__(self):

 "iterates through the full paths of all messages in the maildir"

 return iter(self.list)



 def __len__(self):

 return len(self.list)



 def __getitem__(self, i):

 return self.list[i]



 def deleteMessage(self, filename):

 index = self.list.index(filename)

 os.remove(filename)

 del(self.list[index])



class IMAPMailbox(object):

 implements(imap4.IMailbox)



 def __init__(self, path):

 self.maildir = ExtendedMaildir(path)

 self.metadataFile = os.path.join(path, '.imap-metadata.pickle')

 if os.path.exists(self.metadataFile):

 self.metadata = pickle.load(file(self.metadataFile, 'r+b'))

 else:

 self.metadata = {}

 self.initMetadata()

 self.listeners = []

 self._assignUIDs()



 def initMetadata(self):

 if not self.metadata.has_key('flags'):

 self.metadata['flags'] = {} # dict of message IDs to flags

 if not self.metadata.has_key('uidvalidity'):

 # create a unique integer ID to identify this version of

 # the mailbox, so the client could tell if it was deleted

 # and replaced by a different mailbox with the same name

 self.metadata['uidvalidity'] = random.randint(1000000, 9999999)

 if not self.metadata.has_key('uids'):

 self.metadata['uids'] = {}

 if not self.metadata.has_key('uidnext'):

 self.metadata['uidnext'] = 1 # next UID to be assigned



 def saveMetadata(self):

 pickle.dump(self.metadata, file(self.metadataFile, 'w+b'))



 def _assignUIDs(self):

 # make sure every message has a uid

 for messagePath in self.maildir:

 messageFile = os.path.basename(messagePath)

 if not self.metadata['uids'].has_key(messageFile):

 self.metadata['uids'][messageFile] =

self.metadata['uidnext']

 self.metadata['uidnext'] += 1

 self.saveMetadata()



 def getHierarchicalDelimiter(self):

 return MAILBOXDELIMITER



 def getFlags(self):

 "return list of flags supported by this mailbox"

 return [r'Seen', r'Unseen', r'Deleted',

 r'Flagged', r'Answered', r'Recent']



 def getMessageCount(self):

 return len(self.maildir)



 def getRecentCount(self):

 return 0



 def getUnseenCount(self):

 def messageIsUnseen(filename):

 filename = os.path.basename(filename)

 uid = self.metadata['uids'].get(filename)

 flags = self.metadata['flags'].get(uid, [])

 if not r'Seen' in flags:

 return True

 return len(filter(messageIsUnseen, self.maildir))



 def isWriteable(self):

 return True



 def getUIDValidity(self):

 return self.metadata['uidvalidity']



 def getUID(self, messageNum):

 filename = os.path.basename(self.maildir[messageNum-1])

 return self.metadata['uids'][filename]



 def getUIDNext(self):

 return self.folder.metadata['uidnext']



 def _uidMessageSetToSeqDict(self, messageSet):

 """

 take a MessageSet object containing UIDs, and return

 a dictionary mapping sequence numbers to filenames

 """

 # if messageSet.last is None, it means 'the end', and needs to

 # be set to a sane high number before attempting to iterate

 # through the MessageSet

 if not messageSet.last:

 messageSet.last = self.metadata['uidnext']

 allUIDs = []

 for filename in self.maildir:

 shortFilename = os.path.basename(filename)

 allUIDs.append(self.metadata['uids'][shortFilename])

 allUIDs.sort()

 seqMap = {}

 for uid in messageSet:

 # the message set covers a span of UIDs. not all of them

 # will necessarily exist, so check each one for validity

 if uid in allUIDs:

 sequence = allUIDs.index(uid)+1

 seqMap[sequence] = self.maildir[sequence-1]

 return seqMap



 def _seqMessageSetToSeqDict(self, messageSet):

 """

 take a MessageSet object containing message sequence numbers,

 and return a dictionary mapping sequence number to filenames

 """

 # if messageSet.last is None, it means 'the end', and needs to

 # be set to a sane high number before attempting to iterate

 # through the MessageSet

 if not messageSet.last: messageSet.last = len(self.maildir)-1

 seqMap = {}

 for messageNo in messageSet:

 seqMap[messageNo] = self.maildir[messageNo-1]

 return seqMap



 def fetch(self, messages, uid):

 if uid:

 messagesToFetch = self._uidMessageSetToSeqDict(messages)

 else:

 messagesToFetch = self._seqMessageSetToSeqDict(messages)

 for seq, filename in messagesToFetch.items():

 uid = self.getUID(seq)

 flags = self.metadata['flags'].get(uid, [])

 yield seq, MaildirMessage(file(filename).read(), uid, flags)



 def addListener(self, listener):

 self.listeners.append(listener)

 return True



 def removeListener(self, listener):

 self.listeners.remove(listener)

 return True



 def requestStatus(self, path):

 return imap4.statusRequestHelper(self, path)



 def addMessage(self, msg, flags=None, date=None):

 if flags is None: flags = []

 return self.maildir.appendMessage(msg).addCallback(

 self._addedMessage, flags)



 def _addedMessage(self, _, flags):

 # the first argument is the value returned from

 # MaildirMailbox.appendMessage. It doesn't contain any meaningful

 # information and can be discarded. Using the name "_" is a Twisted

 # idiom for unimportant return values.

 self._assignUIDs()

 messageFile = os.path.basename(self.maildir[-1])

 messageID = self.metadata['uids'][messageFile]

 self.metadata['flags'][messageID] = flags

 self.saveMetadata()



 def store(self, messageSet, flags, mode, uid):

 if uid:

 messages = self._uidMessageSetToSeqDict(messageSet)

 else:

 messages = self._seqMessageSetToSeqDict(messageSet)

 setFlags = {}

 for seq, filename in messages.items():

 uid = self.getUID(seq)

 if mode == 0: # replace flags

 messageFlags = self.metadata['flags'][uid] = flags

 else:

 messageFlags = self.metadata['flags'].setdefault(uid, [])

 for flag in flags:

 # mode 1 is append, mode -1 is delete

 if mode == 1 and not messageFlags.count(flag):

 messageFlags.append(flag)

 elif mode == -1 and messageFlags.count(flag):

 messageFlags.remove(flag)

 setFlags[seq] = messageFlags

 self.saveMetadata()

 return setFlags



 def expunge(self):

 "remove all messages marked for deletion"

 removed = []

 for filename in self.maildir:

 uid = self.metadata['uids'].get(os.path.basename(filename))

 if r"Deleted" in self.metadata['flags'].get(uid, []):

 self.maildir.deleteMessage(filename)

 # you could also throw away the metadata here

 removed.append(uid)

 return removed



 def destroy(self):

 "complete remove the mailbox and all its contents"

 raise imap4.MailboxException("Permission denied.")



from cStringIO import StringIO

import email



class MaildirMessagePart(object):

 implements(imap4.IMessagePart)



 def __init__(self, mimeMessage):

 self.message = mimeMessage

 self.data = str(self.message)



 def getHeaders(self, negate, *names):

 """

 Return a dict mapping header name to header value. If *names

 is empty, match all headers; if negate is true, return only

 headers _not_ listed in *names.

 """

 if not names: names = self.message.keys()

 headers = {}

 if negate:

 for header in self.message.keys():

 if header.upper() not in names:

 headers[header.lower()] = self.message.get(header, '')

 else:

 for name in names:

 headers[name.lower()] = self.message.get(name, '')

 return headers



 def getBodyFile(self):

 "return a file-like object containing this message's body"

 bodyData = str(self.message.get_payload())

 return StringIO(bodyData)



 def getSize(self):

 return len(self.data)



 def getInternalDate(self):

 return self.message.get('Date', '')



 def isMultipart(self):

 return self.message.is_multipart()



 def getSubPart(self, partNo):

 return MaildirMessagePart(self.message.get_payload(partNo))



class MaildirMessage(MaildirMessagePart):

 implements(imap4.IMessage)



 def __init__(self, messageData, uid, flags):

 self.data = messageData

 self.message = email.message_from_string(self.data)

 self.uid = uid

 self.flags = flags



 def getUID(self):

 return self.uid



 def getFlags(self):

 return self.flags



class MailUserRealm(object):

 implements(portal.IRealm)

 avatarInterfaces = {

 imap4.IAccount: IMAPUserAccount,

 }



 def __init__(self, baseDir):

 self.baseDir = baseDir



 def requestAvatar(self, avatarId, mind, *interfaces):

 for requestedInterface in interfaces:

 if self.avatarInterfaces.has_key(requestedInterface):

 # make sure the user dir exists (avatarId is username)

 userDir = os.path.join(self.baseDir, avatarId)

 if not os.path.exists(userDir):

 os.mkdir(userDir)

 # return an instance of the correct class

 avatarClass = self.avatarInterfaces[requestedInterface]

 avatar = avatarClass(userDir)

 # null logout function: take no arguments and do nothing

 logout = lambda: None

 return defer.succeed((requestedInterface, avatar, logout))



 # none of the requested interfaces was supported

 raise KeyError("None of the requested interfaces is supported")



def passwordFileToDict(filename):

 passwords = {}

 for line in file(filename):

 if line and line.count(':'):

 username, password = line.strip().split(':')

 passwords[username] = password

 return passwords



class CredentialsChecker(object):

 implements(checkers.ICredentialsChecker)

 credentialInterfaces = (credentials.IUsernamePassword,

 credentials.IUsernameHashedPassword)



 def __init__(self, passwords):

 "passwords: a dict-like object mapping usernames to passwords"

 self.passwords = passwords



 def requestAvatarId(self, credentials):

 """

 check to see if the supplied credentials authenticate.

 if so, return an 'avatar id', in this case the name of

 the IMAP user.

 The supplied credentials will implement one of the classes

 in self.credentialInterfaces. In this case both

 IUsernamePassword and IUsernameHashedPassword have a

 checkPassword method that takes the real password and checks

 it against the supplied password.

 """

 username = credentials.username

 if self.passwords.has_key(username):

 realPassword = self.passwords[username]

 checking = defer.maybeDeferred(

 credentials.checkPassword, realPassword)

 # pass result of checkPassword, and the username that was

 # being authenticated, to self._checkedPassword

 checking.addCallback(self._checkedPassword, username)

 return checking

 else:

 raise credError.UnauthorizedLogin("No such user")



 def _checkedPassword(self, matched, username):

 if matched:

 # password was correct

 return username

 else:

 raise credError.UnauthorizedLogin("Bad password")



class IMAPServerProtocol(imap4.IMAP4Server):

 "Subclass of imap4.IMAP4Server that adds debugging."

 debug = True



 def lineReceived(self, line):

 if self.debug:

 print "CLIENT:", line

 imap4.IMAP4Server.lineReceived(self, line)



 def sendLine(self, line):

 imap4.IMAP4Server.sendLine(self, line)

 if self.debug:

 print "SERVER:", line



class IMAPFactory(protocol.Factory):

 protocol = IMAPServerProtocol

 portal = None # placeholder



 def buildProtocol(self, address):

 p = self.protocol()

 p.portal = self.portal

 p.factory = self

 return p



if __name__ == "__main__":

 import sys

 dataDir = sys.argv[1]



 portal = portal.Portal(MailUserRealm(dataDir))

 passwordFile = os.path.join(dataDir, 'passwords.txt')

 passwords = passwordFileToDict(passwordFile)

 passwordChecker = CredentialsChecker(passwords)

 portal.registerChecker(passwordChecker)



 factory = IMAPFactory()

 factory.portal = portal



 reactor.listenTCP(143, factory)

 reactor.run()

Run imapserver.py from the command line with the name of the base mail directory as the only argument. This should be the same directory you used for the SMTP and POP3 servers in Examples 8-1 and 8-3:

 $ python imapserver.py mail_storage

Once the server is running, set up your mail client to connect to localhost using IMAP. You should be able to see the Inbox folder, create folders and subfolders, subscribe to folders, view messages, mark messages as read or unread, and move messages between folders, as shown in Figure 8-6.

8.4.2. How Does That Work?

Compared to the other examples in this chapter, there's a lot of code required to make an IMAP server. But don't let that intimidate you. Most of the code in Example 8-4 is in the classes IMAPUserAccount, IMAPMaildir, MaildirMessagePart, and MaildirMessage, which respectively implement the interfaces imap4.IAccount, imap4.IMailbox, imap4.IMessagePart, and imap4.IMessage. These interfaces have a lot of methods, because the IMAP server needs to be able to do a lot of different things. However, most of the methods themselves are pretty simple, taking just a few lines of code. The following subsections go through the interfaces one at a time, to look at how they're used in Example 8-4.

Figure 8-6. Working with messages on the IMAP server

 

8.4.2.1. IAccount

The imap4.IAccount interface defines a user account, and provides access to the user's mailboxes . imap4.IAccount defines methods for listing, creating, deleting, renaming, and subscribing to mailboxes. Mailboxes are hierarchal, with a server-defined delimiter character. In Example 8-4, the delimiter character is a period, so the folder MailingLists.Twisted would be considered a subfolder of MailingLists. In this case, the user's mailboxes are a set of maildir directories kept within a single parent directory. The use of a period as a delimiter makes it easier to keep all the maildirs in a single flat directory structure while still keeping track of their hierarchy. You could feel free to use another delmiter character, such as a forward slash (/), if it were more convenient for your needs.

According to RFC 2060, each IMAP user will have a folder called Inbox available in her account at all times. The name Inbox is case-insensitive, however, so different mail clients may ask for INBOX, Inbox, or inbox. Make sure you account for this in your code.

The select and create methods of IAccount return objects implementing IMailbox. The list method returns an iterable of tuples, with each tuple containing a path and an object implementing IMailbox.

8.4.2.2. IMailbox

The imap4.IMailbox interface represents a single mailbox on the IMAP server. It has methods for getting information about the mailbox, reading and writing messages, and storing metadata. There are a couple of considerations to keep in mind when you write a class to implement IMailbox. First, note that an IMAP mailbox is more than just a collection of messages. It also is responsible for managing metadata about those messages and the mailbox itself. In Example 8-4, the IMAPMaildir class keeps metadata in a dictionary, which is pickled and stored in a hidden file in the maildir directory for persistence between sessions. These are some of the specific kinds of metadata you'll need to track:

 

Message UIDs

Every message in an IMAP mailbox has a permanent unique identifier. These are sequential, and are maintained over the life of the mailbox. The first ever message in a mailbox will have a UID of 1, the second a UID of 2, etc. These are different from simple sequence numbers in that they continue to build over the lifetime of a mailbox. For example, if you added 1000 messages to a new IMAP mailbox, they would be assigned the UIDs 11001. If you then deleted all those messages and added a single new message, it would be given a UID of 1002, even though there wouldn't be any other messages in the mailbox. UIDs are used by the client for caching purposes, to avoid repeatedly downloading the same message. In IMailbox, the getUID function returns the UID of a message, given its sequence number. The getUIDNext function returns the UID number that will be assigned to the next message added to the mailbox.

 

Message flags

Every message in the mailbox can have an arbitrary number of flags associated with it. The flags are set by the client to keep track of metadata about that message, such as whether it's been read, replied to, or flagged. RFC 2060 defines a set of system flags that all IMAP clients should use the same way: Seen, Answered, Flagged, Deleted, Draft, and Recent. The IMailbox getFlags methods returns the list of flags supported by the mailbox. The store method sets the flags on a group of messages.

 

The mailbox UID validity identifier

There can be times when a mailbox keeps the same name, but changes its list of UIDs. The most common example of this is when a mailbox is deleted and a new mailbox with the same name is created. This could cause confusion for a mail client, since it may have a cached copy of the message with UID 1, which is now different from the message on the server with UID 1. To prevent this, IMAP has the concept of a UID validity identifier. Each mailbox has a unique number associated with it. As long as this number remains the same, the client can be confident that its UID numbers are still valid. If this number changes, the client knows to forget all its cached UID information. You should assign a UID validity identifier number for each mailbox, and return it as the result of getUIDValidity.

 

The mailbox subscription status

IMAP clients may not want or need to display all the mailboxes that actually exist on the server. The client can tell the server which mailboxes it is interested in by subscribing to those mailboxes. Subscribing doesn't do anything special other than toggling a subscribed bit on the mailbox. The subscription status of each mailbox is stored on the server, so that it can be maintained from session to session and between different clients.

The second thing to be aware of when implementing IMailbox is that there are two different numbering schemes that clients may use to refer to messages . The first is UIDs, persistent unique identifiers. The second is sequence numbers, which are a simple list of numbers from 1 to the number of messages in the mailbox. The fetch and store methods of IMailbox take an imap4.MessageSet object along with a Boolean uid argument. The MessageSet is an iterable list of numbers, but you have to check the uid argument to determine whether it's a list of UIDs or a list of sequence numbers. The IMAPMaildir class in Example 8-4 uses two methods, _uidMessageSetToSeqDict and _seqMessageSetToSeqDict, to take either type of MessageSet and normalize it into a dictionary mapping sequence numbers to the filenames it uses internally to identify messages. Note that the lists returned by fetch and store always use sequence numbers, whether or not the uid argument was true.

The last property of a MessageSet may be set to None, which means that the MessageSet covers all messages from its start to the end of the mailbox. You can't actually iterate through a MessageSet in this state, though, so you should set last to a value just beyond the last message in the mailbox before you attempt to use the MessageSet.

The IMAP protocol includes support for server-to-client notification . A client will generally keep an open connection to the IMAP server, so rather than having the client periodically ask the server if there are any new messages, the server can send a message to the client to alert it as soon as messages have arrived. The IMailbox interface includes methods for managing these notifications. The addListener method registers an object implementing imap4.IMailboxListener. If you keep track of this object, you can use it later to notify the client when a new message arrives. The IMAPMaildir class in Example 8-4 keeps track of listeners, but doesn't actually use them. Example 8-5 shows how you might use listeners to notify the client of new messages.

Example 8-5. Using listener objects for notification

# run this code to let the client know when a new message

# appears in the mailbox

for listener in self.listeners:

 listener.newMessages(self.getMessageCount( ), self.getRecentCount( ))

 

8.4.3. IMessagePart and IMessage

In IMAP, an email message is not just a blob of data: it has a structure that can be accessed one part at a time. In its simplest form, a message is a group of headers and a body. More complex messages can have a multi-part body, with alternative versions of the body content (such as HTML and plain text) and attachments (which may themselves be other messages with their own structure).

The IMessagePart interface provides methods for investigating the structure of one part of a multi-part message. IMessage inherits from IMessagePart and adds the getUID and getFlags methods for retrieving metadata about the message. Implementing IMessagePart would be a chore if not for the excellent email module in the Python standard library, which contains a complete set of classes for parsing and working with email messages. In Example 8-4, the MaildirMessage class uses the email.message_from_string function to parse the message data into an email.Message.Message object. MaildirMessagePart takes an email.Message.Message object and wraps it in the IMessagePart interface.

8.4.4. Putting It All Together

Once you've implented IAccount, IMailbox, IMessagePart, and IMessage, just tie the pieces together to get a working IMAP server . Example 8-4 uses the same classes, MailUserRealm and CredentialsChecker, as the POP3 server in Example 8-3, except that the realm is set up to return IMAPMaildir avatars when the imap4.IMailbox interface is requested. The IMAPFactory object creates IMAPServerProtocol objects and sets their portal attribute to a Portal wrapping the realm and credentials checkers. The IMAPServerProtocol class is another example of how you can inherit from a Twisted Protocol class and add some print statements for debugging.