Section 14.8. The mailtools Utility Package


14.8. The mailtools Utility Package

The email package used by the pymail example of the prior section is a collection of powerful toolsin fact, perhaps too powerful to remember completely. At the minimum, some reusable boilerplate code for common use cases can help insulate you from some of its details. To simplify email interfacing for more complex mail clients, and to further demonstrate the use of standard library email tools, I developed the custom utility modules listed in this sectiona package called mailtools.

mailtools is a Python modules package: a directory of code, with one module per tool class, and an initialization module run when the directory is first imported. This package's modules are essentially just a wrapper layer above the standard library's email package, as well as its poplib and smtplib modules. They make some assumptions about the way email is to be used, but they are reasonable and allow us to forget some of the underlying complexity of the standard library tools employed.

In a nutshell, the mailtools package provides three classesto fetch, send, and parse email messages. These classes can be used as superclasses to mix in their methods to an application-specific class or to create standalone or embedded objects that export their methods. We'll see these classes deployed both ways in this text.

One design note worth mentioning up front: none of the code in this package knows anything about the user interface it will be used in (console, GUI, web, or other), or does anything about things like threads; it is just a toolkit. As we'll see, its clients are responsible for deciding how it will be deployed. By focusing on just email processing here, we simplify the code, as well as the programs that will use it.

As a simple example of this package's tools in action, its selftest.py module serves as a self-test script. When run, it sends a message from you, to you, which includes the selftest.py file as an attachment. It also fetches and displays some mail headers and contents. These interfaces, along with some user-interface magic, will lead us to full-blown email clients and web sites in later chapters.

The next few sections list mailtools source code. We won't cover all of this package's code in depthstudy its listings for more details, and see its self-test module for a usage example. Also, flip ahead to the three clients that will use it for examples: the modified pymail2.py following this listing, the PyMailGUI client in Chapter 15, and the PyMailCGI server in Chapter 17. By sharing and reusing this module, all three systems inherit its utility, as well as any future enhancements.

14.8.1. Initialization File

The module in Example 14-21 implements the initialization logic of the mailtools package; as usual, its code is run automatically the first time a script imports through the package's directory. Notice how this file collects the contents of all the nested modules into the directory's namespace with from * statementsbecause mailtools began life as a single .py file, this provides backward compatibility for existing clients. Since this is the root module, global comments appear here as well.

Example 14-21. PP3E\Internet\Email\mailtools\_ _init_ _.py

 ################################################################################ # interface to mail server transfers, used by pymail, PyMailGUI and PyMailCGI; # does loads, sends, parsing, composing, and deleting, with attachment parts, # encoding, etc.;  the parser, fetcher, and sender classes here are designed # to be mixed-in to subclasses which use their methods, or used as embedded or # standalone objects;  also has convenience subclasses for silent mode, etc.; # loads all if pop server doesn't do top;  doesn't handle threads or UI here, # and allows askPassword to differ per subclass;  progress callback funcs get # status;  all calls raise exceptions on error--client must handle in GUI/other; # this changed from file to package: nested modules imported here for bw compat; # TBD: in saveparts, should file be opened in text mode for text/ contypes? # TBD: in walkNamedParts, should we skip oddballs like message/delivery-status? ################################################################################ # collect modules here, when package dir imported directly from mailFetcher import * from mailSender  import * from mailParser  import * # export nested modules here, when from mailtools import * _ _all_ _ = 'mailFetcher', 'mailSender', 'mailParser' # test case moved to selftest.py to allow mailconfig's # path to be set before importing nested modules above 

14.8.2. MailTool Class

Example 14-22 contains common superclasses for the other classes in the package. At present, these are used only to enable or disable trace message output (some clients, such as web-based programs, may not want text to be printed to the output stream). Subclasses mix in the silent variant to turn off output.

Example 14-22. PP3E\Internet\Email\mailtools\mailTool.py

 ############################################################################### # common superclasses: used to turn trace massages on/off ############################################################################### class MailTool:                    # superclass for all mail tools     def trace(self, message):      # redef me to disable or log to file         print message class SilentMailTool:              # to mixin instead of subclassing     def trace(self, message):         pass 

14.8.3. MailSender Class

The class used to compose and send messages is coded in Example 14-23. This module provides a convenient interface that combines standard library tools we've already met in this chapterthe email package to compose messages with attachments and encodings, and the smtplib module to send the resulting email text. Attachments are passed in as a list of filenamesMIME types and any required encodings are determined automatically with the module mimetypes. Moreover, date and time strings are automated with an email.Utils call. Study this file's code and comments for more on its operation.

Example 14-23. PP3E\Internet\Email\mailtools\mailSender.py

 ############################################################################### # send messages, add attachments (see _ _init_ _ for docs, test) ############################################################################### import mailconfig                                      # client's mailconfig import smtplib, os, mimetypes                          # mime: name to type import email.Utils, email.Encoders                     # date string, base64 from mailTool import MailTool, SilentMailTool from email.Message       import Message                # general message from email.MIMEMultipart import MIMEMultipart          # type-specific messages from email.MIMEAudio     import MIMEAudio from email.MIMEImage     import MIMEImage from email.MIMEText      import MIMEText from email.MIMEBase      import MIMEBase class MailSender(MailTool):     """     send mail: format message, interface with SMTP server     works on any machine with Python+Inet, doesn't use cmdline mail     a nonauthenticating client: see MailSenderAuth if login required     """     def _ _init_ _(self, smtpserver=None):         self.smtpServerName  = smtpserver or mailconfig.smtpservername     def sendMessage(self, From, To, Subj, extrahdrs, bodytext, attaches,                                             saveMailSeparator=(('='*80)+'PY\n')):         """         format,send mail: blocks caller, thread me in a GUI         bodytext is main text part, attaches is list of filenames         extrahdrs is list of (name, value) tuples to be added         raises uncaught exception if send fails for any reason         saves sent message text in a local file if successful         assumes that To, Cc, Bcc hdr values are lists of 1 or more already         stripped addresses (possibly in full name+<addr> format); client         must split these on delimiters, parse, or use multiline input;         note that SMTP allows full name+<addr> format in recipients         """         if not attaches:             msg = Message( )             msg.set_payload(bodytext)         else:             msg = MIMEMultipart( )             self.addAttachments(msg, bodytext, attaches)         recip = To         msg['From']    = From         msg['To']      = ', '.join(To)              # poss many: addr list         msg['Subject'] = Subj                       # servers reject ';' sept         msg['Date']    = email.Utils.formatdate( )      # curr datetime, rfc2822 utc         for name, value in extrahdrs:               # Cc, Bcc, X-Mailer, etc.             if value:                 if name.lower( ) not in ['cc', 'bcc']:                     msg[name] = value                 else:                     msg[name] = ', '.join(value)     # add commas between                     recip += value                   # some servers reject ['']         fullText = msg.as_string( )                  # generate formatted msg         # sendmail call raises except if all Tos failed,         # or returns failed Tos dict for any that failed         self.trace('Sending to...'+ str(recip))         self.trace(fullText[:256])         server = smtplib.SMTP(self.smtpServerName)           # this may fail too         self.getPassword( )                                       # if srvr requires         self.authenticateServer(server)                      # login in subclass         try:             failed = server.sendmail(From, recip, fullText)  # except or dict         finally:             server.quit( )                                    # iff connect OK         if failed:             class SomeAddrsFailed(Exception): pass             raise SomeAddrsFailed('Failed addrs:%s\n' % failed)         self.saveSentMessage(fullText, saveMailSeparator)         self.trace('Send exit')     def addAttachments(self, mainmsg, bodytext, attaches):         # format a multipart message with attachments         msg = MIMEText(bodytext)                 # add main text/plain part         mainmsg.attach(msg)         for filename in attaches:                # absolute or relative paths             if not os.path.isfile(filename):     # skip dirs, etc.                 continue             # guess content type from file extension, ignore encoding             contype, encoding = mimetypes.guess_type(filename)             if contype is None or encoding is not None:  # no guess, compressed?                 contype = 'application/octet-stream'     # use generic default             self.trace('Adding ' + contype)             # build sub-Message of appropriate kind             maintype, subtype = contype.split('/', 1)             if maintype == 'text':                 data = open(filename, 'r')                 msg  = MIMEText(data.read( ), _subtype=subtype)                 data.close( )             elif maintype == 'image':                 data = open(filename, 'rb')                 msg  = MIMEImage(data.read( ), _subtype=subtype)                 data.close( )             elif maintype == 'audio':                 data = open(filename, 'rb')                 msg  = MIMEAudio(data.read( ), _subtype=subtype)                 data.close( )             else:                 data = open(filename, 'rb')                 msg  = MIMEBase(maintype, subtype)                 msg.set_payload(data.read( ))                 data.close( )                            # make generic type                 email.Encoders.encode_base64(msg)         # encode using base64             # set filename and attach to container             basename = os.path.basename(filename)             msg.add_header('Content-Disposition',                            'attachment', filename=basename)             mainmsg.attach(msg)         # text outside mime structure, seen by non-MIME mail readers         mainmsg.preamble = 'A multi-part MIME format message.\n'         mainmsg.epilogue = ''  # make sure message ends with a newline     def saveSentMessage(self, fullText, saveMailSeparator):         # append sent message to local file if worked         # client: pass separator used for your app, splits         # caveat: user may change file at same time (unlikely)         try:             sentfile = open(mailconfig.sentmailfile, 'a')             if fullText[-1] != '\n': fullText += '\n'             sentfile.write(saveMailSeparator)             sentfile.write(fullText)             sentfile.close( )         except:             self.trace('Could not save sent message')    # not a show-stopper     def authenticateServer(self, server):         pass  # no login required for this server/class     def getPassword(self):         pass  # no login required for this server/class ################################################################################ # specialized subclasses ################################################################################ class MailSenderAuth(MailSender):     """     use for servers that require login authorization;     client: choose MailSender or MailSenderAuth super     class based on mailconfig.smtpuser setting (None?)     """     def _ _init_ _(self, smtpserver=None, smtpuser=None):         MailSender._ _init_ _(self, smtpserver)         self.smtpUser = smtpuser or mailconfig.smtpuser         self.smtpPassword = None     def authenticateServer(self, server):         server.login(self.smtpUser, self.smtpPassword)     def getPassword(self):         """         get SMTP auth password if not yet known;         may be called by superclass auto, or client manual:         not needed until send, but don't run in GUI thread;         get from client-side file or subclass method         """         if not self.smtpPassword:             try:                 localfile = open(mailconfig.smtppasswdfile)                 self.smtpPassword = localfile.readline( )[:-1]                 self.trace('local file password' + repr(self.smtpPassword))             except:                 self.smtpPassword = self.askSmtpPassword( )     def askSmtpPassword(self):         assert False, 'Subclass must define method' class MailSenderAuthConsole(MailSender):     def askSmtpPassword(self):         import getpass         prompt = 'Password for %s on %s?' % (self.smtpUser, self.smtpServerName)         return getpass.getpass(prompt) class SilentMailSender(SilentMailTool, MailSender):     pass   # replaces trace 

14.8.4. MailFetcher Class

The class defined in Example 14-24 does the work of interfacing with a POP email serverloading, deleting, and synchronizing.

14.8.4.1. General usage

This module deals strictly in email text; parsing email after it has been fetched is delegated to a different module in the package. Moreover, this module doesn't cache already loaded information; clients must add their own mail-retention tools if desired. Clients must also provide password input methods or pass one in, if they cannot use the console input subclass here (e.g., GUIs and web-based programs).

The loading and deleting tasks use the standard library poplib module in ways we saw earlier in this chapter, but notice that there are interfaces for fetching just message header text with the TOP action in POP. This can save substantial time if clients need to fetch only basic details for an email index.

This module also supports the notion of progress indicatorsfor methods that perform multiple downloads or deletions, callers may pass in a function that will be called as each mail is processed. This function will receive the current and total step numbers. It's left up to the caller to render this in a GUI, console, or other user interface.

14.8.4.2. Inbox synchronization tools

Also notice that Example 14-24 devotes substantial code to detecting synchronization errors between an email list held by a client, and the current state of the inbox at the POP email server. Normally, POP assigns relative message numbers to email in the inbox, and only adds newly arrived emails to the end of the inbox. As a result, relative message numbers from an earlier fetch may usually be used to delete and fetch in the future.

However, although rare, it is not impossible for the server's inbox to change in ways that invalidate previously fetched message numbers. For instance, emails may be deleted in another client, and the server itself may move mails from the inbox to an undeliverable state on download errors (this may vary per ISP). In both cases, email may be removed from the middle of the inbox, throwing some prior relative message numbers out of sync with the server.

This situation can result in fetching the wrong message in an email clientusers receive a different message than the one they thought they had selected. Worse, this can make deletions inaccurateif a mail client uses a relative message number in a delete request, the wrong mail may be deleted if the inbox has changed since the index was fetched.

To assist clients, Example 14-24 includes tools, which match message headers on deletions to ensure accuracy and perform general inbox synchronization tests on demand. These tools can be used only by clients that retain the fetched email list as state information. We'll use these in the PyMailGUI client in Chapter 15. There, deletions use the safe interface, and loads run the synchronization test on demand; on errors, the inbox index is automatically reloaded. For now, see Example 14-24 source code and comments for more details.

Note that the synchronization tests try a variety of matching techniques, but require the complete headers text and, in the worst case, must parse headers and match many header fields. In many cases, the single previously fetched message-id header field would be sufficient for matching against messages in the server's inbox. However, because this field is optional and can be forged to have any value, it might not always be a reliable way to identify messages. In other words, a same-valued message-id may not suffice to guarantee a match, although it can be used to identify a mismatch; in Example 14-24, the message-id is used to rule out a match if either message has one, and they differ in value. This test is performed before falling back on slower parsing and multiple header matches.

Example 14-24. PP3E\Internet\Email\mailtools\mailFetcher.py

 ############################################################################### # retrieve, delete, match mail from a POP server (see _ _init_ _ for docs, test) ############################################################################### import poplib, mailconfig       # client's mailconfig: script dir or pythonpath print 'user:', mailconfig.popusername from mailParser import MailParser                  # for headers matching from mailTool   import MailTool, SilentMailTool    # trace control supers # index/server msgnum out of synch tests class DeleteSynchError(Exception): pass            # msg out of synch in del class TopNotSupported(Exception): pass             # can't run synch test class MessageSynchError(Exception): pass           # index list out of sync class MailFetcher(MailTool):     """     fetch mail: connect, fetch headers+mails, delete mails     works on any machine with Python+Inet; subclass me to cache     implemented with the POP protocol; IMAP requires new class     """     def _ _init_ _(self, popserver=None, popuser=None, poppswd=None, hastop=True):         self.popServer   = popserver or mailconfig.popservername         self.popUser     = popuser   or mailconfig.popusername         self.srvrHasTop  = hastop         self.popPassword = poppswd  # ask later if None     def connect(self):         self.trace('Connecting...')         self.getPassword( )                          # file, GUI, or console         server = poplib.POP3(self.popServer)         server.user(self.popUser)                    # connect,login POP server         server.pass_(self.popPassword)               # pass is a reserved word         self.trace(server.getwelcome( ))             # print returned greeting         return server     def downloadMessage(self, msgnum):         """         load full raw text of one mail msg, given its         POP relative msgnum; caller must parse content         """         self.trace('load '+str(msgnum))         server = self.connect( )         try:             resp, msglines, respsz = server.retr(msgnum)         finally:             server.quit( )         return '\n'.join(msglines)                 # concat lines for parsing     def downloadAllHeaders(self, progress=None, loadfrom=1):         """         get sizes, raw header text only, for all or new msgs         begins loading headers from message number loadfrom         use loadfrom to load newly arrived mails only         use downloadMessage to get a full msg text later         progress is a function called with (count, total);         returns: [headers text], [mail sizes], loadedfull?         """         if not self.srvrHasTop:                    # not all servers support TOP             return self.downloadAllMsgs(progress)  # naively load full msg text         else:             self.trace('loading headers')             server = self.connect( )                # mbox now locked until quit             try:                 resp, msginfos, respsz = server.list( )   # 'num size' lines list                 msgCount = len(msginfos)                   # alt to srvr.stat[0]                 msginfos = msginfos[loadfrom-1:]           # drop already loadeds                 allsizes = [int(x.split( )[1]) for x in msginfos]                 allhdrs  = []                 for msgnum in range(loadfrom, msgCount+1):          # poss empty                     if progress: progress(msgnum, msgCount)         # callback?                     resp, hdrlines, respsz = server.top(msgnum, 0)  # hdrs only                     allhdrs.append('\n'.join(hdrlines))             finally:                 server.quit( )                          # make sure unlock mbox             assert len(allhdrs) == len(allsizes)             self.trace('load headers exit')             return allhdrs, allsizes, False     def downloadAllMessages(self, progress=None, loadfrom=1):         """         load full message text for all msgs from loadfrom..N,         despite any caching that may be being done in the caller;         much slower than downloadAllHeaders, if just need hdrs;         """         self.trace('loading full messages')         server = self.connect( )         try:             (msgCount, msgBytes) = server.stat( )          # inbox on server             allmsgs  = []             allsizes = []             for i in range(loadfrom, msgCount+1):          # empty if low >= high                 if progress: progress(i, msgCount)                 (resp, message, respsz) = server.retr(i)  # save text on list                 allmsgs.append('\n'.join(message))        # leave mail on server                 allsizes.append(respsz)                   # diff from len(msg)         finally:             server.quit( )                                    # unlock the mail box         assert len(allmsgs) == (msgCount - loadfrom) + 1  # msg nums start at 1        #assert sum(allsizes) == msgBytes                  # not if loadfrom > 1         return allmsgs, allsizes, True     def deleteMessages(self, msgnums, progress=None):         """         delete multiple msgs off server; assumes email inbox         unchanged since msgnums were last determined/loaded;         use if msg headers not available as state information;         fast, but poss dangerous: see deleteMessagesSafely         """         self.trace('deleting mails')         server = self.connect( )         try:             for (ix, msgnum) in enumerate(msgnums):   # don't reconnect for each                 if progress: progress(ix+1, len(msgnums))                 server.dele(msgnum)         finally:                                      # changes msgnums: reload             server.quit( )     def deleteMessagesSafely(self, msgnums, synchHeaders, progress=None):         """         delete multiple msgs off server, but use TOP fetches to         check for a match on each msg's header part before deleting;         assumes the email server supports the TOP interface of POP,         else raises TopNotSupported - client may call deleteMessages;         use if the mail server might change the inbox since the email         index was last fetched, thereby changing POP relative message         numbers;  this can happen if email is deleted in a different         client;  some ISPs may also move a mail from inbox to the         undeliverable box in response to a failed download;         synchHeaders must be a list of already loaded mail hdrs text,         corresponding to selected msgnums (requires state);  raises         exception if any out of synch with the email server;  inbox is         locked until quit, so it should not change between TOP check         and actual delete: synch check must occur here, not in caller;         may be enough to call checkSynchError+deleteMessages, but check         each msg here in case deletes and inserts in middle of inbox;         """         if not self.srvrHasTop:             raise TopNotSupported('Safe delete cancelled')         self.trace('deleting mails safely')         errmsg  = 'Message %s out of synch with server.\n'         errmsg += 'Delete terminated at this message.\n'         errmsg += 'Mail client may require restart or reload.'         server = self.connect( )                       # locks inbox till quit         try:                                            # don't reconnect for each             (msgCount, msgBytes) = server.stat( )      # inbox size on server             for (ix, msgnum) in enumerate(msgnums):                 if progress: progress(ix+1, len(msgnums))                 if msgnum > msgCount:                            # msgs deleted                     raise DeleteSynchError(errmsg % msgnum)                 resp, hdrlines, respsz = server.top(msgnum, 0)   # hdrs only                 msghdrs = '\n'.join(hdrlines)                 if not self.headersMatch(msghdrs, synchHeaders[msgnum-1]):                     raise DeleteSynchError(errmsg % msgnum)                 else:                     server.dele(msgnum)                # safe to delete this msg         finally:                                       # changes msgnums: reload             server.quit( )                             # unlock inbox on way out     def checkSynchError(self, synchHeaders):         """         check to see if already loaded hdrs text in synchHeaders         list matches what is on the server, using the TOP command in         POP to fetch headers text; use if inbox can change due to         deletes in other client, or automatic action by email server;         raises except if out of synch, or error while talking to server;         for speed, only checks last in last: this catches inbox deletes,         but assumes server won't insert before last (true for incoming         mails); check inbox size first: smaller if just deletes;  else         top will differ if deletes and newly arrived messages added at         end;  result valid only when run: inbox may change after return;         """         self.trace('synch check')         errormsg  = 'Message index out of synch with mail server.\n'         errormsg += 'Mail client may require restart or reload.'         server = self.connect( )         try:             lastmsgnum = len(synchHeaders)                      # 1..N             (msgCount, msgBytes) = server.stat( )                    # inbox size             if lastmsgnum > msgCount:                           # fewer now?                 raise MessageSynchError(errormsg)               # none to cmp             if self.srvrHasTop:                 resp, hdrlines, respsz = server.top(lastmsgnum, 0)  # hdrs only                 lastmsghdrs = '\n'.join(hdrlines)                 if not self.headersMatch(lastmsghdrs, synchHeaders[-1]):                     raise MessageSynchError(errormsg)         finally:             server.quit( )     def headersMatch(self, hdrtext1, hdrtext2):         """"         may not be as simple as a string compare: some servers add         a "Status:" header that changes over time; on one ISP, it         begins as "Status: U" (unread), and changes to "Status: RO"         (read, old) after fetched once - throws off synch tests if         new when index fetched, but have been fetched once before         delete or last-message check;  "Message-id:" line is unique         per message in theory, but optional, and can be anything if         forged; match more common: try first; parsing costly: try last         """         # try match by simple string compare         if hdrtext1 == hdrtext2:             self.trace('Same headers text')             return True         # try match without status lines         split1 = hdrtext1.splitlines( )       # s.split('\n'), but no final ''         split2 = hdrtext2.splitlines( )         strip1 = [line for line in split1 if not line.startswith('Status:')]         strip2 = [line for line in split2 if not line.startswith('Status:')]         if strip1 == strip2:             self.trace('Same without Status')             return True         # try mismatch by message-id headers if either has one         msgid1 = [line for line in split1 if line[:11].lower( ) == 'message-id:']         msgid2 = [line for line in split2 if line[:11].lower( ) == 'message-id:']         if (msgid1 or msgid2) and (msgid1 != msgid2):             self.trace('Different Message-Id')             return False         # try full hdr parse and common headers if msgid missing or trash         tryheaders  = ('From', 'To', 'Subject', 'Date')         tryheaders += ('Cc', 'Return-Path', 'Received')         msg1 = MailParser( ).parseHeaders(hdrtext1)         msg2 = MailParser( ).parseHeaders(hdrtext2)         for hdr in tryheaders:                          # poss multiple Received             if msg1.get_all(hdr) != msg2.get_all(hdr):  # case insens, dflt None                 self.trace('Diff common headers')                 return False         # all common hdrs match and don't have a diff message-id         self.trace('Same common headers')         return True     def getPassword(self):         """         get POP password if not yet known         not required until go to server         from client-side file or subclass method         """         if not self.popPassword:             try:                 localfile = open(mailconfig.poppasswdfile)                 self.popPassword = localfile.readline( )[:-1]                 self.trace('local file password' + repr(self.popPassword))             except:                 self.popPassword = self.askPopPassword( )     def askPopPassword(self):         assert False, 'Subclass must define method' ################################################################################ # specialized subclasses ################################################################################ class MailFetcherConsole(MailFetcher):     def askPopPassword(self):         import getpass         prompt = 'Password for %s on %s?' % (self.popUser, self.popServer)         return getpass.getpass(prompt) class SilentMailFetcher(SilentMailTool, MailFetcher):     pass   # replaces trace 

14.8.5. MailParser Class

Example 14-25 implements the last major class in the mailtools packagegiven the text of an email message, its tools parse the mail's content into a message object, with headers and decoded parts. This module is largely just a wrapper around the standard library's email package, but it adds convenience toolsfinding the main text part of a message, filename generation for message parts, saving attached parts to files, and so on. See the code for more information. Also notice the parts walker here: by coding its search logic in one place, we guarantee that all three clients implement the same traversal.

Example 14-25. PP3E\Internet\Email\mailtools\mailParser.py

 ############################################################################### # parsing and attachment extract, analyse, save (see _ _init_ _ for docs, test) ############################################################################### import os, mimetypes                               # mime: type to name import email.Parser from email.Message import Message from mailTool import MailTool class MailParser(MailTool):     """     methods for parsing message text, attachments     subtle thing: Message object payloads are either a simple     string for non-multipart messages, or a list of Message     objects if multipart (possibly nested); we don't need to     distinguish between the two cases here, because the Message     walk generator always returns self first, and so works fine     on non-multipart messages too (a single object is walked);     for simple messages, the message body is always considered     here to be the sole part of the mail;  for multipart messages,     the parts list includes the main message text, as well as all     attachments;  this allows simple messages not of type text to     be handled like attachments in a UI (e.g., saved, opened);     Message payload may also be None for some oddball part types;     """     def walkNamedParts(self, message):         """         generator to avoid repeating part naming logic         skips multipart headers, makes part filenames         message is already parsed email.Message object         doesn't skip oddball types: payload may be None         """         for (ix, part) in enumerate(message.walk( )):    # walk includes message             maintype = part.get_content_maintype( )      # ix includes multiparts             if maintype == 'multipart':                 continue                                  # multipart/*: container             else:                 filename, contype = self.partName(part, ix)                 yield (filename, contype, part)     def partName(self, part, ix):         """         extract filename and content type from message part;         filename: tries Content-Disposition, then Content-Type         name param, or generates one based on mimetype guess;         """         filename = part.get_filename( )                # filename in msg hdrs?         contype  = part.get_content_type( )            # lower maintype/subtype         if not filename:             filename = part.get_param('name')          # try content-type name         if not filename:             if contype == 'text/plain':                # hardcode plain text ext                 ext = '.txt'                           # else guesses .ksh!             else:                 ext = mimetypes.guess_extension(contype)                 if not ext: ext = '.bin'              # use a generic default             filename = 'part-%03d%s' % (ix, ext)         return (filename, contype)     def saveParts(self, savedir, message):         """         store all parts of a message as files in a local directory;         returns [('maintype/subtype', 'filename')] list for use by         callers, but does not open any parts or attachments here;         get_payload decodes base64, quoted-printable, uuencoded data;         mail parser may give us a None payload for oddball types we         probably should skip over: convert to str here to be safe;         """         if not os.path.exists(savedir):             os.mkdir(savedir)         partfiles = []         for (filename, contype, part) in self.walkNamedParts(message):             fullname = os.path.join(savedir, filename)             fileobj  = open(fullname, 'wb')             # use binary mode             content  = part.get_payload(decode=1)       # decode base64,qp,uu             fileobj.write(str(content))                 # make sure is a str             fileobj.close( )             partfiles.append((contype, fullname))       # for caller to open         return partfiles     def saveOnePart(self, savedir, partname, message):         """         ditto, but find and save just one part by name         """         if not os.path.exists(savedir):             os.mkdir(savedir)         fullname = os.path.join(savedir, partname)         (contype, content) = self.findOnePart(partname, message)         open(fullname, 'wb').write(str(content))         return (contype, fullname)     def partsList(self, message):         """"         return a list of filenames for all parts of an         already parsed message, using same filename logic         as saveParts, but do not store the part files here         """         validParts = self.walkNamedParts(message)         return [filename for (filename, contype, part) in validParts]     def findOnePart(self, partname, message):         """         find and return part's content, given its name         intended to be used in conjunction with partsList         we could also mimetypes.guess_type(partname) here         we could also avoid this search by saving in dict         """         for (filename, contype, part) in self.walkNamedParts(message):             if filename == partname:                 content = part.get_payload(decode=1)          # base64,qp,uu                 return (contype, content)     def findMainText(self, message):         """         for text-oriented clients, return the first text part;         for the payload of a simple message, or all parts of         a multipart message, looks for text/plain, then text/html,         then text/*, before deducing that there is no text to         display;  this is a heuristic, but covers most simple,         multipart/alternative, and multipart/mixed messages;         content-type defaults to text/plain if not in simple msg;         handles message nesting at top level by walking instead         of list scans;  if non-multipart but type is text/html,         returns the HTML as the text with an HTML type: caller         may open in web browser;  if nonmultipart and not text,         no text to display: save/open in UI;  caveat: does not         try to concatenate multiple inline text/plain parts         """         # try to find a plain text         for part in message.walk( ):                        # walk visits message             type = part.get_content_type( )                 # if nonmultipart             if type == 'text/plain':                 return type, part.get_payload(decode=1)     # may be base64,qp,uu         # try to find an HTML part         for part in message.walk( ):             type = part.get_content_type( )             if type == 'text/html':                 return type, part.get_payload(decode=1)    # caller renders         # try any other text type, including XML         for part in message.walk( ):             if part.get_content_maintype( ) == 'text':                 return part.get_content_type( ), part.get_payload(decode=1)         # punt: could use first part, but it's not marked as text         return 'text/plain', '[No text to display]'     # returned when parses fail     errorMessage = Message( )     errorMessage.set_payload('[Unable to parse message - format error]')     def parseHeaders(self, mailtext):         """         parse headers only, return root email.Message object         stops after headers parsed, even if nothing else follows (top)         email.Message object is a mapping for mail header fields         payload of message object is None, not raw body text         """         try:             return email.Parser.Parser( ).parsestr(mailtext, headersonly=True)         except:             return self.errorMessage     def parseMessage(self, fulltext):         """         parse entire message, return root email.Message object         payload of message object is a string if not is_multipart( )         payload of message object is more Messages if multiple parts         the call here same as calling email.message_from_string( )         """         try:             return email.Parser.Parser( ).parsestr(fulltext)       # may fail!         except:             return self.errorMessage     # or let call handle? can check return     def parseMessageRaw(self, fulltext):         """         parse headers only, return root email.Message object         stops after headers parsed, for efficiency (not yet used here)         payload of message object is raw text of mail after headers         """         try:             return email.Parser.HeaderParser( ).parsestr(fulltext)         except:             return self.errorMessage 

14.8.6. Self-Test Script

The last file in the mailtools package, Example 14-26, lists the self-test code for the package. This code is a separate script file, in order to allow for import search path manipulationit emulates a real client, which is assumed to have a mailconfig.py module in its own source directory (this module can vary per client).

Example 14-26. PP3E\Internet\Email\mailtools\selftest.py

 ############################################################################### # self-test when this file is run as a program ############################################################################### # # mailconfig normally comes from the client's source directory or # sys.path; for testing, get it from Email directory one level up # import sys sys.path.append('..') import mailconfig print 'config:', mailconfig._ _file_ _ # get these from _ _init_ _ from mailtools import MailFetcherConsole, MailSender, MailSenderAuthConsole if not mailconfig.smtpuser:     sender = MailSender( ) else:     sender = MailSenderAuthConsole( ) sender.sendMessage(From      = mailconfig.myaddress,                    To        = [mailconfig.myaddress],                    Subj      = 'testing 123',                    extrahdrs = [('X-Mailer', 'mailtools')],                    bodytext  = 'Here is my source code',                    attaches  = ['selftest.py']) fetcher = MailFetcherConsole( ) def status(*args): print args hdrs, sizes, loadedall = fetcher.downloadAllHeaders(status) for num, hdr in enumerate(hdrs[:5]):     print hdr     if raw_input('load mail?') in ['y', 'Y']:         print fetcher.downloadMessage(num+1), '\n', '-'*70 last5 = len(hdrs)-4 msgs, sizes, loadedall = fetcher.downloadAllMessages(status, loadfrom=last5) for msg in msgs:     print msg[:200], '\n', '-'*70 raw_input('Press Enter to exit') 

14.8.7. Updating the pymail Console Client

Finally, to give a use case for the mailtools module package of the preceding sections, Example 14-27 provides an updated version of the pymail program we met earlier, which uses mailtools to access email instead of older tools. Compare its code to the original pymail in this chapter to see how mailtools is employed here. You'll find that its mail download and send logic is substantially simpler.

Example 14-27. pymail2.py

 #!/usr/local/bin/python ########################################################################## # pymail2 - simple console email interface client in Python; this # version uses the mailtools package, which in turn uses poplib, # smtplib, and the email package for parsing and composing emails; # displays first text part of mails, not entire full text; # fetches just mail headers initially, using the TOP command; # fetches full text of just email selected to be displayed; # caches already fetched mails; caveat: no way to refresh index; # uses standalone mailtools objects - they can also be superclasses; ########################################################################## mailcache = {} def fetchmessage(i):     try:         fulltext = mailcache[i]     except KeyError:         fulltext = fetcher.downloadMessage(i)         mailcache[i] = fulltext     return fulltext def sendmessage( ):     from pymail import inputmessage     From, To, Subj, text = inputmessage( )     sender.sendMessage(From, To, Subj, [], text, attaches=None) def deletemessages(toDelete, verify=True):     print 'To be deleted:', toDelete     if verify and raw_input('Delete?')[:1] not in ['y', 'Y']:         print 'Delete cancelled.'     else:         print 'Deleting messages from server.'         fetcher.deleteMessages(toDelete) def showindex(msgList, msgSizes, chunk=5):     count = 0     for (msg, size) in zip(msgList, msgSizes):      # email.Message, int         count += 1         print '%d:\t%d bytes' % (count, size)         for hdr in ('From', 'Date', 'Subject'):             print '\t%s=>%s' % (hdr, msg.get(hdr, '(unknown)'))         if count % chunk == 0:             raw_input('[Press Enter key]')         # pause after each chunk def showmessage(i, msgList):     if 1 <= i <= len(msgList):         fulltext = fetchmessage(i)         message  = parser.parseMessage(fulltext)         ctype, maintext = parser.findMainText(message)         print '-'*80         print maintext                # main text part, not entire mail         print '-'*80                  # and not any attachments after     else:         print 'Bad message number' def savemessage(i, mailfile, msgList):     if 1 <= i <= len(msgList):         fulltext = fetchmessage(i)         open(mailfile, 'a').write('\n' + fulltext + '-'*80 + '\n')     else:         print 'Bad message number' def msgnum(command):     try:         return int(command.split( )[1])     except:         return -1   # assume this is bad helptext = """ Available commands: i     - index display l n?  - list all messages (or just message n) d n?  - mark all messages for deletion (or just message n) s n?  - save all messages to a file (or just message n) m     - compose and send a new mail message q     - quit pymail ?     - display this help text """ def interact(msgList, msgSizes, mailfile):     showindex(msgList, msgSizes)     toDelete = []     while 1:         try:             command = raw_input('[Pymail] Action? (i, l, d, s, m, q, ?) ')         except EOFError:             command = 'q'         if not command: command = '*'         if command == 'q':                     # quit             break         elif command[0] == 'i':                # index             showindex(msgList, msgSizes)         elif command[0] == 'l':                # list             if len(command) == 1:                 for i in range(1, len(msgList)+1):                     showmessage(i, msgList)             else:                 showmessage(msgnum(command), msgList)         elif command[0] == 's':                # save             if len(command) == 1:                 for i in range(1, len(msgList)+1):                     savemessage(i, mailfile, msgList)             else:                 savemessage(msgnum(command), mailfile, msgList)         elif command[0] == 'd':                # mark for deletion later             if len(command) == 1:                 toDelete = range(1, len(msgList)+1)             else:                 delnum = msgnum(command)                 if (1 <= delnum <= len(msgList)) and (delnum not in toDelete):                     toDelete.append(delnum)                 else:                     print 'Bad message number'         elif command[0] == 'm':                # send a new mail via SMTP             try:                 sendmessage( )             except:                 print 'Error - mail not sent'         elif command[0] == '?':             print helptext         else:             print 'What? -- type "?" for commands help'     return toDelete def main( ):     global parser, sender, fetcher     import mailtools, mailconfig     mailserver = mailconfig.popservername     mailuser   = mailconfig.popusername     mailfile   = mailconfig.savemailfile     parser     = mailtools.MailParser( )     sender     = mailtools.MailSender( )     fetcher    = mailtools.MailFetcherConsole(mailserver, mailuser)     def progress(i, max): print i, 'of', max     hdrsList, msgSizes, ignore = fetcher.downloadAllHeaders(progress)     msgList = [parser.parseHeaders(hdrtext) for hdrtext in hdrsList]     print '[Pymail email client]'     toDelete   = interact(msgList, msgSizes, mailfile)     if toDelete: deletemessages(toDelete) if _ _name_ _ == '_ _main_ _': main( ) 

This program is used interactively, the same as the original. In fact, the output is nearly identical, so we won't go into further details. Here's a quick look at this script in action; run this on your own machine to see it firsthand:

 C:\...\PP3E\Internet\Email>pymail2.py user: pp3e loading headers Connecting... Password for pp3e on pop.earthlink.net? +OK NGPopper vEL_6_10 at earthlink.net ready <5870.1139382442@pop-savannah.at... 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 load headers exit [Pymail email client] 1:      876 bytes         From=>lumber.jack@TheLarch.com         Date=>Wed, 08 Feb 2006 05:23:13 -0000         Subject=>I'm a Lumberjack, and I'm Okay 2:      800 bytes         From=>pp3e@earthlink.net         Date=>Wed, 08 Feb 2006 05:24:06 -0000         Subject=>testing 3:      818 bytes         From=>Eric.the.Half.a.Bee@yahoo.com         Date=>Tue Feb 07 22:51:08 2006         Subject=>A B C D E F G 4:      770 bytes         From=>pp3e@earthlink.net         Date=>Tue Feb 07 23:19:51 2006         Subject=>testing smtpmail 5:      819 bytes         From=>Eric.the.Half.a.Bee@aol.com         Date=>Tue Feb 07 23:34:23 2006         Subject=>a b c d e f g [Press Enter key] [Pymail] Action? (i, l, d, s, m, q, ?) l 5 load 5 Connecting... +OK NGPopper vEL_6_10 at earthlink.net ready <9564.1139382467@pop-savannah.at... -------------------------------------------------------------------------------- Spam; Spam and eggs; Spam, spam, and spam -------------------------------------------------------------------------------- [Pymail] Action? (i, l, d, s, m, q, ?) s 1 load 1 Connecting... +OK NGPopper vEL_6_10 at earthlink.net ready <5266.1139382521@pop-savannah.a... [Pymail] Action? (i, l, d, s, m, q, ?) m From? pp3e@earthlink.net To?   pp3e@earthlink.net Subj? test pymail2 send Type message text, end with line="." Run away! Run away! . Sending to...['pp3e@earthlink.net'] From: pp3e@earthlink.net To: pp3e@earthlink.net Subject: test pymail2 send Date: Wed, 08 Feb 2006 07:09:40 -0000 Run away! Run away! Send exit [Pymail] Action? (i, l, d, s, m, q, ?) q 

As you can see, this version's code eliminates some complexities, such as the manual formatting of composed mail message text. It also does a better job of displaying a mail's textinstead of blindly listing the full mail text (attachments and all), it uses mailtools to fetch the first text part of the message. The messages we're using are too simple to show the difference, but for a mail with attachments, this new version will be more focused about what it displays.

Moreover, because the interface to mail is encapsulated in the mailtools package's modules, if it ever must change, it will only need to be changed in that module, regardless of how many mail clients use its tools. And because this code is shared, if we know it works for one client, we can be sure it will work in another; there is no need to debug new code.

On the other hand, pymail2 doesn't really leverage much of the power of either mailtools or the underlying email package it uses. Things like attachments and inbox synchronization are not handled at all, for example. To see the full scope of the email package, we need to explore a larger email system, such as PyMailGUI or PyMailCGI. The first of these is the topic of the next chapter, and the second appears in Chapter 17. First, though, let's quickly survey a handful of additional client-side protocol tools.




Programming Python
Programming Python
ISBN: 0596009259
EAN: 2147483647
Year: 2004
Pages: 270
Authors: Mark Lutz

Similar book on Amazon

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net