Hack 83 Downloading POP Mail with fetchmail

You're running your own POP or IMAP mail server but don't have any way of getting your mail from your ISP to your local machine. fetchmail, a popular Unix utility, will fetch your mail for you.

In our ever-connected world, people are rabid about email. "Did you get my email?" they clamor. "Did you read my message?" they beg. "I just sent you an attachment!" they announce. Granted, Apple has thoughtfully included its own nice email program with its latest OS, but you can also check your mail via the Terminal[Hack #48] using a popular utility called fetchmail.

By itself, fetchmail does nothing more than fetch your mail. For those not familiar with the shell, it does even this not so simply, as you've got to know about mail directories and how things work together. Where fetchmail really shines, however, is when it's used in combination with other hacks in this email chapter combining fetchmail with a local IMAP or POP mail server [Hack #81] creates a powerful one-two punch for email independence.

Getting fetchmail running is easy when you have some hand-holding. The first thing we've got to do is create our storage file; this is the place fetchmail will fetch our mail to. This, conveniently enough, is called a mailbox. In OS X and most operating systems like it, this mailbox file is located under the var directory something you normally wouldn't see through the Finder. To set up your mailbox file so you have adequate permissions, perform the following, replacing username with your own:

% sudo touch /var/mail/username % sudo chown username  /var/mail/username % chmod 600 /var/mail/username

These commands will first create an empty file with the touch shell utility and then modify the permissions and ownership until it's accessible only by yourself. This ensures privacy as well as stopping a few warnings from various utilities you may use during your exploration. The file we've just created will be where our incoming mail will be saved, and messages will be stored in plain-text format, where they can easily be imported or used by other utilities.

Our next step is to actually use fetchmail. fetchmail is one of the larger utilities available and has an insane number of options enough so that it's easy to look at the manual and run away screaming in arcane tongues. Once you have an example though, fetchmail is pretty easy. Take a look at the following command:

% fetchmail  check mail.example.com

This command is safe to run; it'll merely check (not download) the supplied mail server to see if there's any mail waiting for you. It'll automatically figure out what type of server you're running (POP3, IMAP, etc.) as well as prompt for your password, automatically assuming your current Mac OS X username as the POP3 username. But what if it's not? How about:

% fetchmail --verbose --check --protocol=pop3 --username=morbus RETURN mail.example.com

This command will spit out more information about what fetchmail is doing, as well as specifying the POP3 protocol (which should save a few seconds on your initial connection) and your preferred username.

With this command, we're only checking to see if we have new mail; we still haven't initiated any downloads. Let's swap out --check for --keep:

% fetchmail --keep --protocol=pop3 --username=morbus mail.example.com

After running this and filling in our password, we'll see some output telling us what mail is being downloaded and then delivered into our /var/mail/username file. Again, even though we're downloading the mail to our local drive, that --keep flag will save the messages on the POP server until we're good and ready; removing the --keep flag will download and then remove the messages from the server.

Typing that long command line each and every time can be pretty pesky and, thankfully, fetchmail will read a configuration file that controls what fetchmail should do. To create one, start a file in your home directory called .fetchmailrc (it'll be invisible to the Finder because of the leading dot, but fetchmail will see it just fine). In this file, add the following, tweaking to your settings:

set postmaster "morbus" poll mail.example.com proto pop3  user "morbus" password "xxxxxxx" fetchall keep mda "/usr/bin/procmail -d %T"

We start off the file with set postmaster "username", which is really just a precaution; it says "if at anytime, fetchmail can't figure out where the mail should go, send it to this username." This option is primarily useful in multiuser systems (or servers).

The poll option is where the going gets good. This is where we start defining which mail servers we want to check. Until fetchmail sees another poll (or skip, but we won't discuss that here) line, it'll assume that all further directives are for the server we've just defined. That's fine with us, since this example shows only one server, using the protocol (proto) of pop3. This is the same thing as using -p or --protocol on the command line.

Our next two lines are obvious; they're the username and the password for the mailbox we want to check. Unlike the command line, we can set the password here (in plain text . . . be sure your file permissions are tight!) so that fetchmail will run without our input. If you're panicky about leaving your password in the open, just remove the password line and fetchmail will go back to prompting you.

The next three lines aren't required but merely tweak the way fetchmail is run. Here, we always want to fetch all the messages on the server (fetchall), as opposed to just new ones, and we want to keep them on the server after we've delivered them to our local box. We then set our mail delivery agent (mda) explicitly to procmail something we won't be discussing in this hack (by default, procmail is configured correctly, so this line is harmless).

As mentioned, an insane number of options are available within fetchmail, and you can find out more by issuing a man fetchmail at the Terminal. This hack should give you enough to start downloading your mail, and later you can hook it up to more powerful environments.