Using Fetchmail

Fetchmail is an unusual mail program. It is neither a mail server nor a mail reader, although it contains elements of both. Fetchmail's job is to fetch mail from a pull mail server and then to pass it on to some other programmost commonly, Fetchmail uses a push mail protocol to forward the mail it retrieves from a pull mail server. This last mail push often occurs entirely on the same computerthat is, Fetchmail acquires mail from another computer, then mails it to a local user of the computer on which it runs. There are many variant ways to configure Fetchmail, though. The program may be set up by individual users by editing its configuration file, .fetchmailconf . There's also a GUI Fetchmail configuration tool available, known as fetchmailconf , that can be used to configure the program for many common tasks .

NOTE

Earlier sections of this chapter deal with running a pull mail server. Fetchmail, though, is generally used to retrieve e-mail from a pull mail server that's run by somebody else.

Fetchmail's Place in Mail Delivery Systems

Pull mail protocols were designed to allow mail reader programs to retrieve e-mail from mail server computers. The idea is that a computer capable of running a mail server is likely to be available at all times, and so can run a push mail server to accept mail at the sender's convenience. Only the end user's connection is likely to be transient or variable in some other way, such as in having a dynamic IP address. Although this generalization is true of many traditional networks, it's not true of all of them. There are situations in which it may be desirable to transfer mail with a pull mail protocol and then continue sending it through additional mail server programs:

• Dial-up Linux systems Linux computers generally run mail servers locally, even when the computers are connected to the Internet only through dial-up Point-to-Point Protocol (PPP) connections. The local mail servers on such systems can exchange mail between multiple users or can be used to notify a single user of messages generated by the computer. One way to integrate these messages with those that arrive at the user's ISP mail account is to pull the mail using POP or IMAP and then inject the pulled mail into the local mail queue. The user can then read all mail (local and remote) using a single mail program configured to read the mail locally rather than using POP or IMAP.

• Dial-up with a LAN Some small businesses and even homes use local area networks (LANs) on which e-mail passes between users. Such LANs may connect to the Internet via a dial-up account with one or more associated e-mail accounts. You can retrieve the dial-up e-mail accounts' messages and route them to appropriate individuals on the LAN's e-mail system.

• Multiple pull mail accounts If you have multiple mail accounts, managing them all can be a nuisance. As with the dial-up Linux system just described, it's often possible to use a pull mail protocol to collect mail from these multiple sources and send them on to a single account. (This account could be local to the computer on which Fetchmail runs, but it might also be an account on a remote server.) Alternative ways to handle this situation include using the forwarding features of many mail accounts or using a mail reader that can handle multiple accounts.

• One mailbox, multiple users Sometimes multiple users may share a single e-mail account. If you have some way to filter the mail intended for different users, you can retrieve it, filter it with features built into Fetchmail or available in separate programs, and send the mail to separate mail queues that can be used by appropriate individuals.

• Converting POP to IMAP Your main external mail account might use POP, but if you want to use IMAP and if you control a Linux computer with appropriate network connections, you can retrieve the mail from the POP account and send it to a local mail queue. If you then run an IMAP server on this computer, you can read your mail using IMAP.

Put another way, pull mail accounts are usually intended for single mail users using fairly simple local configurations. This is especially true of POP, which works best when the user reads mail from a single computer using just one mail program. When the mail setup associated with an account becomes more complex, it may be helpful to add another step to mail delivery in order to merge mail from multiple sources or to separate mail into separate streams. Fetchmail is a tool that allows you to do this.

It's important to realize that Fetchmail functions as a client to both a pull mail server and a push mail server. (Alternatively, it can deliver mail in ways that are unrelated to a push mail server, but such configurations are rarer.) Because Fetchmail fills a dual-client role, it's in charge of the scheduling of e-mail acquisition and delivery, assuming both servers to which it connects are available. Fetchmail can run in both a batch mode, in which it runs, checks one or more pull mail accounts, forwards any mail it retrieves, and exits; or in a daemon mode, in which it runs constantly and performs a check for new mail at some interval you specify. The latter mode sounds convenient , but Fetchmail occasionally crashes or runs into problems when run in daemon mode for long periods, which can result in a failure to retrieve mail. I therefore favor running Fetchmail in batch mode except for configurations in which daemon mode is used for limited periods of time. If you want to schedule regular checks of the pull mail account, you can use cron to schedule Fetchmail runs on a regular basis. You can also run Fetchmail manually, checking for mail only when you decide to do so.

Scheduling Mail Retrieval

Whether you run Fetchmail as a cron job, in daemon mode, or manually, the pull nature of the protocols used for mail retrieval means that you must schedule checks of your mail; you can't simply wait for mail to arrive, as you can when your system receives mail directly via a push protocol. How often you do this, and under what circumstances, is a matter that you'll have to decide based on your needs. If your mail is very time-sensitive, you might want to use a tight schedule for mail checks, such as every five minutes. This has the drawback of consuming more network bandwidth and resources on the server, though. It also makes it more likely that your password will be intercepted and abused, especially if you're checking a mail server on the Internet, rather than on your local network. Checks at wider intervals (say, once every six hours) place less load on the servers and may slightly reduce the risk of your password being intercepted, but your mail may be delayed in reaching you for up to the interval you've selected. If you use cron to schedule mail checks, you can adjust your check interval based on the time of day. For instance, you might check for mail every half hour during business hours, but less frequently or not at all at night.

If the network link between your Linux computer and your pull mail server is transient (such as a PPP connection), you may not want to have Fetchmail check for mail at all times. Instead, you might want to add a call to Fetchmail to your network startup scripts. For instance, you might add a call to Fetchmail in the ppp-on-dialer PPP dialing script described in Chapter 2, TCP/IP Network Configuration. If you configure Fetchmail to run in daemon mode, you might then add a line to kill the program in the ppp-off script. Alternatively, you can use the interface or monitor options, described in the upcoming section "Configuring .fetchmailrc," to have Fetchmail attempt message retrieval only when the connection is up and in use.

NOTE

Fetchmail can handle challenge/response password authentication, as used by Microsoft's Exchange mail server. Thus, if your upstream provider uses this feature, Fetchmail can take advantage of it.

Using fetchmailconf

Fetchmail configuration involves editing a text file that's traditionally stored in the home directory of the user who runs the program. Thus, several users can run Fetchmail on a single computer, if desired. If you prefer GUI configuration tools, though, Fetchmail supports one, known as fetchmailconf . I describe this tool in this section; the next section, "Configuring .fetchmailrc," describes the Fetchmail configuration file that fetchmailconf modifies.

Most distributions include fetchmailconf in a separate package from Fetchmail proper, so you may need to install two packages to get both programs. As an X program that's based on Tcl/Tk, fetchmailconf also requires that you have additional libraries installed. Once you've installed it, you can configure Fetchmail to retrieve e-mail from a pull mail account as follows :

1. As an ordinary user, type fetchmailconf in an xterm window. This action opens the Fetchmail Launcher window, which allows you to click one of four buttons to configure Fetchmail, test Fetchmail, run Fetchmail, or quit.

NOTE

You can configure and run Fetchmail as root , but doing so offers no advantages over running the program as an ordinary user. Thus, to avoid the risk of damage should a security problem be found in Fetchmail, it's best to configure and run Fetchmail as an ordinary user. One ordinary user can retrieve mail intended for several different users, although in this case, the mail-retrieving user must have access to the passwords for all the pull mail accounts. This may be acceptable in some situations, but in others you may prefer to allow all users to create Fetchmail configurations for their own mail accounts.

2. Click the Configure Fetchmail button in the Fetchmail Launcher window. This action produces a Fetchmail Configurator dialog box that allows you to choose from novice or expert configuration modes. The former produces dialog boxes that contain a subset of the options in the latter. Although I don't emphasize the expert-only options in this procedure, I do describe this path so that you can see what some of these options are.

3. Click the Expert Configuration button in the Fetchmail Configurator dialog box. The result is the Fetchmail Expert Configurator dialog box shown in Figure 11.2. If you want to run Fetchmail in daemon mode, enter the number of seconds between mail checks in the Poll Interval field (for instance, 1200 for a check every 20 minutes). Leave this field at 0 if you intend to run Fetchmail in batch mode. The Postmaster field is the name of the local mail account that's to receive reports of problems with Fetchmail's activities. It defaults to the user who runs the program. You can leave most of the other options in the middle section of the window at their defaults, or click the upper Help button to learn more about them.

   Figure 11.2. The Fetchmail Expert Configurator dialog box lets you enter global options and name specific mail servers from which Fetchmail will retrieve mail.

   

4. The most important item in the Fetchmail Expert Configurator dialog box is the bottom panel, in which you enter the name of the mail server from which you want to retrieve mail. Type in the hostname and press the Enter key, and a new dialog box will appear called Fetchmail Host Hostname, as shown in Figure 11.3. The hostname should also appear in the scrollable list below the New Server data entry field in the Fetchmail Expert Configurator dialog box. If you want to retrieve mail from multiple servers, you may enter multiple server hostnames, although you may want to do so after configuring the first server.

   Figure 11.3. You enter most information about specific pull mail servers in the Fetchmail Host Hostname dialog box.

   

5. The most important sections of the Fetchmail Host Hostname dialog box are the Protocol, User Entries for Hostname, and Security sections. The Run Controls section sets options related to the timing of retrieval and the server name if it's different from what you specified earlier. The Multidrop Options area allows you to enter rules with which to duplicate or redirect mail based on mail headers that you specify. You can use this to let multiple users share a single pull mail account, although this use can result in problems with certain types of mail (such as mailing lists), and so is discouraged.

6. In the Protocol section of Fetchmail Host Hostname, you specify the pull mail protocol you want to use. Auto is the default, and works with some servers, but you may want to specify the protocol if you know what's supported on your server. You can click Probe for Supported Protocols to check the server for the protocols that Fetchmail supports, but this option doesn't always work correctly. If necessary, you can ask your ISP or try using telnet manually, as illustrated in Listings 11.1 and 11.2.

7. The Security section is particularly helpful if you use a dial-up network interface that's not always active. Enter an interface name, such as ppp0 , in the Interface to Monitor field to have Fetchmail poll a server only when that interface has been used by another program since the last poll. (This behavior prevents Fetchmail from keeping a link up unnecessarily if it would otherwise be brought down automatically.) The IP Range to Check Before Poll field can be used to check for the presence of an IP address on a given interface. Enter an interface name, IP address, and netmask , separated by slashes ( / ), and Fetchmail will run only if an IP address within the specified range and on the specified device exists. For instance, ppp0/172.20.0.0/255.255.0.0 requires that the computer have an address in the 172.20.0.0/16 network on ppp0 before Fetchmail will poll the host.

8. In the User Entries for Hostname section, enter your username on the mail server in the New User field and press the Enter key. This produces the Fetchmail User Username Querying Hostname dialog box shown in Figure 11.4. As when you enter a pull mail server name, you may finish with this dialog box and enter additional accounts to query on the same server.

   Figure 11.4. Many Fetchmail options apply to specific pull mail server accounts.

   

9. The most important item to specify in the Fetchmail User Username Querying Hostname dialog box is the Password field in the Authentication section. This information is required for retrieving mail from most servers. You should also be sure that the Local Names field lists all the local users who are to receive the mail retrieved from this account. The default is to use the same username locally as is used on the pull mail server, but you may need to change this. The Forwarding Options field lets you change the host that's to receive the mailthe default is to use the local mail system, but you can use Fetchmail to retrieve mail from one system and forward it to another. The Forwarding Options, Processing Options, and Resource Limits sections all let you set assorted options, most of which are self-explanatory, the rest of which are required only in some circumstances. When you first set up Fetchmail, you may want to select the Suppress Deletion of Messages After Reading option in Processing Options, so you can test Fetchmail without fear of losing mail. You'll want to remove this option once you're convinced Fetchmail is working, though. Remote Folders lets you set the names of IMAP folders that Fetchmail is to check in addition to the default INBOX .

WARNING

Fetchmail stores the mail retrieval password in cleartext in its configuration file, .fetchmailrc . Fetchmail refuses to run if the permissions are set more loosely than 0600 ( rw---- ), but this is still a potential security risk should there be a local security breach. At the very least, you should ensure that the passwords used for remote e-mail retrieval aren't used for any other purpose, including access to the same account on the local computer or on any other computer.

10. Click OK in the Fetchmail User Username Querying Hostname dialog box, then in Fetchmail Host Hostname. You can then click Save in the Fetchmail Expert Configurator dialog box to save your changes to .fetchmailrc .

11. To test your setup, click Test Fetchmail in the Fetchmail Launcher window. This runs Fetchmail with debugging options, so you can see the commands that Fetchmail sends to the pull and push mail servers, and the replies that Fetchmail receives in return. This information can be invaluable in tracking down problems. When your configuration works, exit from Fetchmail by clicking Quit.

WARNING

The Test Fetchmail button, although useful for debugging, does not preserve mail on your pull mail server. Thus, if there's a local configuration problem that causes Fetchmail to discard messages (for instance, if your local mail server isn't running or refuses delivery of the messages), you'll lose mail. This is why you may want to configure Fetchmail to not delete mail from the pull mail server in Step 9 for your initial tests.

For many people, Fetchmail configuration via fetchmailconf is quite adequate. Particularly in Expert mode, fetchmailconf can set the options you're most likely to need for retrieving mail from one or more mail accounts, and even for delivering mail from multiple pull mail accounts to multiple local users. Some people, though, prefer editing text files to using GUI configuration tools; or you might want to edit a text file to make a quick change or to create a particularly complex configuration. For whatever reason you want to do it, manual editing of .fetchmailrc is certainly possible.

Configuring .fetchmailrc

When you run fetchmailconf , as just described, the program converts your selections into entries in the .fetchmailrc file in your home directory. The structure of and options in this file therefore mirror those in fetchmailconf . Listing 11.3 shows a sample .fetchmailrc file.

Listing 11.3 A sample .fetchmailrc file

 # Fetchmail file for retrieving mail from mail.abigisp.net # and imap.asmallisp.com set postmaster rodsmith set bouncemail set daemon 1800 set syslog poll mail.abigisp.net with proto POP3     user rodericksmith there with password abc123     is rodsmith here fetchall forcecr     smtphost speaker.rodsbooks.com poll imap.asmallisp.com with proto IMAP     user rodsmith there with password A1B2C3     is rodsmith here 

As with many configuration files, a pound sign ( # ) denotes a comment, so Fetchmail ignores the first two lines. The remainder of Listing 11.3 is broken into two parts . First, a series of set statements set global options, most of which relate to features you can set in the Fetchmail Expert Configurator dialog box's Fetchmail Run Controls section (Figure 11.2) of fetchmailconf . Many of these options can be set or overridden using command-line arguments, as well. Second, Listing 11.3 includes two poll statements, each of which defines a remote e-mail account to be accessed, including how the mail retrieved from that account should be delivered. (This information appears in the Fetchmail Host Hostname and Fetchmail User Username Querying Hostname dialog boxes in fetchmailconf , as shown in Figures 11.3 and 11.4.) These poll statements may exceed one line in length, without using special characters to indicate a line continuation, as some other file formats require. Line breaks in Listing 11.3 are arbitrary.

Fetchmail supports a huge number of options, so I can't describe all of them here. For more information, consult the Fetchmail man page or other Fetchmail documentation. Some options take a string, such as a username, as an argument. If this string contains spaces, you should enclose it in quotes ( " ). Some of the more important global options include the following:

• set postmaster username This option sets the username used for mail delivery if other options fail to specify one. This user also receives certain types of error messages. It's normally an ordinary user, but you can use postmaster or root for username , if you like. (All SMTP mail servers are supposed to support a user called postmaster to handle queries concerning the computer's mail system. The Fetchmail postmaster may or may not be the same as the SMTP server's postmaster .) The --postmaster username command-line option overrides this configuration option.

• set bouncemail This option tells Fetchmail to send error messages to the mail's sender. You can alter this option by using set no bouncemail instead; in this case, the Fetchmail postmaster receives the error messages.

• set daemon interval This option tells Fetchmail to run in daemon mode, and to query mail servers every interval seconds. If you want to run Fetchmail in batch mode, omit this option. You can override this value with the --daemon interval command-line argument. Using the --daemon 0 command-line argument causes Fetchmail to do one fetch operation, even if it's configured to use daemon mode in .fetchmailrc .

• set logfile filename You can tell Fetchmail to log its activities to a particular file with this option.

• set syslog If you want Fetchmail to log its activities through the system logger, you can do so with this option.

The poll statements can be fairly simple or quite complex. The format of a poll statement is as follows:

 poll  server.name   server-options user-descriptions  

The keyword server is synonymous with poll . You can replace either of these with skip to tell Fetchmail to skip this entry. You can use this feature to easily disable an entry without deleting it from the .fetchmailrc file. The server options define how Fetchmail is to interact with the server, and the user descriptions define accounts on the server and locally. You can intermix options within each of these two categories, but not between them. (A common source of problems in manually configuring Fetchmail is to intermix these two option classes.) The words and , with , has , wants , and options are all ignored, as are the punctuation symbols colon ( : ), semicolon ( ; ), and comma ( , ). You can place these anywhere within the server options or user descriptions to make the poll entry easier to understand.

Some of the more common or important server options include the following:

• proto name or protocol name These synonymous options specify the pull mail protocol to be used. The name will most commonly be POP3 or IMAP , but Fetchmail supports several others, including some fairly obscure pull mail protocols. You can override this value with the -p command-line argument.

• interface interface / IP.address / net.mask You can specify a network interface that must be up before Fetchmail will poll a given server with this option. You specify the interface with a device name ( interface ), such as eth1 or ppp0 ; an IP address, such as 192.168.1.0 ; and a netmask that determines the range of acceptable IP addresses. For instance, interface eth1/192.168.1.0/255.255.255.0 specifies that the computer must have an address between 192.168.1.1 and 192.168.1.254 on eth1 before Fetchmail will try to contact the server. You can also provide this information with the -I command-line option.

• monitor interface This option causes Fetchmail in daemon mode to watch a network interface for activity. If Fetchmail detects no activity between the previous poll and a scheduled one, Fetchmail skips the poll. This behavior can keep Fetchmail from keeping an auto-dial device up with its own polling. You can use the -M command-line argument to provide this information, as well.

Some of the more common user options include the following:

• user name or username name These options usually mark the start of the user description portion of the poll option, and they specify a username. This is normally the remote username, but if the sequence is followed by here , the username is local. The keyword there forces interpretation of the user as a remote account. The -u command-line argument overrides this option.

• pass pw or password pw These options specify the password ( pw ) to be used on the pull mail account. You provide the password in cleartext (unencrypted), and it's stored that way in the configuration file.

• is name or to name These options link a pull account to a local user. One of these options normally appears after the description of the pull mail account (using a sequence such as user name with pass pw ). When the remote account description precedes the local account description, here follows these options to identify the account as being local. The keyword there identifies a remote account.

• smtphost host.name Normally, Fetchmail attempts to use the address localhost , which corresponds to the computer on which it runs, to send mail. You can have the program use another computer's mail server for sending mail on its way, though, by using this option. You can also use this option and specify your computer's regular hostname, in which case the addresses in your mail headers corresponding to Fetchmail transfers will include that hostname rather than localhost . The -S command-line argument overrides this setting.

• keep By default, Fetchmail deletes messages from the server once it's retrieved them. This option causes Fetchmail to leave messages. You might use this when testing a new configuration. The -k command-line argument is another way to specify this option.

• fetchall Normally, Fetchmail doesn't retrieve messages that it has already retrieved. You can force the program to retrieve all messages with this option, which has a command-line equivalent of -a .

• forcecr Technically, mail messages should have lines that end with a carriage return/line feed pair of characters. Most mail programs tolerate messages that lack the carriage return, and a few systems send such messages. The qmail push mail server can't tolerate such messages, though, so the forcecr option corrects matters for users of qmail.

When you specify more than one local name, Fetchmail examines the mail headers to try to determine who should receive the mail. For instance, if you specify local accounts of jack and jill , and if incoming mail is addressed to jill , Fetchmail delivers the mail to jill . This is multidrop mode, in which mail from a single account may be processed in different ways according to the headers in the mail.

TIP

For many additional options relating to local delivery of mail after Fetchmail has done its job, look into Procmail. This package is described in Chapter 19. You can use Procmail to help identify and delete unwanted messages, move incoming mail into mail folders according to type, and more.