Using Leafnode

Using Leafnode

INN is a tool that's used mainly by large ISPs or other organizations that want to provide full news feeds, or by organizations that want to provide limited local newsgroups in support of their own activities, but not full news feeds. As described earlier, though, in the section "When to Run a News Server," some people want to run local news servers in order to facilitate their own local news reading. Such a server should retrieve messages from just a few newsgroups, make them available whether or not the main Internet connection is active, and upload the user 's postings during the next round of message downloads. These scheduled message exchanges could take place just once or twice a day, at times when the news server is not heavily loaded or when a Point-to-Point Protocol (PPP) dialup connection is inexpensive or easy to obtain. In principle, INN could serve this function, but considered as a server for just a few local users reading a handful of newsgroups, INN is overkill. The program is also designed to work as a peer on the Usenet network, so chances are you'd have trouble finding an ISP that would agree to feed a limited number of newsgroups to you on your own schedule. For these reasons, other tools exist to help users perform offline news reading. One of the more popular of these is Leafnode (http://www.leafnode.org).

NOTE

Leafnode isn't alone in the class of NNTP servers for small or offline sites. Others that fill similar roles include NNTPCache (http://www.nntpcache.org), Noffle (http://noffle. sourceforge .net), sn (http://infa.abo.fi/~patrik/sn/) and NewsCache (http://www.infosys.tuwien.ac.at/NewsCache/).

Understanding Leafnode's Capabilities

Like INN, Leafnode is actually composed of several programs. The most important of these are:

• leafnode ” This is the NNTP server program. It's launched from a super server, as described in Chapter 4, and communicates with your news reader program on the same or a different computer.

• fetchnews ” This program, as its name implies, is responsible for retrieving news from your upstream news server. It also delivers your posts to the upstream news server. You can run it as a cron job to retrieve news on a regular basis, run it in a PPP dialup script, or run it manually.

• texpire ” Just like a normal news server, Leafnode retains news messages in subdirectories of /var/spool/news . Because of this, it's necessary for Leafnode to delete old messages, lest your hard disk fill up. Doing this is texpire 's job. It's normally run as a cron job on a daily basis.

• newsq ” This is a local information display program; it reports a summary of news articles that have been posted locally but not yet uploaded to your upstream news server.

Leafnode is designed for dynamic newsgroup configuration. If a user tries to start reading a newsgroup, Leafnode will begin fetching that newsgroup with the next run of fetchnews . If Leafnode detects no attempt to access a newsgroup for some period (a week, by default), then it stops retrieving the group. This means that you can provide the illusion that you're carrying all the newsgroups available on your upstream news provider without actually doing so. Your local users will, however, experience a delay of one fetch cycle between attempting to access a new group and seeing the posts that are available in it.

One of the more unusual characteristics of Leafnode is that it doesn't require a full news feed arrangement with the upstream provider; Leaf node (or fetchnews , to be precise) presents itself as an ordinary news reader to the feed site. This means that you can use Leafnode in conjunction with your ISP's ordinary news server.

NOTE

Leafnode uses both greater and fewer resources than an ordinary news reader. Fetching all the posts in a typical text newsgroup will probably consume far less time than would a news-reading session in which you're constantly connected to the remote server. This can reduce consumption of connections on the server (which are sometimes limited), as well as your online connect time. On the other hand, this process consumes far more bandwidth, in terms of bytes transferred, than would a typical online reading session, since Leafnode downloads all the articles ”even those you ordinarily wouldn't read. If Leafnode serves several users who normally read much the same set of newsgroups, though, the total amount of data transferred might not be more with Leafnode than with individual direct connections.

In early 2002, the current version of Leafnode is 1.9.19. Work is underway on a 2.0 release, which will add support for local newsgroups, making Leafnode suitable for running a small news server for local newsgroups. The 1.9. x versions don't support this feature, though.

It's important to remember that Leafnode was designed as a news reader for small sites. It doesn't scale well, meaning that as you add users and large numbers of newsgroups, Leafnode's performance suffers. Leafnode works well for up to a few dozen clients , but it's not very useable beyond that point. If you're experiencing performance problems that are caused by a heavy server load, you should seriously consider switching to INN or some other conventional news server and obtaining a full news feed.

Another problem with Leafnode is that it may drop messages if it encounters a problem. This may result in posts that aren't posted to Usenet, or in existing Usenet posts that aren't readable. Some fetchnews options attempt to work around some of these problems, as described shortly, but you may lose posts if your connection to your upstream news server is unreliable.

Configuring Leafnode

Configuring Leafnode requires setting up three programs: The leafnode server itself, the fetchnews program, and texpire to expire old news. Functions relating to all three programs are contained in a general configuration file, but you'll need to set up each program to run in its own way. If you use a Leafnode package from your Linux distribution, you shouldn't need to edit many files to get it all running.

General Configuration Settings

The main Leafnode configuration file is called config , and it's usually stored in /etc/leafnode . Aside from comments, which begin with a pound sign ( # ), this file consists of lines of the following form:

  parameter  =  value  

A minimal Leafnode configuration needs to set just a couple of parameters: server and expire . Other parameters are optional; they take on reasonable default values if you don't set them. The more important parameters are as follows :

• server ” This parameter sets the name of your upstream news server, as in server = news.abigisp.net . You can configure Leafnode to fetch news from multiple news servers by including more than one server line.

• expire ” This parameter sets the number of days Leafnode retains messages before deleting them.

• username ” If your upstream news server requires you to enter a username, specify it with this parameter.

• password ” If you need a password to access your upstream news server, enter it with this parameter.

WARNING

Your password is stored in a plain text file, and so could potentially be stolen. The default permissions on the config file in most distributions make it accessible only to root , so the risk is minimized. It is also sent unencrypted to the news server. If possible, you shouldn't use the password for anything except access to your news server.

• port ” Most news servers operate on port 119 (the default), but you can specify another port with this parameter.

• nodesc ” Most news servers provide a newsgroup descriptions file when asked for it. A few don't, though. Leafnode works best with these when you use the nodesc = 1 parameter.

• timeout ” When connecting to a news server, fetchnews normally waits ten seconds, then stops trying. You can set another timeout interval with this parameter.

• groupexpire group.name ” If you want to set different article retention intervals for different newsgroups, you can adjust the value set via expire for specific groups or collections of groups with this parameter. The group.name may contain wildcards, such as comp.os.linux.* for the entire comp.os.linux hierarchy.

• maxfetch ” Leafnode limits the number of new articles it will fetch from any one group with this option. Setting this value too low can cause problems, because it might regularly fail to fetch all the new articles, and thus fall behind the current postings.

• initialfetch ” You can set a lower number of articles to fetch for a newly-subscribed group with this parameter, the idea being that you might not want to fetch all the posts in a new group from a server with retention of several days because this might take a long time.

• delaybody ” Leafnode normally fetches all new message headers and message bodies with each run, within constraints such as those set by maxfetch and other options. You can have Leafnode fetch only headers, however, and then fetch message bodies only after a user has clicked on a message in a news reader, indicating a desire to read the message. When an unfetched message is read, the message body indicates that Leafnode has marked it for download. After the next fetch, the message title appears again as an unread message, which then contains the true message. Setting this parameter to 1 will delay your ability to read new messages, but this option can greatly reduce the amount of data that must be transferred.

• maxcrosspost ” Leafnode offers this option as an anti-spam measure. It kills messages that have been cross-posted to more than the specified number of groups. The default is to accept unlimited cross-posting.

• maxage ” Occasionally, misconfigured news servers repeatedly bounce the same messages onto Usenet, resulting in massive numbers of repeat postings. Using this parameter causes Leafnode to ignore messages that are more than a specified number of days old. This might limit the severity of such problems. The default is to not use this criterion.

• maxlines ” You can have Leafnode reject messages that have more than the specified number of lines. By default, this limit isn't used.

• minlines ” You can have Leafnode reject messages that have fewer than the specified number of lines. By default, this limit isn't used.

• maxbytes ” You can have Leafnode reject messages that have more than the specified number of bytes. By default, this limit isn't used.

• timeout_short ” By default, Leafnode retrieves posts for two days after a newsgroup has been opened accidentally . You can change this value with this parameter.

• timeout_long ” By default, Leafnode retrieves posts for seven days after a newsgroup is no longer being read. You can change this value with this parameter.

• timeout_active ” Leafnode periodically refreshes its list of available newsgroups from the upstream server. This parameter sets how frequently this occurs, in days. The default is 90 days.

• filterfile ” This parameter sets the path to the filter file, described in the upcoming section, "Filtering Articles." The default is not to use a filter file.

• hostname ” Most news readers create message IDs for their messages, but if they don't, Leafnode does it for them, using the hostname of the computer on which it runs. If this hostname is inappropriate, you can change the value with this parameter.

These parameters set options that relate to all three of the major Leafnode component programs ( leafnode , fetchnews , and texpire ). These programs don't have separate configuration files, but they must be run in different ways.

Setting Up the Server

As mentioned earlier, the leafnode server program is designed to be run from a super server, such as inetd or xinetd . This topic is covered in Chapter 4. A typical inetd.conf entry for Leafnode resembles the following:

 nntp stream tcp nowait news /usr/sbin/tcpd /usr/sbin/leafnode 

Distributions that use xinetd usually include an appropriate startup file in their Leafnode packages that go in /etc/xinetd.d , so you shouldn't need to create such a file. You might need to enable it, though. Whether you use inetd or xinetd , chances are you'll need to restart the super server before your Leafnode server will become available to your clients. Once you've done this, the Leafnode server computer will respond to requests from news readers as if it were running INN or some other full news server package.

WARNING

Leafnode's main configuration file doesn't include access-control features. You should add such controls to your xinetd or TCP Wrappers configuration to prevent unwanted access to your Leafnode system.

Fetching News

A single run of the fetchnews program will retrieve news from your upstream news server, assuming you've configured the appropriate news server hostname in /etc/leafnode/config . The same process uploads any pending messages to your upstream news server. (You can use the newsq command to see how many outgoing messages you have waiting, as well as their titles, newsgroups, and senders.) The first time you run fetchnews , chances are the operation will take some time, as the program will fetch the active newsgroups list from the upstream server.

The fetchnews program accepts several arguments, which are as follows:

• -v ” This option increases the verbosity of the program's output. You can increase the number of v s, up to four (as in -vvvv ) to further increase the verbosity of output. This can be a useful diagnostic tool when fetchnews isn't working as you expect.

• -x number ” If you ran into a problem on the previous fetch operation, this option will attempt to re-fetch articles up to number back.

• -l ” As noted earlier, Leafnode supports fetching from multiple servers. Using this option forces it to fetch news from just the first one.

• -n ” This option causes Leafnode to not automatically unsubscribe from newsgroups that aren't being read.

• -f ” If your newsgroup list seems stale, you can force fetchnews to update the list using this option. (The program does so automatically every 90 days, by default.) This operation can take some time, though.

• -P ” This option causes fetchnews to post articles it's accumulated in its outgoing queue without retrieving new ones.

TIP

Normally, you won't see your own posts in your Leafnode newsgroups until you've called fetchnews twice. If you call fetchnews with its -P option prior to calling fetchnews , your posts will appear as soon as you make this dual call. The Leafnode 2.0 beta doesn't suffer from the delayed appearance problem, so the workaround has no effect.

The real trick to fetchnews operation is in determining when and how to run it. Two common options are to run it as a cron job and to run it in a PPP dialup script, such as ppp-on-dialer (described in Chapter 2, TCP/IP Network Configuration). Running fetchnews as a cron job makes sense when you have a connection that's constantly active, or if you want to have your computer automatically dial out and retrieve news when unattended, such as early in the morning. How often you should do so is a matter of personal preference and practical matters specific to your configuration and your upstream provider's availability. Running fetchnews as part of a PPP connect script ensures that you'll be able to read the latest news soon after you connect to the Internet.

Expiring Old News

The texpire program peruses the news messages stored in the news spool directory and deletes old messages according to default values or those set in /etc/leafnode/config . This task should be performed on a regular basis to ensure that your hard disk doesn't overflow with old news postings. The usual way to accomplish this task is to run texpire from a cron job on a daily basis. Some distributions' Leafnode packages include an appropriate script that's dropped in /etc/cron.daily or some other location, but you shouldn't assume that your distribution maintainer has seen to this detail; you should check for such a script and, if necessary, create one or use the crontab utility to do the job.

When texpire runs, it expires articles based on threads. (A thread is an original posting and associated replies.) Specifically, texpire deletes a message only if the thread hasn't been read by local users in more than the expire time. Thus, articles may be kept around for longer than you might expect, if the thread in which they reside has been recently read by Leafnode clients.

Like fetchnews , texpire accepts one to four -v options to increase its verbosity. Another potentially important texpire option is -f . Ordinarily, texpire relies upon the last-access dates that Linux stores with files to determine when a thread has gone unread. Adding -f causes texpire to ignore this information. You might do this if you regularly back up your news spool directory with a program like tar , which modifies the last-access date as part of its operation, causing texpire to erroneously believe that messages are being read when they aren't.

Filtering Articles

Leafnode includes the capability to delete messages that match arbitrary criteria based on the contents of the messages' headers. For instance, suppose you read a newsgroup that's regularly disrupted by a person who posts from the account [e-mail] . This username will appear in the From header, so Leafnode can filter messages based on this information. To tell it to do so, you must edit a filter file, such as /etc/leafnode/filters , which contains filtering information in regular expression form. For instance, to remove posts from [e-mail] , you might include a line like the following:

 ^From:.*obnoxious@annoying\.edu 

Each expression in this file begins with a carat ( ^ ), then the name of the header line that's to be filtered ( From: , in this case). The period and asterisk characters ( .* ) indicate that an arbitrary number of characters may intervene. The string obnoxious@annoying can be represented directly, and so is; but the period has special meaning, so the period in .edu must be escaped with a backslash ( \ ) character.

NOTE

This description barely scratches the surface of what regular expressions can do. The "Using a Procmail Filter" section of Chapter 19, Push Mail Protocol: SMTP, provides more information on regular expressions.

You must tell Leafnode where to find the filter file you use. You do this with the filterfile parameter in /etc/leafnode/config , as described earlier. Although the filter file often goes in /etc/leafnode/filters , you can select any other filename you like.