A.5 Ongoing Operations

only for RuBoard - do not distribute or recompile

Before Vineyard.NET, I always handled the security of my own computer system with a few simple and reliable policies: don't run any services that might be unsecure; don't give out accounts to people you don't know; disconnect your computer from the Internet when you are not around. The monitoring that I did, insofar as I did any, was haphazard.

Those techniques wouldn't work for a commercial service such as Vineyard.NET. Instead, I needed to design and deploy a system that could be readily maintained, defended, and scaled.

A.5.1 Security Concerns

From the start, Vineyard.NET's security was one of my primary concerns. As the coauthor of several books on computer security, I knew any Internet service that I was involved with might be a target.

But there were other reasons that I was concerned about security. Vineyard.NET was a company that depended on computers that were connected to the Internet. If these computers were broken into and compromised, our reputation would be damaged, we would lose customers, and we would lose time required to put our systems back in order. We might lose so much in the way of customers, reputation, and time, that we might even go out of business.

Because of these problems, we followed a few simple rules for our system and networks: minimize vulnerabilities and plan for break-ins.

A.5.1.1 Lesson: Don't run programs with a history of security problems.

From the beginning, we avoided running programs that had a history of security problems. The primary offender here is sendmail, the Unix SMTP server and mail delivery agent. sendmail is an especially dangerous program because it runs as the superuser, has full access to the Unix filesystem, implements a complicated command language, and talks to any other computer on the Internet.

Organizations like the CERT/CC are continually sending out notices of new security problems discovered with sendmail. People are advised to download upgrades immediately from their vendors, if possible. Vineyard.NET ran the BSDI operating system, BSD/OS, and BSDI was very good about responding to security problems in a timely manner. But a much better way to deal with the problem is to use smap, a sendmail proxy, so that programs on the outside world never connect directly to our sendmail program.

Over time, we customized smap, adding features to deal with Vineyard.NET's authentication system, and then features to deal with spam. Somewhere in there, though, we ran into problems and smap started losing mail. Although we tried to find what was wrong with our fixes, in the end we had to abandon the smap/sendmail combination and move over to a different mailer. (We checked out both postfix and qmail.)

A.5.1.2 Lesson: Make frequent backups.

Another thing we did to protect ourselves was to make frequent backups of our system. These backups protected us against both break-ins and accidents on the part of myself or Eric.

At Vineyard.NET we make two kinds of backups. The first are tape backups. We started out using the GNU tar facility, but switched over to Unix dump because it ran at nearly twice the speed. Then we switched back to GNU tar, because the dump manpages were filled with warnings about the dangers of backing up an active filesystem. Vineyard.NET's disks are always active.

Each day we make a full dump of every file on the system. To accommodate all of the files, we got a DDS-II DAT tape drive. That held 8 GB with compression, and, as we had only 6 GB online, things were okay for a while. Our tapes are labeled Monday, Tuesday, Wednesday, and so on. At the end of each month, we make a "permanent" backup that is taken offsite. (Eventually we had to get a DDS-III drive, then a DDS-IV drive, as we bought more disks. We always tried to keep one tape per disk for backups.)

We also make daily backups of the entire /usr/local/etc/httpd/htdocs directory. These backups are kept online in a single compressed tar file. The backups are designed to protect us from accidentally deleting a file.

It turns out that we've used all of the backups with more or less successful results. The backups have been quite useful when files have been accidentally deleted. The only time we had a problem that our backups couldn't solve was when we were changing a disk formatting parameter on a disk drive that was being used to store Usenet files. We backed up the entire filesystem to the DAT drive twice, reformatted the disk, and then tried to restore it. It wouldn't. It turns out the Berkeley restore command has some hardcoded limits inside it, and it simply couldn't restore 100,000+ files at once. The result: we lost all of our netnews. In retrospect, it was no big deal.

A.5.1.3 Lesson: Limit logins to your servers.

From the beginning, I decided that Vineyard.NET would not offer "shell" accounts that is, accounts that allowed people to log into our Unix server. We also resolved that all administration of the web server would be done through the FTP command. Although this was a little awkward for some people, and I think it cost us a few customers, in the end it was the right decision. It is far easier to run a Unix machine securely if you do not allow people to log into it.

A.5.1.4 Lesson: Beware of TCP/IP spoofing.

TCP/IP spoofing is a rather arcane attack, but it is one that has been successful in at least one widely publicized attack. To protect ourselves against it, we configured our external routers to reject any incoming packets that indicated they were being sent from our internal network.

A.5.1.5 Lesson: Defeat packet sniffing.

As soon as we could, Eric and I set up ssh, the secure shell, so we could log on from one computer to another with the knowledge that our password was not being sent over the network. We also required that friends on the Internet who wanted guest accounts on our machines use ssh to access those accounts as well.

As for our home machines, we installed Kerberos on our BSDI box and started using a Kerberized-version of the Macintosh Telnet program. About a year later, we switched to a commercial version of ssh sold by Data Fellows. The program was terrible, but it was the only thing available. When Van Dyke came out with SecureCRT, we switched to that on our PCs; by then, we had stopped using the Macs.

A.5.1.6 Lesson: Restrict logins.

We configured our Cisco routers so that they could only be logged into from a particular computer on our internal network. Although this slightly increased the difficulty of our administering the system, it dramatically improved the security of the routers. Prior to the configuration change, a person could have broken into our router by trying to guess thousands or millions of passwords. (Cisco's operating system makes this procedure easier than that statement makes it sound.) After the change, the only way to break into the routers was first to identify and break into the one host on the subnet that could telnet to the routers. Still, it would have been better if Cisco had simply added support for Kerberos, S/Key, or ssh into their products. (Note that Cisco has since added support for some of these protocols.)

A.5.1.7 Lesson: Tighten up your system beyond manufacturer recommendations.

In the interest of improved security, we started reducing the access permissions of many directories on our Unix system. Directories that contained programs were changed from mode 755 (read/write access for the superuser, read and chdir access for everybody else) to mode 711 (read/write access for the superuser, but only search chdir for others). Configuration files were changed from mode 644 (world-readable access) to mode 600 (access only for superuser). The idea of reducing permissions in this manner was to make it harder for somebody who had an account on the system to use that account to probe for further weaknesses.

The revised permissions also protected the privacy of our users. For example, with the original permissions, it was possible for a user to open an FTP connector to our server and list the contents of the /users directory, effectively giving that user a list of all our customers' email addresses. With the revised permissions, this was no longer possible.

One day we noticed that somebody named http had logged into our system and was running an xterm, with the DISPLAY variable set to be somewhere in Texas. Apparently the person had broken into our main computer through the web server and had convinced the program to run an X terminal of his choosing. We immediately modified our firewall to block all packets to or from the attacker's ISP, then used the gcore command to look at the memory map of the attacker's shell. This gave us, among other things, a list of every command that the attacker had typed.

We noticed that the attacker had spent two hours uploading various small programs to our computer, compiling them, and then running the programs with various command-line arguments like "./a 72", "./a 80", "./a 432". We surmised that the attacker had uploaded programs that probe for buffer-overflow exploits and had then tried them with the proper offsets for various operating systems. Poor guy! He thought he had broken into a Linux box, and he was actually on a BSD/OS system. After four hours, he had given up.

We were pleased that he hadn't been able to able to ratchet up his permissions and turn a http login into a superuser access. Clearly, something about our security worked. But we shouldn't have made it so easy for him. The next day, we modified our system further, placing the compiler, the /usr/include files, the linkable libraries, and all of the other program development tools into a separate group. People who weren't in the group would be unable to run those programs.

A.5.1.8 Lesson: Remember, the "free" in "free software" refers to "freedom."

Although I am an advocate of free software, there were many times in the development of Vineyard.NET that we went with commercially available versions of free programs. For example, we chose to use BSD/OS, a commercial operating system from BSDI, rather than using FreeBSD, NetBSD, OpenBSD, or Linux. We wanted BSDI to track security problems for us and provide us with bug fixes, rather than having to do it ourselves. Likewise, we chose to purchase our web server from Community ConneXion, rather than simply using the freely available Apache web server. Yes, our primary reason for purchasing the Community ConneXion server was to obtain an SSL-enabled web server. But we were also eager to have somebody else tracking reported security problems with the web server and providing us with patches.

The "free" in "free software" refers to freedom, not to price. The actual price that you pay for a program frequently is only a tiny part of your total investment. This became clear to us when we spent many hours trying to get both "free" and commercial software to work. Frequently with the commercial software, we were able to get a service contract and get somebody to help us. The same was not always true with the free stuff.

On the other hand, we were more than happy to use free software on occasions where it was clearly better than the commercial alternative. We replaced the Perl 5 interpreter that came with BSDI with one that we downloaded from the Internet because BSDI's version of Perl 5 was out of date. We used GNU emacs as our text editor and GCC as our compiler, rather than purchasing expensive development tools that might not have worked as well.

As time progressed in the history of Vineyard.NET, our attitude towards free versus commercial software changed. We slowly phased out the BSD/OS operating system in favor of FreeBSD, and we switched from Stronghold to Apache with mod_ssl. In contrast to others who made similar switches in the late 1990s, the reason for our switch didn't have to do with the apparent quality of the free software, but instead with the drop in both quality and service of the commercial vendors. As both FreeBSD and BSD/OS matured, we discovered that there were many features in FreeBSD that were simply unavailable in BSD/OS. Likewise, the version of Apache that C2 software distributed fell significantly behind the version of Apache that was being distributed from www.apache.org. We left the commercial vendors because we wanted advanced features that they were not offering.

A.5.2 Phone Configuration and Billing Problems

We had continual (and continuing) problems with the configuration of our telephone lines. Prudent security practices dictate that lines used for incoming telephone calls should not be able to make outgoing calls and they certainly should not be able to place long distance calls. Otherwise, somebody who breaks into your computer might use the ability to place long distance calls to charge tens of thousands of dollars in phone calls to your account.

It turns out that Vineyard.NET is particularly vulnerable to this sort of toll fraud, because we have placed Centrex lines in locations such as schools and libraries, so these organizations could call the Vineyard.NET phone lines without having to pay message units. Every few months, we notice between $10 and $30 of phone calls on these lines to other phone numbers.

We have repeatedly asked our phone company, NYNEX, to disable 9+ dialing on both the dialup telephone lines and the phone lines that are located on customer premises. Each time we've been assured by the phone company that the problem has been corrected, yet every time we check, we discover that the phones still can dial local and long distance calls.

Frankly, this is still an unsolved problem. Right now, I'm trying to get a letter from NYNEX saying that our lines cannot place outgoing calls. Then I'm going to wait for another $100 phone bill and refuse to pay it.

Another problem that we had with NYNEX is billing disputes. For several months we were not billed for one of our high-speed serial lines. Then we were double-billed. Payments made were not credited. We were billed for lines after they were disconnected. New phones that were installed were not billed for many months, then we had months of back invoices suddenly posted to our account. And we have never been able to decipher our Centrex bills to our satisfaction.

After having our office manager spend more than 100 hours trying to resolve these problems with NYNEX, we came to the realization that some amount of incorrect billing is probably unavoidable when dealing with NYNEX. So we are careful about what charges we dispute. Some are challenged right away. Others we let slide, because there are only so many hours in the day. (If our billing is that far off, we wonder what it is like for more substantial commercial firms?)

A.5.3 Credit Cards and ACH

One of the biggest problems for any business is getting paid; after we had been operating for a few months, we had more than 200 customers and we were owed more than $10,000 in late fees. We didn't want to make things worse by charging people finance charges. We just wanted our money.

It turned out that a lot of our customers wanted to pay us, they were simply lazy. Writing a check is, after all, a pain. Receiving them was difficult, too; entering each check into the computer, putting it on a deposit ticket, and taking it to the bank all took a considerable amount of time.

Fortunately, we found our savior: a technique for billing people's credit cards and for withdrawing money directly from people's credit cards using a system called ACH (the U.S. Automated Clearing House).

These days, it's quite easy for a legitimate merchant to get a credit card merchant account. You go to your bank, fill out some forms, and wait. A few days later you'll get a phone call from an investigator, who wants to stop by your place of business and make sure that you are legitimate. If that goes well, and it almost always does, you get a machine.

I had a credit card merchant account once before, when I was running a company that was selling computer software. But having to run people's credit card numbers through a machine and having to do it every month is a real pain. We probably would have had to hire somebody to do the job, and that would have cost time and money.

Instead I started looking around for software that could charge people's credit cards automatically, and submit all of the charges by modem. My bank gave me a list of companies that sell this software. Most of it runs on PCs, some of it runs on Macs, and perhaps one or two programs run on a Unix-based system. But nothing was quite right.

Then, by chance, I visited a trade show and found out about CheckFree's merchant services. Based in Ohio, CheckFree has made a name for itself in the wireless bill payment business. In fact, I've used CheckFree for years to pay my various bills. I simply type all of the bills into Quicken, type a code, and my computer makes a phone call. CheckFree electronically removes the money from my account and transfers it into my merchant's. If the merchant isn't set up for ACH, CheckFree can write a check on my account. If several CheckFree customers try to pay the same merchant during the same payment period, CheckFree may transfer the money out of my account and into a CheckFree account, then send the merchant a single check for all of the CheckFree customers (with an accompanying statement indicating who paid what).

It turned out that CheckFree also had merchant services and could process credit card transactions for us. Also, CheckFree could do Merchant ACH: we could type in the checking account numbers of our customers and pull the money out directly. Even better, Checkfree had set up a system that allowed merchants to submit charges over the Internet encrypted using PGP.

Clearly, CheckFree's Gateway system was exactly what Vineyard.NET had been looking for. It would allow us to take credit card numbers on our web server, store them online, and then automatically use credit cards and ACH charges to settle our customer's accounts. Unfortunately, when I asked CheckFree to send me a program that implemented their protocols, they sent me a three-ring binder instead. There was no off-the-shelf code that they could provide me.

A.5.3.1 Lesson: If you have the time to write it, custom software always works better than what you can get off the shelf.

Over the next four months, I wrote a complete implementation of CheckFree's Gateway 3.0 protocols. Although it was a fun experience and I learned a lot about Perl, bank card processing, and the intricacies of the nation's ACH networks, I wished that I had been able to buy off-the-shelf the programs that I was looking for. In fact, there was a freelance computer consultant who sold such a package, but he wanted $1,000 for the software. It was all written in C and I didn't feel comfortable working with the person. So I ended up writing it all myself. Doing this had the added advantage of teaching me a lot about how credit cards are processed, which was useful for writing Chapter 25 of this book.

A.5.3.2 Lesson: Live credit card numbers are dangerous.

One of the scariest things was working with real live credit card numbers. If we screwed up, we might charge somebody too much money. If we got a sign reversed, we might transfer money from our account into a customer's, rather than the other way around. If we had a break-in, somebody might steal the credit card numbers of our customers, and we could be liable. For the first six months, I went over every credit card batch for half an hour before submitting it to our bank.

A.5.3.3 Lesson: Encrypt sensitive information and be careful with your decryption keys.

We tried to protect the live credit card numbers the best we could. Perhaps the most important thing we did was to store them encrypted on our web server. To do the encryption, we used PGP. The credit card encryption system has a special public key/private key. The web server and CGI scripts know the public key, but not the secret key. So when people enter their credit card numbers, the numbers are encrypted with the public key. The numbers are only decrypted when an actual billing run happens. We do this once or twice a month. To initiate the billing run, either Eric or I must type the PGP passphrase that decrypts the secret key that, in turn, decrypts the credit card numbers.

A.5.3.4 Lesson: Log everything, and have lots of reports.

As an added measure of protection, we set up numerous logs and reports that would be created every time a credit card run was processed. Not only did this help us in writing and debugging the software, it also made it easier for us to track down problems when they occurred.

Each time the software runs, Eric and I get sent numerous reports that are issued from CheckFree. We're used to seeing this sequence of reports, and when we don't get them, we know that there is something amiss. The reports have also notified us when the other person runs the billing system.

A.5.3.5 Lesson: Explore a variety of payment systems.

When customers pay by credit cards, merchants are assessed a surcharge between two and three percent. (Don't think about passing this surcharge on to your customers: that's a violation of your merchant agreement.) Still, despite this surcharge, it's often cheaper for businesses to accept payment by credit cards than by check or cash. That's because it's difficult, and therefore expensive, to handle large amounts of cash or large numbers of checks. And when customers pay by cash or check, there's always a chance that something will go wrong: the check will bounce, or it will get credited to the wrong account, or you'll simply lose the money. (All of these have happened to us at Vineyard.NET.) None of these problems happen when you are charging somebody a monthly fee for a credit card that you've got on file. A further advantage is that you are immediately told if the charge is approved or not. If the charge is approved, it's nearly a sure thing that you will get the money.

We prefer ACH to credit cards. Instead of being charged a fee between two and three percent, we are charged a flat fee of roughly 27 cents. But ACH is not without its problems. Transactions can fail for a variety of reasons, and sometimes we don't hear about failed transactions for up to 10 days. Our ACH system requires more by-hand intervention than our credit card system. Sometimes we receive a Notification Of Change (NOC) on ACH, which means that we have to change somebody's account number. Other times a NOC means that the account has been deleted.

Our enthusiasm for these new electronic payment systems was not shared by our customers. After all, they had been able to pay by check whenever they wanted and not pay when they didn't. We had essentially been giving our customers zero percent interest loans on the overdue balances.

A.5.3.6 Lesson: Make it easy for your customers to save you money.

After nearly eight months of asking our customers to pay by ACH because it saved us money, we hit upon a better approach. We started giving a $1 monthly discount for customers who paid by credit card, and a $2 monthly discount for those who paid by ACH. Soon customers were calling us up, volunteering their credit card numbers and their bank account information. They thought they were getting a terrific deal. As far as we were concerned, the $2/month discount was a lot cheaper for us than typing the checks into two accounting systems, taking the checks to the bank, and hounding customers who forgot to make their payments.

A.5.3.7 Lesson: Have a backup supplier.

Eric and I were so pleased with the CheckFree system, and so hard-pressed for time, that we never seriously explored getting an alternative credit card processor. After all, why look for more work?

Then one day, we got a notice from CheckFree that they were selling their merchant card systems to FirstUSA Paymentech. CheckFree promised us that FirstUSA would continue to offer all of the same services that CheckFree had in the past, but shortly after the acquisition was completed, we received a letter in the mail that said that FirstUSA would no longer be offering the Gateway 3.0 features. We had the choice of moving to a few other systems, including CyberCash and IC Verify. If we wanted to keep using ACH, we had no choice but to move to CyberCash.

Grudgingly, we downloaded the CyberCash software and rewrote our processing system to use the CyberCash gateway. Immediately we had problems with documentation that was incomplete or did not accurately reflect the software that we had downloaded. We were halfway through implementing the system when we discovered that CyberCash had no way to automatically handle charge-backs. We called up CyberCash and asked them what the system did when a consumer reversed a credit card charge. "Oh, somebody will send you a fax or give you a phone call," we were told.

Once we had the CyberCash system finally working, we discovered a serious limitation: we wanted to process transactions in batch, but the CyberCash system couldn't handle batch sizes of more than a few dozen cards FirstUSA Paymentech took so long to process the cards under these circumstances that the CyberCash software actually timed out. We started lowering the batch size first to 100 transactions, then to 50, and eventually to 20. Months later, Paymentech complained to us that we were sending too many batches through and demanded that we increase our batch size, even though that made the system less reliable.

Working with both CyberCash and Paymentech proved to be problematic at best. For every transaction, we would create a unique transaction ID and give it to CyberCash, which forwarded it to Paymentech. But Paymentech didn't use this CyberCash ID: they created their own. Tracking transactions through the system thus became more complicated; it was an incredibly difficult and very manual exercise every time we got one of those faxes from Paymentech telling us that a credit card had been declined or reversed to actually figure out which customer had initiated the charge-back.

Over time we discovered another problem with the CyberCash system. The software wasn't stable. In an effort to deal with its many problems, CyberCash was continually bringing out new releases to its software. That's good. Unfortunately, many of these releases broke our existing code. Every so often, we would get an email from CyberCash about a mandatory upgrade. We ignored those emails at our peril.

Then on January 1, 2000, we discovered that we were unable to process credit cards. We called up FirstUSA; they denied that they had a Y2K problem. We called up CyberCash; they said the same thing. We decided to wait for a week or so to see if the problem would resolve itself, but of course, it did not. When we finally started going through our code line by line, we discovered that CyberCash had made an undocumented change to their API on January 1. When we called CyberCash to complain, they denied it. But their own code proved otherwise.

A.5.4 Monitoring Software

Vineyard.NET had been operational as a commercial service for about two weeks when I started wondering if we had enough phone lines. Unfortunately, there wasn't any way to know for sure: neither our Unix operating system nor our Cisco routers came with monitoring software that would accurately report usage over time.

One way to plan for capacity is simply to count how many users you have and make an educated guess about how many resources each person requires. For instance, one rule of thumb in the ISP industry is that you need 1 dialup phone line for every 10 customers, for a ratio of 1:10. But another rule of thumb is that you need 1 modem for every 20 customers. And some ISPs have 1 for every 6.

There is not much in the way of good hard data behind these numbers. It depends on a lot of things, such as whether your customers are dialing in from work or from home, what time of the year they are dialing in (usage is higher in the winter, when the nights are longer), whether your customers are married, whether they have children, and whether there is anything especially interesting on the Internet. We might have wanted a lower ratio because there isn't a whole lot to do on Martha's Vineyard after the sun goes down, but have a higher ratio because a lot of the computer users on the island aren't very sophisticated and thus aren't likely to stay online for extended periods of time. We saw a strong correlation between daylight savings time and modem usage; in the spring, when clocks were set forward an hour, our modem usage dropped dramatically. We guessed that people had better things to do than go online.

Another approach is simply to buy more phone lines when people start getting busy signals. But adding capacity isn't cheap or fast. The phone company charges us $50 to install each new phone line. Off-the-shelf 28.8 modems can be had for as low as $160 today, but if you don't want a maintenance nightmare you'll buy rack-mounted modems, which cost $600 for the rack (holds 16) and $240 for each modem. Connected to the modems you need a terminal concentrator such as a Livingston Portmaster or a Cisco 2511. We had more experience with the Cisco, so we bought one of those for $3000; it can handle 16 modems at a time. Crunch these numbers and you get a total cost of $515 to install a new port. Then there is the monthly cost of $20 per phone line. And every time you add another phone line, you increase the amount of capacity that's needed for the ISP's connection to the rest of the Internet.

So there is this constant dance between wanting to expand facilities to give better performance and the problem that expanding facilities makes the whole enterprise more expensive to operate. This is the difficulty of managing growth. It's one of the hardest things for a company to do properly.

A.5.4.1 Lesson: Monitor your system.

It turns out that the only sensible way to gauge your capacity is to monitor how much you are using and make your decisions accordingly. We ended up developing a number of reports to do this.

The first report is a modem report. This report is sent nightly and includes the number of users who dialed up during the last 24-hour period, the number of calls per modem, a histogram of the number of users logged on at a time, and a hourly graph that shows the number of people logged on by time of day. This report is built by examining the login/logout records.

The second report we developed shows the usage of our high-speed connection to the Internet. This report is built by querying our external router every 5 minutes and recording the results.

When we started looking at our reports over time, we were somewhat surprised by the results:

Dialup usage was steadily increasing, but not because people were staying on significantly longer. Instead, people were calling up more frequently and staying on for roughly the same amount of time.
Peak usage dropped considerably after daylight savings time kicked in.
The need for bandwidth may be illusory; we ran an ISP with 32 people dialed up simultaneously and never came near to saturating our connection to the outside Internet.

Example A-1 shows what one of Vineyard.NET's early daily reports looked like.

Example A-1. Vineyard.NET's daily modem report standard example format

From: "Mr. Logger" <logger> Subject: Logger Generating report for last 24 hours Total number of calls: 530 Average time per call: 24 minutes Longest call:          472 minutes (ericx) (16:06 - 23:59) *** Bad Lines: A-14  (1) Samples with 0 callers: 0 Samples with 1 callers: 3 Samples with 2 callers: 7 Samples with 3 callers: 27 Samples with 4 callers: 13 Samples with 5 callers: 14 Samples with 6 callers: 19 Samples with 7 callers: 11 Samples with 8 callers: 6 Samples with 9 callers: 19 Samples with 10 callers: 18 Samples with 11 callers: 20 Samples with 12 callers: 27 Samples with 13 callers: 19 Samples with 14 callers: 21 Samples with 15 callers: 15 Samples with 16 callers: 18 Samples with 17 callers: 11 Samples with 18 callers: 7 Samples with 19 callers: 4 Samples with 20 callers: 4 Samples with 21 callers: 1 Samples with 22 callers: 2 Samples with 23 callers: 1 Sample size: 5 minutes Line Reports                 Short     Total        Average      Longest  Line    Calls  Calls     Min.         Length         Call ________________________________________________________________  _ A-01        21      2      7'55'40''      22'39''    2'14'34'' A-02        22      0      8'43'29''      23'47''    1'36'53'' A-03        17      1     10'29'33''      37'01''    7'52'33'' A-04        25      0      6'15'32''      15'01''    1'42'24'' A-05        22      1     10'03'05''      27'24''    2'49'48'' A-06        28      3      7'12'31''      15'26''    1'58'29'' A-07        23      0      7'09'47''      18'41''    1'27'24'' A-08        28      2      9'03'33''      19'24''    1'50'38'' A-09        17      1      6'41'55''      23'38''    1'14'20'' A-10        22      2      7'54'02''      21'32''    1'57'00'' A-11        15      1     11'15'53''      45'03''    6'36'20'' A-12        16      0     11'45'13''      44'04''    3'52'53'' A-13        24      2      7'05'54''      17'44''    1'20'37'' A-15        24      1      7'42'27''      19'16''    1'48'53'' A-16        21      0      8'30'44''      24'19''    1'47'59'' B-01        25      3      6'46'32''      16'15''    1'57'28'' B-02        17      0      7'23'25''      26'05''    2'18'28'' B-03        26      0      6'59'04''      16'07''    1'17'53'' B-04        21      1      9'37'57''      27'31''    3'15'40'' B-05        21      1      6'10'08''      17'37''    1'11'24'' B-06        18      0     10'04'32''      33'35''    2'32'54'' B-07        24      2      8'04'27''      20'11''    1'56'46'' B-08        23      2      8'16'24''      21'34''    3'18'23'' C-01        14      0      8'53'42''      38'07''    3'20'50'' C-02        16      1     12'53'02''      48'18''    4'28'25'' Total lines: 25 Short calls are calls that are less than 30 seconds. Usage by hour: Time           average number logged in  0:00 -  0:59      7.6   **************  1:00 -  1:59      7.1   *************  2:00 -  2:59      4.7   ********  3:00 -  3:59      2.2   ***  4:00 -  4:59      3.1   *****  5:00 -  5:59      3.1   *****  6:00 -  6:59      4.2   *******  7:00 -  7:59      5.2   *********  8:00 -  8:59     10.1   *******************  9:00 -  9:59     12.9   ************************ 10:00 - 10:59     15.1   ***************************** 11:00 - 11:59     13.0   ************************* 12:00 - 12:59     10.3   ******************* 13:00 - 13:59     11.2   ********************* 14:00 - 14:59     12.5   ************************ 15:00 - 15:59     17.8   ********************************** 16:00 - 16:59     14.2   *************************** 17:00 - 17:59     16.2   ******************************* 18:00 - 18:59     13.4   ************************* 19:00 - 19:59     13.5   ************************** 20:00 - 20:59      9.8   ****************** 21:00 - 21:59      9.4   ***************** 22:00 - 22:59     15.4   ***************************** 23:00 - 23:59     15.2   *****************************

Vineyard.NET also set up a system called MRTG (Multi Router Traffic Grapher) by Tobias Oetiker and Dave Rand to monitor the bandwidth utilization of its external Internet connection (see Figure A-1). Over time we modified MRTG to track the CPU of our servers, our dialup utilization, the amount of free disk space, and the utilization of each of our leased lines. MRTG has since been superseded by RRDTool and MRTG2. We highly recommend these programs.

Figure A-1. Vineyard.NET monitors the utilization of its network connection using MRTG; the holes in the graph represent times when the monitoring system was not running.

only for RuBoard - do not distribute or recompile