Hack39.Manage Bandwidth

Hack 39. Manage Bandwidth

Obviously it's great to get a lot of listeners, but it's not great if you get a big

bandwidth bill at the end of the month. Find out several approaches to

avoiding monster bandwidth bills.

Well folks here's a story about a man name Jed, / He wrote a funny blog and was Slashdotted, / When he first got the news he was one happy dude, / Then he opened his ISP bill, and was totally screwed.

The Ballad of Slashdotted Jed

Bandwidth is a serious problem for people with popular blogs. And considering that the average HTML page is 30 KB and the average podcast size is 5 MB, it quickly can become a big problem for you, too.

Let's do a few calculations, starting with an example monthly bandwidth limit of 100 GB. If my average podcast is around 5 MB, I can get 200 shows in 1 GB (roughly). With a 100 GB monthly limit, I can get 20,000 total downloads before I go over the limit. In practice, I would hit that limit before that because of the transfer protocol cost and the RSS feeds.

For a single podcast, 20,000 downloads a month sounds like a lot. But this is a subscription service, and if you figure on 20 podcasts in a month, the number of downloads drops to 1,000 per podcast, which looks a lot like a small fan base.

What's worse is that many hosting sites have far less than a 100 GB bandwidth limit on their accounts. This means the size of the audience you can support will drop dramatically.

The troubles aren't going to come from your fan base, though. When you look at your server logs, you will be able to see well ahead of time if you are going to go over your bandwidth allocation from the day-to-day downloads of your fan base. The problem comes from the spike in bandwidth caused by services such as Slashdot.

Slashdot is the geek capital of the Internet, and it's the reason you want to be sure of what happens in an overage situation. Any site referenced from an article off the Slashdot home page invariably will be hammered with hundreds of thousands of requests in the first few hours. The effect is so well known that it has been dubbed Slashdotting. Sites brag about being Slash-dotted and withstanding the load.

If you say something interesting enough to get on Slashdot, chances are good that your podcast will have a sudden and dramatic rise in the number of downloads. And the only thing you will notice is that your site is responding much slower, or not at all. If you are lucky, your site will be disabled due to bandwidth overage and that will be the end of it. If you haven't planned for this and checked out your ISP contract, you could be in for a nasty surprise.

There are four solutions to the bandwidth problem: using a free content hosting service, using a pay-for file service, using your own ISP, and having your own server and a dedicated connection.

Hack 6.4.1. Free Content Service

You can avoid bandwidth and storage problems altogether by using a free content service such as the Internet Archive (http://www.archive.org/). They will pay the hosting costs and the bandwidth charges for you, and will give you a link that you can use in your blog. In exchange, you need to conform to the Creative Commons license and use their minimum MP3 encoding format (64 Kbps).

You start with getting an account on the Internet Archive. Then download ccPublisher from http://mirrors.creativecommons.org/software/publisher/. Use ccPublisher to upload your file to the server. At the end of the process, you will get a link to the content description page. And from that page you get a link to the media itself. You will use that link on your blog and in your RSS enclosure tags.

The Ourmedia service (http://ourmedia.org/) is a free blogging and podcasting system built on top of the Internet Archive storage system [Hack #38].

At the time of this writing, none of the top 10 podcasts was using this technique for file storage.

Hack 6.4.2. Hosted File Services

For $99.95 a year, you can get a 250 MB file store on .Mac, which you can use to store your podcasts (for $50 more per year, you can get 1 GB). The fine print on the contract states that bandwidth overages will result in the site being disabled, though it doesn't mention the exact bandwidth restrictions. To create a .Mac account, go to http://www.mac.com/.

This solution is ideal if you already have an account on a service such as .Mac. But with more than 1 GB of file space going for around $7 a month, you can do better with other ISPs, and have complete control of your hosting environment.

At the time of this writing, two of the top 10 podcasts were using .Mac for hosting, including Adam Curry's Daily Source Code.

Hack 6.4.3. Using Your ISP

The dreaded scenario I painted at the beginning of the hack referenced a hosted ISP service. With this type of service, you pay a small monthly fee to get a portion of a server on the Internet, and a small amount of disk space and bandwidth. DreamHost (http://www.dreamhost.com/) and BlueHost (http://www.bluehost.com/) are two examples of this type of service.

So, what happens when you go over the bandwidth limit with an ISP? That depends on the hosting service. Some will allow reasonable overages. Others will warn you and then start denying the transactions. Still others will charge you for overages on a per-megabyte basis. The lesson is, read the fine print before signing up with a hosting company. And if the company doesn't specify anything, get something in writing from an authority at the company about how bandwidth overages will be handled.

When interacting with the hosting company, be clear what you intend to do with the service; you plan on hosting a blog, that blog will reference MP3 files, created by you, on the server, and it's likely that people will be downloading them quite often. If you go way over and you were cagey or deceptive about your intentions with their service, you could be held in breach of contract anyway. Honesty is the best policy.

If you plan to do just a few podcasts a month, you will probably be fine. But if you plan on podcasting exclusively and often, and you are a popular person with a popular topic, you will need to take steps to ensure you don't run into bandwidth problems.

Hack 6.4.3.1 Staying under the limit.

One way to combat excessive traffic is to reduce the number of items in your RSS feed. If you feed only the 3 most recent pod-casts, when new subscribers show up they will download just those 3 and not the 10 you might have listed on the home page. To get more they would have to go to the home page and click the links manually.

Another way is to host your MP3s on a different server. There is no hard and fast rule that says the blog server also has to serve the MP3s. If you can find a hosting site that will host just the audio, you can use a link to that in the blog and in your enclosure tag.

Hack 6.4.3.2 In an emergency, pull this lever.

If you do go over your bandwidth limit and you want to stop traffic yourself, simply rename the RSS feed file, and rename your podcast files temporarily. The server will start responding with "Resource Not Found" responses, which are a lot smaller than your podcasts.

Hack 6.4.3.3 Find a hosting service.

If you don't have a hosting service and you are going to start looking for one, I recommend spending a least a couple of hours on the task. Keep some notes on what hosts have what plans. Shop around. In particular, make sure you know the bandwidth limits and overage policy. If you have the time, ask for a recommendation on a podcast forum or mailing list. Sometimes it's only after a few months that a service's flakiness becomes apparent. You can leverage the community to get some honest feedback about good services.

When people ask me, I recommend DreamHost (http://dreamhost.com/), BlueHost (http://bluehost.com/), or WebIntellects (http://webintellects.com/). All of these provide a great service at competitive rates.

At the time of this writing, most of the podcasts in the top 10 were hosted using an ISP service. However, no particular ISP service dominated the field.

Hack 6.4.4. Hosting Your Own Servers

When it comes to hosting your own servers, you have two choices. You can get a commercial-grade DSL connection with a static IP address to host your web server. Commercial DSL lines have a guaranteed bandwidth and have a fixed monthly fee. With this, you can set up a server and publish as many podcasts as you want, without worrying about bandwidth overage charges.

If you are looking for something even larger, you can do what ISPs do. What is an ISP after all? It's a company or co-op that has leased some T3 or larger lines from a larger network provider. Then it set up shop with some firewalls, routers, and server hardware. Some individuals have done this for themselves and they donate or rent space very cheaply to their friends and family.

On the upside, the costs look appealing if you have a very large web service in mind. You can put as much cheap equipment as you want behind the firewall as a one-time cost, and the only monthly cost is the connection.

The downside of both methods is that maintenance is a nightmare. Worms, viruses, denial-of-service attacksall of these things that once fell on your ISP's plate now fall on yours. This leaves you less time for podcasting.

If you only want to get better bandwidth rates for your podcasts, this is not the avenue to take. However, if in your travels, you find a person or co-op that is doing this as a friendly service, you should try to see if you can get involved. As with an ISP, you should be up front with what you are going to do with the service, and have your current bandwidth numbers in mind to give them a sense of the impact you will have on their service.

Hack 6.4.5. BitTorrent

BitTorrent is a peer-to-peer (P2P) file-sharing network. The value of posting your podcasts on the BitTorrent network is that you will share the bandwidth load with everyone who seeds a copy of your file. Unfortunately, using BitTorrent is not as simple as just uploading your file to a server. First you have to understand the basics.

Using a BitTorrent client, a user can download a .torrent file to start the Bit-Torrent download. This .torrent file contains a pointer to a tracker application hosted somewhere on the Internet that the client then logs into. The tracker tells the client that several other clients are seeding that file. The download client then connects to those seeding clients to get chunks of the file.

You have several BitTorrent clients to choose from. On the Macintosh you have the Tomato Torrent (http://sarwat.net/bittorrent/) and original BitTorrent (http://bittorrent.com/) clients. On Windows you have the original Bit-Torrent among other native clients. But the one that seems to get the most praise is the cross-platform Java client named Azureus (http://azureus.sf.net/).

One particularly nice feature of Azureus is the graphical display that shows you what portions of the file are still left to download.

This file chunking is one of the reasons BitTorrent is so fast. You don't download the entire file from one source. You download segments of the file from a variety of hosts and assemble them into one big file as you go.

The moment you download a segment of the file your client is also seeding that segment to the rest of the world. That's how BitTorrent distributes the bandwidth load. Every client that has a copy of the file distributes portions of that copy to everyone else.

The downside is that at least one client must be seeding the whole file for it to be downloadable. This is why web seeding is so important. In the web seeding model you have a server on the Internet that both holds the .torrent file and acts as a continuous seed for the whole file. If some clients come along and get the file, they too will act as seeds, but in the meantime, at least the file will be on the network. Some web hosting services such as Hurricane Electric (http://he.net/) provide web seeding services.

If you don't have a web seeded file, you will have to host that seed yourself by having a computer on the network that runs the BitTorrent client continuously. Unfortunately, this is not so easy. Most people don't even notice when their client isn't seeding the file it's downloading. And in most cases, for home machines connected through a router, your client will not be able to seed for two reasons: the BitTorrent ports are blocked by the firewall in the router or in the operating system of the computer; and the host is on an internal IP address provided by the Network Address Translation (NAT) system in the router (in other words, your broadband provider's network configuration renders your computer effectively invisible to the outside world).

Now that you understand the basics, let's talk about how to put this into practice.

Hack 6.4.5.1 Finding a tracker.

You can't BitTorrent a file without first having a tracker to assign the file to. Trackers on the Internet come and go, primarily because they are being used to exchange copyrighted material. But some are restricted to material where there is no copyright infringement, and one in particular, http://downloadradio.org/, is designed specifically for podcasting.

Another option is to use a hosting service that provides tracker and torrent services such as Hurricane Electric (http://he.net/). And still another option is to use a machine that is always on the Web on your home network to provide tracking services. This will require altering your firewall and installing tracker software. Look for the tracker software on http://bittorrent.com/.

One last option is Blog Torrent (http://blogtorrent.com), which is a PHP-based tracker. As of this writing, the home page advertises that the software will support web seeding soon. There are potential security issues with this software, so make sure you read the documentation fully when you install it.

Hack 6.4.5.2 Creating the torrent.

Once you have a tracker, you can create a .torrent file to reference your podcast .mp3 file. In the File menu of your BitTorrent application, select Create Torrent File. Then specify the location of the .mp3 file and the URL of the tracker.

Upload the .torrent file to the tracker. Then reference the .torrent file in the enclosure tag of your RSS 2.0 file:

   <enclosure url="http://www.mytracker.com/bit/mypodcast.torrent "     length="12782" type= "application/x-bittorrent"/> 

Notice that the length of the file is the size of the .torrent file and not the size of the target .mp3 file.

Podcatchers that support BitTorrent (e.g., iPodder) will initiate a torrent session to download the file when they see this MIME type or extension.

Hack 6.4.5.3 Seeding the file.

The podcatcher won't get very far with downloading the file unless that file is being seeded actively by a client somewhere. An ISP that does web seeding can do that seeding, or you, the podcaster, can do it, keeping your BitTorrent running with a copy of the podcast. If you remove your computer from the network, and you are the only one seeding the file, no client will be able to download the file.

Another problem is that you might not be able to seed the file from your computer. This happens when you are behind a firewall or have an IP address provided by NAT. To configure your router properly, you will need to do three things. First, get a static IP address from your broadband provider, or set up an account with a dynamic DNS service such as DynDNS (http://www.dyndns.org/), which allows you to have a permanent hostname, even when your IP address is shifting constantly. Second, assign that static IP address to a set of ports on the serverthis is called port forwarding. Third, disable any filtering of the forwarded ports on your own computer.

Another option that doesn't require a static IP address is to combine port forwarding with port triggering if your router supports it. With port triggering, the act of attaching the client to the BitTorrent tracker triggers the router to open the forwarding ports and assign them to your computer dynamically. This has the advantage of being more secure, since the ports aren't opened until you are serving files, and when you stop serving files the ports are closed.

The manner of configuring your router is specific to the router's make, model, and even firmware version number. Full details on configuring each type of router are available on http://portforward.com.

Hack 6.4.5.4 Is it worth the time?

BitTorrenting your podcasts has several significant downsides. First, it's hard to set up. Second, in the case where only you are seeding your file from your personal computer on your home network, you will have to hack your firewall and it will eat your bandwidth. Third, not all podcatchers are compatible with BitTorrenting. Fourth, not all blog software and RSS tools support building BitTorrent feeds. So, you might have to do some hacking.

On the upside, if your podcast is very popular, you will save on bandwidth.

Hack 6.4.5.5 Trackerless BitTorrent.

As this book was going to press, the BitTorrent client was being upgraded to work without a tracker application. This 4.1 beta version is certainly worth looking at, though it remains to be seen how long it will take for this to become ubiquitous and be supported by podcatchers.

Hack 6.4.6. The Future

Podcasting is developing rapidly, and as it grows, so do the bandwidth problems that podcasters face. Wherever there are problems, there are going to be people looking to make a dollar from solving those problems. An example is Liberated Syndication, which created a whole new business model around the requirements of podcasters.

Keep subscribed and current with some podcasting mailing lists, and with the forums on the major podcasting sites (http://ipodder.org/ and http://podcastalley.com/), to watch for announcements from hosting services.

Hack 6.4.7. See Also

• "Construct Your MP3s" [Hack #18]