Section 23.7. September 11


23.7. September 11


Department of Defense money originally funded the research that became the architecture of the Internet. That research mandate was based on the perceived needs of a late-'60s Cold War nation. One of the design constraints was this: in the event of a selective, possibly nuclear attack on the United States, could a data network be designed that would continue to function despite outages to significant grid sections of the network? In other words, could the architecture of the network allow for graceful degradation and an opportunity to route around outages?

Two principles that underly the Internet architecture are decentralization and redundancy. Think, for example, about the way in which the Domain Name System (DNS) is implemented. This is the protocol by which a computer knows how to associate a human-readable address such as Slashdot.org with a computer-readable IP number. DNS has no single, canonical server to provide an authoritative list of these mappings. Such an approach would create a single point of failure that would serve as a bottleneck under high-traffic conditions and would bring down all DNS-dependent traffic in the event of a server failure.

Instead, DNS is more of a peer-to-peer network, with thousands of servers across the Internet functioning as DNS servers. Any server can update its records, and its updates will gradually be propagated to other DNS servers. In the course of a couple of days, any change to one DNS server can reach all others. Nor does one need "permission" to put up a DNS server. The requisite software is open source, and the Internet architecture is designed to automatically accept new servers or respond when encountering missing or offline servers. The system is highly distributed and redundant.

Even the basic network rules about how packets are routed from one destination to another follow these principles. Any computer sends a "test packet" first, attempting to establish a route to its destination. Once a route is established, actual data packets are sent. If at any point, the originating computer fails to get an authenticating response for that route, it explores for a new route and continues sending packets along the new route. There are no canonical, authoritative routes from Point A to Point B; each network route is a process of discovery based on current network conditions.

The principles are simple enough: avoid single points of failure by relying on a highly distributed network of peers rather than one or a few hubs around central, authoritative servers. The network protocols that employ these principles, however, are only as robust as the applications that use them. All of that redundancy and flexibility in routing does no good once email is queued up at an unresponsive destination server. If millions of requests are all headed for the same web server, that becomes the de facto center of an unresponsive hub.

In other words, to benefit from the design features of the Internet architecture, an application must be specifically tailored to use those features. In fact, relatively few applications do make use of this underlying structure. One application that does is IRC, the staple of online communication in the open source community.

A look at the network list in a default setup of XChat (a common IRC client) reveals dozens of IRC networks. Some are based around a common interest, like QuakeNet; some are based around a geographical location, like OzNet. Many, like Freenode, are general purpose. Within each of these networks will be dozens, or even hundreds of channels, each of which represents a particular community or topic of interest. The more popular networks easily have tens of thousands of users connected simultaneously at any given time.

IRC puts very slight demands on a server; all of the transmissions are short strings of text. Many universities and a large number of commercial sites volunteer server space to run an IRC server. All of the servers that are part of a given network work together to mirror the activity on the network. Typically servers in a network are partitioned into groups, with each group responsibile for mirroring a subset of the channels on that network. In the event that a given server goes offline, clients connected to channels for which that server had responsibility automatically reconnect to another server in that group. The view of a channel conversation that a particular user has, remains the same even through several reconnects to different servers.

As early as 1998, the Slashdot staff had set up an IRC network, called Slashnet. Initially this included a work channel for the staff to communicate with each other. This made sense since the staff was not always together in one place, but it was also just a natural form of communication for those with a Linux/open source background. A public channel was also added, for members of the Slashdot audience to communicate with the staff. The work channel quickly split into two channels, one for actual work communication, and another "water cooler" channel for idle conversation among staff members. Over time, other channels appeared, many from users treating Slashnet as just another IRC network, who were unaware that Slashnet and Slashdot were in any way affiliated.

By September of 2001, Slashnet had become an indispensable form of communication for the Slashdot staff. By this time, the staff was very distributed: Rob Malda and a core group of programmers remained in Holland, Michigan, but editors Timothy Lord and Robin Miller worked remotely; Timothy from various midwest locations, and Robin from Maryland. Jeff Bates had moved to Boston, working out of the offices of the parent company that had acquired Slashdot.[5]

[5] Originally this was online media company Andover.net. Andover was acquired by VA Linux Systems, and reformed as the wholly owned subsidiary OSDN, the Open Source Development Network. Since then, VA Linux Systems has changed its name to VA Software, and OSDN has changed its name to OSTG, the Open Source Technology Group.

Slashnet was, in many ways, the last refuge for Slashdot's original core audience. As the web site itself had become more mainstream, more about culture and less about technology, Slashnet represented a technical hard core of the site's open source roots. The barrier was not a very rigid one. While IRC channels can be moderated, and access can be password restricted, Slashnet, like most networks, was wide open for anyone to participate. In fact, though, the more mainstream online audience tended to gravitate to one-to-one IM systems like AIM or Yahoo! Instant Messenger, rather than the more text-based, more complex, and less user-friendly IRC.

Jeff Bates began the morning of September 11 at home in Boston before heading to the company office. He started with a call to Northwest Airlines, hoping to rearrange some business travel scheduled for later in the month. The call to Northwest was the first he knew that anything out of the ordinary was transpiring that day.

The woman at customer service told Jeff that a plane had flown into the World Trade Center. She had not seen or heard a news report directly, but was instead repeating what she had heard from other customers calling in that morning. Still on the phone, Jeff turned on the television to watch events unfolding on CNN, all the while describing what he was seeing to the woman at Northwest.

After talking to Northwest, Jeff called a friend's cell phone in Manhattan to make sure he was OK. This call went through; many others, from many other people that day, would not. Jeff left for the office wondering, as many people did in those early hours, if this was some sort of freak accident or something more sinister.

Nine hundred miles away, in Holland, Michigan, Rob Malda was also beginning his workday. For Rob, this involved logging on to the Slashnet staff IRC channel, checking his email, and reviewing the Slashdot submissions bin. Rob's first word of the World Trade Center attacks came from monitoring discussions on IRC. With no radio or television at hand, he attempted to look at the CNN and MSNBC web sites, but both sites were already struggling under heavy load, and other than one small, grainy photo from the CNN web site, Rob was unable to get any information. Only the first plane had hit at this point, but the Slashdot submissions bin was already filling with related submissions, and Rob quickly realized this was not going to be an ordinary news day.

Slashdot reviews submissions 24 hours a day, 7 days a week. To provide this coverage, the staff rotates who is in charge of the submissions bin. While any staff online at a given time can review submissions and make suggestions, one person has to be the final authority: only one person at a time can wear the pants in this family. That's an official Slashdot job description: "Daddy Pants." On the morning of September 11, Rob was wearing Daddy Pants. He made the decision that they would depart from their normal coverage and focus exclusively on the World Trade Center story.

By the time Jeff arrived in the Boston office, he had heard on the radio that the second plane had hit, and everyone knew that some form of terrorist attack was underway. Rob's decision to focus the coverage was the right one. By 9:30 A.M.. EST, Slashdot was serving 30-40 pages per second off of its six mirrored web servers, roughly double the usual traffic load.

It's significant to note the range of communication media the Slashdot staff was involved with during the first 90 minutes of that morning: land line telephone, cellphone, television, radio, web sites, email, and IRC. In his 1991 book Virtual Reality Howard Reingold described cyberspace as where you are when you're on the phone. His point, in part, was that we live in a vast and evermore pervasive telecommunications network, and that oftentimes our location within that network is more significant than our geographical location.

All of the Slashdot staff described that working day as one of feeling intimately connected to others at work, despite the fact that they were operating from at least five different geographical locations. In fact, neither Rob nor Jeff can recall, and probably never knew, where Timothy was that day. It could have been Texas, but it could just as easily have been Tennessee. All that mattered was that he was there in channel on IRC to contribute and help out.

It's also significant to note what part of the telecommunications network suffered that day. Cellphone calls in and out of New York and Washington became increasingly difficult, though some calls went through under remarkable and tragic circumstances. Cellphone calls provide our most intimate historical record of what happened within the World Trade Center itself, as well as what happened on the doomed Flight 93 that crashed in Pennsylvania. Land-line long-distance calls would experience bottlenecks nationally throughout the day. Television and radio provided important early reports, but these became less effective later in the day as reporters had difficulty getting on the scene.

News web sites suffered the most, many completely unprepared for the deluge of traffic. Slashdot began a daylong battle to stay up and stay on top of events. The Columbine experience had alerted them to the need to overhaul the site infrastructure. Many changes had been made; now those changes would be put to the test.

Thirty page views per second was well above Slashdot's normal load, but also about the limit of what the site architecture was designed to handle. Around 10:00 that morning, the backend database crashed, and the site was temporarily down. In fact, Slashdot had a backup database server on hand, one with a more current version of the database software (MySQL). This server was not yet online only because the staff didn't want to take the site offline to make the switch. The database crash provided an opportunity to quickly make that switch.

The new database server performed well, and now the bottleneck shifted to the web servers themselves. The staff made some on-the-fly adjustments to caching limits on the servers, and for the moment everything was functioning. It was now noon EST, and the site was serving 50 pages per second. Rob took a short break, and for the first time saw the actual video footage of the two crashes that morning.

As the staff struggled to keep the Slashdot web site up, a parallel phenomenon was emerging. Traffic on Slashnet was swelling, as more and more people turned to IRC as a way to communicate. The 200,000 pages per hour Slashdot's web site was now serving was impressive enough. Yet at the same time, Slashnet had thousands, and perhaps as many as 20,000 simultaneous users sharing information even more rapidly. The Slashdot staff set up a moderated channel to bring some organization to the process, but unmoderated public channels were springing up as well. There was one channel for communicating with the staff, and another for just general discussion.

Many of the links Slashdot posted that day, and many of the inline comments and quotes that appeared in stories, were pulled directly out of IRC on Slashnet. At a time when major news networks had difficulty getting reporters to the scene, Slashdot had eyewitness accounts coming in over IRC. At a time when major news web sites could not keep up with the traffic or rapidly changing information, Slashdot persevered. Some it was information, but some of it was a matter of dispelling misinformation ("I heard a truck bomb went off outside the State Department".... "No, my dad works at State, I just spoke to him, and nothing like that's going on"). Some of the concerns were global ("Who's behind the attacks?"). Some of the concerns were terribly personal ("I have a friend/loved one/family member who works in midtown Manhattan...").

Behind the scenes, the Slashdot staff frantically stripped down functionality on the site to keep the bare minimum of processes running and the maximum number of pages flowing. Dynamic content was turned off. Reverse DNS lookup was turned off. Eventually the ad server was turned off. Logfiles were turned off. After the initial database failure early in the day, however, the site stayed up. At the peak, Slashdot was serving 70 pages per second. For the day, it served more than 3 million page views.[6]

[6] Rob Malda has nicely summarized the day's behind-the-scenes work in his piece, "Handling the Loads," at http://slashdot.org/article.pl?sid=01/09/13/154222.

As a nation, we've never faced the kind of global telecommunications breakdown that the Internet was architected to handle as gracefully as possible. We have, however, seen episodes like September 11 that put a sudden and unexpected strain on the telecommunications infrastructure, and where the graceful degradation for which the Internet allows, matters a great deal. Taking advantage of that inherent robustness, however, requires a communication medium that follows the same architectural design and a group of communicators comfortable using that medium. Slashdot's successful coverage that day would not have been possible without IRC, a protocol as distributed and robust as the Internet itself, and without Slashnet, a community of users who knew how to make the most of that medium.

The word disintermediation has been much abused in the online world. The idea is simple enough: where traditional news media "mediates" between audience and events, the directness of the Internet should make this traditional mediation unnecessary. In practice, disintermediation happens far less often and far less effectively than one might think. Several elements need to be in place. First, there must be a genuine community of like-minded communicators looking to interact directly. Second, the medium through which they interact must be, to the degree possible, both responsive and transparent. Finally, those enabling the medium must have the humility to do nothing more than facilitate.

With Columbine, the Slashdot staff learned very quickly that they were not presenting a story, but were instead in the midst of a story that was happening all around them. The most useful thing they could do was get out of the way and let the story happen, let the Slashdot community connect to each other. September 11 proved again the value of facilitating rather than mediating. By and large, the audience that day did not notice or care that it was Slashdot they were using as their medium. Any channel of communication that was real time, up-to-date, and available would have sufficed. September 11 revealed which communications channels were up to that challenge; which could be effective but informative; which could disintermediate when disintermediation was needed most.



Open Sources 2.0
Open Sources 2.0: The Continuing Evolution
ISBN: 0596008023
EAN: 2147483647
Year: 2004
Pages: 217

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net