Why Do We Need to Scale?

Although I did say that the approach to this book would not be a collection of Stephen King short stories, perhaps that approach is best employed to answer the question of why we need to scale. The need to scale can be easily demonstrated by a story or two of "scaling gone wrong."

The inherent need for scalability is due to changes in business that can cause the size of a problem to dramatically ebb or swell. Obviously, building an architecture that can serve millions of users when your current user base is only 10,000 is a waste of money. The goal is to build a system for 10,000 users but to architect it in such a fashion that scaling it up to serve millions will not require application or architectural redesign.

Scaling Up Gone Wrong

I once encountered an architecture used to service online shopping malls. The site's purpose was to be a cookie-cutter shopping center portal service with strong online editing (Wiki) for stores, shopping center managers, and corporate managers. Users (mall shoppers) could visit online to register and learn of new sales, events, and online coupons from their favorite stores; use an online, dynamic gift guide; and subscribe to weekly email bulletins that contained information about their personally selected stores and special email-only offers.

It sounds simple enough. However, when I arrived at the architecture, the site was offline regularly due to capacity issues that occurred often, and the weekly email, which was supposed to be delivered all in one morning, was taking more than 24 hours to get out the door. Why? Very bad design issues both on an application level and a system level.

What was so bad about this design? Well, the one requirement was for 24x7 uptime, and that was not being achieved. The system was built on commodity hardware, and that hardware occasionally failed. The application was designed with no caching infrastructure and a data model not suited for the online transaction processing required to drive the website or the bulk querying required to construct the weekly emails.

The site originally started with only a few tens of thousands of users and infrequent visits. The ill effects of the poor design were not made immediately apparent due to the minimal stress on the architecture initially. However, when more traffic was driven to the site through marketing and advertising, the site capsized. The ramifications of shoddy development, design, and construction were clear.

Fortunately, we were able to take over the architecture and were given free reign to perform a complete reimplementation. As the foundation of the architecture was quite small, it was rebuilt in about a month, coming in at 30,000 lines of Perl code for application business logic and presentation. As always, the code change was the most profound but the least noticeable by the owner because software and development are such abstract things. Also, during the process of a normal application reimplementation, the presentation, creative and copy, does not change, so the only visible effects of the work are performance and reduction of long-term maintenance costs.

The architecture itself changed as well. The site was and still is run on commodity hardware, but the preexisting environment was riddled with single points of failure. The new hardware architecture was marginally more expensive, doubling up on specific devices in the content delivery pipeline to address both performance and availability demands.

Although it sounded easy enough to do, and from a technical point of view it was, it took a month of labor (and costs) to reengineer an architecture that had a considerable (and wasted) capital investment. In addition to the sunk and unrecoverable costs of the first failed attempt at developing the site, and the costs of reengineering the entire solution from scratch, a month was lost as well.

During the month in which reimplementation and deployment took place, all Internet-based business initiatives that required "new development" were drastically slowed and often completely derailed. Time is moneyusually much more than any techie will ever realize.