17.3. Running the Numbers OK, so here's the big question: what is information architecture actually worth? The best source of numbers is white papers created by such analyst firms as Forrester Research and The Gartner Group. These numbers often don't focus on the ROI for information architecture per se, but they do address similar or overlapping areas of practice (e.g., user experience) or a hot technology (e.g., portals) that may involve a specific architectural approach. For intranets, most utilize an opportunity cost approach to assessing ROI, drawing on a technique that was popularized in the web design community by Jakob Nielsen.[ ] Table 17-1 shows the basic calculation. [] "Intranet Portals: The Corporate Information Infrastructure" (http://www.useit.com/alertbox/990404.html).
Table 17-1. ROI case for investing in the Sun intranet's information architecture| Factor | Cost |
|---|
| Time lost due to a design-related problem (determined through user testing) | 10 seconds/occurrence | | Time lost over course of a year per employee (10 seconds/occurrence x3 occurrences/dayx200 days/year) | 6000 seconds (1.67 hours)/year | | Cost per employee (e.g., $50/hour/employee, including benefits) | $83.33/employee | | Number of employees that experience this problem | 5,000 | | Total cost due to this design-related problem | $416,667/year |
For example, if the design problem at hand is a confusing labeling system, and you feel confident that investing $150,000 will make it go away, then you can claim an ROI of 178 percent ($416,667$150,000 / $150,000). Not bad, especially if you consider that this particular design problem may be just one of many that can be addressed. Here are some more examples of this opportunity cost approach: Bay Networks invested $3 million into organizing 23,000 documents for its 7,000 users. Among other benefits, Bay estimated that each member of the sales staff would save a minimum of two minutes a day searching for documents, or roughly $10 million a year.[ ] Thats a 233% return on investment. [] Fabris, P. "You Think Tomaytoes, I Think Tomahtoes" (http://www.cio.com/archive/webbusiness/040199_nort.html).
[§] Agency.com surveyed 543 employees from different companies regarding their use of portals. Respondents reported that portal use saves on average 2.8 hours per week, or 7 percent of their time. Assuming $55,000/year per employee (fully loaded), a well-designed portal would save employers $3,908 per employee. A 5,000-person company would therefore save about $20,000,000/year. [§] See http://research.agency.com/.
Applying this approach to intranet portals, Nielsen states that "The cost of poor navigation and lack of design standards is . . . at least ten million dollars per year in lost employee productivity for a company with 10,000 employees."
The last two examples don't provide investment costs, so we can't determine the actual ROI. Regardless, the number jockeys will be extremely impressed. These examples focus on ROI for intranets, which is measured primarily in cost savings. What about external sites, such as e-commerce sites, that are geared toward increasing revenue? The most powerful numbers come from examining sales lost due to sites that confuse and frustrate customers. For example, Creative Good tested the BestBuy.com e-commerce site and found that over 78 percent of customers' purchase attempts failed.[||] Creative Good then designed a prototype of the BestBuy.com site with improvements made to, among other things, some aspects of the information architecture. Among customers who used the prototype, 88 percent could complete a purchase, exactly quadruple that of the live site's rate. [||] "Holiday 2000 E-Commerce: Avoiding $14 Billion in 'Silent Losses'" (http://www.creativegood.com/holiday2000).
It's not clear what the improvements would cost, but Creative Good estimated that they would require less than one month to develop and implement. Let's be conservative and assume that this effort cost $1,000,000 (a reasonably high number). If BestBuy.com's current sales are $100,000,000, and the improvements only doubled (not quadrupled) sales to $200,000,000, the ROI would still be quite healthy: ($100,000,000$1,000,000) / $1,000,000 = 9,900%! There are many other similar and exciting numbers for e-commerce sites.[#] For example, IBM spent millions over a 10-week/100+ employee effort to improve ibm.com's information architecture, resulting in a 400 percent increase in sales.[**] And Tower Records was able to double the rate of purchases made by visitors to its site by improving its search system.[] [#] Another good article: Najjar, L. J. "E-commerce user interface design for the Web." (http://mime1.gtri.gatech.edu/mime/papers/e-commerce%20user%20interface%20design%20for%20the%20Web.html).
[**] Tedeschi, B. "Good Web site design can lead to healthy sales." ( http://www.nytimes.com/library/tech/99/08/cyber/commerce/30commerce.html).
[ ] Guernsey, L. "Revving up the search engines to keep the e-aisles clear." Many of the metrics used to judge a site's success can be positively impacted by an improved information architecture. And for each of those architectural improvements, there is likely an exciting number that matches it. If LL Bean is trying to sell more ties, better contextual navigation from the shirts area to connect to matching ties might raise revenue. If the Sierra Club is trying to increase awareness of environmental issues, perhaps a more prominent link to its mailing lists and feeds would raise subscription levels. If American Express is drowning in costs associated with printing, maintaining, and distributing product literature to financial advisors across the country, a well-architected extranet might save them big bucks. And if Dell is trying to reduce technical-support call volumes, perhaps reconfiguring its site's search system will result in higher usage levels and allow for a reduction in technical support staff. Ultimately, certain aspects of information architecture, like any other UCD-influenced improvement, should have a direct and quantifiable impact on just about any site's performance. Because the cost of information architecture work can be measured, ROI calculations should be attainable. And you should therefore be able to have fruitful and productive conversations with the "by the numbers" people. 17.3.1. Debunking the ROI Case By now, you should be getting nervous because we italicized "should" three times in the last paragraph. Unfortunately, it's almost always impossible to calculate true ROI for an information architecture. We can discuss it as theory, but information architects must be careful not to fall into the trap of false claims of attaining proven ROI numbers. There are three major reasons why ROI measurements of information are, at best, unreliable:
The benefits of a complete information architecture cannot be quantified -
It's generally possible to measure the value (and ROI) of some of an architecture's individual components. For example, we may be able to determine how well users navigate a broad and shallow hierarchy versus a narrow and deep one. Or we might measure how users respond to one way of presenting search results versus another. However, an information architecture is made up of many such components. And it's generally wrong to measure an individual architecture component, as there's a good chance that its performance will be impacted by that of another component. As mentioned earlier, users often integrate tools for both searching and browsing in a single effort to find information. Although the natural tendency is to separate these tools for testing purposes, it makes more sense to measure searching and browsing performance togetherafter all, that's how the site is used. But measuring both concurrently is exceedingly difficult; it soon becomes apparent that you can't isolate the impact on performance that each component makes. Measuring the performance of a component of an information architecture is useful as long as such measurements are not confused with the measure of the overall architecture.[ ] [] A good source of evaluation techniques for information architecture components is the November 2000 ACIA white paper "Evaluating Information Architecture" by Steve Toub (http://argus-acia.com).
The benefits of many information architecture components can't ever be quantified -
Though an information architecture is greater than the sum of its parts, the performance of many of its parts can't ever be quantified. For example, many efforts to measure search performance focus on how long and how many clicks it takes users to find the answer. This is reasonable if users are performing only known-item searches, where there is a "right" answer to their question and a consistent and measurable endpoint to their search sessions. But as discussed earlier in this book, the majority of many sites' users are not performing known-item searches. Instead, they're looking to perform comprehensive research, learn a few tidbits about a topic, pick up some news, or be entertained. These types of searches usually don't have an endpoint. If there is no endpoint, it's not possible to confirm (and therefore quantify) success. Another consideration is that many users don't find what they need from a site. There are potentially huge numbers associated with this cost, but how would it ever be measured? In these situations, you might ask subjects if they were satisfied with their results. And their answers might suggest that they were indeed pleased. But when it comes to finding information, ignorance is often bliss: users don't know what they don't know. They may miss out on the best, most relevant content, but they simply have no idea that it exists.
Most claims for quantified information architecture benefits can't be validated -
Most quantifications of information architecture, like those discussed above, can't be proven. When we read about how many minutes per day an employee would save, or how many more sales a redesigned shopping cart would convert, we are essentially reading predictions. We ultimately have no way of proving that those minutes are used productively and not for playing Tetris, or that customers bought more or less due to the redesign. It's unfortunate, but efforts at validation are rarely made because they're too expensive and time-consuming. And many, many factors might influence a before-and-after outcome besides the redesign. Would an e-commerce site's numbers go up because the information architecture was better, because more redundant connections were added to the site's servers, or because the overall number of web users had grown? There are an incredible number of uncontrollable and, at times, unknown variables to consider that make it difficult, if not impossible, to validate such measurements. Information architecture is a human issue. For that reason, it doesn't lend itself to the type of quantification that one might expect of other areas, such as determining what type of router to purchase to accommodate more network traffic. Unfortunately, it is often confused with such technical areas by those who have insufficient knowledge of information architecture. Numbers associated with information architecture should be seen for what they are: predictions based on soft numbers that haven't been or can't be validated. That doesn't mean they're not useful. ROI cases are simply one of many tools that, if they sound reasonably valid, make people feel comfortable with an unknown. And after all, we do have to survive, and sometimes the only way to convince a "by the numbers" person is to show them numbers. But if you do provide ROI numbers to a manager or potential client, be honest that you're not really proving anything; you're simply predicting value that probably can never be measured but is real nonetheless. It's our responsibility, not to mention in our own interest, to educate our market. After all, our work will always be easier and more effective if we're selling to and working with smart people. If we continue to hammer away at the honest truth about ROI numbers, perhaps information architecture will eventually be broadly accepted as a valuable (but not quantifiable) field such as public education or psychotherapy. Or, for that matter, management, marketing, human resources, and IT.
|
