17.2. The Two Kinds of People in the World
Now that we've covered the need to sell it, just what is involved in making the case for information architecture? That depends on the person to whom you're making the case. To grossly overgeneralize, we've found that business people typically fall into two groups: "by the
The "by the numbers" people require data to help them make their decisions. They need to see figures: "If we invest X dollars into this information architecture thing, we'll make or save 2X dollars." They rationally consider return on investment (ROI) as the basis for their business decisions. Makes sense, right? Well, as we'll see, it doesn't. But you still need to understand this mindset because you'll encounter it again and again.
"Gut reactionaries" do what
right. They trust their
Ultimately, when you're making the case for information architecture, you don't know which of these narrow and extremely unfair stereotypes will describe your client. So be prepared to discuss both the numbers and the intangibles.
17.3. Running the
Time lost due to a design-
|Time lost over course of a year per employee (10 seconds/occurrence x3 occurrences/dayx200 days/year)||6000 seconds (1.67 hours)/year|
Cost per employee (e.g., $50/
|Total cost due to this design-related problem||$416,667/year|
For example, if the design problem at hand is a confusing labeling system, and you feel confident that investing $150,000 will make it go away, then you can claim an ROI of 178 percent ($416,667$150,000 / $150,000). Not bad,
Here are some more examples of this opportunity cost approach:
Bay Networks invested $3 million into organizing 23,000 documents for its 7,000 users. Among other benefits, Bay estimated that each member of the sales staff would save a minimum of two minutes a day searching for documents, or
[ ] Fabris, P. "You Think Tomaytoes, I Think Tomahtoes" (http://www.cio.com/archive/webbusiness/040199_nort.html).
[§] Agency.com surveyed 543 employees from different companies regarding their use of portals. Respondents
[§] See http://research.agency.com/.
Applying this approach to intranet portals, Nielsen states that "The cost of poor navigation and lack of design standards is . . . at least ten million dollars per year in lost employee productivity for a company with 10,000 employees."
The last two examples don't provide investment costs, so we can't determine the actual ROI. Regardless, the number jockeys will be extremely impressed.
These examples focus on ROI for intranets, which is measured primarily in cost savings. What about external sites, such as e-commerce sites, that are geared toward increasing revenue? The most powerful numbers come from examining sales lost due to sites that confuse and frustrate customers. For example, Creative Good
 "Holiday 2000 E-Commerce: Avoiding $14 Billion in 'Silent Losses'" (http://www.creativegood.com/holiday2000).
It's not clear what the improvements would cost, but Creative Good estimated that they would require less than one month to develop and implement. Let's be conservative and assume that this effort cost $1,000,000 (a reasonably high number). If BestBuy.com's current sales are $100,000,000, and the improvements only
There are many other similar and exciting numbers for e-commerce sites.
For example, IBM spent millions over a 10-week/100+ employee effort to improve ibm.com's information architecture, resulting in a 400 percent increase in sales.
And Tower Records was able to double the rate of purchases made by
[#] Another good article: Najjar, L. J. "E-commerce user interface design for the Web." (http://mime1.gtri.gatech.edu/mime/papers/e-commerce%20user%20interface%20design%20for%20the%20Web.html).
[**] Tedeschi, B. "Good Web site design can lead to healthy sales." ( http://www.nytimes.com/library/tech/99/08/cyber/commerce/30commerce.html).
[ ] Guernsey, L. "Revving up the search engines to keep the e-
aislesclear." Many of the metrics used to judge a site's success can be positively impacted by an improved information architecture. And for each of those architectural improvements, there is likely an exciting number that matches it. If LL Bean is trying to sell more ties, better contextual navigation from the shirts area to connect to matching ties might raise revenue. If the Sierra Club is trying to increase awareness of environmental issues, perhaps a more prominent link to its mailing lists and feeds would raise subscription levels. If American Express is drowning in costs associated with printing, maintaining, and distributing product literature to financial advisors across the country, a well-architected extranet might save them big bucks. And if Dell is trying to reduce technical-support call volumes, perhaps reconfiguring its site's search system will result in higher usage levels and allow for a reduction in technical support staff.
Ultimately, certain aspects of information architecture, like any other UCD-influenced improvement, should have a direct and quantifiable impact on just about any site's performance. Because the cost of information architecture work can be measured, ROI calculations should be attainable. And you should therefore be able to have fruitful and productive conversations with the "by the numbers" people.
17.3.1. Debunking the ROI Case
By now, you should be getting nervous because we italicized "should" three times in the last paragraph. Unfortunately, it's almost always
impossibleto calculate true ROI for an information architecture. We can discuss it as theory, but information architectsmust be careful not to fall into the trap of false claims of attaining proven ROI numbers.
There are three major reasons why ROI measurements of information are, at best, unreliable:
The benefits of a complete information architecture cannot be
generallypossible to measure the value (and ROI) of some of an architecture's individual components. For example, we may be able to determine how well users navigate a broad and shallowhierarchy versus a narrow and deep one. Or we might measure how users respond to one way of presenting search results versus another.
However, an information architecture is made up of many such components. And it's generally wrong to measure an individual architecture component, as there's a good chance that its performance will be impacted by that of another component. As mentioned earlier, users often integrate tools for both searching and browsing in a single effort to find information. Although the natural tendency is to separate these tools for testing purposes, it makes more sense to measure searching and browsing performance togetherafter all, that's how the site is used. But measuring both concurrently is exceedingly difficult; it soon becomes apparent that you can't isolate the impact on performance that each component makes.
Measuring the performance of a component of an information architecture is useful as long as such measurements are not
confusedwith the measure of the overall architecture. [ ]
[ ] A good source of evaluation techniques for information architecture components is the November 2000 ACIA white paper "Evaluating Information Architecture" by Steve Toub (http://argus-acia.com).
The benefits of many information architecture components can't ever be quantified
Though an information architecture is greater than the sum of its parts, the performance of many of its
partscan't ever be quantified.
For example, many efforts to measure search performance focus on how long and how many clicks it takes users to find the answer. This is reasonable if users are performing only known-item searches, where there is a "right" answer to their question and a consistent and measurable endpoint to their search sessions. But as discussed earlier in this book, the majority of many sites' users are not performing known-item searches. Instead, they're looking to perform comprehensive research, learn a few tidbits about a topic, pick up some news, or be entertained. These types of searches usually don't have an endpoint. If there is no endpoint, it's not possible to confirm (and therefore quantify) success.
Another consideration is that many users don't find what they need from a site. There are
potentiallyhuge numbers associated with this cost, but how would it ever be measured? In these situations, you might ask subjects if they were satisfied with their results. And their answers might suggest that they were indeed pleased. But when it comes to finding information, ignorance is often bliss: users don't know what they don't know. They may missout on the best, most relevant content, but they simply have no idea that it exists.
Most claims for quantified information architecture benefits can't be
Most quantifications of information architecture, like those discussed above, can't be proven. When we read about how many minutes per day an employee would save, or how many more sales a redesigned shopping cart would convert, we are
essentiallyreading predictions. We ultimately have no way of proving that those minutes are used productively and not for playing Tetris , or that customers bought more or less due to the redesign. It's unfortunate, but efforts at validation are rarely made because they're too expensive and time-consuming. And many, many factors might influence a before-and-after outcome besides the redesign. Would an e-commerce site's numbers go up because the information architecture was better, because more redundant connections were added to the site's servers, or because the overall number of web users had grown? There are an incredible number of uncontrollable and, at times, unknown variablesto consider that make it difficult, if not impossible, to validate such measurements.
Information architecture is a human issue. For that reason, it doesn't lend itself to the type of quantification that one might expect of other areas, such as determining what type of router to purchase to accommodate more network traffic. Unfortunately, it is often confused with such technical areas by those who have insufficient knowledge of information architecture.
Numbers associated with information architecture should be seen for what they are: predictions based on soft numbers that haven't been or can't be validated. That doesn't mean they're not useful. ROI cases are simply one of many tools that, if they sound reasonably valid, make people feel comfortable with an unknown. And after all, we do have to survive, and sometimes the only way to convince a "by the numbers" person is to show them numbers.
But if you do provide ROI numbers to a manager or potential client, be honest that you're not really proving anything; you're simply predicting value that probably can never be measured but is real nonetheless. It's our responsibility, not to mention in our own interest, to educate our market. After all, our work will always be easier and more effective if we're selling to and working with smart people. If we continue to
hammeraway at the honest truth about ROI numbers, perhaps information architecture will eventually be broadly accepted as a valuable(but not quantifiable) field such as public education or psychotherapy.
Or, for that matter, management, marketing, human resources, and IT.