Competitive Research Methods

Competitive user experience research focuses largely on how people use and perceive a product without focusing on its popularity, business model, or revenue stream. Traditional competitive analysis or brand analysis tries to understand what makes companies successful by looking at their revenue and other "fundamentals." Such analyses rarely focus on the consumer's or user's view apart from the occasional feature grid or brand perception study. User experience research can work in concert with more traditional competitive analyses, of course, but its focus is from the bottom up rather than from the top down.

Apart from this fundamental difference, the process is quite similar to traditional competitive analysis. In both cases, the analyst must determine who the competitors are, what aspects should be compared, and how to compare them. The general sequence of steps is largely the same one that a financial analyst would follow when understanding the competitive landscape.

• Identify the competition.

• Profile them for the key attributes that make them competitive.

• Compare them to each other (and to your product).

• Use the comparisons to create "actionable intelligence."

Identify the Competition

This may seem obvious, but before you can start competitive analysis, you need to know who your competitors are. Unfortunately, this is harder than it may first appear. Although the obvious competition is from companies and products that are in the same sector as yours, the exact definitions of that sector are often fuzzy.

You can start researching potential competitors by looking in online directories. If your product is listed in a category in Yahoo! or the Open Directory, look at the other sites in the same category. They're probably your competitors. Go one level up in the hierarchy. Take a likely competitor and search for it in the directory. What other categories does it appear in? The members of these categories are probably your competitors, too. Look at the sister categories at the same level as the one where you would put your product. These may contain competitive products, too.

Moreover, as discussed in Chapter 8 on contextual inquiry, the obvious competition may make up only part of the competitive landscape. There may be other competitors in other media. A bicycle touring site's competition (at least for someone's leisure time) is not just other bicycling information sites, but bicycling itself, since the site user's end goal is not just to find out information about bicycles, but to go out and ride. A health advice product competes with the Physician's Desk Reference, the family doctor, and even grandma. Likewise, although Barnes and Noble is an obvious competitor to Amazon, so is the corner bookshop. What does the corner bookshop have that Amazon doesn't have? How is the bookshop's user experience better or worse than Amazon's? Take into account what other tasks people could be doing instead of using your product. Unless they're being paid to use it, if it's not more useful, more fun, or more informative than competing tasks, they're not going to do it.

Make a list of your competitors. List the most likely ones from the directories, and brainstorm ones from other media and in other situations. If you want, you can write them on index cards or Postit notes to make them easier to organize.

Then sort them. You can sort them any way you want, of course, but the following is a typical organization that I have found useful:

• Tier 1 competitors are those whose product most directly competes with yours. Barnes and Noble competes directly with Amazon's bookselling business, Yahoo! Auctions competes directly with Ebay. These products try to capture the same audiences in the same way, offering the same (or very similar) service, and these are the primary targets of your analysis. There should be no more than five of these in your list. There may be more than that in actuality (there are a lot of places that books are sold, for example), but five is probably enough to give you a good idea of the competitive landscape. There will always be a competitor. Even if you perceive your product as completely new and innovative, it's still competing for your audience's time with something. That something is the competition. Cars and horses look and work differently, but in the early 20th century, they were tier 1 competition for each other.

• Tier 2 competitors are products that are from the same category as the tier 1 competitors, but either they are not as directly competitive as those or they are so similar that researching a tier 1 analogue is sufficient. They're products you should look at to check your understanding of the tier 1 products, but understanding them is not as critical. There shouldn't be more than five or ten of these. For example, Half.com and Alibris.com both sell used books online, and they may be competitors for each other, but they're not really direct competitors to Barnes and Noble.

• Niche competitors compete directly with part of your product, but not the whole thing. In the 1880s, horses and trains competed on long-distance travel, but not in terms of local transport. As of summer 2002, Barnes and Noble's site doesn't carry toys so it doesn't compete with Amazon's Toys and Games section. It does, however, sell books, so its book and music offerings compete with Amazon, and the only aspect of its toy section that it needs to worry about are its children's books.

Profile the Competition

To focus your competitive research, you need to know something about whom you are dealing with. A complete competitive analysis will begin with competitive profiles of all the competitors. These consist of two elements: a description of the product and a profile of its audience.

Product Description

A product description should not just be a list of features. It should be a statement from the users' perspective of the value it brings them. From a user perspective, Amazon's primary value does not lie in big discounts, huge selection, fast access, and its highly relevant recommendations. Few of those boil down to single features or precise quantities.

Product descriptions should not be more than a paragraph long. How would a user describe it at a cocktail party to someone who had never heard of it? For example,

ZDnet is a daily technology magazine. It offers hardware and software reviews, the latest news on the high-tech world, in-depth analysis, and columns written by insightful industry commentators. Its articles provide an in-depth understanding of the changes in the technology world and are written in a direct and lively style. It provides a number of email newsletters on popular sectors of the technology world.

Audience Profile

Chapter 7 lists attributes that your user profile can have. Using this list, look at the competitor's product and try to assign values to as many attributes as you can, creating separate profiles when you feel that the product serves multiple audiences. As with your own product, you may want to create three or more different profiles that represent the different kinds of customers. Highlight the differences between these people and your audiences. If they're direct competitors, the profiles could be quite similar to yours, but each of them will have at least one key difference: they don't use your product. Why not? What makes the competition's user market different from yours?

Identify Features and Attributes

The most important step in profiling your competition is creating a list of things to compare. These can be specific features, or they can be abstract qualities. Feature identification should come from the user's perspective. Your product may run on state-of-the-art hardware in a climate-controlled data center, whereas your competitor's runs out of a dusty closet on 10-year-old Amigas, but your users don't care. They care about things such as how quickly they can get their work done and whether they can find what they're looking for.

There are two ways to collect lists of attributes: by asking users and by looking for yourself. You should do a combination of both.

The users' perception of what constitutes a feature may be different from yours, so you may want to start by doing some light contextual inquiry or a small focus group series. Identify what people consider to be the key qualities of your service and what they consider to be the important functionality. These may be as complex as "I like the way the features are integrated so that my numbers carry over from one section to another," or as simple as "The search button is easy to find."

In addition, do a feature audit. Sit down and make a list of the prominent features of your product. From a user's perspective, what does your product do? What are the prominent functions? Then look at each of your competitors' products and check which ones have the same or similar features (but, again, similarity is measured from the perspective of the user, not how you know the site works). As you run into new features that your competition offers, add them to the list. You can also look at product reviews to see which features reviewers highlight and what they have to say about them.

A list for a search engine could look like this.

• Fast response.

• Provides category and specific site matches.

• Corrects simple spelling errors.

• Sorts results by relevancy.

• Compatible with AltaVista search language (, link:, etc.).

• Etc.

Next, combine the two lists and prioritize the features into tier 1 and tier 2 lists. Prioritization is important when concentrating your research. Prioritize based on the areas that mean the most to the product's functionality, to the company's success, or to the users' satisfaction. You will probably want to discuss the competitive research plan with management since they may have questions of their own that your process can answer.

While you are collecting features, you should also be collecting the important attributes of the competitive products. Attributes aren't necessarily related to functionality, but they are things that make a product memorable, likable, or interesting. They are the adjectives that people use to describe the product to friends or co-workers. Although not as important to the usability of a product as the actual features, attributes describe the feeling and environment of a user experience. A product's vibe can change the way people experience and use it; it should not be ignored. For example, the user experience of a product that is perceived as "bland" and "boring" will certainly be affected by people's opinion even if they are not directly related to how well it works. Form may follow function, but it shouldn't be ruled by it.

Competitive Analysis Techniques

The same techniques that provide insight into your product—usability tests, focus groups, surveys, and the like—work as well when applied to your competitors as when done for your own products. The major difference in approach is the amount of focus you place on creating a balanced perspective. When researching your own products, it's common to focus on the problems that the product is having—after all, those are more immediately actionable than knowing where change is unnecessary—but in competitive research it's as important to understand what your competitors have done right as it is to know where they've faltered. In addition to knowing where their failings are, which parts of their product do users have no problems with? What aspects do they like?

Recruiting

Just as you should recruit real users (or potential users) when trying to understand how people perceive your product, you need to recruit real users when working competitively.

The process of recruiting is the same as that described in Chapter 6, but with the additional consideration of the target audience's experience with your competitors' products. Whether such experience is important depends on the kind of research you plan on doing. If you're usability-testing advanced features, you'll probably have to find people who have experience with the more common features, so it's not necessary to familiarize the participants with basic functionality. Likewise, focus groups of current users can tell you a lot about what attracts them to the product. However, research with inexperienced users—with people who have no experience with your product or your competitors'—often provides some of the most valuable competitive information because the first experience with a product determines a lot about users' future relationship to it and understanding of it.

Warning

Keeping the identity of your company anonymous is important when competitively recruiting. People will almost certainly behave differently if they know that the research is for the purpose of comparison. Some feel defensive about their choices; others choose to use the testing opportunity to vent their frustrations. Both cases are an exaggeration of how they really respond and should be avoided.

The logistics of recruiting people who match a competitive recruiting profile is no different from the process described in Chapter 6, but how do you find your competition's users?

One method is to do a broad public invitation and include in your screener for it a list of products that includes both your product and your competitors', but does not reveal that you (or the recruiter) represent one of them. The list should contain your product, your competition's product, and several others. Invite those who use your competition's products but don't use yours.

Competitive Contextual Inquiry

One-on-one interviews with users of a competitive product can reveal much about what makes that product functional and where it fails. Watching people use your competition's product reveals usage patterns that your product can emulate or avoid.

For example, while watching people reading news online for a technology news competitive analysis, a researcher observed that frequent news readers would first go to CNN or CNBC and read a couple of stories, then they would jump to a smaller news site. When questioned, they consistently said that they use the major news sources for general headlines and the specialized news site for a comprehensive view of the topics that most interested them. The specialized sites provided better views of their favorite topics, whereas the general news sites were better for "hurricanes and stuff." The technology news site was planning to carry general news content from a news wire, but seeing this they decided that a wire feed was unlikely to compete with CNN on the "hurricanes and stuff" front, so they decided to highlight breaking technology news instead.

Focus Groups

Inviting the users of a competitive product to participate in a focus group can reveal why they use it and which aspects of it attract and repel them. Such groups can also reveal people's views of the brand and the identity of the product, and the qualities they assign those brands. Anecdotes collected during the focus group can be analyzed to see what the initial impetus for using product B (the leading brand) versus product A (your product) was.

For example, several focus groups with the users of a furniture sales site were invited to talk about how they buy furniture online and offline. The goal was to understand what information they found most valuable and what presentation most compelling. Quickly and consistently, it was seen that there were certain classes of furniture that most participants would buy only through a trusted name. Unfamiliar with most manufacturer brands and unable to touch the furniture, they needed the assurance of a reputable seller. They complained about the quality of the navigation and pictures on the site, but the trust they placed in the name overrode their apprehension.

Note

For the record, both of the furniture sites—the one discussed in the focus group and the one for which the research was done—failed. The focus groups also revealed that people weren't likely to buy much furniture online at all, but both sites clearly thought that this would change. It didn't.

More than one product can be discussed in a competitive focus group. Introducing several products and actively comparing them with a group helps further reveal what people value and find interesting. These groups can be made up of people who have experience with one of the products or with people who have no experience with any of them (though it's probably not a good idea to mix the two groups). When presenting multiple products, the order in which they're discussed should be varied so as to minimize the bias that can come from having the same product shown first every time or having the same two products appear in the same order every time.

Usability Tests

Competitive usability tests are one of the more useful competitive research techniques. Watching people use your competition's product can show you where its functionality differs from yours (something that's not always apparent when examining the interface), where it succeeds, and where it fails.

The features you concentrate on will drive the development of the script and the tasks. Focus on the features that most differentiate your competition's product from yours, and isolate the features that will best describe whole classes of functionality. For example, it may be more valuable to investigate how well your competitor's navigation tabs work than whether people understand the icons used in the shopping cart. Even though the icons may be a flashy differentiator, the tabs are more likely to affect more people's experiences with the site.

The tasks you create will likely differ little from those you use when testing your own products. In fact, you can use the same tasks. In addition to being less work, this lets you compare how different interface organizations can create different usage patterns.

An occasional problem occurs when you don't have access to your competitor's product. For publicly available sites, it's easy to just point a browser at them or pay the nominal registration fee for an account or two. But for products that you don't have access to or for which the registration costs are outside your budget, testing them can be challenging. In such situations, many questions can be answered by prototypes that you build or, in the case of software, demo versions (which are often freely downloadable, though you should carefully read the demo's user agreement to make sure that competitive analysis is an appropriate use of the product).

It's also possible to test multiple products at the same time. Compare approaches by giving evaluators the same task on several different interfaces (but only if the tasks are actually doable on all the interfaces). Again, you can reduce bias by changing the order in which you present the competing products and not recruiting users who have experience with any of the products under examination.

For example, a general-purpose Web directory wanted to understand what worked in its competition's interfaces. Many of the category-based directories were based on Yahoo!, and that site had already been studied, so it was decided to choose two of the lesser-known directories. LookSmart and snap.com were chosen because they offered similar services to similar audiences (at the time). People who had not used either site were asked to search for similar things with both services. Localization features (either by specifying a zip code or a region) were considered to be of tier 1 importance and were the primary focus of the usability test. Most of the participants said these features were interesting, and everyone understood the two sites' different methods of communicating this functionality (in snap's case, it was an "Enter Zip Code" field, in LookSmart's, a "My Town" navigation tab). However, the ability to use the two sites' implementations of the features was different. LookSmart's "My Town" tab attempted to anticipate people's local needs, almost always incorrectly, whereas snap's option dropped people into a different directory, which was treated by the users as a separate service and understood as well as they understood the general snap interface. The Web directory decided to go with a snap-like model.

Warning

Bias easily creeps into the testing process. A subconscious nod toward a preferred solution or a moderator's slip of the tongue can skew a participant's perceptions. Although this can happen in any research situation, it's more likely to happen when the moderator has strong competitive allegiances. Focusing on behavior and current usage, rather than opinion, reduces the chances that an opinionated moderator will skew the results.

Bias is also the reason third-party consultants are particularly useful when doing third-party analysis.

Surveys

Before you run special competitive surveys, look at the customer and user research you already have. Useful competitive information can often be extracted from surveys not specifically designed for it. You can filter responses to pull out users of competitors' products and tabulate the responses of just those users, treating them as a subset of the whole survey population. Although the results will likely not be significant relative to the general user population, analyzing them may be able tell you something about what differentiates your users from your competitors'.

The process of writing a survey specifically to understand your competition follows the same steps as for a regular survey. Create reasonable objectives about what your survey is to accomplish, focus most of the questions on the tier 1 issues, and present the features in a neutral way. From the respondent's perspective, the questions should read as if an impartial third party wrote them.

Some popular topics of competitive investigation in surveys include

• Other products your users use. This question allows you to estimate the popularity of your competitors and, using published third-party audience numbers, can give you a ballpark figure for the total size of your market.

• How much they use the products. If you are using dynamic survey generation software, leverage its capabilities for creating customized follow-up questions to ask in-depth rating questions about your competition. How often do people use their product? For how long do they use it? When did they start using it? Have they ever switched between competing products? How often? Why?

• Their loyalty to the competition. What attracted your respondents to your competitor in the first place? How satisfied are they? You can create a grid of Likert rating scales that lets the respondents quickly fill out their views about specific features and attributes of the products (see Chapter 11 for more information on Likert scales).

• The features they use. These questions can be generated from a database of known competitor features, or it can be free text, letting people discuss how the product is useful to them. It can also include a feature satisfaction component.

Surveying just your competitors' users is difficult. You don't even necessarily know who they are. Unlike surveying the users who come to your door, you have to actively find your competition's. You can either field the survey to your own users and filter out those who use competitive products or you can field it to a broad range of users (say, through a banner ad on Yahoo! or a site that you feel will attract a lot of competitive users) and hope that enough of the respondents will be competitive users to justify the cost. Of course the best way to get a representative survey is to do a random telephone survey, but those are cost prohibitive in most cases.

Like other kinds of competitive research, surveys can be easily biased if they're revealed to be competitive. The knowledge that they're discussing one product for the benefit of another can change the way that people respond to questions, especially satisfaction and preference questions. It's probably OK to reveal your identity at the end of the survey (or at the end of any of these procedures), after they've submitted their responses, but it's generally better to do the whole thing as a third party.