Organize and Analyze Them | Real-World .NET Applications

If the system keeps statistics, start your exploration by looking there. This can set your initial expectations and give you some quick insight into what your users feel is worthy of comment. The numbers and proportions of comments should not be taken as a literal representation of the severity of the problems they describe, but they begin to give you a flavor for your audience and their concerns.

Formal comment analysis begins with organization. Organization is the process of grouping comments by the subject of the comment. The clusters allow you to reduce the flurry of people's words to something more manageable.

One quick way to organize comments is by the subject of the comment and its severity. The subject is the part of the service that the comment is directed toward, whereas the severity represents the analyst's judgment of the impact that the comment's subject has on the user experience. A more formal way of organizing comments is by coding them.

Coding Comments

Coding any kind of content is a process for creating an organizational scheme for a list of relatively unstructured information. The process described here is based on one described by Carol A. Hert and Gary Machionini in their work for the Bureau of Labor Statistics (ils.unc.edu/~march/blsreport/mainbls.html). It requires the participation of two people.

One person begins by working through the messages and creating categories until no obviously new categories appear. This will probably not require the whole set of comments. Comments are generally categorized "at the sentence level" so that the analyst looks each sentence in the comment. Categories can be anything—the subject of the comment, the nature of the comment (a question, a feature request, etc.), its tone ("anger," "praise," etc.), or whatever else makes sense for the analysis. There can be multiple categories inspired by a single sentence. Each category is then assigned a name, which is its code.
Using the list of codes, the first person and a second person independently categorize the same subset of the messages. The subset should be fairly large (several hundred messages would be a typical size for customer service analysis).
The two people then compare their categorizations. Did they interpret the code names in the same way? Are there any that are too general? Too specific? Are there some that don't make sense? Do two codes mean the same thing? In comparing the way that the two subsets were coded and discussing the codes, the two people define the meaning of the codes and their scope. Once decided, the two people should create guidelines for using the codes.
When the coding scheme has been established and documented, the rest of the comments—however many there are— can be divided and categorized by as many people as necessary.

Coding Software

There are several pieces of software specifically designed to assist in coding interview data. These allow you to tag specific passages in a transcript with codes and then treat the coded document as a database, extracting content based on specific codes and combinations of codes. This can be useful if you have huge quantities of text or if you're doing fine-grained analysis.

A guide to the available products can be found in www.textanalysis.info/qualitative.htm.

The U.S. Center for Disease Control (CDC) provides a free piece of analysis software, EZ-TEXT (www.cdc.gov/hiv/software/ez-text.htm).

Tabulating Comments

When the comments have been coded and organized, they should be tabulated.

Counting the number of items that fall into each code shows you where there are areas of interest or friction. In addition, once a coding scheme has been established, you can apply it to groups of comments collected over a certain period of time or from a group of people, and compare how different groups perceive your service or how perceptions change over time. For example, if the number of complaints the month after a change to the product is double that of the month before the change, there's probably a qualitative change in the experience of a sizable chunk of the audience. The doubling doesn't mean that twice as many people are unhappy across your whole user base, but it's an indicator that there is a change and that the change is for the worse.

Analyzing the Comments

Once support comments have been analyzed and an organizational scheme created, then it should be shared with the people who are in the front line of collecting user feedback. If customer service has an existing classification for their questions, these content-based categories may help organize future comments, while insights into user profiles and mental models can help them formulate answers. Once the analysis has been done one time, regularly revisiting support comments makes it easier to keep tabs on how the audience changes and how they react to site modifications.

But people's perceptions of the problems they face may not reflect the actual core problems in their experience. Knowing how they actually behave, and comparing that to how they're expected to behave, can expose the places where things are really going wrong. For this, you can turn to log files.