Section 8.2. Criteria for Building and Evaluating a Challenge Question System

8.2. Criteria for Building and Evaluating a Challenge Question System

We begin our introduction to the design of challenge question systems by introducing criteria that are helpful in both their design and evaluation. These criteria relate to the privacy, security, and usability of the challenge question system.

8.2.1. Privacy Criteria

In environments that use personal information, it should be common practice to follow recognized privacy principles to protect answers to challenge questions.^[1] For the use of challenge questions and answers to authenticate users, one principle in particular seems relevant: collection limitation. This criterion serves to limit the collection of personal user information to what is necessary for the purpose of authenticating an individual. Adherence to this principle helps to ensure that only information necessary to support a suitable level of security and usability is maintained.

^[1] "Guidelines on the Protection of Privacy and Transborder Flows of Personal Data," Organization for Economic Co-operation and Development (OECD), 1980.

Designers should give particular caution to using questions that ask for personal information, such as "What is your mother's maiden name?", because the answer, while possibly obscured (hashed), will be stored at the account server. Preference should be given to asking nonpersonal questions, provided that they offer sufficient security and usability.

In addition, answers to challenge questions should be used only for the purpose of recovering user access to one's accountconforming to a use limitation principle. If challenge questions are to be used for other purposes, individuals should be notified and their consent obtained. Furthermore, care should be taken when asking for answers that users may find sensitive. Therefore, best practice involves offering as much choice as possible (while maintaining a suitable security level) to individuals for question selection, allowing individual control over the answers that are provided.

8.2.2. Security Criteria

The security of a challenge question system is related directly to the confidentiality of the challenge question answers. Other properties such as integrity and availability are also important to the security of the overall system, but are not the focus of our framework. The following security criteria apply primarily to the content of individual questions and answers:

Guessing difficulty

Answers should be difficult to guess and have an answer space with a fairly uniform distribution. Questions that can be guessed successfully in a small number of attempts (for example, "What is your eye color?") do not make good challenge questions.

Observation difficulty

The answers to challenge questions should be difficult for an attacker to retrieve or observe easily. In particular, the answers should not be available from public sources. Questions that individuals are often asked to answer, such as "What is your mother's maiden name?", do not pose much observation difficulty. Unlike guessing difficulty, a determination of a question's observation difficulty is more subjective, as the difficulty of determining the answer is dependent upon a number of factors (e.g., the availability of the answer). In addition, observation difficulty will differ for individuals that have different relationships with the userfor example, family, friends, acquaintances, colleagues, or strangers.

For an authentication system that consists of multiple questions, additional criteria should be considered, including the total guessing and observation difficulty for the entire set of questions. In addition, answers should be unrelated so that both their availability and entropy can be maintained independently when multiple questions are used. One way to support answer independence is to use independent questions (questions that would encourage the submission of independent answers).

8.2.3. Usability Criteria

The usability of a challenge question system is concerned with providing a user-friendly experience at the stages of both answer registration and subsequent answer presentation. The following usability criteria should be used when evaluating a challenge question system:

Applicability

The applicability criterion attempts to characterize the size of the target population for which a question might be applicable. For example, a question about pets would not apply to those individuals who have never owned a pet. Attempts should be made to support highly applicable questions, although not at the expense of other criteria. A sufficiently large quantity of possible questions can be used to ensure enough coverage across the population of individuals.

Memorability

An answer is memorable as long as the user is able to recall the answer. This generally implies that the answer would be personally significant. Information that is used frequently will be more memorable, indicating that answers reflecting the habits, activities, or practices of users provide suitable answers. For an answer with high recall, only the likelihood of recalling an answer rather than the likelihood of knowing some answer is considered for the memorability criterion. For example, although many people may not know their high-school locker combination, those who do know the combination are likely to be able to continue to recall the answer. However, such a question would be applicable (as discussed for the applicability criterion) to only a smaller set of individuals that would know of an answer.

Repeatability

There are at least two aspects of answer repeatability to consider. First, answers should have few syntactic representations. For example, a question involving an address might be answered with "St." or "Street," and the word contractions (e.g., "St." versus "Street") may cause a discrepancy. Second, answers should have a semantic value that remains the same over time. For example, questions about favorites may be susceptible to the answer changing over time. For this reason, questions asking for "favorites" should be avoided in favor of "first time" or perhaps "memorable" qualifiers.

Additional usability issues include the number of questions and answers stored and the number of answers required to authenticate. These issues are discussed further in the next section.