13.1 Measuring Accuracy

Recognition accuracy is typically described by a number of measurements. The first step is to separate data into two groups: in-grammar and out-of-grammar. The in-grammar data are things callers say that are in the defined grammar for the relevant dialog state. Out-of-grammar data are not in the defined grammar.

In-grammar data fall into one of three categories:

1. Correct accept: The recognizer returned the correct answer.

2. False accept: The recognizer returned an incorrect answer.

3. False reject: The recognizer could not find a good match to any path in the grammar and therefore rejected rather than return an answer.

Out-of-grammar data fall into one of two categories:

1. Correct reject: The recognizer correctly rejected the input.

2. False accept: The recognizer returned an answer that is, by definition, wrong because the input was not in the grammar.

From the point of view of callers, the distinction between in-grammar and out-of-grammar is irrelevant. All they know is that they said something, and the dialog with the system is not proceeding as they expect. Especially in the case of systems that are mainly single use (rather than called repeatedly by the same person), the user is often unaware that there has been an error and is confused by how the dialog is progressing.

On the other hand, from the point of view of the designer or developer trying to improve system performance, two things the distinctions between in-grammar and out-of-grammar data and the categories of errors are important, as you will see.