Applicable stages: design, code, test, and deployment.
Personnel needed for the evaluation: Usability experts: 1 Software developers: 0 Users: 6	Usability issues covered: Effectiveness: Yes Efficiency: Yes Satisfaction: No
Can be conducted remotely: No	Can obtain quantitative data: Yes

 Overview

This technique is to used to obtain quantitative data about test participants' performance when they perform the tasks during usability test. This will generally prohibit an interaction between the participant and the tester during the test that will affect the quantitative performance data. It should be conducted in a formal usability laboratory so that the data can be collected accurately and possible unexpected interference is minimized. Quantitative data is most useful in doing comparative testing, or testing against predefined benmarks. To obtain dependable results, at least 5 user participants are needed, while 8 or more participants would be more desirable. The technique can be used in combination with retrospective testing, post-test interview or questionnaires so that both quantitative and qualitative data are obtained. The technique can be used in the following development stages: code, test, and deployment.

Define the Goals

• Define the goals for the usability testing in terms of usability attribute (e.g. easy to learn, efficietn to use, easy to remember, few errors, subjectively pleasing).
• Balance the various components of the goals and decide on their relative importance.
• Quantify these usability issues by measurements such as those given below:
  • The time users take to complete a specific task.
  • The number of task of various kinds that can be completed within a give time limit.
  • The Ratio between successful interactions and errors.
  • The time spent recovering from errors.
  • The number of user errors.
  • The number of commands or other features that were never used by the user.
  • The number of syste features the user can remember during a debriefing after the test.
  • The frequency of use of the manuals and/or the help system, and the time spent using them.
  • The proportion of users who say that they would prefer using the system over some specified competitor.
  • The proportion of users using efficient efficient working strategies in case there are multiple ways of performing the tasks.

Conduct the Test

Make sure that there won't be unexpected interruption during the test. Conduct pilot test to make sure that the tools and the techniques for data collection work well. When possible, the test should be video-recorded to support data collection, so that some data can be collected or verified after the test by reviewing the video recording.

Even though this technique is aimed to collect quantitative data, it should be noticed that it's very important to collect qualitative data to uncover the user's mental process and other information behind the quantitative data and take them into account while drawing the conclusions.

Analyze the Data to Draw the Conclusions

To compare with a benchmark value (for ordinal, interval, or ratio data), mean or median can be calculated, together with standard deviation, standard error of the mean, and the confidence intervals (for details see reference 2).

To compare the data from different user interfaces, some kind of inferential statistics test can be performed (for details see reference 2).

1. J. Nielsen "Usability Engineering", pp.191-194, Academic Press, 1993.
2. N. Soken, B. Reinhart, P. Vora, & S. Metz "Methods for Evaluating Usability" (Section 5B), Honeywell, Dec. 1993.