This is an old revision of the document!
There are two ways to calculate effectiveness:
Effectiveness measures the ratio of tasks completed versus those that are not completed. It is usually reported as a simple percentage such as 80% of the tasks are completed. The data is collected using a simple binary (i.e., yes = '1' and no = '0') for each user for each task.
Mathematically:
Here is an example where the system uses 10 tasks and all the tasks are completed by 5 users. The results are:
user | Completed Tasks | Incomplete Tasks | Total Tasks | Effectiveness |
---|---|---|---|---|
user 1 | 8 | 2 | 10 | 80% |
user 2 | 9 | 1 | 10 | 90% |
user 3 | 10 | 0 | 10 | 100% |
user 4 | 5 | 5 | 10 | 50% |
user 5 | 6 | 4 | 10 | 60% |
total | 38 | 12 | 50 | 76% |
This efficiency ratio is referred to as the Completion Rate or Fundamental Usability Metric. As a result of its simplicity and ease of understanding, the Completion Rate is a popular way of reporting efficiency. However, in order to determine the efficiency a working system with real users need to be available. This is a major advantage of using the Agile Model since the efficiency can be calculated during each Sprint and used as a guidance for future sprints.
Another way to understand an application or system efficiency from a usability perspective is to count the errors instead of counting the success fo completing tasks. Every error can have more consequences requiring more work to “undo” thus negatively impacting Effectiveness and efficiency. The following is a partial list that users can make while completing tasks1) :
next
operation without intending.return
escape
key accidentallyRecording each instance of an error along with a description can help refine the system to prevent these errors in the future. For example, “user entered last name in the first name field”.
In many ways, this metric can be considered similar to the Task Complete Rate metric but instead of counting completed tasks the errors are counted. Note, many there can be many errors required to complete a single task.For example, a data entry form required for one task can have errors associated with each field. Each of these errors can have a negative impact on the overall efficiency.
Another enhancement to this metric wold be to assign a weight to the error. For example, data entry errors are less dramatic and important than unintended actions. Therefore, unintended actions would receive a higher weighting than data entry.
TBD