What is usability testing?
Usability and user-experience design researchers employ a variety of
empirical research methodologies which most often include direct observation of actual users attempting to use a product to complete realistic tasks in a
natural environment. Usability tests generally seek to address two primary
objectives:
- To learn more about users' needs and the conditions in which they work
so that product designers can create more efficient and functional product
designs.
- To evaluate whether or not a specified group of users can complete a specified task under natural conditions. These tests are designed to
identify problems with products before they are released to the market to
ensure that only the most usable products are manufactured.
The DK Handbook usability test results
Why did Longman commission a major scientific usability study?
- To demystify how students use handbooks and where and why they tend to get stuck or frustrated when using handbooks on their own.
- To ensure The DK Handbook’s format was not just visually appealing, but a more useful resource for students.
- To get a student/user perspective on how The DK Handbook compared to the traditional handbook model.
Methodology
Dr. Tharon Howard, Director of the Clemson University Usability Testing Facility* conducted one-on-one, direct observation tests with 12 college freshmen (6 from two-year schools and 6 from four-year schools). Student/users were given tasks that required the use of the same chapters from two handbooks—The DK Handbook and a market-leading handbook from another publisher. Students were videotaped as they performed the tasks, first using one handbook’s chapter then the other handbook’s chapter, talking aloud and saying what they were thinking as they worked. Each test lasted 2-3 hours, for a total of approximately 30 hours of direct one-on-one observation of student users.
The study focused on three sections of The DK Handbook: one grammar (Commas), one research (Evaluating Sources), and one documentation (MLA). Spiral-bound, four-color booklets containing these chapters and a second one containing the same sections from a market-leading handbook were tested. Realistic scenarios were created that would require students to use the handbooks to perform specific tasks so their progress could be observed. Nearly 30 hours of video was recorded, and the users’ behaviors were subsequently coded and analyzed.
Summary of Major Findings
Nine out of the twelve students tested preferred The DK Handbook. DK also came out a full point higher than the market leader on a four-point usability scale. These findings were used as a jumping off point for the development of the rest of the handbook. Aspects of the presentation that worked especially well in the tested chapters were used as models for subsequent chapters.
- The overall strategy suggested by the usability test was to help students “set goals for completing a task and then step them through the process needed to accomplish the task.”
The DK Handbook’s section on Evaluating Sources succeeded best in laying out the goal for the chapter and then stepping students through the process, and was thus used as the model for other sections in the handbook. The Evaluating Sources section starts out with an overview of the topic/process, then lead the students to “detail” pages that gave them more in-depth information on individual steps in the process. Detail pages then lead to application pages, where students see examples and apply what they’ve just read. This “layered” presentation (overview > detail > application/examples) imitates what students experience when using the Web and was highly praised by the student users.
- Students look for examples first, and navigate handbooks looking for examples or patterns that match the problem they are having. When using handbooks students feel successful when they “find a match” for their problem, but get frustrated quickly when they cannot. DK’s example-driven approach and design that imitates patterns and processes helps students find the right matches quickly.
- The text needs to be clearly organized around patterns, and its internal cross-references and navigational aids must be clear and consistent. The Comma section was deemed most successful in organizing itself around patterns and was used as a model for subsequent chapters.
- The findings for the MLA Documentation section were perhaps the most valuable and exciting.
The study showed that students often do not understand the type of source they are documenting which can lead them to follow the wrong citation examples. Therefore, a section was added to teach students how to identify the type of source they are documenting so they can correctly distinguish for example, an anthology from an edited collection.
*The reported usability test was conducted by Dr. Tharon Howard, Director, Usability Testing Facility, Clemson University under Contract with Longman Publishers. This does not constitute an endorsement by Clemson University or by Dr. Howard.
How to conduct your own handbook usability study
The great “black hole” for most educators is understanding exactly how students use the books they are assigned. What do students really do when we ask them to use a textbook? For instance, do students read prose explanations of grammatical rules found in most handbooks or do they focus on examples and learn patterns? The best way to truly answer this question and to be able to say that a text is “usable” is through direct observation of students actually attempting to use a textbook to complete a realistic task. To acheive this, you can try conducting your own usability study.
To help you conduct your own, informal usability study, here are the materials you’ll need:
|
