Identifying User Requirements Through Prototyping and Usability Testing:
The Light is Yellow, Proceed With Caution!

by John Harris

Introduction

One of the biggest challenges we in the usability community face is getting project teams to involve us early in the project cycle. We have come a long way from the days when usability was just the final checkmark on a project plan. We are now getting usability involved early and often throughout the project lifecycle. The emergence of paper prototyping as a valued design technique has helped this progress. More and more project teams are being exposed to the value of paper prototyping as an early design technique. Prototyping is a fast and cheap way to incorporate user feedback early in the design process and to explore several design concepts. As project teams jump on the bandwagon of paper prototyping, we have a new challenge--ensuring that paper prototyping is not abused!

What am I talking about? I'm talking about using paper prototyping for the wrong reasons. Let me explain. In his article, "Users First: Prototyping Web Designs," Jakob Nielsen gives us his official blessing to take it easy in our next web design project. In fact, he says "lazy is good." His point is that you don't have to work yourself to death, just remember his golden rule: "The more frequently you get usability data and iterate your design, the better the usability of the end result." Lazy may be good from a usability perspective; however, when the project team gets lazy you'd better watch out! Especially if they get lazy and want to use paper prototyping for the wrong reason. What is the wrong reason? It's simple, here is golden rule number 2: Do not use paper prototyping or any other form of usability testing as a means to gather user requirements. There are other techniques more suitable to do that (e.g., task analysis, surveys, focus groups, contextual inquiry, etc.). Paper prototyping should be used to collect usability data, not user requirements.

The Scenario

Often a project team will not do a thorough user analysis study. Instead, they will gather some high-level user requirements and then they will call us. We think, "Great, the team is getting me involved earlier than they used to. I can help them with conceptual design and we can do some paper prototyping on the design, etc. etc." Granted, we could have helped them even earlier in the process with requirements gathering, but at least they didn't wait to call us right before implementation!

So off we go with the project team to create a paper prototype. We get out our paper, pens, post-it notes, and create a prototype. We then test our prototype by bringing in real users and having them do real tasks and we collect the data. What should we do if we find most of the data we are collecting is really user requirement data? For instance, we find users telling us they need a way to do this or that or they need to be able to perform this function or that function. Even if the project team is thinking, "Wow, this paper prototyping really is a great idea, look at all this feedback we are getting," we should be concerned. We should be concerned because it appears that an adequate user requirements analysis was not completed. Instead of letting the team accept this type of data at face value, we may want to suggest that the team revisit the user analysis phase. If we don't raise this issue, teams will continue to break golden rule number 2: Do not use paper prototyping or any other form of usability testing as a means to gather user requirements!

Some of you may be thinking that there is nothing fundamentally wrong with gathering user requirements while testing using a paper prototype. In fact, many of you may even welcome it. In practice, it is almost impossible to test a paper prototype without uncovering some user requirements. Depending on the method and extent of the initial requirements gathering, you may find yourself uncovering a large number of user requirements or hardly any at all. Regardless of the number of user requirements you uncover, they should be used with caution. Be especially cautious if significant design changes will be made based on those requirements because they may not be representative of the entire user population.

Usability Issues vs. Requirements

When we test a system using a paper prototype we often blur the line between a usability issue and a user requirement. What we should be looking for are usability issues, not user requirements. Let me give a simple example of the distinction between the two. Let's say your prototype requires the user to enter a time (e.g., hours and minutes). When you test your prototype, you find that all 4 users are having difficulty figuring out how to enter the time because of the type of interaction you chose for your design. Let's also say that 3 of the 4 users comment that it would be easier if they only had to enter the hours and not the minutes. They tell you that they don't need to be that specific on the time. In this case, the usability issue is the difficulty the users had entering the time (probably due to the type of interface interaction being used). Whether or not minutes need to be entered is the user requirement .

Let's take this example one step further. Let's say that based on the feedback from these users, the project team concludes that they need to change the time interaction so that users only have to enter hours. The project team may think they are killing two birds with one stone: eliminating the usability issue and implementing a user requirement. So what happens when this system is implemented and the helpdesk or support area is flooded with calls from users saying they need to be able to enter minutes.....oooops!

This example illustrates why I caution you about using paper prototypes to gather user requirements. Actually, the same can be said about any form of usability testing from low-fidelity paper prototypes to high-fidelity online prototypes. I'm focusing on paper prototypes in this article because of the increasing possibility and probability of project teams getting lazy in gathering user requirements and expecting to use paper prototyping to do it for them. It could become easy for them to fall into the trap of thinking, "Let's get some high-level requirements and then slap a paper prototype together and iron out the details with it." We have to be cognizant of and educate project teams about this inappropriate use of paper prototyping.

Usability and Statistics

Now for the hard part. Besides the time example I used previously to illustrate the danger of using paper prototypes to gather user requirements, there is also a more fundamental and scientific reason for not doing it. Usability research tells us that we can identify approximately 80% of the major usability issues by using 4 to 6 evaluators. Anyone who has conducted usability studies can probably attest to the validity of this research. After a few evaluators you start to see the same usability issues over and over. This research, however, cannot be generalized to include user requirements. In other words, we cannot say that we can uncover 80% of user requirements from 4 to 6 users.

Usability issues and user requirements are different data and should be collected separately using different methods. User requirements should be collected with a much more statistical approach than provided by a typical usability evaluation. You often hear people in the usability community say that usability is not a statistical science. This is true when refering to usability tests used to uncover usability issues. For instance, if we tested a system with 4 users and found that one user had a problem we would not claim that 25% of all the users will have the same problem. In addition, we would not claim we are 90% confident plus or minus 5% in our results. We would, however, try to determine the root cause of the problem because it could be a problem for others. In that sense usability is not a statistical science. On the other hand, when it comes to gathering user requirements, we should strive to be more statistical and not use usability testing for user analysis.

When a thorough user analysis is completed up front and incorporated into the design, the prototype can be used for its intended purpose - - early user feedback on the usability of the system. Many usability professionals who claim that usability is not a statistical science would also argue that a statistically valid user analysis is not required. They would argue that you only need a small number of users (10 to 20) who are representative of the user community. This argument works well when we are trying to sell the concept of a user analysis to upper management because collecting data from a sample of 10 to 20 users is usually cheaper than a statistically valid sample. This argument doesn't work well from a scientific point of view nor does it work well from a design point of view.

From a scientific point of view, a sample of 10 to 20 users cannot be representative of many user populations, unless the population size is relatively small. The laws of statistics tell us that if we want to ensure a high level of confidence that our sample represents the population, we need an unbiased and sufficiently large sample. To obtain an unbiased sample we must use a probability sample that is randomly selected. A sufficiently large sample is determined by the population size, the level of confidence you want, and the amount of error you will accept. Many people believe that you always have to have a large sample (in the hundreds) to achieve statistical significance. This is not always the case. For instance, a sample size of 63 would be statistically significant for a population size of 1,000 (90% confidence and plus or minus 10% error).

From a design point of view, functionality decisions based on requirements gathered from a sample of 10 to 20 users increase the potential for error because this small sample size may not be representative of the user population. The risk is even higher when significant functionality changes are made based on requirements gathered during a usability test that uses only 4 to 6 users. When we collect requirements data we should, as much as possible, ensure that this data can be generalized to the user population. In contrast, we don't need to be able to generalize usability data to the user population. When we find usability issues based on a sample of 4 to 6 users, we resolve those issues because we believe other users may have the same problems. It is not important for us to conclude that all the users will have those same problems. When it comes to user requirements, it is important for us to know how those requirements affect the entire user population. In other words, we need to be able to generalize user requirements to the user population.

Conclusion

So what do we do? We must educate project teams on the importance of conducting a thorough, up-front user analysis and not let them think that they can (or even should) rely on usability testing to do it. The analysis should include as many users as practical from a business standpoint. You'll notice that I didn't say we had to achieve statistical significance (even though I would like to say that). I realize that we don't operate under unlimited resources or freedom and it is often not feasible to collect data from a large enough sample to have statistical confidence in our results. However, that doesn't mean we should not advocate the importance of adhering to the laws of statistics as much as possible when determining the appropriate number of users to involve during the user analysis phase. When we strive to reach statistical significance (even if we don't), what we are really doing is reducing the probability of error in our data. This can have a significant impact on our design and ultimately the usefulness of the system.

When we do find user requirements from paper prototyping or any other type of usability testing, we should proceed with caution. This is especially true when the requirement will result in a significant design change. In those cases, you may want to conduct further analysis of the user population before making changes. Often we uncover a user requirement during usability testing that seems so obvious that we don't see a need to do further analysis (e.g., users request a spell check function in an application that requires a lot of text entry). In those cases, the risk of making a mistake is not as high and the team might decide to forgo further analysis. Regardless of the nature of the requirements found, we should treat them with caution and not accept them at face value because they may not be representative of the user population. The light is yellow!

References

Nielsen, Jakob, Jan 21, 1999. "Users First: Prototyping Web Designs", http://www.zdnet.com/devhead/stories/articles/0,4413,2190669,00.html

About the Author

John Harris is a Usability Specialist at Edward Jones. John works with development teams to ensure systems are easy to use and useful. Prior to joining the firm, John was a human factors engineer in the United States Air Force. In the Air Force, John was responsible for the operational test and evaluation of Air Force systems ranging from software systems to cockpits. John received a bachelor's degree in human factors engineering for the Unites States Air Force Academy and a Master's degree in Industrial/Organizational Psychology from St. Mary's University ---- San Antonio Texas.