Usability Testing Tutorial: A Complete Getting Started Guide

This comprehensive guide to Usability Testing explains what is Usability Testing, why we need it,  its methods, examples, process, and tools:

We test a software product, website, or service to measure the quality of the deliverables. This test consists of both functional and non-functional testing. Usability testing is one of the many non-functional testing types.

Usability Testing is a realm of the IT world that tries to incorporate how “people” would like their applications to be– which is never easy. It is like trying to read minds. What sounds like common sense, is a profound science when it comes to testing usability.

Why is Usability Testing Necessary

In this article, we will learn and understand the necessity of usability testing, understand its various categories. We will also see the different tools and methods. Learn about different phases of Usability testing and ways to analyze the results.

[image source] 

We create websites or products or services for users or customers to provide solutions for their requirements. By performing usability testing, we can understand whether the created solution satisfies the customer or user requirement or not.

It may happen that the service or product created can be working as expected in functionality, but is not user-friendly. Hence, it cannot attract the users or customers of the product and provide a better user experience. This will eventually impact the business of the company, as users would find other products or services much easier to use.

What Is Usability Testing

It is a non-functional type of software testing. It is broadly divided into understandability, learnability, operability, attractiveness, and compliance. Usability testing is to determine the extent to which we understand the software product, easy to learn, easy to operate, and attractive to the users under specified conditions and requirements.

This type of testing is usually executed by real-life users and not the development team. The development team is the one who has created the product, and hence they fail to find fewer defects that are related to user experience.

Usability testing is mainly divided into three categories.  These are:

• Explorative
• Assessment
• Comparative

Let us understand these categories in detail:

#1) Explorative

We usually consider this category in the early stages of the software testing process. The earlier usability testing is executed in the testing process, the less the risk in the product. The design of the product and concepts related to the product or service is usually considered for usability testing in this stage.

#2) Assessment

This category describes the assessment of the end-to-end execution of the test and also analyzes the effectiveness of the product and user satisfaction.

#3) Comparative

In this category, two or more products of similar technology are compared at different attributes as the design of the product, advantages, disadvantages of the product or services, and which helps in selecting the product that provides a better user experience.

As the name suggests in the comparative category of usability testing, we compare two or more similar kinds of products and understand the features, working, pros, and cons of the product. This helps to decide the areas of improvement in the product build to attract customers and rule the market.

Usability Testing Methods

Following are the usability testing methods that are most commonly used:

#1) Hallway Testing

This method is one of the most effective and cost-efficient methods available when compared with others. In this method, a few random people are given the website or product to test rather than trained professionals.

As random people test the service without prior knowledge of the product, they test it more effectively and provide more accurate results and honest feedback for improvement, if any.

#2) Remote Usability Testing

As the name suggests, remote usability testing takes place by people who are located at remote locations, i.e. who are located in several states or sometimes in several countries for testing purposes. This type of testing is done remotely and issues can be reported if found.

Feedback can be recorded and can be submitted by random people and not a technology expert. Sometimes remote testing is executed using video conferencing. This type of usability testing adds less cost value as compared to other types of testing.

#3) Expert Review

The expert in the field is asked to test the product or service and provide feedback and then submit the results. This type of testing is usually expensive, as the company needs to hire a technology expert. An expert review can also be done remotely to submit the results.

This type of usability testing is executed quickly and takes less time. The expert quickly finds the loopholes and detects the flaws in the product or service. Such testing is usually expensive, hence clients avoid this option.

This testing is reliable because of the experience possessed by the expert. Expert review is worth the expense when the product has a critical feature, and the company needs to find the user’s reaction before launching the product.

#4) Paper Prototype Testing

Paper prototype testing is one of the most traditional approaches for usability testing. This method involves a dry run of the test execution, hand sketch, drawing of the models or prototype.

Discussing the flow and drawing them on paper and considering all the possible inputs and scenarios and conditions is the objective of this type of testing. This is one of the primary types of testing which is mostly observed across all the projects to eliminate primary issues. By performing paper prototype testing, one can have more clarity on the execution process.

The paper prototype testing is usually done within the project team. Hence it is considered in the earlier stages of the testing process. This is relatively a cheaper method of usability testing but is not the most effective way of testing as it is more time-consuming at times and there is a higher possibility that even after testing we may lose a few issues.

#5) Automated Usability Testing

As the name suggests, this testing method is performed by writing automation scripts. We write scripts for automation and when the scripts are triggered, test cases are executed. After the execution of the test, results are recorded and submitted.

For this type of testing method, a company needs to hire a resource who is well acquainted with writing scripts and building an automation framework. This is one of the most used testing methods as this is not as expensive as the expert review type of testing and not as primary as the paper prototype testing method.

This is one of the effective types of usability testing as there is less human intervention because of automated scripts and less likely to lose any issues.

Usability Test Tools

There are many tools that help this process along. All these tools can be roughly categorized as follows:

Category #1) Create tasks/tests and give them to users (finding the users and giving them tasks is a manual activity, outside of the tool). While they are performing these tasks, the facilitator could watch their screen and interact with them. This could be in the lines of how you would “Skype”.

Category #2)  Tool provides users or you can pick your own users. You can submit your page/design and the tasks to be performed. The tool, in turn, will provide you with the videos of the user interaction plus the user’s comments. You can make your own analysis.

Category #3) Tools that use eye-tracking and heatmap methods to determine which part of the page the user has spent the most time on. Some of the tools in this category also record the user’s clicks, scrolls, mouse moves, etc.

Category #4) Tools that provide you with feedback based on the website, page or wireframe that you submit as input. Some tools of this type also provide surveys that help in giving conclusive evidence regarding usability issues.

Category #5) Tools that recruit users for your usability test.

The above is a very broad classification. There are many other tools. And also, the division into a certain category is not always so clean. Sometimes the tools employ multiple methods at the same time.

Here is a list of tools category wise and the link to find more info about them. 

Using different tools for usability testing can help the user complete the task in a more effective and efficient way. These tools help in the identification of a better user experience. Not all the projects use tools for usability testing as they are expensive and clients do not want to invest in tools for this purpose.

CrazyEgg is one of the most popularly used usability testing tools. It is a click-based user experience tool which makes it easy to use.

Here is the list of most popularly used tools:

1. UXtweak
2. CrazyEgg
3. Userzoom
4. Loop11
5. Optimizely
6. TryMyUI
7. UserTesting.com
8. Qualaroo
9. Usabilla
10. UserFeel
11. Userlytics
12. Validately.com

Further reading => List of Best Usability Testing Tools

Advantages and Disadvantages

Advantages of Usability Testing:

• By executing a usability test, we can improve on the flaws of the product, if found any, and rectify them before the launch of the product. Executing tests will thereby help in boosting product quality.
• Many times internal discussions are headed towards debates that can be resolved after performing usability testing, which helps in understanding the user experience better.
• Sometimes it is observed that the development team fails to detect a few minor defects which can be found by executing usability testing.
• As the feedback is directly from the user side, it becomes beneficial for the business to improve the product according to the target audience.

Disadvantages of Usability Testing:

• Financial constraint is one of the major disadvantages as one needs to recruit and manage the resources for usability testing.

Different Participants or Elements

The participants of usability testing depend on the complexity and budget of the project. If the project is highly complex and contains crucial and critical features, then more participants are considered. Whereas a small team of participants is considered for testing the mid-level complexity project.

In most cases, there is a researcher and a few participants for usability testing. The number of participants ranges between three to twenty-five, while any team of participants with five to ten members is mostly observed.

The researcher is also known as a facilitator or moderator, where the researcher is the one who observes the participants and moderates the task performed by them and also records the feedback.

Mainly there are three elements, which are as follows:

Researcher, Participant, and Task to be performed.

The participant is the one who executes the task whereas the facilitator guides the participants performing the task. The facilitator and the participants have brainstorming sessions among themselves so that participants can test the service more effectively and in the right direction.

The facilitator also checks if the results submitted are accurate. The researcher passes the instructions to the participants via task sheets or verbally or on video conferencing if testing is performed remotely. The facilitator can ask cross-questions to the participants just to check that the task performed by participants is on track.

If the researcher found any deviation in the task performed, then the researcher can control the execution flow.

Different Phases Of Usability Testing

Let us understand how to start usability testing? OR what are the different phases of usability testing?

Same as for other testing techniques, usability testing also should be started as early as possible. Different stages of includes planning, recruiting, execution, data analysis, and reporting.

[image source] 

#1) Planning

This stage is the first stage in software testing. The planning stage is when the objective of the project is defined. The documentation of the project is done in the planning stage. This is one of the most fundamental and critical stages in software testing.

In this stage, we plan the road map to proceed with the testing process. We plan for questions like what to test? how to test? who would be testing which features? which different scenarios should be considered for testing?. In the planning stage, we gather the business requirement and prepare the plan.

#2) Recruiting

In this phase, we recruit the facilitator and the participants according to the budget and complexity of the project.

#3) Execution

In this stage, the participants perform usability testing and perform the assigned task.

#4) Data Analysis

During the data analysis stage, the feedback received from usability testing is analyzed. Results are categorized and patterns are identified. The conclusion is generated from the feedback submitted to improve the product and thereby improve the business.

#5) Reporting

The feedback and findings received are shared with the customers and the development teams across the project. Discussion on the areas of improvement and its implementation is considered in this stage.

[image source] 

Factors To Consider While UX Testing

First and foremost, the earlier you test the better it is to prevent risk, wastage of time, and budget of the project.

1. Areas of improvement can be handled more easily if testing has started at an early stage.
2. Select a team of users who would be executing the task.
3. We need to understand the target audience to know more about the user experience and the product. This will help us understand who will be using the product, what are the needs of the user of these categories. Purpose of using the product. Is the product able to satisfy the users?
4. Possible ways of using the product and additional features which the user might look for in the future.
5. Discuss the improvement areas with the development team and work with them accordingly.
6. We also need to consider the performance and the rate of response while testing the product.

Usability Testing Examples

For analyzing testing results, we should always think through users’ points of view. We need to keep in mind the features that will impact and improve the user experience. To understand this better, let us discuss a few examples below.

Example 1: Usability Testing Of A Travel Website.

Consider the below scenario for testing a travel website.

#1) Booking a flight

While considering scenarios for booking a flight, the user should be able to select one-way and return flight options. One option which can improve the user experience is if the user could select the multiple cities option.

#2) Booking of hotels

While considering that the user is traveling to a city, he/she may also look for accommodation. If a user could find both flight booking and hotel accommodation in one place, can add up to user satisfaction. As the user does not have to browse ten different websites. Everything is taken care of in one place and hence more users will be attracted towards the website.

#3) Renting a car/two-wheeler

While the user is planning to travel to a city for vacation or for small business, they may require a vehicle for traveling locally. If the website has the option of renting the vehicle, it may attract the users to the website positively.

#4) Tracking the flight

This is one of the important features which customers look for before arriving for the flight.

This screen should show the flight status of both from arrival and departure as per the source city mentioned. Also, for a better user experience, the page should contain search options for a quick search based on Airline, Time, or Destination.

#5) Things to do/best places to visit

This could be an extra feature that will interest the users of the website. Through this feature, the user gets to know about the place better and things to explore around. This can add the possibility that the user would rent a vehicle for local transport. If the website has the feature of renting a vehicle and places to visit around, this will add an advantage.

#6) Managing past booking

This feature is of great help for both the user and the business owner as the user can easily repeat the recently traveled option if the user has similar requirements. Because of this feature, the business owner can understand the type of user and the type of bookings made, which can help in providing better deals and offers to the user to improve the business.

This feature is of great help in case if the user needs to submit a receipt of the bookings made for reimbursement, which just got misplaced. Because of this feature, the person can get a duplicate receipt from the website.

As this feature helped the person, in future the person could consider this website and would be the mouth of a good word to others.

#7) Customer Service

Besides all the features we have on the website, if the user comes across any difficulty or needs any help, customer service is the feature that can be looked for. This is a very critical feature as it directly relates to customer satisfaction, which is our objective while usability testing.

Getting connected to customer service via call, email, or chat. Posting frequently asked questions will also help customers.

Let’s discuss another application

Example 2: Getting Transportation Online Using Uber

#1) Booking a ride

This is the primary functionality of the application that even if 100’s users are trying to book the ride at the same time, the user should be able to book the ride.

#2) Types of ride

The user should be able to view and book all the types of rides available starting from micro, mini, prime, Sweden, carpool, etc. The number of available seats in the ride is also displayed with the type of ride.

#3) Estimated time of arrival

While booking it should display the estimated time of arrival so that the user can decide between the different ride options available.

#4) Estimated cost while booking the ride

While booking it should display the estimated cost so that the user can book from the different ride options available.

#5) Ride later option

If the user wants to pre-book the ride, so the person should be able to book a ride of personal desire.

#6) Share ride option

If the user has booked a ride for his friend and wants to share the ride details with his friend. The person should be able to share the ride details.

#7) Adding different payments

The user should be able to pay in all the modes. Like a credit card, debit card option, UPI, wallet, or cash payment. The application should ask for confirmation before saving the credit card, debit card credentials.

#8) Customer support

Users should be able to connect to customer service via call, email, or chat. Posting frequently asked questions will also help customers. It is also helpful in case if the person by mistake forgot any belongings, or if the driver has charged him incorrectly, also if the driver tried to misbehave or if the user wants to file a complaint or seek any kind of help.

This feature proved to be one of the most important features for users seeking help.

#9) Cancel the booked ride

If the user booked a ride just by mistake and wishes to cancel, then the user should be able to cancel the booked ride. 

#10) Call the driver

It may happen that the driver is not able to locate the rider at the pickup location, or the user is at the pickup location and not able to find the driver. This feature reduces confusion and saves time if the driver is not aware of the location, he can contact you and reach you easily.

What Happens After Usability Testing

While considering the above example, the features mentioned are the ones received as feedback from the user while using the travel website and online vehicle booking application. There could be few more required features depending upon the customer requirements and usage of the website.

After completion of the usability testing, the first thing to do is to bring together all the feedback and conclusions and then to discuss what all we can be accommodated as improvement of the product/service. How much time will be required to incorporate the changes? If the project team is not clear about the changes to be implemented, then they can discuss again and clear the ambiguities.

The impact of the enhancements depends on the target audience who is going to use the application, the purpose of the application, and the features that need to be added or improved on the website. After all of the above points are discussed, it is a management call while considering the features of the product. This also depends on the business impact of the feature.

More About Usability Testing

[image source]

A usability test makes sure that the interface of an AUT is built in a way that fits the user’s expectations with respect to meeting requirements (effectiveness) easily (efficiently) in a simplistic satisfying manner.

The primary focus is on:

• Ease of use
• Ease of Learning or familiarizing with the system
• The satisfaction of the user with the entire experience

Usability has many dimensions to it. It is all about the user’s ‘experience’ during their interaction with an application and their ‘feeling’ towards it. A structured Usability Test translates this experience/feeling into a Validation Process.

Why It Is Performed

The Web and mobile applications have ruled the business world in recent times. These apps being efficient, effective, easy, simple, appealing, engaging, etc. is very critical for them to be embraced by the customers. The Usability Test is all about determining if a site is what the user would want to use and come back to or not.

This not only applies to software systems. Any machine/interface that has human interaction has to satisfy these rules. How do you ask? Democracy would suffer if the voting machines were not usable.  I wouldn’t vote if I had to click more than one button to choose my candidate, would you? Exactly!

For a more software-specific example, check out this 300-million-dollar article by Jared Spool that will clearly explain how the placement of a button has caused the business to be impacted.

When Is Usability Testing Conducted

As testers, we know that the earlier a defect is found in the SDLC the cheaper it is to fix it. The same concept holds true for testing Usability also.

Usability Test results affect the design of the product. So, ideally, the usability test should start at the design level. But that is not all; software undergoes many changes/interpretations/implementations throughout the SDLC process. To make sure that we do not make usability-related mistakes at any of these steps – this testing should be conducted often and continuously for maximum results.

Who Performs This Testing

It can be done as an internal process, when the designers, developers, and anyone else can sit down and analyze their system and get the results. Based on these results, the design and/or code can be modified to be in accordance with the changes they all agree on.

A more advanced approach is to hire real-time users and give them particular tasks. A facilitator/s can devise these tasks and get the results from the users.

The users can then provide information, on whether:

• The task was successful or not
• The task could be performed easily
• Was the experience interesting, engaging, or annoying – their feeling towards the software

How To Conduct Usability Testing – Process

Testing is a validation of software against its requirements.  A usability test is not different – The only requirement, in this case, is to validate if the software is as per a mental map of how a user would want the software to be like, what makes it comfortable for them to use, what kind of holistic experience is the user going to take away from the interaction, etc.

These are just a few of the ways in which this testing is carried out.

#1) During the design phase, you could just take draw your website/application design on a piece of paper and evaluate whether it is going to work or not.

#2) An exploratory method would be to build the site and perform some random tests (by the development/design/QA- any or all internal teams) to determine usability factors.

#3) Hire a set of real-time users to work on the site and report results.

#4) Use a tool that would provide statistics based on the input wireframes and designs submitted.

#5) Hire a third-party usability team that specializes in this field.

#6) Submit your site design and wireframes to an external evaluator and get results from them.

The structured Usability Testing process contains the following steps:

Step #1) Identifying the users to perform the Usability Test – it helps to choose the set of users that is close to how the real-time users are going to be. Care has to be taken not to pick experts or complete newbies. The experts are going to simply run through the entire process and the novices need lots of background training to even get started- neither situation is optimum.

Step #2) Designing the tasks that the users are going to perform on the application –A list of situations which the users are going to use the application form is to be made prior to starting the test. This can include something like: ‘Search for an X-box and buy it” or ‘submit a customer care question” etc. on an eCommerce site. The tasks should closely represent the real transactions the users would use the site for.

Step #3) Facilitating the testing – The usability team will have the users perform the tasks on the site and are going to gather information regarding the test progress and results. It really makes a huge difference when the design team participates while the users are working on the application. It gives them a better picture of how the app was used and where it did not deliver what the user wanted etc, firsthand.

Step #4) Analyze results – At the end of the test, we might end up with the time it took to perform tasks, whether the task was successful or not, etc, so basically raw data. The results have to be presented to all the stakeholders and analyzed for the identification of potential problem areas.

From what we have discussed so far, these are my impressions:

#1) Testing Usability is not an out-and-out QA task. A role that traditional QA teams could play in this context is that of a facilitator who would create tasks, conduct the test, and interpret the result. The ‘tester’ in this case is actually the user.

#2) The end results or outcome of usability testing are not defects. It is simple information on how the user felt, what they liked, what they did not – raw data. The usability tester’s job does not end with reporting the results (in the case of traditional QA jobs, we report bugs and do not get involved in fixing them), it includes providing suggestions on how to make the software more conducive to user interaction.

#3) By understanding these concepts, we can actually make a difference as functional testers. To add more value to our role as QAs we can apply these usability concepts and provide insightful suggestions on how to improve the usability of an application.

Frequently Asked Questions

Conclusion

In the above tutorial, we have learned in detail about Usability testing in the software testing life cycle. The tutorial also covers topics like why is usability testing necessary, its advantages and disadvantages, different participants or elements of usability testing, etc.

We have also covered how to analyze testing results, and different Usability testing tools in detail with examples.

Usability testing is necessary as it gives the chance for improvement before the launch of the product. This improvement can thereby boost the business positively. Hence usability testing is extremely important and should be performed to provide better deliverable products to the customers.

Also, read A complete guide to testing web applications.

Author: This article is written by STH team member Swati S.

I hope you find this article helpful in understanding usability testing. Please add your comments and queries in the comments below.