Usability Testing from Test Plan to Final Deliverable
BRIEF
A usability testing plan has been developed to evaluate the usability of the WeChat mobile app. WeChat is a versatile app developed by Tencent, offering messaging, social media, and mobile payment functionalities. It has been recognized as China's "App for Everything," WeChat includes a wide range of features and platforms within a single interface. While it has already made inroads into the North American market through partnerships such as WeChat Pay's collaboration with Citcon to enable businesses in Canada to accept WeChat Pay, it is essential to assess the app's usability specifically for Canadian users. Therefore, the purpose of this plan is to conduct usability testing on the WeChat app in Canada and gather feedback to improve its usability in this market.
To ensure that users could successfully complete the app’s primary task and to measure performance. During the testing, I observed and recorded how well the participants were able to accomplish twelve predefined tasks within the app. This allowed me to assess their levels of satisfaction, identify any sources of frustration, understand their design concerns, and determine the strengths of the app.
PROJECT INCLUDED
UX Research | Usability Testing Method
Goal #1
To assess the learnability of the app’s various core functionalities among first-time participants including the length of time and number of clicks it takes a participant to complete the tasks.
Goal #2
To identify participants’ frustration, challenges, problems, and concerns with the design of the interface and information architecture.
After conducting the usability test, the findings and insights gathered from the participants were analyzed. Based on the test results, the app would be improved to better align with the needs and expectations of users in Canada. This would ensure that the app provides a tailored and user-friendly experience to the Canadian user base.
The usability test involved various activities, such as task performance assessments, post-task questionnaires, and post-usability test interviews.
To assess the learning process of first time users with no experience using the app and measure performance success, the usability test would assess the learnability of the app’s various core functionalities.
By testing a common set of tasks and workflows across a range of participant types, specific performance-based patterns and trends could begin to emerge. These patterns could reveal how well participants performed a given task, the length of time it took to complete the task and whether or not a participant took a direct path to satisfy the function.
By collecting data on performance time and determining areas where a participant felt frustrated and challenged, I could identify which tasks have a steep learning curve and look for ways to reduce completion time, limit errors and improve performance.
Objectives:
• Identify length of time it took to complete each of the twelve tasks
• Calculate the average number of clicks/taps it took to finish a required task
• Locate indirect or unintended task completion sequences (e.g. extra or unneeded steps)
• Discover where participants got lost in the process of completing a task
Since participants directly interacted with the app’s interface, it was important to consider accessibility obstacles and their needs. Through the task, I could identify problem areas within the app’s user interface and information architecture and recorded the frustration and challenges participants had for the future improvements.
Objectives:
• Identify the frustration and challenge that participants had in the process of completing a task
• Record the failed tasks
• Record the problems participants met and questions they came up with
Research Questions
How easily do participants understand what the app is?
How easily do participants register for the app?
How easily do participants start to explore the core functionalities?
How well do participants understand the symbols and icons?
Is there an appropriate balance of ease of use and ease of learning?
I recruited FIVE participants who have not used or not quite familiar with WeChat.
Also, all participants should be intermediate and advanced mobile technology users.
Recruitment
In general, the recruiting criteria were based on the familiarity of the app, experiences of mobile technology and level of comfort. I approached all five participants. Participants were ideally required to have zero or less experience with WeChat app, and the were mostly from my classmates or coworkers.
Additionally, they must own a smart phone (iPhone or android) and have known how to download apps.
By recruiting people with a wide range of experience using mobile technology, I was able to more successfully understand the needs, expectation and motivation from them. Moreover, the results helped to determine if the design of has been enjoyable, meaningful and functional.
Procedure
I conducted the usability test at different locations based on the situation of participants. Participants may be comfortable with their own phones, so they were instructed to use their own smartphone to do the usability test. I sat next to the participant in order to provide instruction. I was also the note taker who observed the whole process and recorded behaviours. All participants were video recorded entirely when he/she were doing tasks. After I provided instructions, I asked the participant to sign a consent form. Once signed and all questions had been answered, the test could begin. I started by providing a brief overview of the app and explaining how the test would be measured, as well as what the participant should be doing during each task.
Once the instructions had been shared and the participant agreed to begin, I read off the first task. At the conclusion of each task, I asked the participant to complete a short questionnaire. Following the completion of all required tasks, the participants participated in a one-on-one interview with me.
Data to be collected
To collect data in this test, I, as a note-taker, would document participant’s responses, noting comments and actions. I would be familiar with WeChat App and he/she would be provided a sheet of tasks’ list. The percentage of tasks that users completed correctly would also be calculated.
Since the WeChat App is already built so I would provide a rubric for testing basic usability metrics. I would also be the moderator to read the instruction for participants and guided the test.
To test the accessibility and learnability of the app, as well as Identifying participants’ frustration, challenges, problems and concerns with the design of interface and information architecture, the five participants completed a series of task based workflows and scenarios.
Measuring Criteria
In order to monitor, collect and document the usability test results for each participant, I would use a pre-determined criteria set. The participant results from within each documented criteria addressed one or more of the original usability test goals.
Specifically, the users were asked to
Task #1 Download the app
(Successful Completion Criteria: Participant finds the app from within the app store and successfully downloads it on their device)
Task #2 Sign up
(Successful Completion Criteria: Participant opens the app, clicks on the sign up button and completes both steps required to create a new account)
Task #3 Get into the home page
(Successful Completion Criteria: Participant signs in the app and get into the home page)
Task #4 Build the profile
(Successful Completion Criteria: Participant finds “My Profile”, uploads an image and fills out the questions under profile)
Task #5 Add new contact (add my WeChat)
(Successful Completion Criteria: Participant finds the “Add Contacts” icon, and adds my WeChat number which will be given during the oral instruction. )
Task #6 Start to send various messages
(a recorded verbal msg, an image, and an emoji) (Successful Completion Criteria: Participant successfully added the contact, and sends the messages required above after finding the corresponding icons.)
Task #7 Post a moment (an image with a description)
(Successful Completion Criteria: Participant finds the “Moment” section, upload an image and types a short description and posts it.)
Task #8 Look my Moments/Album
(Successful Completion Criteria: Participant navigates to the Contacts, finds my WeChat that was just added, taps it and taps the “Album” and opens it. )
Task #9 Leave a comment on any one of my postings
(Successful Completion Criteria: Participant stays at my Album/Moments page, and goes through a few postings, and leaves a comment of any one of my postings.)
Task #10 Find the function “Message in a bottle”
(Successful Completion Criteria: Participant finds the function “Message in a bottle” and pick up one bottle, and throw it back or reply.)
Task #11 Set up the Privacy by turning on “Friend Confirmation”
(Successful Completion Criteria: Participant navigates to account settings, finds the Privacy, and turn on the Friend Confirmation)
Task #12 Find the Language setting
(Successful Completion Criteria: Participant navigates to General setting, and finds
Language setting. )
At the conclusion of the usability test, participants were provided with a multiple choices and open-ended questionnaire allowing them to share their overall experience with the app and rate the level of satisfaction. This questionnaire provided the participant with an opportunity to share their thoughts on whether the app is functional, meaningful and useful. Also, I conducted short interviews after participants completed the tasks, so participants were able to have direct conversation with me about their experiences of this test and how they felt about the WeChat app. I planned to measure how long it took for each participant to complete each of the twelve tasks. This data could be used to measure task completion speed and identified the errors, frustration and challenges.
Reporting Results
At the conclusion of the usability test, a report was generated.
The report/final poster was developed which includes all findings and results, an evaluation of usability goals and objectives and recommendations for future improvements.
Participants had 12 tasks, and they performed effectively in most of the cases. Some tasks caused troubles and confusion.
80% participants failed the task of sending a voice message because it requires
users to speak closer to the phone’s speaker so it would capture voice correctly.
Most of them didn’t realize that, and they got frustrated when failed sending voice
message.
Hidden features in WeChat gave participants a bit hard time to find them.
Some features are labelled similarly and it caused confusion, e.g. voice call, voice input and voice message.
Half participants landed on wrong pages many times before successfully completing tasks.
Participants felt WeChat is easy to navigate, and they were pretty satisfied with the interface. They gave positive feedback about some features that other apps don’t have.
