Grocery Shopping App
Usability Study
Overview
The Kroger Application redesign project undertaken at Indiana University was an integral part of our coursework, aimed at honing our research skills and deriving insights to inform redesign decisions. Through various research methods, we evaluated the current user experience, identifying areas for improvement and proposing design enhancements to ensure a seamless and user-friendly experience for app users.
Methodology
Interviews, Think Aloud, Affinity Mapping, Thematic Analysis, Codes, Performance Metrics, Error Rates, Task Success/Failure Criteria, System Usability Scale, Correlation, Usability Evaluation
Team
Khushi Shah , Rahi, Kashvi, Devanshu, Rutika, Devraj
My Role
UX Research, Literature Review, Methods Planning, Pain Point Discovery, Qualitative Analysis, Quantitative Analysis, Research Planning, Recruiting Participants
Duration
Fall 2023 (Aug2023- Oct 2023)
Tools
FigJam, Miro, Microsoft Excel, Power point, Word
Goal:
-
Assess onboarding process for ease and efficiency.
-
Evaluate awareness and usage of 'My List' feature for convenience and satisfaction.
To reach there
We did this,
Research Questions:
Our research was focused on finding answers of below questions:
RQ1: How easily and successfully do users start the adding items to 'My List' and then to Cart?
RQ2: Does the onboarding process deliver a remarkable experience for novice users?
RQ3: What is the level of user awareness regarding the 'My List' feature and its functionality within the application?
RQ4: Do users encounter obstacles while completing the process of buying items from Kroger's App?
To Find Answers
of these questions
We did this,
Interviews
Think Aloud
Usability Testing
12
Qualitative Analysis
Quantitative Analysis
Recommendations
for Improvement
7
Targeted Participants:
Our user base is diverse, comprising individuals aged 21 to 57 with varied cultural backgrounds, including American and Asian. These users are Kroger customers, representing a wide demographic range.
Introduction:
The Kroger grocery app has been downloaded over 10 million times and is a favorite among shoppers. With 2,719 supermarkets across the US, this app makes grocery shopping easy. It's user-friendly and lets you explore, plan, and buy from a wide range of products available at Kroger's stores.
Features:
Ability to browse and purchase groceries
Create and manage shopping lists.
Access digital coupons and deals
Schedule home
deliveries
Overall, to understand the extent to which the Kroger's app delivers a consistent experience for grocery shoppers, we conducted a battery of tests – driven by our research questions – such as interviews and Think-Aloud sessions. Through these tests, we aimed to explore the expectations of the sub-30-year-olds, discover the limitations and failures of the app, and identify obstacles the users face with the app. Specifically, we captured the subjects’ comments, concerns, frustrations, satisfaction, errors, and general feedback.
Methodology Rationale:
Interviews:
To understand user sentiments, experiences, and expectations regarding the Kroger Application.
Rationale: Interviews provided a rich source of qualitative data, allowing us to understand users' attitudes and gather feedback about their experiences. This method enabled us to uncover the emotional aspects of user interactions and uncover pain points that might not be apparent through quantitative data alone.
Think Aloud:
To observe user while performing tasks like adding items to "My list " feature, aiming to uncover pinpoints challenges and issues encountered during the interaction with Kroger App.
Rationale: Think Aloud sessions for the Kroger app were employed to observe and understand users' real-time interactions, capturing frustrations, errors, and feedback, thereby revealing cognitive processes and highlighting usability issues for improved app functionality and user experience and provide quantitative data.
System Usability Scale:
System Usability Scale (SUS) lies in its effectiveness as a validated and widely accepted questionnaire for evaluating the overall usability of systems. It consists of ten items, each rated on a five-point scale.
Rationale: Quick, reliable, and standardized assessment of user perceptions, providing quantitative data that helps in gauging user satisfaction, identifying usability issues, and comparing the app's performance against benchmarks or iterations over time.
Interviews
The interview protocol was designed with a semi-structured approach to investigate users' engagement with Kroger's online grocery shopping. For those who had previously purchased groceries digitally, the interview centered around their recent purchase timing, the specific platform they utilized for shopping, and a comprehensive inquiry into their overall shopping experience, aiming to capture their likes, dislikes, and encountered challenges.
Similarly, participants without a history of online grocery purchases were probed regarding their shopping preferences and reasons for abstaining from digital purchases. This allowed for an understanding of their preferred shopping methods and the factors influencing their decisions.
We recruited 12 participants who helped us address our research questions. They were a mix of Asians and Americans, aged 21 to 57 years old. Some of them were new to using the Kroger mobile application. We followed an interview protocol designed to answer our research questions effectively.
Recorded issues with the existing user experience:
The onboarding process was found to be lengthy and time-consuming.
New users find the interfaces confusing, with the application containing jargon.
Think Aloud
The Think Aloud method serves as the gold standard test for comprehending and observing user thoughts and actions as they navigate through tasks. Our objective was to uncover any obstacles or difficulties users faced while performing given tasks, their perceptions of the app, and any general complaints they might have had.
We asked user to perform 2 Task,
Onboarding
Adding items to "My list" and then add to cart
Task 1: Onboarding and subtasks
We assigned users the task of downloading the Kroger app if they hadn't already done so, followed by creating a new account and completing the onboarding process. This involved selecting their preferences, inputting their phone number, ALT ID, and entering their address.
The rationale behind selecting this particular task was aligned with our research objectives aimed at understanding the experiences of new or novice users. Our focus was to ascertain whether this process was seamless for them and to identify any potential frustrations or challenges encountered during the onboarding phase. This examination aimed to provide valuable insights into the user experience, enabling us to improve and optimize the app's usability for newcomers.
Table 1: Subtasks for task 1
Task 2: Add items to "My List" and manipulate it
The "My List" feature in the Kroger application allows users to create and manage personalized lists of grocery items they frequently purchase or plan to buy. This feature enables users to easily organize their shopping needs, making it convenient to locate and purchase items during their shopping trips.
The decision to focus on this task was driven by its ability to address key research questions. By analyzing user interactions and measuring errors, we can assess the usability of the feature and understand why users may not be utilizing it. Additionally, we aim to determine user awareness of the feature and its perceived advantages for those who do use it.
Table 2: Sub tasks for task 2
Each team member conducted two think-aloud sessions, during which observations were recorded. Before performing the task, success/failure criteria were defined, focusing on factors such as time spent on each task. Micro interactions were also observed, including subtasks, to gain a deeper understanding of user errors and interactions. Success and failure criteria were defined by the peroforming the task by expert.
Qualitative Analysis from Interview & Think Aloud
To synthesize data from interviews and think aloud session, we conducted an affinity mapping exercise to identify gaps in the Kroger mobile application where user expectations are not met. Thematic analysis and affinity diagramming facilitated the identification of 12 distinct categories encompassing user grievances with Kroger's mobile app. Subsequently, personas were crafted to establish a comprehensive understanding of our target users.
Codes Identified:
Uncommon fields, confusion, unnecessary data, feature accessibility, update list items, feedback mechanisms, user expectation misalignment with app functionality, the usefulness of the 'My List' feature, icons, navigation, text, and Kroger Plus functionality.
Affinity Map and User Personas
Problems Identified
01. The onboarding process was found to be lengthy and time-consuming.
02. Participants encountered confusing text fields during onboarding.
03. Participants were not familiar with some of the jargon and icons used in the app.
04. Participants had difficulty locating and understanding the "My List" feature.
05. Participants found it challenging to navigate and manipulate multiple lists, leading to confusion about where items were added.
06. Participants faced difficulties when attempting to remove items from their lists.
Observation/ Note Taking findings from Think Aloud
-
The user found the onboarding process lengthy and suggested improvements for the “Suggested Address” option, proposing it appear before users start writing their address. Additionally, users recommended linking Google accounts to streamline the process instead of manual entry.
-
Users initially confused the "Cart" and "My List" features, leading them to inadvertently add items directly to the cart.
-
Users attempted to locate the "My List" feature using the search bar, but found no results, causing confusion. Eventually, they discovered the feature within the Hamburg menu bar after multiple attempts.
-
Within the "My List" feature, Users encountered challenges in creating his own list due to its overwhelming interface. They mistakenly added items to the wrong list and faced difficulty in removing them until he found the red "Remove item from the list" button.
-
Users encountered visibility issues when attempting to add items from "My List" to the cart, preventing successful completion of the task. Most of the users expressed dissatisfaction with the usability of the feature and suggested the implementation of a tutorial or tour guide.
-
Users recommended keeping the "Add to cart" button visible when selecting items and suggested making list features more prominent with a highlighted/solid background.
Quantitative Analysis from Think Aloud
We were unable to conduct in-depth quantitative data analysis due to the small size leading to poor power in the data to discover any meaningful causational relationships. However, we did find some interesting trends.
Once the data was consolidated, we identified user issues and assessed their frequency using the probability of detection method.
Probability of Detection
#1
During testing, 9 users were uncertain about the list to which their items were being added when creating new lists.
#2
7 users were unaware of the "ALT ID" term during the onboarding process.
#3
58% of users encountered challenges in locating the "My List" feature within the application, while an equal percentage struggled to identify the icon for the feature. Additionally, the same proportion of users found difficulty in creating lists within the application.
Performance Metrics
We looked at how well users completed tasks and whether they met specific success or failure criteria, which we set beforehand and listed in Table 1 & Table 2. These criteria were crucial for us to measure performance. Additionally, we tracked the time users spent on each task to see if there was a connection between how long they took and whether they succeeded or failed. This data helped us understand user behaviors and answer questions about task success, patterns in task completion times, and overall performance.
System Usability Scale and Quantitative Data
After each think-aloud session, we asked participants to fill out the system usability scale questionnaire and gathered quantitative data for their user experience during the study.
Considering an SUS score greater than 80, a hypothesis test was conducted. Given that the calculated t-score (-2.85) falls below the critical value (-1.36), there is substantial evidence to reject the null hypothesis (H0) suggesting that the perceived usability of the interface is not high (SUS score <= 80).
Mean SUS Score: 64.17 (less than the industry standard which is 68)
T-score: -2.85
Through our method, we identified a significant problem with how users perceive the "My List" feature in the application. The SUS score of 64.17 indicates a dissatisfaction with the feature, suggesting the need for major improvements. To validate this finding, testing with more users could provide further insight.
With a 90% confidence interval, we estimate that the true population mean SUS score falls within the range of 56.6 to 71.74, meaning we are reasonably confident about this assessment.
Recommendations
Reduce user frustration by unnecessary information from the onboarding process:
Requirements of address input and Kroger’s pay setup should be added on to a later stage, i.e. asking for the address during checkout.
Make My List more evident to the user:
This could be added in the bottom navigation bar on the home screen or as a bubble button on the bottom right of the home screen.
The My List feature does not match with a common conceptual idea people have about favorites:
The feature can either be matched with the user’s perception or can be designed in a way to deploy the concept clearly to the user.
The concept of checklist and adding to cart should be distinct and clearly identifiable:
Give a toggle button on the home screen to change between in-store and online shopping. In-store would activate the ‘checklist’ feature while online shopping would activate the ‘add to cart’ feature.
The user should have control of where the product is being added:
When adding a product to my list the user should be given the option of all the existing lists to select where the product has to be added.
Navigation can be made easier:
A navigation system should be implemented where user can achieve their desired goals without having to go through unnecessary steps.
My Learnings
#1 Choosing the right method:
I've learned the crucial importance of flexibility in research plans and the necessity of selecting appropriate methods considering the time and recruitment of participants. However, the most significant takeaway is the imperative of having well-supported findings and justifications for any proposed changes or recommendations. Whether backed by qualitative or quantitative data, these findings are essential for driving meaningful improvements in user experience.
#2 Presenting an effective report:
This was perhaps the most important part of successful research - to present findings in an easily understandable way. Since not only UXRs but also PMs and engineers were reviewing my work, it was crucial that study impact and recommendations were highlighted in my slide deck. I used simple graphs, kept a clean layout, and kept my presentation short and succinct.