Pokemon Go

Overview

Our team remotely conducted usability testing on Niantic Labs’ game, Pokémon Go during the period from February 2022 to April 2022 (during this time, there were still procedures put in place to protect others from COVID-19). We launched 3 tests (which allowed 3 responses each due to financial constraints) on usertesting.com on April 19th, 2022, and received results back during the next 12 hours. Alongside the tests, our team conducted a Heuristic Evaluation, PURE research, and a competitive audit.

My Role: UX Researcher

Duration: 2 months

Deliverables:

• Usability Test Plan

• Presentation & Findings Report

Problem Statement

The Pokémon Go app lacks a cohesive onboarding experience and clear in-game instructions for new users. There is almost no assistance in learning about the multitude of features in Pokémon Go including but not limited to how to catch a pokemon and spin a pokestop. At the moment, there are too many features in the app that users do not know exist or where they are.

Methods

Heuristic Evaluation:

A heuristic evaluation will be conducted by our team in order to find the main pain points of the game. To do this, our team will utilize Nielsen’s ten usability heuristics as a guideline to determine how POGO is succeeding or failing in terms of usability. Since a heuristic evaluation allows us to obtain both qualitative and quantitative data, it is considered to be mixed-method.

Nielsen’s 10 Usability Heuristics:  

1. Visibility of System Status

2. Match Between System and the  Real World

3. User Control and Freedom

4. Consistency and Standards

5. Error Prevention

6. Recognition Rather Than Recall

7. Flexibility and Efficiency of Use

8. Aesthetic and Minimalist Design

9. Help Users Recognize, Diagnose, and Recover from Errors

10. Help and Documentation

PURE Research:

The PURE research method will also be conducted by our team to assess the difficulty of various tasks. As our team is comprised of experts in terms of usability as well as Pokémon players at various stages of familiarity with different titles we are uniquely equipped to test Pokémon Go based on varying criteria. This will allow our team to gain quantitative data which we will turn into an overall score to visually show how Pokémon Go stacks up in terms of usability.     

Card Sort:

We used OptimalSort.com to create our card sorting test. Users participated in a card sort, where they were tasked with organizing 20 features in the game into three columns which represented sections of the game’s menu. we obtained valuable qualitative data about the navigation hierarchy in the game.

Task-Based- New Users

We used UserTesting.com to recruit new users for a usability test and got 3 responses. The users were tasked with:

• Going through the onboarding process of the game

• Putting on a specific hat (Magikarp Hat) during the apparel customization part of the character creation.

• Catching a Pokémon on screen.

• Find the AR feature, toggle it on, and catch a Pokémon with it enabled. (Used usertesting.com - 3 responses) 

Task-Based- Experienced Users

We used UserTesting.com to recruit experienced users for a usability test and got 3 responses. The users were tasked with:

• Finding features of the game that our team found to be difficult via the PURE research method. These features were:

  • Find the postcard book,

  • Find the journal, 

  • Set up your Pokémon party for battles and raids,

  • Identify a PokéStop/gym on their map, and if accessible, spin it.

Through the data we gathered from our usability testing and the findings we had from our heuristic evaluation and our PURE research method, we found weaknesses surrounding the hierarchy of Pokémon Go’s menus and features. We found that a majority of our users organized features differently from the pre-existing structure of the app (optimalsort.com). We also found that users had a difficult time finding various features in the game. These features (ranging from accessories to sections of content) were predicted to be challenge points in our tests (PURE research method) and were shown to be difficult, and in some cases, unable to complete. (usertesting.com) 

Emergent Findings

Emergent findings are things that our team did not intentionally test for, but came up in our findings. While we were conducting our usability tests, we identified a number of emergent findings:

• Order the contents of various menu features in a way that makes sense to users

• Include more on-screen instructions regarding the basic functions of the game during the tutorial

• Include descriptions of basic accessories so that they can be observed in a way that reduces cognitive load

• Show users where and how to set up their party (this is a new feature that isn't obvious to legacy and veteran players who are used to set up the party during a raid)

Key Findings

Along with our emergent findings, we discovered a list of key insights that we want to emphasize:

Reccommendations

 Taking into consideration our team's research findings, we propose the following items as recommendations and action items that could be utilized in improving Pokémon Go:

1. Improve understanding of categories and hierarchy

   • Many participants were confused by the categories presented to them in-game. The category titles appear to be too broad and are too similar to each other which leads to users not knowing where to find certain in-game features. For example, users looked under “Profile” for features that were under “Battle”. We recommend changing the category names, creating more categories, or otherwise clarifying what can be found under what category possibly by using in-game tips.  

2. Improve onboarding experience

   • Participants noted how there was a lack of information in the onboarding experience. They mentioned how they did not know you had to throw the Pokéball at a certain time, and the rings and their colors were not explained. Many of the features of the game are also not shown to the users in the onboarding experience which led to many users not knowing of their existence and never using them. Hence, we recommend improving the onboarding experience and making it more in-depth in order to give users a solid base of knowledge of how to play Pokémon Go to the fullest by utilizing all of its features.    

3. Provide accessible “help” 

   • During testing, we observed that many of the users would get lost or confused while playing Pokémon Go, however, none of them were able to find any in-game “help” to remedy their problems. This led to them giving up on some testing tasks or skipping them altogether. While the team found an official help section in the game, it was difficult to locate and hard to use. Thus, we recommend adding a more in-depth onboarding experience or some type of easily accessible in-game “help” in order to teach the players all of the features of the game and help them find features when they get lost.    

4. Improve in-game instructions

   • Participants noted how there was a lack of instructions in the game. Participants expected to get help right away if they were lost, but they were not able to find any. Also, the experienced users specifically mentioned that they did not know some features existed such as the feature to create your battle party before starting a battle. We recommend adding some kind of tips into the game in order to draw users’ attention to features that may be overlooked or help them when they are lost.