CNBC Alexa Skill
Testing voice user interfaces for needs assessment and usability
CLIENT: CNBC
TYPE: Team project
TEAM: Paryn Assavanop (MBA), Maria Natalia Costa (MBA), Megan St. Andrew (Information Science), Shannon Weber (Health Informatics)
ROLE: UX Researcher
COURSE: Needs Assessment & Usability Evaluation | UMSI
TIMESPAN: Winter 2019
CHALLENGE: Use various research methods to better understand the needs and current perceptions of CNBC users in regards to the Alexa skill, ultimately providing suggestions for improvement and new features.
METHODS:
User testing
Usability testing
Survey
Heuristic evaluation
Comparative analysis
Card sorting
What was CNBC looking to understand?
The client originally approached us with the following:
“…We would like your team to conduct usability tests on the CNBC Alexa Skill , particularly the voice interaction (VUI), the flow through the experience, discoverability within the skill, and the personality of the skill.”
After an initial interview the client to discuss and better manage scope, we established the following target user and usability focus:
Target user
Financially active and finance-savvy professionals
Scope
Understand the unmet needs of users when using CNBC Alexa skill
Understand how users currently perceive and discover features within the skill
Explore opportunities to improve and make the skills more relevant to target users
Related resources
Because most modern methodologies regarding needs assessment and usability testing are structured around web and mobile devices, our team conducted further research to better plan and adapt our methods for VUI-specific testing. These resources can be found below:
Designing a VUI — Voice User Interface by Frederik Goossens
Mental Models for Intelligent Assistants by Raluca Budiu
Heuristic Evaluation — Voice Design by Arun George
Do you understand me? by Sara Marler
10 Usability Heuristics for Voice User Interface Design by Deepika Mittal
Methodology
For the purpose of this project, our team used a variety of methodologies to explore the problem and solution space.
Due to issues of privacy, I am unable to publicly reveal materials and reports generated as a result of this research but am happy to discuss further upon request.
Recommendations
After many rounds of hypothesis testing and analysis, the team offered a final report with recommendations for improvement and new features. These improvements and new features were separated into three overarching categories, with supporting evidence represented by methodology iconography:
Each category has more specific recommendations. Due to privacy concerns, I am unable to explicitly share these recommendations but am happy to discuss further upon request.
Final thoughts
Usability in terms of voice user interfaces is relatively new and unexplored. Our team adapted most usability tests to better serve a VUI, leading to further insights not just for CNBC’s Alexa Skill, but generally for all voice interfaces. Nielsen’s 10 Usability Heuristics for User Interface Design especially highlighted base usability issues that have not yet been addressed in all voice systems. Some heuristics may be impossible to achieve with voice alone, but perhaps adding secondary responses (such as visual or tactile feedback) or improving other heuristics would be a way to improve the usability and integration of VUI into the greater technological landscape.