Community

Researcher Don't Let Friends Use Unreliable Task Ratings

Usability studies often include tasks for participants to complete, which are then ranked for perceived difficulty. However, these rankings are often subjective and non-standardized across studies conducted within the same team.

To learn how we might improve task ratings' reliability, we will welcome a special guest speaker Ann M. Aly, a social scientist and UX researcher. Ann will share a 5-level rubric her team created to assess task difficulty for task factors contributing to (non)completion. Interrater reliability testing used Cohen's Kappa (which measures the agreement between raters) to ensure rigor and consistency in rankings.

The presentation will include best practices and practical tips, like:

• How to make task assessments less subjective
• How to keep ratings rigorous, eligible for quantitative assessments, and increase research maturity among team members
• The benefits of a standardized rubric across studies and the importance of developing a shared language into a cross-functional team
• Using detailed task ratings to justify which domains or features require the most resources before release, like a mini heuristic evaluation
• A how-to guide for conducting a quantitative assessment and sample rubric

Ann is a social scientist and mixed methods UX researcher at Agile 6 Applications. She is currently teamed with Fearless Solutions to support Human-Centered Design at CMS' Office of Information Technology. Ann firmly believes in keeping the social in social science by centering the perspectives of those whose research her work serves and by making her insights accessible to non-specialists. Before joining Agile 6, Ann conducted research and consulted on higher education, bilingual communities, skill acquisition, and culturally relevant healthcare applications. Ann enjoys caffeinated beverages, remote beaches, and entry-level woodworking when not lost in a spreadsheet or transcript.

Register today to join the discussion and to collaborate with a community of professionals. Attend via Zoom.

CCSQ strives to design, build, and continuously improve human-centered and innovative IT solutions to advance quality initiatives. This work is managed in the center's Information Systems Group (ISG). In keeping with this mission, CCSQ will celebrate its 2nd Annual World Usability Day on November 12, 2020.

This year's theme is Human-Centered Artificial Intelligence (AI). AI is not a futuristic concept. AI systems surround us today, impacting parts of our lives that we may not even realize. During World Usability Day, we will explore the unique design challenges of AI.  

The CCSQ event will feature an open-house format on Zoom so you can join the sessions that align with your interests and schedule. Check out the speakers and schedule today. 

August 14, 2020

Designing World Usability Day

February 2020 Meeting Overview and Video

Using Personas for Effective Decision-Making: A Panel Discussion