Page History

Although small tests give you ample insight into how to improve design, such tests do not generate the sufficiently tight confidence intervals that traditional metrics require. Think-aloud protocols are the best way to understand users' thinking and thus how to design for them, but the extra time it takes for users to verbalize their thoughts contaminates task time measures. Plus, qualitative tests often involve small tweaks from one session to the next, and, because of that metrics, collected in such tests are rarely measuring the same thing.

Thus, the best usability methodology is the one least suited for generating detailed numbers.

Measuring Success

One of the more common metrics used in user experience is task success or completion. This is a very simple binary metric.  When we run a study with multiple users, we usually report the success (or task-completion) rate: the percentage of users who were able to complete a task in a study.  

Like most metrics, it is fairly coarse — it says nothing about why users fail or how well they perform the tasks they did complete.

Nonetheless, success rates are easy to collect and a very telling statistic. After all, if users can't accomplish their target task, all else is irrelevant. User success is the bottom line of usability.

Levels of Success

Success rates are easy to measure, with one major exception: How do we account for cases of partial success? If users can accomplish part of a task, but fail other parts, how should we score them?

Let's say, for example, that the users' task is to order twelve yellow roses to be delivered to their mothers on their birthday. True task success would mean just that: Mom receives a dozen roses on her birthday. If a test user leaves the site in a state where this event will occur, we can certainly score the task as a success. If the user fails to place any order, we can just as easily determine the task a failure.

But there are other possibilities as well. For example, a user might:

• order twelve yellow tulips, twenty-four yellow roses, or some other deviant bouquet
• fail to specify a shipping address, and thus have the flowers delivered to their own billing address
• specify the correct address, but the wrong date
• do everything perfectly except forget to specify a gift message to enclose with the shipment, so that mom gets the flowers but has no idea who they are from

Each of these cases constitutes some degree of failure.

If a user does not perform a task as specified, you could be strict and score it as a failure. It's certainly a simple model: Users either do everything correctly or they fail. No middle ground. Success is success, without qualification.

However, we sometimes grant partial credit for a partially successful task. It can seem unreasonable to give the same score (zero) to both users who did nothing and those who successfully completed much of the task. How to score partial success depends on the magnitude of user error.

In the flower example, we might define several levels of success:

• complete success: the user places the order with no error, exactly as specified
• success with one minor issue: the user places the order but omits the gift message or orders the wrong flowers
• success with a major issue: the user places the order but enters the wrong date or delivery address  
• failure: the user is not able to place the order

Of course, the precise levels of success would depend on the task and your and your users’ particular needs. (For example, if you did a survey and determined that most mothers would consider it a major offense to get tulips instead of roses, you may change the rating accordingly).

Reporting Levels of Success

To report levels of success, you simply report the percentage of users who were at a given level. So, for example, if out of 100 users, 35 completed the task with a minor issue, you would say that 35% of your users were able to complete the task with a minor issue.  Like for any metric, you would have to report the confidence interval for that number.

Image Removed

(*) In this table, the ranges represent 95% confidence intervals calculated using the Adjusted Wald method.

Image Removed

Note that this method simply amounts to using multiple metrics for success instead of just one — each level of success is a separate metric.

You can also use other metrics such as number of errors; for example, you could define different error types (e.g., wrong flowers, wrong shipping address) and track the number of people who made each of these errors. Doing so may actually give you a more nuanced picture than using levels of success because you might be able to say precisely which of the different errors is more common and, thus, focus on fixing that one.

Do Not Use Numbers for Success Levels

A common error that people make when working with success levels is to assign numbers to them; for example, they may say:

• complete success = 1
• success with one minor issue = 0.66
• success with a major issue = 0.33
• failure = 0

And then, instead of reporting success, they simply average these success levels for their participants. In our example, they might say that the success rate is:

(20*1+35*0.66+ 30*0.33+0*15)/100 = 0.53 = 53%

This approach is wrong! The numbers that we assigned to the different levels of success are simply labels and they form an ordinal scale, not an interval or ratio scale. That means that, even though there is an order established across these levels of success (e.g., failure is worse than success with major issue), there is no mathematical meaning to these numbers and we cannot average them because we cannot truly guarantee that these numbers are evenly spaced on a 0 to 1 scale (or whatever other scale we’re using between complete success and complete failure). In other words, we don’t know and have no reason to assume if the difference between complete success and success with minor issue is the same as the difference between failure and success with major issue.

Since the temptation of averaging numbers is so big in real life, we strongly recommend that you assign word labels to levels of success instead of numbering them.

Why these stages matter

HCD practitioners can better understand their own learning process and set appropriate expectations if they know the typical stages of the learning journey. 

NN/g concluded that:

• Most learning journeys feel frustrating at some point. Having insight into the HCD learning journey and the end goal can provide encouragement during “what’s the point” moments. If you’re learning HCD independently, awareness of the journey can help you feel less alone.
• Identifying one’s current phase can help predict future progress, staying focused on the goals of the current learning phase rather than jumping ahead. Jumping ahead runs the risk of creating experiences that may be confusing, and thus less motivated to continue to learn. 

Course facilitators (or managers or mentors), must empathize, create an effective HCD learning experience for others, and enable sustainable, long-term success. 

• Understanding that different people will be at different stages in the learning process is a key part in being an effective educator. It allows for the delivery of effective learning experiences without overwhelming the audience with too much complexity and also preemptively mitigate learners’ pain points at each phase. 

Educating and activating a group of people takes a lot of resources (time, money, and effort). The education process should be intentionally designed, in order to maximize resources and return on investment. Mapping participants’ phases and their progression through the learning journey allows an educator to benchmark progress, and indirectly, success of the training. 

Image Added

As practitioners progress through the stages, their mastery increases in a nonlinear fashion — experiencing fluctuations due to various factors, especially, lack of self-confidence. Note that mastery is a combination of competence and confidence; both are required to effectively use HCD. If a practitioner is good but still feels insecure, they won’t deviate from the strict steps of HCD and won’t teach others.  The goal is to create alignment between competence and confidence in order to master HCD.

The Structure

Rather than thinking of each phase as a discrete checklist, NN/g created a three-component framework for characterizing each learning stage: criteria, primary activities, and educator goals.

1. Criteria are observable qualities that can help the learner (or an external observer, such as a trainer) identify the learner’s current stage along the learning journey. It includes awareness of one’s own competence, level of exposure, and confidence.
2. Primary activities are the learner’s actions and use of HCD methodologies.
3. Educator goals and obstacles summarize learner’s pain points at any given stage and corresponding educator goals that can help learners overcome them. 

Phase 1: Newcomer 

This phase is the first in the learning journey. For the majority of HCD practitioners, this exposure occurs via a university or institution, a place of work, or an online resource. The amount of time spent at this first stage depends on how motivated the learner is. Interestingly, many practitioners immediately perceive HCD as useless and never leave this stage.

Criteria

Individuals in this stage have been introduced to HCD, but have limited experience with it. Practitioners in this phase fall into 2 buckets:

1. Individuals who are committed to learn HCD
2. Individuals who are not interested to learn more about HCD. They’ve been exposed to it, but that is where their learning journey halts. Newcomers may remain in this stage indefinitely until they encounter a deeper exposure to HCD that broadens their perspective, experience, or acceptance. 

Newcomers’ knowledge is minimal; they have a surface-level understanding of HCD, often rooted in the definition they received during their first exposure. They may be able to provide a definition, but are not familiar with the details of a framework or its value. They have topical associations with HCD — sticky-notes, phase models, and whiteboards. Newcomers are unaware of their HCD incompetence. In other words, they don’t yet know what they don’t know.

Primary Activities

Newcomers’ primary goal is to understand the basics — what HCD is and why it’s useful. Often, this stage’s activities are self-initiated: browsing articles, reading books, or signing up for a course. In other cases, HCD is learned by necessity or requirement at work or school through onboarding programs, collaborative workshops, required courses, or mandated trainings. Most newcomers have not yet actively practiced HCD activities. If they have practiced HCD at all, their participation is limited and surface-level. 

Educator’s Goals and Obstacles

The goal of an HCD educator at this stage is to communicate the purpose and potential value of HCD and to motivate the newcomer to pursue learning, then make time for it and move on to the next phases. Common obstacles are overcoming learners’ negative sentiments: annoyance with “yet another thing to learn,” unsuccessful previous attempts or experiences, capped bandwidth, and realization of their own limits (in this case, how little they know about HCD). 

Phase 2: Adopter

Individuals in phase 2 have adopted HCD and begun to practice it. They may have had some ups and downs in their limited experience with HCD. It is common for adopters to flip-flop between overconfidence and self-doubt and to feel both confused and successful at the same time. Successful adopters have discovered how relevant HCD is to their work or life and, thus, make a commitment to continue learning.

Criteria

Increased (passive or active) exposure and familiarity with HCD has made adopters aware of their knowledge limits. They understand the potential of HCD, but still have a lot to learn. Adopters have invested time, effort, and energy into HCD and have started applying it to their work with mixed success. The commitment to HCD at this phase can be self-initiated or dictated by an authority with an invested interest (leadership, organization, or supervisor). 

Primary Activities

Adopters practice HCD in a linear way — by the book. They rely heavily on checklists — for example an HCD Phases Model, as well as for the activities associated with each of its steps. Many adopters lean towards a prescribed, branded version of HCD, often provided by their institution, company, or a reputable external organization.  

It is common for HCD practitioners to encounter failure in this phase — especially if they are learning predominately on their own — because of their incomplete understanding of the HCD framework. Common types of failure include jumping to conclusions, using the wrong activity at the wrong time, and lack of buy-in or support from others. These failures push some practitioners to abandon HCD altogether. However, most learning in this phase occurs through failure. Practitioners who embrace failure tend to develop a better understanding of HCD in future phases.

Some adopters might learn primarily by actively participating in HCD training, coaching and activities together with experienced HCD practitioners. While this group of adopters may not have the opportunity to experience a sense of failure because of the support received from their peers, it is still likely they will question the value and legitimacy of HCD from time-to-time.   

Educator’s Goals and Obstacles

Adopters are on the bike, but still need training wheels and coaching. Adopters’ confidence drops as they realize the indirect ways in which HCD can be applied and how much they have to learn — for example, how straightforward they originally thought the process was versus how abstract (and potentially overwhelming) it really is. The goal of the educator at this stage is to help learners through hands-on practice and assistance until they can comfortably use HCD on their own. Many individuals can become confused in this phase when they cannot make a direct connection to the relevance of HCD. Thus, it is imperative that adopters find relevance and application to their everyday work.

Phase 3: Leader 

This is the proficiency stage of HCD. Leaders can articulate HCD succinctly to others, steadily growing their confidence, with varied experiences and continued exposure. Leaders take an active, independent role in their learning journey and begin to think adaptively about HCD; starting to explore new applications and may even have established a reputation as a subject matter expert. 

Criteria

Leaders practice HCD with general ease, confidence, and independence. Leaders become more and more aware of their new knowledge and comfort as they mature through this phase. They often teach others earlier in the learning journey. They are able to consistently and somewhat adaptively perform HCD activities without thinking too much about them. Leaders don’t try to apply HCD by the book, rather use it as needed depending on the goals. 

Primary Activities

HCD practitioners in this phase lead HCD activities with others or perform HCD without coaching, but still apply preparation and focus. While they were previously participants, they now facilitate, initiate, and even advocate for collaborative HCD activities. Activities that leaders are involved in gradually increase in complexity, ranging from involving users in learning opportunities to fostering stakeholders into the process and prototyping. 

Educator’s Goals and Obstacles

The goal of the educator in this phase is to continue to instill confidence and help sustain learner commitment. The goal should be to empower leaders to transition into the role of HCD educators and facilitators. As much as they’ve progressed, they still may not be aware of weaknesses or potential improvements (even though they often recognize a mistake after they made it). Promoting reflection in this phase is the key to helping leaders continue to grow (and progress to grandmasters). They must take an active role in adapting the HCD practice to fit their contextual goals and needs, be sustainable over time, and maximize potential benefits.  

Phase 4: Grandmaster

If you are familiar with the game of chess, the title of grandmaster is only given to the most exceptional players—those who have spent countless hours perfecting their skills and techniques.

HCD practitioners at this stage have not only become teachers of HCD, but create new ways of applying it, thinking about it, and adding to it. The practice of HCD is so embodied in their behavior that they seldom have to think about applying it. Grandmasters view HCD as a flexible, dynamic toolkit. They’ve long departed from the concept of a prescribed process and rather view it as scaffolding to solve both organizational (often internal) and end-user (external, product-related) problems. However, this phase doesn’t come without downsides. Grandmasters are more likely to doubt HCD than leaders, often when early-stage learners fall victim to mismanaged HCD marketing and thus misinterpret, misapply, or undercut the practice as a whole. 

Criteria

Grandmasters’ defining characteristic is the ability to critically reflect on their HCD practice. This enables them to judge what is useful and potentially depart from the traditional ways and activities of HCD. Grandmasters also know how to help others to this same stage of enlightenment. Grandmasters are aware of their competence. They have an in-depth, intuitive understanding and can blend HCD skills together to meet specific needs. 

Primary Activities

Traditional HCD activities are still used by grandmasters, but are altered, adapted, and applied in complex ways, depending on the goal, audience, and potential obstacles. Grandmasters don’t stick to prescribed or branded versions of HCD and more often pull tools and/or activities from other realms, like service design and business strategy. While basic HCD activities are still carried out in this phase, they differ from those in earlier stages because they are inputs or alignment strategies for more complex, involved activities (compared to previous stages where the basic activities are the end goal). 

Educator’s Goals and Obstacle

Grandmasters have very likely surpassed their original teachers. Their goal becomes not just activating and educating individual learners, but rather organizations as a whole. As practitioners themselves, grandmasters face an increasing likelihood of (re)questioning the value of HCD. Increased knowledge and mastery are a blessing and curse; this pessimistic view is often rooted in the realization of what HCD can and cannot solve (contrary to earlier naive ideas that HCD can be a cure-all). However, even grandmasters can get better, by self-reflection, by learning from their peers, and also by learning from their juniors: one of the skills of supreme mastery is the ability to discern which of the hundreds of ideas generated by eager newcomers is actually a stroke of genius.

Other Considerations 

• Trained designers experience the HCD learning journey too, just differently. Many designers view HCD as simply a way to articulate and communicate a creative approach to problem solving. Thus, many designers are likely to feel as if they bypassed this learning journey altogether because this is how they instinctively think. However, designers experience this learning journey too, albeit much earlier in their education or career, and likely not packaged or branded as HCD. This does not mean that all grandmasters are designers, but rather that many successful designers likely are. 
• Individual practitioners can be at multiple levels at the same time. Some skills may fall into one phase, while others into a lower different phase. For example, a practitioner’s mindset and reflection may fall into Grandmaster, while their hands-on experience and exposure to activities into Leader. The goal is to identify this imbalance and invest in experiences that bolster weaker dimensions of our practice. 

The HCD learning journey is a high-level, distilled representation of the most common learning phases observed in the NN/g research. Learning and teaching HCD is messy; not all experiences will fit squarely into this model. 

Regardless, it is imperative to frame and articulate learning HCD as an experiential journey. Doing so can help all practitioners become more effective learners and educators. Learners can gain insight and awareness into the greater journey and goals, while educators can thoughtfully and successfully execute the HCD learning experience they aim to create.  

Image Added

This article is based on a research study conducted by the Nielsen Norman Group. Originally published January 2021.