Page History

Although small tests give you ample insight into how to improve design, such tests do not generate the sufficiently tight confidence intervals that traditional metrics require. Think-aloud protocols are the best way to understand users' thinking and thus how to design for them, but the extra time it takes for users to verbalize their thoughts contaminates task time measures. Plus, qualitative tests often involve small tweaks from one session to the next, and, because of that metrics, collected in such tests are rarely measuring the same thing.

Thus, the best usability methodology is the one least suited for generating detailed numbers.

Measuring Success

One of the more common metrics used in user experience is task success or completion. This is a very simple binary metric.  When we run a study with multiple users, we usually report the success (or task-completion) rate: the percentage of users who were able to complete a task in a study.  

Like most metrics, it is fairly coarse — it says nothing about why users fail or how well they perform the tasks they did complete.

Nonetheless, success rates are easy to collect and a very telling statistic. After all, if users can't accomplish their target task, all else is irrelevant. User success is the bottom line of usability.

Levels of Success

Success rates are easy to measure, with one major exception: How do we account for cases of partial success? If users can accomplish part of a task, but fail other parts, how should we score them?

Let's say, for example, that the users' task is to order twelve yellow roses to be delivered to their mothers on their birthday. True task success would mean just that: Mom receives a dozen roses on her birthday. If a test user leaves the site in a state where this event will occur, we can certainly score the task as a success. If the user fails to place any order, we can just as easily determine the task a failure.

But there are other possibilities as well. For example, a user might:

• order twelve yellow tulips, twenty-four yellow roses, or some other deviant bouquet
• fail to specify a shipping address, and thus have the flowers delivered to their own billing address
• specify the correct address, but the wrong date
• do everything perfectly except forget to specify a gift message to enclose with the shipment, so that mom gets the flowers but has no idea who they are from

Each of these cases constitutes some degree of failure.

If a user does not perform a task as specified, you could be strict and score it as a failure. It's certainly a simple model: Users either do everything correctly or they fail. No middle ground. Success is success, without qualification.

However, we sometimes grant partial credit for a partially successful task. It can seem unreasonable to give the same score (zero) to both users who did nothing and those who successfully completed much of the task. How to score partial success depends on the magnitude of user error.

In the flower example, we might define several levels of success:

• complete success: the user places the order with no error, exactly as specified
• success with one minor issue: the user places the order but omits the gift message or orders the wrong flowers
• success with a major issue: the user places the order but enters the wrong date or delivery address  
• failure: the user is not able to place the order

Of course, the precise levels of success would depend on the task and your and your users’ particular needs. (For example, if you did a survey and determined that most mothers would consider it a major offense to get tulips instead of roses, you may change the rating accordingly).

Reporting Levels of Success

To report levels of success, you simply report the percentage of users who were at a given level. So, for example, if out of 100 users, 35 completed the task with a minor issue, you would say that 35% of your users were able to complete the task with a minor issue.  Like for any metric, you would have to report the confidence interval for that number.

Image Removed

(*) In this table, the ranges represent 95% confidence intervals calculated using the Adjusted Wald method.

Image Removed

Note that this method simply amounts to using multiple metrics for success instead of just one — each level of success is a separate metric.

You can also use other metrics such as number of errors; for example, you could define different error types (e.g., wrong flowers, wrong shipping address) and track the number of people who made each of these errors. Doing so may actually give you a more nuanced picture than using levels of success because you might be able to say precisely which of the different errors is more common and, thus, focus on fixing that one.

Do Not Use Numbers for Success Levels

A common error that people make when working with success levels is to assign numbers to them; for example, they may say:

• complete success = 1
• success with one minor issue = 0.66
• success with a major issue = 0.33
• failure = 0

And then, instead of reporting success, they simply average these success levels for their participants. In our example, they might say that the success rate is:

(20*1+35*0.66+ 30*0.33+0*15)/100 = 0.53 = 53%

This approach is wrong! The numbers that we assigned to the different levels of success are simply labels and they form an ordinal scale, not an interval or ratio scale. That means that, even though there is an order established across these levels of success (e.g., failure is worse than success with major issue), there is no mathematical meaning to these numbers and we cannot average them because we cannot truly guarantee that these numbers are evenly spaced on a 0 to 1 scale (or whatever other scale we’re using between complete success and complete failure). In other words, we don’t know and have no reason to assume if the difference between complete success and success with minor issue is the same as the difference between failure and success with major issue.

Image Added

Nielsen Norman Group defines a design system as “a set of standards to manage design at scale by reducing redundancy while creating a shared language and visual consistency across different pages and channels.” 

Brad Frost defines a design system as “The official story of how an organization designs and builds digital interfaces.”

What a design system is not?

A design system is a collection of multiple elements such as pattern libraries, style guides and UI toolkits and more. Only having one singular element does not mean you have a design system. If you have a Sketch or Figma file but don’t have any code examples or patterns you don’t have a design system. If you have code examples but no design artifacts you don’t have a design system. 

A design system requires design, code and guidelines and not having each part of it makes a design system incomplete.

So now that we have an understanding of what a design system is and is not, let’s get into the state of the CMS Design System, how we got to where we are today, as well as what’s planned for the future.

Creation of the CMS Design System

The CMS Design System was created in March of 2017 and adopted in several Healthcare.gov products. Upon adopting the CMS Design System, these applications needed more specific Healthcare.gov style and component customizations, which were originally supported through site packages. 

Site packages contained styles and components unique to a particular CMS site, and were built on top of the CMS Design System. This was the first introduction of a sub-brand or theme of the design system. After creating and using site packages for over a year, we discovered that there were many challenges with how the site packages were constructed.

Site packages had a large technical overhead and were difficult to use and maintain. They had complex dependencies that required four separate dependencies. This required product teams to use and track four different code libraries. They were also difficult to build, publish and test, due to no standard practices and each site package being a little bit different. The documentation sites for these site packages were also difficult to update due to a high technical proficiency required.

We made some improvements by consolidating dependencies from four down to one. We also created shared scripts and more standard practices to be used by both the CMS Design System and site packages. Documentation site source was organized code making it easier to find and modify the Design System and increase consistency between the CMS Design System and site packages for Healthcare and Medicare.

Because the “site package” terminology was unique to the CMS Design System and was confusing for users, we renamed site packages to Child Design Systems. This created a stronger mental model of the relationship between the CMS Design System and a Child Design System like Healthcare or Medicare. This also tied into larger goals at the time of increasing design system ownership and autonomy within Healthcare and Medicare.

When version 2 of the CMS Design system was released, Medicare, Healthcare, and the CMS Design System each had different components, styles, and guidance. The concept of child design systems supported this need to have differing components, guidance and styles as well as more autonomy at the Medicare and Healthcare level.

Current state of the CMS Design System 

If we look at how our design system family is set up we can see how CMS is currently working and thinking about design systems. 

CMS is designing and building products in 3 and soon to be 4 separate design systems and there is a lot of similarity between these systems that could be consolidated. This siloed approach is not the ideal state, and we’ve experienced many pain points because of it.

The current siloed setup results in duplication of components, guidance, and styles. This leads to updating the same UI component 3 times over for each design system as well as verifying guidance across many different sites both in GitHub and the InVision Design System Manager. 

Having documentation and guidance in two separate places also splits our users into 2 separate camps: design and development. This is not a good approach to facilitate a consistent and unified approach of how to use the design system. 

For example, I should be able to find documentation, code examples and design artifacts for an alert component in one place. To fully understand the alert component a user would need to go to 3 separate places (GitHub, InVision DSM and Sketch Cloud) to get all the information about the alert component. It shouldn’t be this hard to do the right thing. 

Because it requires visiting multiple sites to get the full understanding of a design system component, we're seeing inconsistent digital experiences across CMS, in addition to solving the same problems across the organization. We should be solving problems once at a design system level, and then iterating and expanding on those solutions to fit more use cases across the CMS product landscape. 

Our team is also experiencing extremely large overhead in maintaining multiple design systems. Currently we are supporting 3 separate codebases, 3 separate documentation sites for code, 2 documentation sites for design and 3 separate UI Toolkits. That's eleven different properties all providing similar, overlapping or specific content. This makes it difficult to manage as well as difficult for users to know where to go to get the information they need.  

Image Added

With this siloed approach we see the digital sprawl increasing exponentially, as well as growing apart. This leads to technical debt and an increase of duplicative work and maintenance. 

Our current contribution model is complicated and confusing because we’ve got multiple processes and places for contribution. This leads to a lack of contribution from teams. For example product teams might ask “Where do I go to propose something new?”, ”Where do I ask questions?”, “Where do I go to find documentation?”. Unfortunately for each of these questions, the answer currently isn’t straightforward because we’ve got 3 design systems which all have a slightly different process.

We’ve used these pain points to help define what needs to change to help product teams use and interact with the CMS Design System effectively, and we’ve got a future vision that we are working towards throughout 2022 and beyond. 

Our future vision, a unified CMS Design System

Our future vision is to unify the entire design system by doing away with the concept of child design systems altogether and creating one themable design system that can support branding, theming and custom guidance. 

Who knows? we could even give this new unified system a new name like Lyndon Design System (a nod to President Lyndon B. Johnson who signed into law legislation that established the Medicare and Medicaid programs).

This vision comes from our hypothesis that:
Healthcare, Medicare, and CMS products are more similar than they are different.

A unified design system would offer documentation on guidance, accessibility, and usability in one place both from the highest CMS level perspective while also providing space for brand specific guidance for Medicare, Healthcare or CMS in the same location. 

Another part of our future vision is to create the concept of themes that can be applied to components and patterns to support Healthcare and Medicare branding along with any future themes for something like CMS.gov. We would also be streamlining and modifying our contribution model to have one set of processes for the Design System no matter which product team you are supporting.

A theme for the CMS Design System would be made up of the following parts:

• Color
• Type
• Spacing
• Icons/Illustrations
• Guidance

Color would contain specific brand colors, as well as any supporting color palette to accompany the theme. These colors would come from a system-level color palette which would be defined at the top design system level as design tokens. 

Type sizing and fonts would come from system-level font settings that are defined at the theme level. Spacing would also be defined in the theme and would contain specific spacing values that map back to design tokens. 

In addition, a theme would contain specific logos, illustrations and imagery that apply to that particular theme. For example, the Healthcare.gov logo would be part of a theme. 

Guidance is the final piece that we envision as being part of a theme. This theme-level guidance would be specific guidance as it relates to the Medicare or Healthcare brand. 

For example, with a button, the CMS Design System would define high level usage and guidance around best practices and accessibility and then in addition there would be specific guidance for how buttons are used within Healthcare products.

We see many benefits of a united CMS Design System and those are:

• Less overhead to maintain a single design system
• A systematized approach to color, type, spacing and other foundational elements
• Energy focused towards a single design system that everyone benefits from
• Over time the design system will strengthen with focus on a unified system
• Contribution model is straightforward and the same for everyone

So now that we have a vision for the unified design system what steps are we taking to get there?

First, we’d like to pull components, styles, guidance and icons from both Healthcare and Medicare child design systems up into the CMS Design System to have one place to reference the CMS Design System and that would be design.cms.gov. This single site would incorporate a theme switcher to allow for switching between themes to see, for example, how an alert would be styled on Medicare vs. Healthcare. 

We’d develop a set of processes that work for anyone wanting to interact with the CMS Design System no matter what product they are supporting including how to contribute, how to get help, where to give feedback, where to get information needed to build a product using the design   system, and other ways to get information or interact with the design system.

We’re currently in the process of upgrading the design system to utilize design tokens to give us a more systematic approach to things like color, spacing, and typography, which will fit nicely into a more unified and themable design system. In addition, we’re also exploring options to improve the documentation experience. Our team is excited to continue maturing the CMS Design System over the rest of 2022 and beyond as we begin to really see the benefits it can bring!

We’d love to hear feedback if you are using the design system or plan to in the future and you can reach out to us in the CMS Slack channel #cms-design-system, view our Jira backlog, or drop us a line through our contact form.

You can find the CMS Design System documentation site at design.cms.gov

Read more about design systems…

• Creating Themeable Design Systems - Brad Frost
• Why you don't need a separate design system for internal services - Amy Hupe   
• How I made a themeable Design System - Oscar Gonzalez
• Design system theming: How and why it works- Adam Sadowski
• An approach to theming Design Systems - Daniel Del Core
• Structuring documentation in multi-brand design systems- Amy Hupe

Design Systems are for user interfaces - Brad Frost Since the temptation of averaging numbers is so big in real life, we strongly recommend that you assign word labels to levels of success instead of numbering them.