Here to introduce our distinguished speakers for our next session, unlocking cross team collaboration with quantitative metrics.
First, I'd like to introduce Chantal Ramirez, a UX researcher with Ventura with a decade of experience in human-centered research.
Holding a PhD in experimental psychology from the University of Texas, Chantal boasts expertise in both quantitative and qualitative research methodologies.
She has previously consulted for industry giants like Amazon and Dell and in her current position Chantal is enhancing quantitative research within the enterprise systems and services group to ensure continual improvement through empathetic research and data driven insights.
Next we have Megan Nichols, a seasoned UX research lead with Ventura.
She started her career in graphic design and discovered her passion for research after observing a usability test.
Megan has worked on a variety of platforms from desktop to mobile web and apps even contributing to an AI and natural language processing powered mobile application for a healthcare organization.
Making connections and bringing people together to create trusting relationships is a core part of why she loves user research.
Megan and Sean Tal's combined wealth of knowledge and experience promises to shed light on the vital topic of unlocking cross team collaboration with quantitative metrics.
Please join us in welcoming Megan and Chantal.
Hello.
Thank you. That was a wonderful introduction.
Okay.
I know it's, I feel like you set the bar very, very high property. We aspire to be as amazing of presenters as you are.
Okay.
I don't know if you're on here, of presenters as you are. I don't know if you're on here, but at least it's recorded so you can refer figure how awesome you are.
Yeah.
So yeah, we're gonna get started talking about numbers now. Thank you so much, Chelsea, for that introduction.
Megan and I are super excited to share this topic with you all. We've made some recent efforts to focus on leveraging quantitative metrics to foster greater opportunities for collaboration within our own program.
So we're excited to share that information with you as well. Alright, so. To set the stage kind of going off of what Dr.
T said we wanna get to know our audience a little bit better. Oh, thank you. That might be cool.
I'm telling you not sharing your screen by the way. Okay.
Okay, well, there's nothing new that you hadn't seen already. Let me see. Share screen.
There we go. Thank you, Megan. Yes. So we wanna start off with a poll just to kind of get to know our audience.
We want to know if your team currently measures user experience. Now there are a lot of different ways to measure user experience.
Now there are a lot of different ways to measure user experience. Now there are a lot of different ways to measure user experience.
We've listed a few here. There are a lot of different ways to measure user experience. We've listed a few here.
Feel free to throw some other ones in the chat if we didn't capture them here. But yes, so we have interviews, focus groups, open-ended survey questions, quantitative, close ended survey questions. We have click stream data. Maybe you use, 2 or more of these that we listed.
Maybe so it looks like. We've got about 40% of answers in already.
Get some more.
Oh, looks like we have a really nice distribution so far.
So we're halfway there. Let's get some more people in.
Anyone else? Anyone else?
And again, feel free to add to the chat. If we didn't capture a good one here.
Yeah, that'd be great.
Almost maybe if we get to like 65%. Almost there. We like to close it around like 70%.
So. If you don't mind answering, we would appreciate it.
This is not a test.
Okay.
Do we have anything in the chat? I'm not sure how to.
No, I'm checking it and I don't see anything right now. We're good.
Okay. We're still waiting down from the excitement of our keynote.
Okay.
Alright, we're at 65% now. I think we can stop there. Keep sending in your answers.
On chat if you'd like to, and maybe we can share out these results.
So everybody can see. Is that an option?
Let's see.
Sorry, getting him shared. Yeah.
Oh, already being shared. Okay. So looks like The majority use open-ended survey questions. In regards to you know an individual one followed by interviews and then quantitative survey questions.
It looks like a lot of folks also use 2 or more of these that were listed above. So we're excited to hear that there's a good diversity.
It sounds like some, folks have experience using quantitative measures already as well as a lot of qualitative in here.
So hopefully, this will bring some, this presentation will bring some new questions and ideas and we can kind of share our experience.
With each other. Awesome. Okay, so close this out.
Alright, so before we dive into the actual talk, I just wanna. Clarify how we're defining quantitative metrics.
So there's some debate around when a metric is considered quantitative versus when a metric is considered qualitative.
Many refer to quantitative metrics as those that are related specifically to behavioral data. So for example the number of clicks, the number of page views, the number of error rates, or rather the error rates, etc.
In contrast, qualitative metrics are often related to more attitudesal data. So a little more abstract.
Think about, you know, user satisfaction, trust in a product, things like that. So for purposes of this talk, we're gonna be combining all of those into one because we don't want to restrict these metrics that we're discussing.
To just a specific type of data. So when we refer to quantitative metrics, we're going to be referring to any quantified measure of user experience.
Which may include behavioral data, attitudinal data, etc. Okay, so. Why do we want to talk about numbers and how do they relate to collaboration, right?
Our focus of this entire event today is around collaboration. Well, as Nathaniel West once said, numbers constitute the only universal language.
So we want to talk about how you can use this universal language to communicate and collaborate across teams.
So to cover our takeoff like Dr. T mentioned, the 3 main takeaways you can expect from this presentation include what a shared quantitative metric is and how it can foster collaboration.
The benefits of cross team collaboration through a shared quantitative metric and how to strategically implement shared quantitative metrics in your program.
And most of all, we hope that you can walk away from this presentation. With more questions and more ideas for discussion.
Alright, so I'm gonna pass it over to Megan so that she can give us, some context for the inspiration for this talk.
Thanks, Chantal. And, like, Chantal said, I'm going to start by giving a background about the ESS program as well as the ESSUX team.
So for those of you who, might not be familiar with ESS. I'm gonna wait till the slide goes so you can see.
There it goes. For those who night not familiar with ESS, ESS stands for the Enterprise of Shared Services.
And it is a collection of digital tools and services, used by internal CMS employers, employees, I'm sorry, contractors and vendors, which is a little bit different from other CMS programs that you might be on your be yourself.
So like think of Ike's or QPP, which are more, public facing.
In fact, one of there are many tools that are supported by ESS. And I have a few examples on the screen such as Slack, Z scalar, file cloud and gear just to name a few.
And so what's cool is that because ESS is used by people like us, it's central to meeting the needs of the CCS community.
And so it includes everything from identity management tools. File management tools, communication and collaboration tools, analytics tools, and even services for ADOs.
And even cooler is that many users actually access multiple services within ESS daily.
And so, diving a little deeper. Now let's talk about the ESS user experience team.
So we are a shared services team. And what I mean by that is that our one team supports multiple products.
So in order to support these multiple products, each team member is assigned one to 2 products, each team member is assigned one to 2 products.
Each team member is assigned one to 2 products really just depends on the needs of the product, each team member is assigned one to 2 products, really just depends on the needs of product. Really just depends on the needs, the product team at that time.
And so naturally what happens as a result of us being split the team members can become kind of hyper focused on the individual products they support, right?
Because we want to do a really good job at the thing that we're working on. And so because like I just mentioned previously, most access multiple products within ESS, which I mean most most users, that means that if I am focused on the angel product, let's say for example, I am on the tools team and I am working as a researcher to support that product.
What may happen is I may become unaware of how my work fits into like the bigger picture of the entire user experience.
And so this current way of working on individual ESS product teams has kind of lent itself to a more siloed approach towards research and design.
And so what do I mean by that by siloed approach and how might this way of working compare if we work any more collaborative way.
Okay, so let's dive in and look at that a little deeper. So when I say siloed research approach, what I mean is that there are separate teams that are focused on separate points of the customer experience.
So in the case of ESS, we are focused on the user experience for each product separately. So kinda like you see there, the ESS heart product team is focused on the Harp user experience.
The, tools team within ESS is focused on the tools. That user experience. And so while we are generating, you know, excellent insights within the experience for that particular product, what we are kind of starting to uncover is that this division is leading to sort of a lack of overall understanding.
Of the ESS user. And so, you might be. Starting to formulate some things like, oh, maybe there's some downfalls or some limitations to this and you're right.
So some of the things that we are sort of uncovering as we've been working this way is that it is leading to a limited view of the overall customer experience.
So different teams have focused on specific like aspects or touch points. And so what happens sometimes is that we miss the bigger picture of the customer journey.
And not only that, but within these individual teams, we might unknowingly investigate the same issues, which could result in wasting time or resources or to kind of even make it more relatable when we're having conversations and our team meetings, we might say, oh, I just spoke to an individual about that.
I didn't realize they used this tool and that tool. And so this more isolated approach means that we're missing these opportunities to sort of like identify connections across the different parts of the ESS user experience.
And so ultimately what happens is while we are sharing information among the individual product team, sometimes the insights get restricted, though not intentionally, but they get restricted to those internal teams and they don't get shared more widely across the organization.
And so if I. Go the flip side and say okay so we know that there's a little problematic with siloed approach if we take a more collaborative approach what might that look like and what do we mean so when I say collaborative research approach what I mean is that where the teams are collaborating not just individually but also amongst other teams and that hopefully if we shift and work in this way, we might be able
to capture more of the entire user experience. And why else might this approach be better? Well, as we were kind of talking and discussing, we started thinking about how it could allow us to really see the whole product of the ESS suite, if you will, and how that impacts the ESS users.
So stop referring to them as the Harp user or as the file management user, but as the ESS user.
And so that should hopefully make it easier. And so that should hopefully make it easier to identify where the products are intersecting.
And like I said, to help like make those clear connections across the whole journey. And then also if we're.
Talking more among other teams, then hopefully we're starting to get more perspective on things and that should allow for more inclusive solutions.
For the user because again if you recall an ESS user uses multiple products and so we're thinking that all of these new connections and perspectives are going to lead to new research and design opportunities.
And if I haven't made it clear enough, let me just really drive home the point. So there is naturally this.
Many benefits that come from this cloud of research approach and they don't extend to just the user. But they extend to the product team and to the whole ESS program team, which ultimately means that the CMR CMS stakeholders are going to also benefit because this more collaborative way of working just really has impact across multiple levels.
And so, you might be wondering why are more organizations doing this?
And well, it's hard. Collaboration is difficult. Most will say, yeah, I wanna get better at it, but they don't really know how to go about it or they know that it's a challenge.
Where do you even start? And so often the collaboration is happening like I was just saying, but it's happening among individual teams.
And so I thought it might be helpful to just. Take away ESS or CMS and just user really simple example.
Like an ecommerce site to kind of, make sure I'm getting my point across.
And so if we take an ecommerce site, let's say that you have a team working on the login experience.
You have another team working on a shopping experience and another team working on the checkout experience. There's probably other teams this is just a few to give an example.
So all of these teams individually are collaborating. They're generating excellent research and insights. They're even having brainstorming sessions.
But they're falling short because they're sharing their findings within and not beyond the team. And so.
You know, that means that each team is only relying on the information that they know. And so how does that compare to ESS?
Well, it's actually pretty similar. On ESS, we are doing a good job of collaborating.
We have those daily stand-ups. We have refinement means we have backlog means and each team is working on their product.
They're on harp. They're on Q not or they're on Q and P and so forth.
But this way of collaborating falls short because the teams again are relying on their own knowledge instead of sharing out.
And so even if we start doing this sharing out and surfacing of knowledge, it doesn't necessarily mean that all the teams are going to align and agree.
But if we are having a better like starting point, we'll have a better chance of working together.
And a loting around what we know. And so now I want to hand things back over to Chantal as she discusses how we might in this case as we kind of hinted as you might use numbers to improve collaboration because like she said earlier, numbers are universal language.
Alright, so as members of the ESSUX team, our goal is helping CMS create a consistent high quality user experience.
So how can we achieve this more collaborative effort like Megan was just discussing? So one way of achieving a more collaborative approach that may offer a more holistic view and understanding of the user is through using shared quantitative metrics.
So again, just a quick reminder that in our talk we're defining quantitative metrics as a quantified measure of user experience, including behavioral and attitudinal data.
This, this sort of measure can include things like number of page views that can be collected via click stream data, user error rates that can be collected during usability testing or even user satisfaction scores that you can collect from a scaled response survey question.
Now keep in mind that not every metric applies to your product. As UX researchers, our job is to ensure that the chosen metrics are valuable and insightful.
So I wanna turn him back to the audience again. With a similar question, I know we, asked about whether or not your team was currently measuring user experience and how at the beginning.
But now we want to know specifically now that we have this definition of quantitative metrics. Is your team currently using quantitative metrics to measure user experience.
Okay. Let's see if we can get as many or more responses on our first.
And again, feel free to throw any comments or, thoughts into the chat.
Anyone just wants to send if you did put that you are using quantitative metrics.
I'd love to see in the chat, which of those.
Yeah, I was gonna ask. We're almost 50%. We'd love more participation.
User error for sure.
I'm happy to see that no one has put no interest in using these. Okay.
Hi, Gary. I would have been very sad.
Okay.
So I see Ethan said user error. Thank you so much. Definitely a good one. Raphael, I need to use more.
Hopefully you can have some inspiration from this talk and some ideas.
Alright, let's see if we can make it to 60. Anyone else? It's okay if you're not sure.
Or if you're not interested. Maybe we'll p your interest.
Okay. Okay.
Alright, doesn't look like we're, moving past 58%. So let's just, stick with that.
It looks like, a third of our audience is currently using quantitative metrics.
So that's exciting. We have about a quarter of the audience that is not currently using, but we would like to.
So that's great to hear. And some of us that is not currently using, but we would like to. So that's great to hear.
And some of us that aren't sure, which is totally fine. Hopefully we can learn a little bit from this presentation and spark some ideas.
So thank you. All so much for giving us that. Feedback. Alright, so.
Why do we wanna talk about quantitative metrics? Why numbers? Well, quantitative metrics provide objective and measurable data.
This makes it easier to draw clear conclusions and make data-driven decisions. Quantitative metrics can also be used to monitor changes in user experience over time.
So you can collect data at multiple points and you can track the impact of things like design improvements, changes, or any other updates.
You can also benchmark your products performance against other things like industry standards or perhaps competitor products by using quantitative metrics.
This can help you find areas where your product may be falling short or may even be excelling. Quantitative metrics can also help in measuring the return on investment of UX improvements.
I know sometimes that's very abstract and hard to communicate. So by having an objective metric, you can then compare those to key performance indicators like conversion rates.
Engagement or even revenue. And finally, stakeholders tend to be especially fond of numbers and we always like to be able to communicate very clearly with them.
Okay, so we've been talking about quantitative metrics, but specifically we're interested in talking about shared quantitative metrics, right?
Because we're talking about collaboration. So what is a shared quantitative metric? A shared quantitative metric is a single quantitative metric that is used across multiple teams or products.
So think back to our spider-man picture. Basically all those Spider-man's would be using the same single quantitative metric.
So for ESS, this would be a shared metrics across all ESS products. However, for your team, this might be a shared metric across various touch points.
Within a product that you have teams for each of those touch points.
Now, how is it that shared quantitative metrics can help foster collaboration? First, it helps with alignment of objectives.
So having a shared metric can ensure that all teams are aligned in what their understanding is of what constitutes a positive user experience.
Right? You're using the same metric and presumably from that metric you can tell with a score of this that's a positive user experience with a score of that things might be in trouble.
So this alignment helps promote a common goal of enhancing that user experience because you have that shared understanding.
It also helps foster collaboration by facilitating clear communication about customer experience. So communication is a lot easier when everyone has the same understanding of the same metrics.
Teams can have a clear and shared understanding of what that metric is actually measuring and what it represents.
This also necessitates an alignment of terminology. So for instance, teams need to agree on the specific components and definitions of the metric.
A shared metric also ensures data consistency which of course can make it easier to compare performance and track progress over time.
This can help minimize discrepancies and miscommunication that can occur when different teams use different metrics.
And finally, a shared metric creates a sense of shared responsibility for customers satisfaction. Teams are more likely to take ownership of their role in delivering a great user experience if they know that their performance is going to impact the same metric.
So how can we develop and maximize the potential of our own shared quantitative metric?
Can't we just make a survey and call it a day? Well, you could, but you probably won't get much out of it.
And you'll likely end up with findings that have Very little direction.
Implementing a shared quantitative metric is actually a very strategic process. It requires cross-functional participation including researchers, designers, product owners, and business stakeholders.
It has to be grounded in a cohesive product vision, a unified set of values, and a dedication to user-centered approach.
And establishing a shared quantitative metric requires a commitment to ongoing enhancement and consistent monitoring of that metric.
So how do we do that?
Well, there are several steps to creating and implementing a successful quantitative metric. First, you'll want to establish unified objectives.
When you're setting your goals, take into account both the user objectives and the business goals. What holds the highest significance to the user?
What aspects of the customer experience or user experience are most important. For the organization.
Next, you'll want to develop your shared metric after you've defined your goals and objectives.
First, identify key metrics, determine those specific metrics that will help you evaluate user experience effectively and that align with your objectives.
That will help you evaluate user experience effectively and that align with your objectives. So for example, if you're interested in user experience effectively and that align with your objectives.
So for example, if you're interested in user experience effectively and that align with your objectives. So for example, if you're interested in, and that align with your objectives.
So for example, if you're interested in, user behavior, you may identify key met If you're interested more in customer, I'm sorry, user attitudes or user experience in that sense, then you might be interested more in user satisfaction.
Scores or things like that. So once you've identified those key metrics, then you want to use them to create a composite metric that combines multiple key metrics into one.
It's hard to get a holistic view of the user experience if you're only considering one key metric.
Like only looking at the number of page views, for example. You wanna remember to also standardize your questions and response scales if you're using a survey so that you can maintain data integrity across teams.
So everyone should be using as close as possible of the same questions and the same scales. And importantly, you want to keep it simple.
It should be very straightforward and not too complex. You want to make it straightforward for team members and stakeholders to be able to grasp an act upon.
If you're using a survey, it should be used friendly enough to encourage participation. Nobody wants to complete a 20 question survey.
At that point you might as well just call it an exam.
So after you've developed your shared metric, now it's time to collect an analyze your data.
Before you do that though, you want to make sure that you standardize data processes. Make sure there's consistency in how that data is being collected.
Stored and analyzed. Ideally, your data should be stored in a shared data repository for transparency and accessibility by all teams.
You'll also want to establish standardized methods for analyzing and reporting your data. If you think of 2 different teams using the same metric but visualizing them in very, very different ways, it can be a little bit hard to compare findings.
Next, now that you have your collected data and you've analyzed it, you'll want to share your insights, share the work that you've done, and communicate it broadly.
So share that in those insights. With all relevant teams and departments within your organization. This will help ensure that everyone is aligned with where the user is now and where they could be.
You also want to try to have a centralized resource where all teams can share their metric results. Something like a metrics dashboard would be great so that you can make insights readily available to all.
And keep that updated.
You'll want to make sure to take action after you've shared your insights. Don't just collect the data.
Sit on it and not do anything about it. As is any good practice of user experience work, you want to make sure to act on that customer feedback to address those pain points, enhance the customer experience.
You also want to set targets, define measurable targets to help guide improvement. And remember that this is a cyclical process.
You want to continueously monitor and improve the performance of your metric. Be prepared to adjust it as needed.
Based on feedback or even environmental factors. We can all think of one really big one, the pandemic.
People's practices and values really changed during that time. And you know, you can't use the same measures to understand user experience.
So I'm gonna turn it back over to Megan so she can give us our some closing thoughts.