Join us as we "dig for gold" in sizeable textual data sets to uncover the stakeholder themes and sentiment, with industry-standard accuracy rates. Using a presentation and demo, we will share the challenges we overcame to minimize text interpretation bias by using Natural Language Processing and Machine Learning (ML). Attendees will journey through our data mining process, which maximized model accuracy when identifying themes and associated sentiment.    

We will compare two thematic/sentiment model visualizations to emphasize the criticality of Human-Centered Design in big data interpretation while highlighting interactive filters to maximize targeted views of subsets of data. 

When talking about Artificial Intelligence (AI) in the past few years, we have focused a lot on Machine Learning (ML), but much like in our school system, learning is only half of the equation. We are missing the other half: Machine TEACHING, which is where the human-centered portion comes in. This session will reveal the findings and intersect a technical discussion with a real business context: Fraud, Waste, and Abuse (FWA).  

For a business to thrive, cutting expenses is just as necessary as increasing income. In industries like Insurance, Finance, and Healthcare, FWA is one of the top costs, costing companies billions of dollars. Traditionally, a group of experts and investigators handle this time-consuming and labor-intensive process, but the rise of AI can assist humans in identifying and preventing FWA.  

This session will show not only “why” we should use AI to fight against FWA but also “how” to achieve it practically in 3 significant sections:  

• Supervised Learning: We will explore how basic algorithms like decision trees, logistic regressions, and support vector machines can help fight FWA. 
• Unsupervised Learning: We will see how techniques like clustering can complement the supervised learning approach.  
• Leveraging Generative Adversarial Networks (GAN): We will investigate the use of GAN (used in image generation) and see how this more sophisticated model can help fight FWA.  
• The session demonstrates by utilizing an open data set, which is available online. We will also include the latest research in this area to strengthen the cases presented. 

The recording of this session and a copy of the presentation slides are posted above.

Artificial intelligence (AI) seems to promise wide-ranging automation of many familiar civic technology components by providing just-in-time help to website visitors, performing audits and calculations, and personalizing services to specific users. However, in practice, these “artificially intelligent” systems generally rely on complex combinations of humans and machines to produce user experiences. This requirement poses unique challenges for service design and system transparency, with profound consequences for overall user trust and user experience.  

In this presentation, we take a sometimes-speculative look at how we have been thinking about “heteromated” systems such as chatbots and measure calculation on Healthcare Quality Reporting and beyond, providing both a theoretical overview and some practical guidance for UX professionals working with artificially intelligent systems. 

The recording of this session and a copy of the presentation slides are posted above.

Imagine a world where Medicaid and Medicare beneficiaries are empowered hand-in-hand with providers focused on improving their care while also managing costs. How can we incentivize clinicians and beneficiaries to participate proactively in the CMS CCSQ mission? 

Machine Learning (ML) can help drive this vision. ML presents all-new capability options for CMS, but it's not a panacea. There are challenges and opportunities to consider when moving from predominantly post-care quality measures and traditional analytics to a proactive ML model-driven approach. Humans are critical to the overall performance of the recommender (ML Models) systems. 

We will discuss the challenges and capabilities of four main topics: 

• How can we train ML models to help drive the CMS CCSQ mission focus: Steadfast focus on improving outcomes, beneficiaries' experience of care, and population health, while also aiming to reduce healthcare costs through improvement." For example, AML recommender systems might include health goals, treatment outliers, fraud, waste, abuse, treatment trends, and anomaly detection. 
• Where is the CMS data? Discuss privacy concerns of training data such as using federated learning. 

• What data/feature engineering challenges are there inherently with Medicaid and Medicare data? 
• Discuss which humans (actors) should participate in ML to ensure safety, reliability, and trustworthiness. How can beneficiaries remain in control of their participation, privacy and continually improve the trustworthiness of AML?  

Conclusion: Human involvement and participation is an essential and necessary element for ML unbiased trustworthiness, reliability, fairness, and safety.

A recording of this session is posted above. Please note there was not a presentation deck for this session. 

Health data requires unique privacy and governance protections. Certain types of health data warrant specific protections based on how and from whom the data is collected. Patient-reported outcomes measures (PROs/PROMs) data, for example, require specific protections, particularly when it is used in or informed by machine learning regimes. A patient/human-centered, federated learning architecture is appropriate for ensuring the privacy of users’ data. However, using a federated learning approach to provide privacy without attention to private data management dimensions may compromise users’ data unexpectedly.  

Users of machine learning to collect and manage PROs/PROMs should ensure that:   

• Choices about machine learning models do not open users to attack or undue influence and thus do not open users to liability for interpretation of false responses,  
• Machine learning (ML) models are not compromised, and valuable machine learning spending lost to competitors; and  
• ML models are tested and validated to ensure the quality of unstructured PROM data versus influencing or skewing PROs concerning safety, symptoms, and other vital outcomes. 

The recording of this session and a copy of the presentation slides are posted above.

This session will serve as a practical walk-through of combining techniques from Human-Centered Design (HCD) into Machine Learning (ML) projects to create usable, trustable, and accurate systems to support real-world data quality and data governance efforts. 

The presentation will connect the dots between the: 

• logical flow of an example system, 
• specific open-source technologies and sub-systems, and 
• specific recommendations to get the right people involved in each step of the process. 

The presentation is targeted at cross-functional teams and provides an example implementation approach and common pitfalls seen along the way. The presentation will utilize a case study focused on improving data quality pipelines and supporting data governance – but the information applies to any HCML project.  

The audience will leave with: 

• A solid understanding of the component parts of a real-world and complex ML project – along with the typical flow through those components. 
• Specific technology examples used in each component area, and an overview of what typically goes into ML architectures that utilize a modular and open-source approach as a precondition. 
• Information on exactly how and where to involve users, whether building or maintaining an analogous ML system.

The recording of this session and a copy of the presentation slides are posted above.