Week of 4/5/2021

EQRS Data Submission (EDI/API):  Errors & Warnings

Prior to the 11/9 implementation of EQRS, the EDSM business validations rules were “relaxed” at the request of the Electronic Data Interface (EDI) submitters, who submit 92% of data in EQRS via EDSM, in order to meet the EQRS implementation timeline. With the relaxation of the business validation rules, there is potential for EDI submitters to submit invalid data, resulting in high data submission errors (e.g. 10x increase of 401s, receiving errors CMS thinks are turned off)

CMS can not confidently use the data for quality reporting and quality improvement initiatives (i.e. QIP, Dialysis Facility Compare (DFC) 5 Star Rating) for public reporting.

Request additional data scientist support to a data validation study to compare the data from CROWNWeb to data in EQRS at a point in time. This would restore community confidence in the data.  
Return the validation rules to the original pre 11/9 state.

6/1/2021 - All validation codes are turned back on and EDIs are able to successfully and confidently submit data.

6/30/2021 - Data ready for measures and scoring calculations.

Michael Kennedy discussed the following with Adrienne Ray as it relates to the issues being reported to Phase 1 Admit Reason Code deployment:

• Types of data issues (duplicate admissions, erroneous admissions with and without attached 2728 forms, incorrectly attributed death events and forms, incorrectly attributed acute statuses that will eventually affect coverage) and the errors allowed by EQRS or the data migration in November.

• The current perception by the ADO and the Helpdesk on adding additional false data to make the system work, but it is still incorrect data (adding acute discharges to duplicate and overlapping admissions).

• What was being allowed in the system with the System Admin role vs. CMS editor and NW editor roles, but can’t since some validations were turned back on in production (deleting erroneous admissions from the incorrect unit).

Adrienne agreed that was never the real purpose of the deletion discussion and the integrity of the registry was more important. Adrienne met with Lisa Rees and Curt Phillips to get their approval.  Lisa Rees signed off on Production support under Semantic Bits being able to perform these updates with their historical and expert knowledge from previous Network and ESRD work.  Curt understood the issues and he gave Adrienne approval for SB to work the tickets as needed currently for the foreseeable future until UI enhancements and EQRS enhancements can be implemented to prevent the need for manual intervention.

Increase numbers of issues being reported to Helpdesk as it relates to the new admit reason codes that were pushed to production as EDIs are submitting their files this week.  These are related to missing discharge dates due to bad data being submitted via batch files with no validations.  Clean up effort analysis is underway and a meeting will be scheduled once SemanticBits is able to determine full scope and best approach for clean up.

Duplicate Patient Record (Ghost Record)

Duplicate patient in the DB with full, partial or no attached data.  Each duplicate record could potentially contain important transactional data changes.   Determined that the issue in EDSM was occurring due to same patient records being submitted within two different files.  Because these files are being processed at the same time, the system is unaware of the potential duplicate record that could be created.  Regarding the duplicate record in the UI, the ADO team was unable to recreate this defect.  

Note:  While the results are the same (duplicate records within the system for the same patient), the root cause of this issue is different then the Patient Merge issue above.  

User Interface (UI Screens/Reports)

There are two reports available through the EQRS User Interface (UI):  Patient Events Report (PER) and Patient Roster Reports (PRR).  There are issues with generating the reports and the system flags an error.  RCA to be provided by 2/23/2021 by ManTech.  

Amazon Redshift imposes a limit of 505 non-super-user connections. The RDS DMS pipelines, the reports microservice, and SAS Viya all require connections from the pool. We've limited the reports microservice to 200 concurrent connections to prevent it from locking out the other services by exhausting the connections. Our production deployment of the microservice depends on 8 containers to provide the reports.  ManTech found that the reports microservice had maxed out its connections. Generally, 200 connections should be sufficient to meet the needs of EQRS and its users because the microservice reuses connections; however, in order to do so, it must be able to release connections back to the pool for reuse. Our current configuration does not allow that. It sets the min and max connections per container to 20, which means that the steady state for connections from the microservice is 160 and that none of those connections are returned to the pool for reuse. 

Network and Facilities are unable to perform patient related operations.  Due to the absence of reports to complete the 2744 in EQRS, the facilities and Networks are relying on the PRR to assist in completion of the 2744.

Renal Data System (RDS) Extract

Due to constraints with the implementation timeline, development of the RDS extract was de-scoped by the Program to support other time sensitive initiatives.

Dialysis Facility Compare (DFC) program utilizes the extracts for rating purposes.

CMS support contractors are unable to meet contract deliverables and perform a variety of CMS functions (i.e. calculate scores, derive data, generate ICH CAHPS survey, perform evaluations.)

Data recipients will access data in Fall 2021. In the interim, the Network program is working to determine what they can provide to data recipients.

Fall 2021

Curt and Scott met with Michele Walton  on Friday afternoon to explore how the work the NCC is doing to provide data for UMKECC might be also be leveraged for HRSA and their contractors, Scientific Registry of Transplant Recipients (SRTR) and Organ Procurement and Transfer Network (OPTN).  Given there are some overlaps in their data needs, they wanted to explore how that may be of assistance to them.  Michelle is the COR for UMKECC.  Her assistance was also solicited given her knowledge of the UMKECC data sets.

We plan to pursue investigating if the UMKECC data set may at least partially solve some of the immediate HRSA challenges.

EQRS Latency

Issue #1 - A long-running query in the Patient Microservice caused other transactions to run slowly or timeout, resulting in Facility users reporting that EQRS is slow and times out

Issue #2 - Issues with incorrect data populating within the Forms and/or not able to view and complete Forms within the application.  RCA is under investigation

Issue #3 - Data job issues where they are taking too long or terminating within the CDR.

Issue #4 - EDI submitters commented that receiving feedback files takes longer than 2 minutes.  Given the robustness of the system, they expect to be dealing within receiving feedback files within micro-seconds.  No tickets have been submitted to the Helpdesk related to this issue.

Network and facilities report slow performance with some operations ‘timing out’ before completion.  This is impacting the completion of the 2744 forms.  NCC reports slow performance data ‘jobs’ that should take 5 minutes are taking over an hour.  The CDR team has terminated ‘jobs’ that took too many resources without informing the NCC.  This impacts the programs ability to provide data reports to the Networks to assist facilities with the 2744 forms, produce reports surrounding data accuracy for the EDI submitters, and provide data extracts for RDS entities.

Issue #1 - ManTech tuned the query to be more efficient.  

Issue #2 - This issue is being tracked in the User Interface (UI Screens/Reports) section below.

Issue #3 - NCC is working directly with CDR to correct issues identified with CDR and RedShift.  

Issue #4 - No tickets have been submitted to the Helpdesk related to this issue.  Upon further conversations with NRAA, this was a "recommendation" for ManTech to look into and not necessarily a problem.  A JIRA ticket will be created to add this item to the backlog of future work to be considered based on prioritization by the SMT.

Issue #1 - Fix was deployed to prod on 12/8/2020.

Issue #2 - See User Interface (UI Screens/Reports) section below.

Issue #3 - Will not track this issue here.  

Issue #4 - Will not track this issue here.