2022-9-1 CCSQ D&A Townhall

1. CMS Support Email Reminder
2. System Updates
3. Data Source and Usage Updates
4. Monthly User Experience Satisfaction Survey – Zoom Poll
5. CDR Persistent Storage Migration
6. Notebook Alternative Update
7. SAS Viya Workbench Integrated with CCSQ QuickSight
8. Q&A

• The CMS Support email (servicecentersos@cms.hhs.gov) will be unavailable between Friday, September 2nd at 8 pm EDT, and ending on Tuesday, September 6th.

What if you need support?

• Utilize CCSQ Support Central “Create a Ticket” to submit CCSQ Service Center tickets
• If you have access to CCSQ QualityNet Slack, you can contact the CCSQ Service Center at #help-service-center-sos

SAS Viya Lagging Issue

• Issue Description: SAS Viya users have reported intermittent lagging and freezing while utilizing the application.
• Resizing efforts will take place during the September 9th maintenance event to help address this issue. More resources will be available for SAS job processing

Scheduled CAP & CDR Maintenance Schedule

• September 9th
• October 7th
• November 4th

All events will begin at 8:00 pm ET and end approximately at 11:00pm ET. Some events may require longer time, in which case users will be notified separately. A communication will be sent out once maintenance is complete. As a reminder, whenever there is maintenance on the environment, you will need to make sure all of your code and table changes are saved.

Known Issue - QMARS Longer Text Fields - 1/12/2021

• Workaround: Consider using healthcare_service_qmars_ng if your DUA supports it

• Data & Analytics Team have started to reach out to CDR users to migrate legacy hive data sets from HDFS to S3 in phases
• Benefits:
  • Frees up disk space within CDR Cluster
  • S3 storage provides unlimited capacity, better persistence, and lifecycle capabilities
  • Supports future migrations of CDR Architecture to better alternatives
• Dependency: Data & Analytics will depend on teams for coordination and validation of migrated data sets

Phases:

1. Pre-migration
   • CDR/CAP users notified that their data sets will be migrated
   • CDR/CAP users verifies that they are able to access databases named <db_name>_s3
2. Migration
   1. D&A Engineering takes multiple backups of databases (one month of weekly back-ups)
   2. D&A Engineering migrates user databases and tables (~2 weeks) (note: during migration you'll see tables appear with <table_name>_CDRTMP)
3. Post-migration
   1. D&A Engineering removes all temporary <db_name>_s3 and <table_name>_CDRTMP tables
   2. CDR/CAP users notified that data migration is complete
   3. CDR/CAP users verifies that all databases and tables are available and accessible 

• Migration will be completed in phases with multiple groups at a time
  • Phase 0: X-small databases (up to 100 GBs) – Completed (Aug 2022)
  • Phase 1: Small databases (100 GBs - 1TB) – In Progress (ETC: Mid-Sep 2022)
  • Phase 2: Medium databases (1-3 TBs) – Upcoming (ETC: EOM Sep 2022)
  • Phase 3: Large databases (3-10 TBs) – Upcoming (ETC: Mid-Oct 2022)
  • Phase 4: X-large databases (10+ TBs) – Upcoming (ETC: EOM Oct 2022)

• The pilots will allow us to confirm with users whether Sagemaker Studio + EMR or Databricks Notebooks + DBX Compute would be a better long-term alternative to replace Zeppelin and Ambari
• This is being done to provide a stable, scalable environment, improve developer productivity, and decrease analysis runtimes
• We are collecting and analyzing user feedback throughout the pilot to ensure the tool meets the needs of the community.

Phases:

1. Design
   • D&A Team and CMS meet to outline technology pilot project goals, requirements, and UAT criteria
   • D&A Product Team begins outreach to recruit users for pilot
2. Implementation
   • D&A Engineering, HARP, and HIDS Teams colalborate to setup, configure, and implement technology solution for the pilot
   • D&A Engineering sets up user accounts for the pilot
3. Evaluation
   • Pilot Users evaluate product and record feedback in User Acceptance Testing (UAT) pages
   • D&A Engineering evaluates product from scalability, extensibility, and maintainability perspective
4. Analysis 
   1. D&A Engineering finalizes engineering assessment and validation of technology product
   2. D&A Product Team synthesizes findings from Pilot User and Engineering feedback and presents to CMS
5. Decision
   • CMS Leadership makes final decision on technology pilot options
   • (post-pilot) D&A Engineering productionalizes product and onboards CCSQ User Community

Tools Under Evaluation: 

1. Databricks – Evaluation Complete (Aug 2022)
2. Sagemaker Studio / EMR – Evaluation In Progress (ETC: End of Sep 2022)

Upcoming: Complete Sagemaker Evaluation and Start Analysis (ETC: Oct 2022)

• SAS Viya Users can now use data directly from their Workbench in CCSQ QuickSight!
• Eliminates exporting data files just to upload into QuickSight to be used in your dashboards
• 2 Steps needed to complete to establish this capability:

  1. Request a dedicated folder in your Workbench here: https://qnetconfluence.cms.gov/x/8IjEBg

  2. Write your SAS Viya data files into a Hive table:
     https://qnetconfluence.cms.gov/x/_7ULC

     • Data in Hive will be accessible in QuickSight within 15mins (background processes run with AWS Glue).
• Once logged into QuickSight:
  • Go to 'Datasets"
  • Select 'New Dataset'
  • Scroll down, under 'From Existing Data Sources' and select 'CCSQ Data Warehouse...'

1. What is an update on the CCSQ Data & Analytics boot camp?
   A - We have been working on finalizing some details, but next Communications Call, we will be able to share more about the date and theme. 

2. Regarding migration....would the team prefer that we drop any unnecessary tables now or will doing so cause issues when validation occurs. For instance, given that migration of tables of a certain size has begun if we drop a table now will that be an issue - cause confusion during validation?

   A - If there are temporary tables that can be cleaned up before we started the migration, please do so to reduce the migration time for your group. 
3. Is the pilot for AWS Sagemaker/Databricks already in progress/completed ? If not, how can one become part of it ?
   A - The pilot is already in progress and it's currently limited to only a few set of users to reduce the maintenance overhead associated with running the pilot. However, if there is interest from users to participate in future pilots, we'll reach out in these communications calls to let you know of the opportunity.
4. We have been trying to do some cleanup of data we have stored on Hive but the %usage_hive macro continues to show tables that we do not see in Hive and the total data usage has remained unchanged even after removal of large tables. Can you please advise how best to check current Hive space usage?
   A - Please create a support ticket to track this issue. There may be reasons for why your HDFS usage is high even though you've dropped tables, including temporary data sets that may have been generated but not cleaned up automatically by hive when an error occurs during processing or if tables are explicitly configured to not drop data objects from the cluster when you drop your table. 
5. Will the Pilot phase allow full access to CDR and Viya S3 datasets using R and Python?
   Will the same resources (CPU / memory) be available in the new notebook as they are available for Viya processing?
   A - As part of the pilot, we are assessing R, Python, and PySpark with PROD CDR Data. The processing of the new tools will be more than SAS Viya currently. We want this new solution to be scalable with personal compute settings. In the new model, users can isolate their workloads so that they do not affect other users (scalability in real time). 
6. We used usage hive macro to get individual table sizes in our hive databases when notified of the migration and several showed size 0 or no tables in the database. Any idea what might have caused that?
   A - Please create a support ticket for this question since we want to evaluate your use case.