- Created by Rachel Dodge, last modified by Gabrielle O'Neal on Jan 10, 2024
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 48 Current »
CDR Data Contribution Standard Operating Procedure (SOP)
This SOP provides an overview of the Centralized Data Repository (CDR) Contribution process for contributors that plan to make data available in the CDR for all organizations with an approved data usage agreement (DUA). These sources and their approved documentation will be posted to the CDR Data Catalog.
Background
The CDR provides access to CCSQ data including claims, provider, beneficiary, and other data within a secure cloud environment. The goal of the CDR is to make data available from source systems with greater accessibility, security, quality, and timeliness of data. The goal of Contribution is to allow CDR Data Contributors to make data available directly to CDR users with less data duplication.
Request Form
Submit a request through the CCSQ Data & Analytics Request Form to start the contribution process. The request will be reviewed and prioritized by the CMS Business Owners of the CDR. Use the following as a guide for what information to include in the request.
Required Information
The following information is required to be included in the request to become a contributor:
- A short description of the data with the minimum required information.
- The data dictionary with the minimum required information.
- The point of contact for the contribution effort.
- The anticipated timeline to load, validate, and release data to end users.
- The list of any validation groups that will require access to the pre-prod data for testing.
- The Enterprise Privacy Policy Engine (EPPE) DUA Entry associated with this data.
- The Access Control Model
The CDR will need one of the following for support purposes to forward user inquiries specific to the data:
- A CCSQ ServiceNow Support Group name
- A support email address
The team will also need to know:
- The access control model for this data (see below).
- Whether a data catalog entry can be made for this data on the public data catalog or if access to the data documentation should be restricted.
Required Data Documentation
All documentation submitted for contributed data must meet the minimum required standards as outlined below.
Short Description
Short description of data must include the following:
- Title - The name of the dataset or resource.
- Domain - The category of the resource such as (eg., Clinical, Provider, Claims, Beneficiary)
- Source - The system or activity that generates or provides the dataset.
- Granularity - The level of data detail that the resource covers, such as Atomic, Aggregated, Hybrid
- Load/Refresh Frequency - How often this data source will be refreshed/updated.
- Geographic Scope - What level of geographic scope this data entails (e.g., national, regional, state, county, zip)
- Timespan Covered - What timespan this data covers (e.g., 2019-2022)
- Personally Identifiable Information (PII)/Protected Health Information (PHI)/Sensitive Information - Whether this data contains PII/PHI/Other Sensitive information or not.
Data Dictionary
At minimum, documentation for the data contributed to the CDR should be in the form of a data dictionary that includes:
- Table name
- Table description
- Field Name (Short Name)
- Field Name (Long Name)
- Data Type/Length
- Field Description (provide details)
- Possible Values/Range or a Reference Link
- Partition Key
- Comments
Data Documentation Format
Submitted data documentation should be in a comma-separated values (CSV) format that meets the required data dictionary template.
Required Data Format
The preferred data format for partnering systems is parquet due to superior performance and lower storage cost. However, CSV or JavaScript Object Notation (JSON) may also be used. This data should be stored in the partnering Application Development Organization’s (ADO's) Simple Storage Service (S3) bucket. It is also recommended to create directory versioning so that if parquet files need to be re-created, they can be created in a different version directory and not impact the current parquet files.
- Preferred Data Format: Parquet
- Accepted Data Formats: CSV, JSON
Access Control Model
The access control model refers to how contributors would like to restrict or grant access to the data that they are contributing. The CDR supports two access control models:
- Automated (DUA Based)
- Access to this data will automatically be granted to any end user organization that has the appropriate DUA entry on their DUA. Contributors will provide the DUA entry that corresponds to the contributed data.
- Manual (Notification Based)
- In addition to the DUA entry, access to this data is manually approved by the Business Owners of the data. End user organizations request access to the data, and we notify the Business Owners for either approval or rejection.
Contributors must indicate which access model they would like to utilize for your data.
CDR Onboarding Steps
Onboarding into the CDR is central to becoming a Data Contributor. If the prospective contributor is not already onboarded, see the training and onboarding page for more information.
Being onboarded into the CDR comes with the following automatically provisioned resources for the contributor's organization/group:
- A production database/schema
- A pre-production (test) database/schema
- A Service Principal/Service Account
Once onboarded, the team will generate a schema specific to the contributor's BYOD data in accordance with CMS Data Taxonomy.
Contribution Workflow
The following is a generalized workflow that outlines the steps involved in contributing to the CDR.
Phase | Step | POC |
---|---|---|
Phase 1 | Request Submitted with Required Information | Contributor |
Phase 1 | Onboarding Initiated (if not already onboarded) | Contributor |
Phase 1 | Service Principal Automatically Generated | Automated upon onboarding |
Phase 2 | Bring Your Own Data (BYOD) Prod and pre-PROD Schemas created | CDR |
Phase 2 | Elevated Service Principal granted r/w/x access | CDR |
Phase 2 | Data Documentation Reviewed | CDR |
Phase 2 | Data Catalog Entry Created (if applicable) | CDR |
Phase 3 (optional) | Data published to pre-PROD schema | Contributor |
Phase 3 (optional) | Green light for Validation Groups | Contributor |
Phase 3 (optional) | Validation groups granted r/x access | CDR |
Phase 3 (optional) | Validation period conducted | Contributor |
Phase 4 | Data published to PROD BYOD Schema | Contributor |
Phase 4 | Green light for end user access | Contributor |
Phase 4 | End users granted r/x access per access model | CDR |
Phase 4 | End user communication released | CDR |
Cross Account Access
A contributor's data (parquet files) are stored in their own Simple Storage Service (S3) bucket so cross-account access must be established to allow CDR necessary access to the S3 bucket. CDR has adapted the resource-based policies and Amazon Web Services (AWS) identity and access management (IAM) policies method for accessing cross-account S3 bucket documented on AWS Support. In this case, the partnering contributor is "Account A", and CDR is "Account B". Data encryption is highly recommended either with the standard AWS server-side encryption or custom key stores.
Below is an example of a resource-based policy with custom key stores configuration from a partnering contributor.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "GrantS3BucketAccessToCDR", "Action": [ "s3:GetObject*", "s3:List*", "s3:SelectObjectContent", "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey"], "Resource": [ "arn:aws:s3:::AccountABucketName/*", "arn:aws:s3:::AccountABucketName/", "arn:aws:s3:::AccoutA:key/AccountAKMSKey"] } ] }
Caution
A resource-based policy must be granted before external tables can be created in the CDR Hive metastore.
Connecting to the CDR
Data Contributors may desire to contribute data to the CDR using either our suite of tools that they gain access to through onboarding or through their own external.
See the following articles for details on making inbound and outbound connections to or from the CDR.
- For accessing an external database through the CDR, see this article on making Outbound Database Connections through the CDR.
- For accessing an external datastore through the CDR, see this article on making Outbound Datastore Connections through the CDR.
- For using an external tool to connect to the CDR Datastore to write data to the CDR, see this article on making Inbound Connections to the CDR Datastore.
Validating Contributed Data
A contributor's service principle is for automated, system to system connections only. Validation should be done through user accounts associated with the designated validation group - a contributor's group will be automatically provisioned validation access.
Ensure that any users performing validation request the HCQIS Access Roles and Profile (HARP) role for the appropriate group so that they can access the data in the CDR.
Changes to Data or Data Documentation
Data Contributors are expected to notify the CDR team of any changes to their data or the data documentation. Submit a request through the CCSQ Data & Analytics Request Form in advance of any changes, outlining the expected changes and with any new documentation attached.
Reporting Issues
Data Contributors are responsible for informing the Data & Analytics team when there is an issue with their data. Notify the CDR team within one business day when an issue is identified with their data. The following information will be needed:
- Impacted Data
- Issue Description
- Current Actions Taken to Resolve
- Planned Resolution Date
- Data Owner Technical Point of Contact
Note that contributors are also responsible for sending follow-ups and updates via email after the issue is identified.
User Support
End users of the CDR may have questions specific to the data that Data Contributors have added that require support from a data subject matter expert (SME). Data Contributors and Data Source Owners are expected to support the end users of their data with any questions or concerns that they may have that cannot be answered through the provided documentation or the CDR support team. In such a case, end user inquiries would be routed to one of the following, to be addressed by the data source SME:
- A CCSQ ServiceNow Support Group name
- A support email address
- No labels