ISG Playbook
APIs
- Created by Angel Tucker, last modified by Nosakhare Eguagie yesterday at 7:57 am
What is QNOD?
The QualityNet Operations Dashboard (QNOD) offers a comprehensive overview of the health of the Center for Clinical Standards and Quality (CCSQ) ecosystem. By aggregating data from multiple reporting sources, QNOD delivers critical insights into the availability and performance of each CCSQ Service. Designed to meet the specific needs of service owners, it showcases essential Key Performance Indicators (KPIs) to assess and manage the operational status of their services effectively.
How Does it Work?
QNOD integrates data from multiple reporting sources to deliver a comprehensive view of system health and service availability across all CCSQ services. This platform aggregates and presents each service's key performance indicators (KPIs), providing an at-a-glance understanding of operational status. Service owners carefully select these KPIs based on what they consider most critical for assessing whether their services are functioning as expected. By curating and submitting precise, meaningful data, service owners can leverage QNOD to its fullest potential, gaining valuable insights into performance trends, identifying areas for improvement, and making informed decisions to enhance service quality and reliability, thereby feeling more informed and knowledgeable.
Users can access the QNOD Dashboard by using the following link:
- QNOD: https://qnod.cms.gov
Getting Started
Requesting a Role for QNOD
Step 1: Register for a HARP Account
- If you do not have a HARP account, you need to register for a HARP ID.
- For detailed instructions on the HARP registration process, refer to the HARP page.
Step 2: Request a Service Entitlement
- Log into your HARP account.
- Navigate to User Roles and select Request a Role in HARP.
- Choose Program > QualityNet Operations Dashboard.
- Select your organization from the list.
- Note: If your organization is not listed and wants early access to the dashboard, please contact our QNOD team via Slack #help-qnod.
- Select the User Role > Viewer.
Step 3: Approval Process
- Your organization's Security Official will review and approve or deny your User Role Request.
- You will receive an email notifications regarding the submission and the approval or denial of your request.
Step 4: Access the Dashboard
- Once your role is approved, log into the QualityNet Operations Dashboard using your HARP credentials.
What is Machine Learning and How Can it Help You?
Machine learning (ML) is a branch of artificial intelligence that uses algorithms to identify patterns and insights from data. This technology is of particular benefit when the datasets are large and complex. At QualityNet Operations Dashboard (QNOD), our machine learning models are trained on historical data from the Center for Clinical Standards and Quality (CCSQ) services to understand how these services operate. Providing QNOD with high-quality data is essential for achieving accurate and reliable machine learning results, which can be used to enhance service performance and detect issues early.
QNOD currently utilizes two types of machine learning models:
- Anomaly Detection
Anomaly detection models are designed to identify unusual patterns in service data. An anomaly does not necessarily indicate a problem with a service but suggests that the model has detected values far outside the typical range. This range is established by training the models on historical Key Performance Indicators (KPIs) data, which each service provides to the QNOD team. By identifying anomalies, we can proactively monitor systems for potential issues.
- Classification Models
Our classification models analyze historical service data to categorize and identify different states or conditions of the systems. These models help us monitor the systems and determine if any issues need attention. By classifying data based on historical patterns, we can better understand the current state of our services and take appropriate action when necessary. These models differ from the Anomaly models in that known service degraded or outage states for a given service may be used to train the model such that future degraded or outage states may be accurately detected.
QNOD’s machine learning engineers are also exploring advanced techniques, such as Generative Artificial Intelligence (AI) and Large Language Models (LLMs), to enhance our capabilities further. Additionally, we are investigating predictive model forecasting to improve our ability to anticipate future trends and performance.
Are you interested in implementing machine learning in your services? Contact the QNOD team via Slack #help-qnod or the QNOD Service Desk.
QNOD takes in values from multiple monitoring tools and applications and brings them into one unified view to make it easy for people to understand how a service performs. This process means that we can add new monitoring tools to QNOD in the future without losing the historical view.
Submitting Requests with the QNOD Service Desk
The QNOD Service Desk is your go-to resource for submitting requests for changes or enhancements to your service. Simply fill out a simple form with the details of your request and click "Create," and your submission process is complete!
Hosted on the Help Center, the QNOD Service Desk allows you to track all your requests for QNOD and other services without making a call, drafting an email, or sending a Slack message.
How to Fill Out the Form:
Step 1: Access the QNOD Service Desk
Click "QNOD Service Request" to complete the form.
Step 2: Fill Out the Form
- Enter Request Details:
- Fill in the "Summary" and "Description" fields with the information regarding your request.
- Select Your Service:
- Choose your Service Name from the "Service" dropdown menu.
- Optional Field:
- The "Expected Completion Date" field is optional.
- Submit Your Request:
- Click "Create" to submit your request.
And that's it! After that you can track your request through the Help Center.
Frequently Asked Questions (FAQs)
Service owners should inform the QNOD team when they make changes to their infrastructure, services, or the source of metrics for the dashboard (e.g., CloudWatch, HealthCheck APIs, HTTP, NewRelic, Splunk, etc.). If data feeds change and the feed details have not been updated in QNOD, services may display as down. This situation can occur with:
- Host Name changes
- AWS tag and environment updates
- New monitoring tools added
- Any other changes to your data feeds
Refer to the QNOD Service Desk page to submit a request for any changes to the service(s).
Step 1: Log into HARP, and you will land on your user dashboard. From there, select "User Roles."
Step 2: Select "Request a Role in HARP".
Step 3: Select a Program "QualityNet Operations Dashboard" and click "Next."
Step 4: Select an Organization (your contract name, CMS federal employee, etc.) and click "Next."
Step 5: Select the Role (Viewer or Security Official) that applies to the user and click "Submit."
Step 6: Enter the reason for your request, including relevant information such as your job title, company name, etc., and click "Submit."
Your new role request will be sent to your Security Officer, who will approve access for your Organization.
Step 7: The Security Official responsible for approving access for your Organization will receive a notification of your request. You will be notified via email when your role request has been approved or rejected.
For instructions on the HARP registration process, refer to the HARP page.
Using your HARP credentials, you can access the QualityNet Operations Dashboard by following this process:
- https://idm.cms.gov/
- Select "CCSQ QNOD2.0", and this will populate the QualityNet Operations Dashboard.
Contact the QNOD team at #help-qnod slack channel for assistance.
QualityNet Operations Dashboard Enhancements for v2.37.0
Planned Actions: On Wednesday, January 22, 2025, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.37.0:
Updates for Version 2.37.0:
- Enhance the accuracy of capturing the source metrics for wide area network (WAN) Connectivity service network devices.
- The Syslog service will now be monitored under the Splunk service, and its configuration will be updated to determine its status accurately.
- The configuration for the Trend Micro Service will be updated to reflect its status accurately, and the service, including subsystem names, will be standardized.
- The configurations for ClamAV, Office 365, QTSO, and Skyhigh Secure Web Gateway (SWG) services will be updated to ensure precise status monitoring.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- Security vulnerabilities identified in SYNK have been addressed for code and dependencies.
- The QTSO Service Anomaly model has been retrained to improve accuracy in monitoring and determining anomalies.
- The New Relic Service Anomaly model has been retrained to reflect changes in metrics for service monitoring.
- The Tenable Relational Database Service (RDS) Scanner anomaly model has been retrained to enhance accuracy.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
For any questions, please contact the QNOD team on the #help-qnod Slack channel.
QualityNet Operations Dashboard Enhancements for v2.36.0
Planned Actions: On Tuesday, January 7th, 2025, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.36.0:
Updates:
User Interface (UI) Updates for the QNOD Public Dashboard:
- A new banner will be added to the Status Summary page specifically for public users.
- The content of the Status Summary page, including the main banner and the Service Status section, will be updated.
- Authenticated content will be hidden from the public view. This includes the Key Performance Indicator (KPI) section, research participation information, and the Service Desk.
- The Line of Business services section will be repositioned to the top of the Summary page.
- A link to CCSQ Support Central will be incorporated into the Status Summary page.
These changes aim to enhance the navigation and the overall user experience for both public and authenticated users.
Technical Updates:
- The application backend code will be updated to filter data based on whether users are public or authenticated.
- The application front-end code will be modified to properly identify and display different views for public and authenticated users.
- The infrastructure (EKS Cluster size) will be updated to accommodate the anticipated increase in visitor traffic based on stress test results.
- Accessibility issues (508 compliance) for the summary and details pages of the site will be addressed.
- Connection rate limiting for the application will be implemented at the F5 level.
- Monitoring code has been updated to enable enhanced Matomo analytics, allowing for better tracking of user engagement and website performance.
- Configuration updates for services including AD, Splunk, Ansible, and Nexus will address missing EC2 status metric data and enhance the efficiency of New Relic queries.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
For any questions, please contact the QNOD team on the #help-qnod Slack channel.
QualityNet Operations Dashboard Enhancements for v2.35.0
Planned Actions: On Tuesday, December 20, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.35.0:
- Updates:
- Update Service metric configuration by updating New Relic queries to reflect the metric streams.
- Update evaluation code to sync with Artificial Intelligence/Machine Learning (AI/ML) anomaly table attribute name changes.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- Resolve the security vulnerabilities to address Snyk findings in code, source dependencies, and container images.
- Retrain the Unified File Management (UFM) Node model to manage sparse data using a new feature in the AI/ML data preprocessing pipeline.
- To improve the model's accuracy, the Domain Name System (DNS) model for Qualnet will be retrained using a classification approach.
- Update the anomaly model for Service QMARS Fax, focusing on the accuracy of the EC2 status metric in the Job Tracking Reporter and Job Tracking Database.
- Remove the references to the DARRT and FIVS models from the sub-services and transition to using stand-alone services.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
For any questions, please contact the QNOD team on the #help-qnod Slack channel.
QualityNet Operations Dashboard Enhancements for v2.34.0
Planned Actions: On Tuesday, December 10, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.34.0:
Updates:
• Update New Relic accounts and address queries for missing metrics to revise the service configuration and accurately determine the service status.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
For any questions, please contact the QNOD team on the #help-qnod Slack channel.
QualityNet Operations Dashboard Enhancements for v2.33.0
Planned Actions: On Tuesday, November 26, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.33.0:
Updates:
- Update the Service configuration to determine Service Status and Uptime accurately.
- Upgrade EKS cluster version from 1.28 to 1.30 for the application components.
Artificial Intelligence/Machine Learning (AIML) enhancements:
- The FAS service model was retrained on Monday, November 12th, using outage data from November 11th to November 12th. This retraining allowed for a more accurate classification model.
- The iQIES Service Anomaly Detection model was retrained on Friday, November 15th, due to the dataset lacking certain key performance indicators (KPIs) essential for the model.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard Enhancements for v2.32.0
Planned Actions: On Wednesday, November 13, 2024, at 4 p.m. ET, the QualityNet Operations Dashboard (QNOD) team will release updates for QNOD. The following changes will be implemented in Version 2.32.0:
Updates:
- Change the favorite icon (favicon) to the new QNOD logo in the User Interface (UI).
- Add a new source input for the Internet Control Message Protocol (ICMP) to accurately evaluate network status and connectivity.
- Update the Wide Area Network (WAN) Connectivity Service configuration for the Centers for Medicare & Medicaid Services (CMS) and Baltimore Data Center (BDC) configuration to evaluate device availability.
- Modify the EQRS Portal Service configuration to include synthetic metrics to accurately track service impact and status.
Impacted Application Development Organizations (ADOs): All QNOD users
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.31.0
Planned Actions: On Tuesday, October 29, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.31.0:
Updates:
- Implement User Interface (UI) changes, including Tooltip styling, to improve readability and the verbiage of outage status tiles.
- Enhance the ingestion and evaluation processes for service metrics to improve the accuracy of Uptime History.
- Integrate the Anomaly model missing data input to determine the service status correctly.
- Update the iQIES, QPP, and QTSO service configuration to reflect the infrastructure changes.
- Update the Skyhigh Secure Web Gateway (SWG) Service configuration to track additional web gateway devices and accurately determine their status.
Impacted Application Development Organizations (ADOs): All QNOD users
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.30.0
Planned Actions: On Wednesday, October 16, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following updates and enhancements will be implemented by the QNOD team for Version 2.30.0:
Updates:
• Enhance the ingestion of Service metrics to reduce missing data and improve the accuracy of Uptime history.
• Updates will be made to the following areas:
o UFM Service configuration will reflect the infrastructure changes.
o QPP Service configuration to reflect changes from Amazon Web Service version 3 to version 4 architecture.
o Confluence and Jira Service configuration to correct false statuses of issues.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
• Update the backend process to stabilize the model artifact across all existing models.
• Enhance the Anomaly service to improve the process of ingesting service data.
• Optimize the AI/ML anomaly detection pipeline to trigger automatically based on a predefined schedule.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.29.0
Planned Actions: On Tuesday, October 1, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following updates and enhancements will be implemented by the QNOD team for Version 2.29.0:
Updates:
- Implement the following changes to the User Interface (UI):
- Add the current day in the Uptime History and remove thresholds to track all
non-operational issues, including missing metric data. - Update the New Relic service to use the Status page and retire the synthetic canary for accuracy.
- Update service configurations (Ansible, Nexus, Splunk, and TrendMicro) to align with the metrics streaming into the New Relic account.
- Improve the process of integrating subsystem references using the metadata database.
- Enable Matamo to collect user analytics.
- Removing the tracking of the retired Barracuda application.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- Update the backend process to stabilize the model artifact across all existing models.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.28.0
Planned Actions: On Tuesday, September 17, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following updates and enhancements will be implemented by the QNOD team for Version 2.28.0:
Updates:
- Display a 'Powered by AI' indicator for services, including subsystems, which leverage Artificial Intelligence/Machine Learning (AI/ML) driven anomaly models to determine the Service Status.
- Enhance the User Interface (UI) for Uptime History to provide improved information and tracking of service disruptions and non-operational issues.
- Integrating Databricks into the dashboard via the Service Status page.
- Updating the UI code to resolve issues with error page flow on the browser refresh.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- Improving the anomaly detection pipeline to report missing metric data.
- Enabling notifications for missing metric data in anomaly models.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.27.0
Planned Actions: On Wednesday, September 4, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The QNOD team will implement the following updates and enhancements for Version 2.27.0:
Updates:
- Integration of anomaly detection with services and subsystems to improve the accuracy of the Service Status.
- The service evaluation query logic will be updated to accurately determine the latest Service Status.
- Updating the service configuration to use synthetics with ServiceNow for determining the Service Status and uptime.
- Updating the service configuration for GitHub and SurveyMonkey to align degraded status with their respective status pages.
- Transitioning McAfee WG service to Skyhigh SWG and implementing synthetics to determine Service Status and uptime.
- Enhancing user login behavior to resolve an issue where login sessions would hang and fail to redirect to the Internet Download Manager (IDM) login page.
- The redundant AWS Canary for ServiceNow will be deactivated.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- Enabling integration of the Anomaly and Classification model with subsystems to improve the enhanced service degradation accuracy for the following:
- AD, DNS, EQRS SF, F5, QDIVS, Tenable, and WAN Connectivity
- Enabling automation anomaly services for Slack notifications in Production.
Impacted Application Development Organizations (ADOs): All QNOD users
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) update will be deployed from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.26.0:
Planned Actions: On Tuesday, August 20, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following updates and enhancements will be implemented by the QNOD team for Version 2.26.0:
Updates:
- Implement User Interface (UI) changes for the QNOD page, including service card hover and scroll enhancements to improve readability.
- Link and display the status page for external services like Slack and SurveyMonkey to determine their Service Status.
- Update Service (ClamAV, Confluence, DELWeb, Jira, MADiE, Tenable) configuration files for metric monitoring changes.
- Relocate CCSQ QuickSight as QuickSight under AWS Services.
- Relocate FAS to Data & Analytics.
- Onboard Lucid to the dashboard using the Service Status page.
- Implement the use of a refresh token for access token generation.
- Update evaluation code to integrate the subsystem's AI/ML anomaly models to improve service status accuracy.
Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- The pipeline now includes the capability of modeling service subsystems within QNOD. Additionally, the newly implemented classification modeling capability enables training models from known system historical degraded or outage data, improving the accuracy of event detection.
- The following service models will be deployed with subsystem models noted per service:
- Anomaly or Classification model deployment for the services to improve service degradation accuracy:
- Services including all subsystems: AD, DNS, EQRS SF, F5, QDIVS, Tenable, and WAN Connectivity.
- Services without subsystems: PRS, QTSO, VPN ASA
- The service models listed below were retrained using classification models to improve accuracy:
- Confluence, DELWeb, Jira, and QCOR
- The service model below was retrained using anomaly models to improve accuracy:
- QNP
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard and AI/ML Enhancements for v2.25.0:
Planned Actions: On Tuesday, August 6, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following updates and enhancements will be implemented for Version 2.25.0:
- Updates:
- Remove the "Help" link from the QNOD footer.
- The "Contact Help Desk" Footer has been replaced with the "QNOD Service Desk."
- Security vulnerabilities addressed for the QNOD components.
- Confluence and Jira Services configuration files updated to accurately determine the service degradation status.
- Artificial Intelligence/Machine Learning (AI/ML) enhancements:
- Update AI/ML pipeline to handle Classification model that can work with ground truth data.
- Set up a notification service code to alert AI/ML engineers of anomalies detected by Anomaly Detection (AD) models.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.24.0 and AI/ML Enhancements for Version 0.4.0.0:
Planned Actions: On Tuesday, July 23, 2024, at 4 p.m. ET, we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following will be implemented:
- Updates for Version 2.24.0:
- Implement User Interface (UI) changes for QNOD content pages for better readability and spacing.
- Update Service status definitions for the dashboard.
- Integration of QNOD JSON Web Token (JWT) with Amazon Web Service (AWS) Secrets Manager.
- Update the Service configuration files to incorporate the New Relic and Splunk metric monitoring changes.
- Update Service configuration files to accurately determine Service Uptime and Status.
- Artificial Intelligence/Machine Learning (AI/ML) enhancements for Version 0.4.0.0:
- Ansible, EQRS SF, F5, FireEye ETP, FireEye vNX, iQIES, McAfee WG, MedTrak, Next Gen Firewall, QCOR, and QSEP.
- Anomaly model deployment for the services below will work with the Service health calculation and service uptime/status to improve service degradation accuracy.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.23.1.0 and AI/ML Enhancements for Version 0.3.0.0:
Planned Actions: On Friday, July 12, 2024, at 4 p.m. Eastern Time (ET), we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following will be implemented:
- Updates for Version 2.23.1.0:
- Enable the data pipeline to support the deployment of additional anomaly models.
- Update Service configuration files to enable Anomaly model input.
- Artificial Intelligence/Machine Learning (AI/ML) enhancements for Version 0.3.0.0:
- API Gateway, CCSQ QuickSight, Certificate Authority, DELWeb, EQRS Portal, iQIES, Mailman, New Relic, PRS, QNP, SAS Viya, ServiceNow, Syslog, and Zscaler
- Anomaly models will be integrated into the backend of the QNOD dashboard for selected services to improve service degradation accuracy.
- Anomaly model deployment for the services below will work with the Service health calculation, service uptime and status to improve service degradation accuracy.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.23.0 and AI/ML Enhancements for Version 0.2.0.0:
Planned Actions: On Tuesday, July 9, 2024, at 4 p.m. Eastern Time (ET), we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following will be implemented:
- Updates for Version 2.23.0:
- Addressing vulnerabilities identified by the Netsparker Scan for QNOD header issues.
- Onboard MADiE service to QNOD, replacing MAT and Bonnie services.
- Update Service configuration files to accurately determine Service Uptime and Status.
- Artificial Intelligence/Machine Learning (AI/ML) enhancements for Version 0.2.0.0:
- Logic to handle NaN data from services was integrated into the pipeline. The first five services (Barracuda, Confluence, Jenkins, Jira, and FAS) models were retrained and deployed with this logic.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.22.0 and AI/ML Enhancements for Version 0.1.0.0:
Planned Actions: On Tuesday, June 25, 2024, at 4 p.m. Eastern Time (ET), we will release updates and enhancements for the QualityNet Operations Dashboard (QNOD). The following updates and enhancements will be implemented:
- Updates for Version 2.22.0:
- Address vulnerabilities identified by the Netsparker Scan for QNOD.
- Add the logic for determining the evaluation code.
- Integrate the Anomaly model output into the evaluation code.
- Update the service configuration to align with the service status determination logic.
- Remove outdated New Relic token references from the environment configuration files.
- Remove tutorial video links from the QNOD homepage footer.
- Artificial Intelligence/Machine Learning (AI/ML) enhancements for Version 0.1.0.0:
- Deploy models for Barracuda and FAS to work with Service health calculations.
- Deploy models for Confluence, Jenkins, and Jira to monitor unusual metrics and service uptime.
- Anomaly models will be integrated into the backend of the QNOD dashboard for selected services to improve service degradation accuracy.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.21.0
Planned Actions: On Tuesday, June 11, 2024, at 4 p.m. Eastern Time (ET), we will release QualityNet Operations Dashboard (QNOD), Version 2.21.0. This version will include the following enhancements:
- Release enhancements to User Service Configuration Key Performance Indicator (KPI) updates and Interface (UI).
- Service Configuration Updates:
- Jenkins Service KPI update to accurately reflect the availability in QNOD.
- FIVS Summary Service KPI queries are updated under QDIVS Service to reflect infrastructure changes on the QNOD dashboard.
- User Interface Updates:
- A new dropdown filter capability to search based on service status and clear filter selections.
- Enhancement to the Search Bar for greater visibility and easier access to specific Services by name.
- Converting the existing Legend into a non-filtering functionality, guiding users towards the new filtering feature.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.20.0
Planned Actions: On Wednesday, May 29, 2024, at 4 p.m. Eastern Time (ET), we will release QualityNet Operations Dashboard (QNOD), Version 2.20.0. This version will include the following enhancements:
- Release enhancements Service configuration model updates.
- Service Configuration Updates:
- Update the Service Metadata model to include an Active and Inactive flag. This change will allow QNOD to retire a Service and maintain a record for tracking purposes.
- Update the ingester and evaluator processes to obtain the active Service List using the Service Metadata model.
- Update the ingester configuration service files to use a new, read-only New Relic Application Programming Interface (API) key for QNOD.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
For any questions, please feel free to contact the QNOD team on the #help-qnod Slack channel.
QualityNet Operations Dashboard v2.19.0
Planned Actions: On Tuesday, May 14, 2024, at 4 p.m. ET, we will release QualityNet Operations Dashboard (QNOD), Version 2.19.0. This version will include the following enhancements:
- User Interface (UI) Changes:
- Add a message on the Search and Filtering results for the Summary page.
- Add Uptime History with an Uptime Chart on the Service Detail and Subsystem Detail pages.
- Add changes to sync the Service Status with the Key Performance Indicator (KPI) state.
- Add a Status banner to be reflected on Service Detail Page and Subsystem Detail Page for each QNOD status.
- Service Configuration Changes:
- Update the Service Metadata Model to include the Category Field and Display Name.
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnod.cms.gov/) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.18.0
Planned Actions: On Tuesday, April 30, 2024, at 4 p.m. ET, we will release QualityNet Operations Dashboard (QNOD), Version 2.18.0. This version will include the following enhancements:
- User Interface (UI) Changes
- Update existing UI code to enhance responsiveness for Header Enhancements and Logic Behavior.
- Service Configuration Changes
- Upload updates to the existing ingester, evaluation, and metadata configuration files for the services in QNOD (MFT, Next Gen Firewall, Office365, QMARS Fax, QPP)
Impacted Application Development Organizations (ADOs): All QNOD users.
What To Expect: The QNOD Dashboard (https://qnetdashboard.cms.gov) deployment will be from 4:00 p.m. until 6:00 p.m. ET.
QualityNet Operations Dashboard v2.17.0
Planned Actions: On Tuesday, April 16, 2024, at 4 p.m. ET, we will release QualityNet Operations Dashboard (QNOD), Version 2.17.0. This version will include the following enhancements:
• Improved accessibility for Service Status Definitions
o For Service and Services Subsystems Summary Page Headers tiles.
• Service Subsystems Summary Page Header Tiles
o Changes were implemented without adding the filter by Status function.
• Summary Page Header Tiles
o Added functionality for filtering by Status.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: Tuesday, April 16, 2024, between 4:00 p.m. and 6:00 p.m. ET.
What To Expect: The QNOD Dashboard (https://qnetdashboard.cms.gov) will be unavailable from 4:00 p.m. and 6:00 p.m. ET.
QualityNet Operations Dashboard v2.16.0
Planned Actions: On Tuesday, April 2, 2024, at 4 p.m. ET, we will release QualityNet Operations Dashboard (QNOD), Version 2.16.0. This version will include the following enhancements:
- Grouping of Services by Categories
- Legends Filter
- Redesign of the KPI Detail Page
- Services Search Functionality
- User Interface (UI) Home and Summary Page Redesign
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: Tuesday, April 2, 2024, between 4:00 p.m. and 6:00 p.m. ET.
What To Expect: The QNOD Dashboard (https://qnetdashboard.cms.gov) will be unavailable from 4:00 p.m. and 6:00 p.m. ET.
QualityNet Operations Dashboard v2.15.2
Planned Actions: QNOD 2.0 Slack integration will be updated to extend the implicit timeout for post-login browser loading. This will raise the duration threshold to 150 seconds and adjust the canary execution frequency.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: Tuesday, March 26, 2024, between 4:00 PM and 6:00 PM ET.
What To Expect: The QNOD Dashboard (https://qnetdashboard.cms.gov) will be unavailable from 4:00 PM and 6:00 PM ET.
QualityNet Operations Dashboard v2.15.1
Planned Actions: QNOD 2.0 updates to address Appliance Availability KPI issues, adjust FireEye vNX's window_size parameter from 60 to 90 seconds in QNOD 2.0. This aligns the window_size with New Relic's update frequency of every 60 seconds, avoiding conflicts with data being written to New Relic and resolving the problem.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: Friday, March 15, 2024, between 4:00 PM and 6:00 PM ET.
What To Expect: The QNOD Dashboard (https://qnetdashboard.cms.gov) will be unavailable from 4:00 PM and 6:00 PM ET.
QualityNet Operations Dashboard v2.15.0
Planned Actions: On Tuesday, March 5, 2024, at 4 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.15.0.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Tuesday, March 5, 2024, from 4 p.m. until 6 p.m. ET.
What To Expect: The following Onboarding services will be added to QNOD:
- QNOD QPP Service Onboarding
- QNOD Logout User Interface (UI) Changes
- QNOD Logging Formatting Changes
- Decommission FileCloud from QNOD
The changes also include the Home and Drill down pages.
Known Issues:
- Kentik Simple Network Management Protocol (SNMP) polling containers were moved to a new host. The change is impacting Key Performance Indicator (KPI) ingestion for the below services:
- Next Gen Firewall
- VPN ASA
- WAN Connectivity
QualityNet Operations Dashboard v2.14.1
Planned Actions: Upgrade the QualityNet Operations Dashboard (QNOD) Production Amazon Elastic Kubernetes Service (EKS) clusters to version 1.25.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: Tuesday, February 27, 2024, between 4:00 PM and 6:00 PM ET.
What To Expect: The QNOD Dashboard (https://qnetdashboard.cms.gov) will be unavailable from 4:00 PM and 6:00 PM ET.
QualityNet Operations Dashboard v2.14.0
Planned Actions: On Tuesday, February 20, 2024, at 6 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.14.0.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Tuesday, February 20, 2024, from 6 p.m. until 8 p.m. ET.
What To Expect: Implementation of the following Subsystem changes for QNOD2:
- Deployment of new Services: HQR
- QNOD UI – 508 Fixes
- UI Detail Page Changes
Known Issues:
- NewRelic queries timeout issue could cause a service (e.g., HQR or AD) to go into an unknown state.
- Migrate New Relic Kentik monitors for Networking devices.
QualityNet Operations Dashboard v2.13.0
Planned Actions: On Tuesday, February 6, 2024, at 4 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.13.0.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Tuesday, February 6, 2024, from 4 p.m. until 6 p.m. ET.
What To Expect: Implementation of the following Subsystem changes for QNOD2:
- Deployment of new Services: QMARS Fax, Splunk
- QNOD UI – Breadcrumb changes
- Fixes for Page Load Performance Issue and Missing KPIs
- Decommission of Services: CDR and Hive
Known Issues:
- NewRelic timeout issues in QNOD logs.
QualityNet Operations Dashboard v2.12.1
Planned Actions: On Thursday, January 25, 2024, at 4 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.12.1.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Thursday, January 25, 2024, from 3 p.m. until 5 p.m. ET.
What To Expect: The following Services changes in QNOD2:
- Ansible to remove the unused synthetic monitor.
- New Relic Query updated for MFT, as MFT has been migrated to a new AWS Account.
- Next Gen Firewall to update incorrect KPI.
Known Issues:
- CDR degraded service due to the Ranger subsystem.
- Hive has been degraded.
- Office365 is degraded due to Microsoft Exchange.
QualityNet Operations Dashboard v2.12.0
Planned Actions: On Tuesday, January 23, 2024, at 4 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.12.0.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Tuesday, January 23, 2024, from 4 p.m. until 6 p.m. ET.
What To Expect: The following Onboarding services will be added to QNOD2:
- DNS, EQRS SF, F5, HARP, Nexus, QDIVS, QIES, Tenable, and TrendMicro.
The changes also include the Home and Drill down pages.
Known Issues:
- Airflow, Hive, Office365, and Zeppelin have been decommissioned.
- Degraded CDR service due to the Ranger subsystem KPIs in QNOD1 and QNOD2.
QualityNet Operations Dashboard v2.11.0
Planned Actions: On Tuesday, January 9, 2024, at 4 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.11.0.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Tuesday, January 9, 2024, from 4 p.m. until 6 p.m. ET.
What To Expect: Implementation of the following Subsystem changes for QNOD2:
- A new Drilldown page to display the subsystems of a service.
- Deployment of new Services: AD, CDR, MFT, WAN Connectivity.
Known Issues:
- Degraded CDR service due to the Ranger subsystem KPIs in QNOD1 and QNOD2.
QualityNet Operations Dashboard v2.10.0
Planned Actions: On Tuesday, December 26, 2023, at 4 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD), Version 2.10.0. This release includes newly added services monitoring.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable on Tuesday, December 26, 2023, from 4 p.m. until 6 p.m. ET.
What To Expect: The following Services will be added to the QNOD Home and drill down pages:
- AWS
- Barracuda
- EQRS Portal
- HARP Automation
- QMARS
QualityNet Operations Dashboard v2.9.1
Planned Actions: On Friday, December 22, 2023, at 4 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD), Version 2.9.1.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable from 4 p.m. until 6 p.m. ET.
What To Expect: Users can expect service monitoring changes and fixes in addition to routine security patches.
QualityNet Operations Dashboard v2.9.0
Planned Actions: On Tuesday, December 12, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2, Version 2.9.0).
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: December 12, 2023, between 8:00 p.m. ET and 10:00 p.m. ET.
What To Expect: The following Services will be added to the QNOD2 Home and drill down pages. They will include new KPI sources from CloudWatch and HTTP apart from New Relic.
- CCSQ QuickSight
- FireEye ETP
- Office365
Note: The above new services drill down page will show complete Health Composite chart data after 24 hours.
Enhancements: Footer has been redesigned to align with other ESS services such as HARP.
Known Issues: "Open Issues" tile on the service drilldown page links to an older Jira project.
QualityNet Operations Dashboard v2.8.0
Planned Actions: On Tuesday, November 28, 2023, at 4 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2, Version 2.8.0).
Impacted Application Development Organizations (ADOs): All QualityNet Operations Dashboard users.
Planned Downtime: November 28, 2023, between 4:00 p.m. ET and 5:00 p.m. ET.
What To Expect: The following Services will be added to the QNOD2 Home and drill down pages:
- GitHub
- New Relic
- ServiceNow
- Slack
- SurveyMonkey
- TestRail
Note: The above new services drill down page will show complete Health Composite chart data after 24 hours.
Enhancements: The y-axis scale on the Health Composite graph has been updated to reflect the service's total possible health dynamically. Additionally, minor spacing adjustments have been made to the graph.
Known Issues: "Open Issues" tile on the service drilldown page links to an older Jira project.
QualityNet Operations Dashboard v2.7.0
Planned Actions: On Tuesday, November 14, 2023, at 4 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2, Version 2.7.0).
Impacted Application Development Organizations (ADOs): All QualityNet Operations Dashboard users.
Planned Downtime: November 14, 2023, between 4.00 PM ET and 5.00 PM ET.
What To Expect: A new version is being released that includes adding the following Services:
- File Cloud
- QSEP
- SAS Viya
Note: The above new services drill down page will show complete Health Composite chart data after 24 hours.
What are some of the enhancements included in the upgrade?
- Users logging in through Identity Management (IDM) will be redirected to the last QNOD page they were on prior to session expiration.
- Adjusted how total health is displayed on the Health Composite graph for services where max health does not add up to 100%.
- Fixed a spacing issue on the Health Composite graph.
Known Issues:
- "Open Issues" tile on the service drilldown page links to an older Jira project.
QualityNet Operations Dashboard v2.6.0
Planned Actions: On Tuesday, October 31, 2023, at 4 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2, Version 2.6.0).
Impacted Application Development Organizations (ADOs): All QualityNet Operations Dashboard users.
Planned Downtime: October 31, 2023, between 4:00 p.m. ET and 5:00 p.m. ET.
What To Expect: A new version is being released that includes adding the following Services:
- Airflow
- FireEye vNX
- IQIES
- Mailman
- McAfee WG
- MedTrak
- Next Gen Firewall
- QTSO
- PRS
- QCOR
- QNP
- VPN ASA
- Zeppelin
Note: The above new services drill down page, will show complete Health Composite chart data after 24 hours.
What are some of the enhancements included in the upgrade?
- ClamAV Service – The following new KPIs are added:
- ECS Service Running Task Count
- ECS Service CPU Used Percent
- ECS Service Memory Used Percent
- Service Drill down Page - Health Composite graph has been modified to include a border, background color, and left-hand side y-axis labels is made like right side labels indicating the numbers (0, 50, 100) for better readability.
- QNOD Footer links section – ‘Share Feedback’ older link was removed.
Known Issues: "Open Issues" tile on the service drilldown page links to an older Jira project.
QualityNet Operations Dashboard v2.5.1
Planned Actions: On Wednesday, October 20, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2, Version 2.5.1).
Impacted Application Development Organizations (ADOs): All QualityNet Operations Dashboard users.
Planned Downtime: None
What To Expect: A new version is being released that includes QNOD metric changes to support the new ClamAV infrastructure.
QualityNet Operations Dashboard v2.5.0
Planned Actions: On Wednesday, October 17, 2023, at 4 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2, Version 2.5.0).
Impacted Application Development Organizations (ADOs): All QualityNet Operations Dashboard users.
Planned Downtime: October 17, 2023, between 4:00 p.m. ET and 5:00 p.m. ET.
What To Expect: A new version is being released that includes adding the following Services:
- Bonnie
- Certificate Authority
- DELWeb
- FAS
- Hive
- MAT
Note: The above new services drill down page, will show complete health composite chart data after 24 hours.
What are some of the enhancements included in the upgrade?
- Landing Page - Clicking on the Service status tile will scroll to the correct accordion section.
- QNOD footer link - Slack channel change from #help-qnod-dashboard to #help-qnod.
Known Issues:
- "Open Issues" tile on the service drilldown page links to an older Jira project.
QualityNet Operations Dashboard v2.4.0
Impacted Application Development Organizations (ADOs): All QualityNet Operations Dashboard users
Planned Downtime: September 29, 2023, between 4.00 p.m. ET and 5.00 p.m. ET.
What To Expect: A new version is being released that includes a User Interface (UI) design overhaul for QNOD 2.0. The overhaul introduces changes to how service status and information are displayed for the following Services:
- Ansible
- Barracuda
- ClamAV
- Confluence
- Jenkins
- Jira
- Syslog
- Zscaler
What are some of the enhancements included in the upgrade?
- Services on the landing page are now organized according to Health status instead of percentage.
- Added status accordions for each Status (Outage, Degraded, Operational, Unknown).
- Removed "Abnormal" and "Open Issues" tiles on the landing page.
- Removed the percentage histogram and the "Services Operating at 100%" accordion on the landing page.
- Removed the "Anomalies" tile on the service drilldown page.
- Removed the “Anomalies - past 24 hours" graph on the service drilldown page.
Note: Some of the removed features above may return in future versions.
Existing Bug Fixes:
- Error 500 issue when accessing QNOD2 has been resolved.
- Barracuda – Network missing data issue and health correlation have been corrected to show valid data.
- Confluence – Removed KPIs that were not used in the Service Health calculation.
Known Issues:
- "Open Issues" tile on the service drilldown page links to an older Jira project.
QualityNet Operations Dashboard v2.3.0
Affected customers: All QualityNet users.
What is happening?
On Wednesday, August 30, 2023 at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD2).
What is this release happening?
New versions of QNOD2 are released each sprint to ensure that QNOD2 users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
- Remediated PEN testing findings
- Updates to service configurations to align with baseline reconciliation findings:
- Confluence
- Jenkins
- Zscaler
QualityNet Operations Dashboard v1.7.1
Planned Actions: On Wednesday, December 20, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD), Version 1.7.1. This release includes security patching only.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable from 8 p.m. until 10 p.m. ET.
What To Expect: Users will not see new functionality with this release as it includes security patching only.
Known Issues
- Syslog – We are investigating a false positive from Syslog Synthetic Availability KPI.
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs.
- QNOD – The time picker for service drill-down dashboards has been disabled to ensure system stability.
QualityNet Operations Dashboard v1.7.0
Planned Actions: On Tuesday, December 12, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD), Version 1.7.0. This release includes security patching and service monitoring adjustments.
Impacted Application Development Organizations (ADOs): All QNOD users.
Planned Downtime: This release will cause QNOD to be unavailable from 8 p.m. until 10 p.m. ET.
What To Expect: Users can expect the following service monitoring changes and fixes in addition to routine security patches.
Service Monitoring Changes
- Barracuda – Updated Volume Queue Length and Avg Health Host Count KPIs. Removed Request Count KPI.
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs.
- QNOD – The time picker for service drill-down dashboards has been disabled to ensure system stability.
- QNOD – Work continues to permanently address recent data ingestion instability. Slack alerts have been disabled in the interim.
QualityNet Operations Dashboard v1.6.0
Planned Actions:
On Thursday, October 26, 2023, at 8 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD) v1.6.0. This release includes security patching and service monitoring adjustments.
Impacted Application Development Organizations (ADOs):
All QNOD users.
Planned Downtime:
This release will cause QNOD to be unavailable from 8 p.m. until 10 p.m. ET.
What To Expect:
Users can expect the following new functionality, service monitoring changes, and fixes in addition to routine security patches.
Service Monitoring Changes
- Confluence – Remove unused Network Request Count Key Performance Indicator (KPI).
- FAS – Rename FAS service to AIMM to reflect recent contract changes.
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs.
- QNOD – The time picker for service drill-down dashboards has been disabled to ensure system stability.
- QNOD – Work continues to permanently address recent data ingestion instability. Slack alerts have been disabled in the interim.
QualityNet Operations Dashboard v1.5.0
Planned Actions:
On Wednesday, October 11, 2023, at 8 p.m. ET, we will release a new version of the QualityNet Operations Dashboard (QNOD) v1.5.0. This release includes security patching and service monitoring adjustments.
Impacted Application Development Organizations (ADOs):All QNOD users.
Planned Downtime:
This release will cause QNOD to be unavailable from 8 p.m. until 10 p.m. ET.
What To Expect:
Users can expect new functionality, service monitoring changes, fixes, and routine security patches.
Service Monitoring Changes
- Jira – Adjust the Network Request KPI to use the new Application Load Balancer.
- MAT – Remove KPIs that are missing data and not indicative of service health.
- All KPIs, except for Synthetic Availability and Synthetic Latency, have been removed.
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs.
- QNOD – The time picker for service drill-down dashboards has been disabled to ensure system stability.
- QNOD – Work continues to address recent data ingestion instability. Slack alerts have been disabled in the interim.
These and past Release Notes can be found on the QNOD Confluence page by clicking the Release Notes 1.0 tab.
QualityNet Operations Dashboard v1.4.0
Affected customers: All QualityNet users
What is happening?
On Wednesday, September 13, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Service Monitoring Changes
- F5 – Remove Client Connections KPIs that are not indicative of service health.
Known Issues:
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – The time picker for service drill-down dashboards has been disabled to ensure dashboard stability.
- QNOD – Work continues to permanently address recent data ingestion instability. Slack alerts have been disabled in the interim.
QualityNet Operations Dashboard v1.3.0
Affected Customers: All QualityNet users.
Why is this release happening?
On Wednesday, August 30, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
What are some of the enhancements included in the upgrade?
Users will not see new functionality with this release as it includes security patching only.
Known Issues
CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs.
QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.2.0
Affected customers: All QualityNet users.
On Wednesday, August 16, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Service Monitoring Changes
- F5
- Add devices that handle non-production traffic.
Next Gen Firewall
Add devices that handle non-production traffic.
Routing
Remove the decommissioned Routing service from QNOD.
Resolved Issues
- HIDS/HARP Automation – Resolved insufficient data for Git(Hub) Status KPI.
Known Issues
CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs.
QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.1.0
Affected customers: All QualityNet users.
On Wednesday, August 2, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Please note that QNOD 1 is adopting semantic version numbers instead of the previous system that was based on the project increment. The previous release was 1.23.2.6. This will be the 1.1.0 release.
Service Monitoring Changes
- DELWeb
- Temporarily monitor only Synthetic Availability and Latency until new DEL infrastructure metrics are available in New Relic
- FireEye vNX
- Adjust the Appliance Availability KPI to better represent transient monitoring gaps
- QMARS Fax
- Update Synthetic Availability and Latency to use new synthetic monitor in New Relic
- Next Gen Firewall
- Remove decommissioned legacy gateways and add new gateways to all KPIs
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.2.6
Affected customers: All QualityNet users.
On Wednesday, July 6, 2023, at 6 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 8 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Service Adjustments and Baseline Reconciliation Improvements:
- McAfee Web Gateway
- Replaced hardcoded device list in queries with New Relic tag.
- Added Memory Used Percent KPI.
- Removed EC2 Status Check, Application Availability, Network Status, and Process Load KPIs.
- MFT
- Updated KPIs to reflect migration to AWS Fargate.
- QMARS
- Removed KPIs for decommissioned “Next Gen” subsystem.
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.2.5
Affected customers: All QualityNet users.
On Wednesday, June 15, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users will not see new functionality with this release as it includes security patching only.
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.2.4
Affected customers: All QualityNet users.
On Wednesday, June 7, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Machine Learning Enablement
- FAS – Anomaly Detection (Released June 2)
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- HQR – Anomaly Detection (Released June 2)
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
Service Adjustments and Baseline Reconciliation Improvements
- Ansible
- Updated KPI and component weights to more accurately represent service health.
- FireEye vNX
- Updated KPI and component weights to more accurately represent service health.
- GitHub
- Replaced monitoring of GitHub Enterprise Server with GitHub Enterprise Cloud via GitHub Status API.
- Next Gen Firewall
- Updated KPI weights to more accurately represent service health.
- TrendMicro
- Updated KPI and component weights to more accurately represent service health.
- Remove KPIs:
- DSM RDS Read Latency
- DSM RDS Write Latency
- DSM RDS Connections Count
- DSM RDS Freeable Memory
- DSM Request Count
- DSM Avg Healthy Host Count
- Relay RDS CPU Used Percent
- Relay RDS Free Storage Space
- Relay RDS Freeable Memory
- Relay Request Count
- Relay Avg Healthy Host Count
- Relay RDS Read Latency
- Relay RDS Write Latency
- Relay RDS Connections Count
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.2.3
Affected customers: All QualityNet users.
On Wednesday, May 24, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Machine Learning Enablement
- FAS – Uptime Prediction
- Synthetic Latency
- Disk Free Percent
- Memory Used Percent.
- This AI/ML capability with deep learning models provides the service owner with a look-ahead of 20 minutes into the future for any potential issues at the KPI level, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Predictions are available for the following KPIs:
Service Adjustments and Baseline Reconciliation Improvements:
- F5
- SWAP Memory Utilization
- Other Memory Utilization
- Synthetic Availability
- Synthetic Latency
- Total Client Bytes Received
- Total Client Bytes Sent
- Inbound Error (%)
- Outbound Error (%)
- Updated thresholds and weights to more accurately represent service health.
- Added KPIs
- Remove KPIs:
- GitHub
- Synthetic First Contentful Paint
- Synthetic First Paint
- Synthetic First On Page Load
- Removed KPIs:
Resolved Issues
- FileCloud -- Process Count is no longer reporting insufficient data.
- Machine Learning Models – Ansible anomaly detection (AD) model has been retrained and alerts enabled.
Known Issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – The time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.2.2
Affected customers: All QualityNet users.
On Wednesday, May 10, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Service Adjustments and Baseline Reconciliation Improvements
- FileCloud
- Removed non-impacting KPIs and updated weights to more accurately represent service health.
- FireEye vNX
- Removed decommissioned appliances.
- EQRS Scoring and Feedback
- Updated KPI weights to more accurately represent service health.
- MFT
- Removed non-impacting KPIs and updated weights to more accurately represent service health.
- PRS
- Updated KPI weights to more accurately represent service health.
- QNP
- Updated KPI weights to more accurately represent service health.
- Zscaler
- Updated KPIs to capture new Zscaler infrastructure after PACE migration.
Resolved Issues
- EQRS SF –User Experience KPIs (“canaries”) have been restored for NGMC and service health alerts have been enabled for EQRS SF.
Known Issues
- FileCloud -- Process Count is reporting insufficient data. The service owner is working with APM to restore this metric in New Relic.
- Machine Learning Models – Ansible anomaly detection (AD) alerts have been disabled while the model is retrained.
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- QNOD – Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.2.1
Affected customers: All QualityNet users.
On Wednesday, April 26, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Service Adjustments and Baseline Reconciliation Improvements:
- Jenkins
- Removed non-impacting KPIs and updated component and KPI weights to more accurately represent service health.
- WAN Connectivity
- Updated New Relic queries to reflect infrastructure changes.
Resolved issues:
- Machine Learning Models – Shifting trends in service metric data may adversely affect the performance of both Anomaly Detection (AD) and Uptime Prediction (UP) models. Retraining is complete for the listed models and AD alerts have been re-enabled where applicable.
- Ansible UP
- HARP AD
- Jira AD & UP
- Syslog UP
Known issues:
- Machine Learning Models – Ansible anomaly detection (AD) alerts have been disabled while the model is retrained.
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- QNOD – Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.1.6
Affected customers: All QualityNet users.
On Wednesday, April 12, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features
- Machine Learning Enablement – Uptime Prediction
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 10 minutes into the future for any potential issues at the KPI level, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Predictions are available for the following KPIs: RDS Freeable Disk, RDS Freeable Memory.
- QTSO
Service Adjustments and Baseline Reconciliation Improvements
- FAS
- Removed non-impacting KPIs and updated component and KPI weights to more accurately represent service health.
- New Relic
- Updated KPI and component weights to more accurately represent service health
- McAfee Web Gateway
- Added additional Synthetic Monitor and removed non-impacting KPIs to more accurately represent service health.
- EQRS Portal
- Added additional KPIs and removed non-impacting KPIs to more accurately represent service health.
- QTSO
- Removed non-impacting KPIs and updated component and KPI weights to more accurately represent service health.
- iQIES & QIES
- Moved monitoring of MDS application infrastructure from QIES to iQIES in preparation for migration cutover on April 17.
Resolved issues
- Machine Learning Models – Shifting trends in service metric data may adversely affect the performance of both Anomaly Detection (AD) and Uptime Prediction (UP) models. Retraining is complete for the listed models and AD alerts have been re-enabled where applicable.
- ClamAV AD
- Confluence AD
- QTSO AD
- ServiceNow AD
Known issues
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- QNOD – Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.1.5
Affected customers: All QualityNet users.
On Wednesday, March 29, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features
- Machine Learning Enablement – Anomaly Detection
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- QTSO
Service Adjustments and Baseline Reconciliation Improvements
- HQR
- Removed non-impacting KPIs and updated component and KPI weights to represent their contribution to service health more accurately.
- QDIVS DARRT
- Updated component weights to represent service health more accurately.
- DEL
- Removed non-impacting KPIs and updated KPI weights to represent their contribution to service health more accurately.
- F5
- Added KPIs that were considered critical to overall service health.
- Presentation Zone
- Decommissioned this service as it was a duplicate of the F5 service.
Resolved issues
- iQIES – Improved New Relic queries to avoid occasional timeouts when fetching metrics.
- Machine Learning Models – Shifting trends in service metric data may adversely affect the performance of both Anomaly Detection (AD) and Uptime Prediction (UP) models. Retraining is complete for the listed models and AD alerts have been re-enabled where applicable.
- Confluence AD
- Jira Anomaly D
- ServiceNow AD
- Syslog AD.
Known issues
- ClamAV Anomaly Detection (AD) Machine Learning Model – Shifting trends in service metric data may adversely affect the performance of both Anomaly Detection (AD) and Uptime Prediction (UP) models. AD alerts have been disabled while this model is monitored and retrained.
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- QNOD – Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v1.23.1.4
Affected customers: All QualityNet users
On Wednesday, March 15, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features:
- New Service Onboarding
- Onboarded initial KPIs: CPU Used Percent and Memory Used Percent.
- VPN ASA
- Machine Learning Enablement – Anomaly Detection (Released March 13)
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- New Relic
- SAS Viya
- Machine Learning Enablement – Uptime Prediction (Released March 13)
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 10 minutes into the future for any potential issues at the KPI level, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Predictions are available for the following KPIs: Queue Depth.
- New Relic
Service Adjustments and Baseline Reconciliation Improvements:
- Barracuda
- Updated component and KPI weights to more accurately represent their contribution to overall service health.
- Added ELB and Email Gateways to those already being monitored.
- Added failing state thresholds to In and Out Queue KPIs.
- Confluence
- Removed non-impacting KPIs and updated component and KPI weights to more accurately represent overall service health.
- iQIES
- Removed non-impacting KPI's to more accurately represent overall service health.
- QMARS Fax
- Removed sunsetting Reverse Proxy subsystem.
- Nexus
- Removed sunsetting IQ Auditor and IQ Firewall subsystems.
- Zscaler
- Removed non-impacting KPIs and updated component and KPI weights to more accurately represent overall service health.
Resolved issues:
- Adjusted New Relic query intervals to avoid gaps in EC2 Status Check data for some Active Directory hosts.
- Resolved an issue where a missing entity in HARP data interfered with Anomaly Detection for that service.
- Resolved minor issues with panel labels and rendering on the QIES drill-down dashboard.
Known issues:
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v1.23.1.3
Affected customers: All QualityNet users
On Thursday, March 2, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features:
- Machine Learning Enablement – Uptime Prediction
- HARP
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 30 minutes into the future for any potential issues at the KPI level, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues. Where possible, predictions are made at the entity level (e.g. per host) to improve accuracy and diagnostic utility.
- Predictions are available for the following KPIs: Hera APM Throughput, Login APM Throughput, SO Tool APM Throughput, Account Recovery CPU Used Percent, Account Recovery Memory Used Bytes, ADO API Memory Used Bytes, HERMES CPU Used Percent, HERMES Memory Used Bytes, HOMER CPU Used Percent, HOMER Memory Used Bytes, Registration CPU Used Percent, Registration Memory Used Bytes, SO Tool Memory Used Bytes, Utility Memory Used Bytes.
- New Relic
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 10 minutes into the future for any potential issues at the KPI level, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Predictions are available for the following KPIs: New Relic Status.
Baseline Reconciliation Improvements:
- Ansible Tower
- Added an additional synthetic monitor, "AT-PROD-URL."
- ClamAV
- Updated component and KPI weights to more accurately represent their contribution to overall service health.
- Improved KPI legend specificity.
- HARP
- Improved resiliency of synthetics queries by using tags instead of synthetic names.
- Updated KPI weights to more accurately represent their contribution to overall service health.
- HIDS HARP Automation
- Updated component and KPI weights to more accurately represent their contribution to overall service health.
- Office365
- Renamed "API Status" to "O365 Status API."
- QIES
- Removed MDS APM Error Rate KPI as it does not impact overall service health.
Known issues:
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v1.23.1.2
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features:
- Machine Learning Enablement – Anomaly Detection
- HARP
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
Resolved issues:
- Updated New Relic account information for the DELWeb service.
- Updated PRS service Compute KPIs to reflect infrastructure migration to AWS Fargate.
Known issues:
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v6.8
Affected customers: All QualityNet users
On Monday, January 9, 2023, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features:
- Machine Learning Enablement – Anomaly Detection
- Ansible Tower
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners in investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
Resolved issues:
- Resolved an issue where selecting a specific KPI for viewing or editing opened the incorrect KPI
Known issues:
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- Time picker for service drill-down dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v6.7
Affected customers: All QualityNet users
On Wednesday, December 21, 2022, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New features:
- Machine Learning Enablement – Anomaly Detection
- Jira
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners with investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- Machine Learning Enablement – Uptime Prediction
- Jira
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 5 minutes into the future for any potential issues at the KPI level, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Predictions enabled for the following KPIs: APM Heap Used Percent, APM Throughput, Disk Free Percent, EFS Data Read IO, EFS Percent IO, Memory Used Percent, RDS CPU Used Percent, RDS Connections Count, RDS Freeable Memory, RDS Free Storage Space, RDS Read Latency, RDS Write Latency, Request Count, Synthetic Availability, Synthetic First Byte, Synthetic First Contentful Paint, Synthetic First Paint, Synthetic Latency, Synthetic On Page Load.
Resolved issues:
- Airflow
- Restored Process Count KPI
- Transitioned CPU and Memory Used Percent KPIs from EC2 to ECS source
- Removed EC2 Status Check and Disk Free Percent KPIs
Known issues:
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v6.6
Affected Customers: All QualityNet users
On Wednesday, December 7, 2022, at 8 p.m. ET, we will be releasing a new version of the QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Features:
- Machine Learning Enablement - Uptime Prediction
- ClamAV
- Added Uptime Predictions to Disk Free Percent and Memory Used Percent KPIs
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 5 minutes into the future for any potential issues at the KPI level for the ClamAV service, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- ClamAV
Issues Resolved:
- Barracuda
- Aligned QNOD queries with New Relic (one hour) polling interval to mitigate intermittent insufficient data periods for various KPIs
- QDIVS (FIVS)
- Restored Network KPIs with insufficient data after QDIVS network infrastructure changes.
Known Issue(s):
- Airflow– No data is available for some KPIs due to an issue with the New Relic data source
- Barracuda – Insufficient data overnight for Network KPIs
- CCSQ QuickSight – Occasional insufficient data false positives for User Experience KPIs
- EQRS SF – No data is available for User Experience KPIs (“canaries”) while the Portal team troubleshoots API endpoints. Service health alerts have been disabled for EQRS SF.
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v6.5
Affected Customers: QNOD CMS Executive Users
On Wednesday, November 28, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
Issues Resolved:
- Barracuda
- Updated QNOD data ingestion after device migration
- Add additional gateways to QNOD
- Office365
- Updated QNOD data ingestion after device migration
- Removed Office365 Process Count KPI
- IQIES
- Updated Web Transaction Time, SQS Oldest Message, and Visible Message
- Corrected Redis Infrastructure unit type from “milliseconds” to “bytes”
- Added ‘displayName’ facet to show all of Redis infrastructure
- Updated the Redshift CPU legend to show hostnames
Known Issue(s):
- Airflow - no data is available for some KPIs due to a data source issue unrelated to QNOD
- Barracuda - intermittent insufficient data periods for Network KPIs
- CCSQ QuickSight - occasional insufficient data false positives for User Experience KPIs
- EQRS SF - no data is available for User Experience KPIs ("canaries") while the Portal team troubleshoots API endpoints
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability
QualityNet Operations Dashboard v6.4
Affected Customers: QNOD CMS Executive Users
On Wednesday, November 9, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Onboarding for Service(s):
- Auth API
- Response Time
- Error Percentage
- ECS CPU Utilization
- ECS Memory Utilization
- ECS Task Running Count
- RDS CPU Utilization
- RDS Memory Utilization
- Synthetic Latency
- Synthetic Result
- Content Management
- 4XX/5XX Error Count
- Request Count
- ECS CPU Utilization
- ECS Memory Utilization
- Analytics Cloud Reporting - full subsystem decomposition
- Application
- Compute
- Network
- User Experience
- QPP
- HQR
Issues Resolved:
- PRS - Resolved insufficient data reported for some KPIs
- SAS Viya - Removed KPIs that are no longer relevant to service health
- Web Transactions Time
- APM Error Rate
- APM Throughput
- PG Database Connections
- EQRS Portal - Resolved insufficient data reported for User Experience KPIs
- EQRS Scoring and Feedback - Resolved degraded service health after New Relic account changes
- QSEP - Resolved insufficient data reported for some KPIs
- HQR - Removed decommissioned services from some KPIs
- Corrected typographical errors in some KPI names
- Switched FireEye_vnx Network Response time metrics provider from Manage Engine to New Relic
- Removed Availability Network component
- Separated Response Time metric into two components
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v6.3
On Wednesday, October 27, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Onboarding for Service(s):
- Clinicians Insights API
- Response Time
- Error Percentage
- ECS CPU Utilization
- ECS Memory Utilization
- ECS Task Running Count
- RDS CPU Utilization
- RDS Read Latency
- Synthetic Latency
- Synthetic Result
- Scoring API
- Response Time
- Error Percentage
- ECS CPU Utilization
- ECS Memory Utilization
- ECS Task Running Count
- RDS CPU Utilization
- RDS Read Latency
- Synthetic Latency
- Synthetic Result
- Front End
- 4XX/5XX Error Count
- Healthy Host Count
- Request Count
- ECS CPU Utilization
- ECS Memory Utilization
- ECS Task Running Count
- Error Percentage
- Response Time
- Web Errors
- Web Throughput
- Web Transaction Time
- Content Management
- RDS – CPU Utilization
- RDS – Free Storage Space
- FSx Metrics
- Removed Subsystems
- QPP
- MedTrak
- Machine Learning Enablement – Uptime Prediction for the following service(s):
- Corrected the QNOD visualization to reflect 5-minute predictions. This was changed in release 6.1, when the Barracuda model had 3 additional KPIs added and the prediction time was changed from 30m to 5m to improve KPI predictive accuracy.
- Barracuda
Known Issue(s):
Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v6.2
On Wednesday, October 12, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Onboarding for Service(s):
- QPP
- Subsystems – Onboarded these subsystem components: Eligibility, Submissions API, Targeted Review & Self Nomination.
- User Experience – Onboarded KPIs: Synthetic Availability, Synthetic Latency.
- Compute – Onboarded KPIs:
- ECS – CPU Utilization, Memory Utilization, Task running count.
- RDS – BinLogDiskUsage, CPUUtilization, DatabaseConnections, DiskQueueDepth, FreeStorageSpace, ReadIOPS, ReadLatency, ReadThroughput, SwapUsage, WriteIOPS, WriteLatency,
- Applications – Onboarded KPIs: Errors, Web Transaction Time, Web Throughput, HealthCheckStatus.
- Network – Onboarded KPIs: Healthy Host Count, Request Count, HTTPCode_Target_4XX_Count, HTTPCode_Target_5XX_Count.
- FAS
- Network – Onboarded KPIs: Request Count, Active Connection Count, Processed Bytes, Response Time.
- QPP
- Machine Learning Enablement – Anomaly Detection for the following service(s):
- ClamAV
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners with investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- ClamAV
- Machine Learning Enablement – Uptime Prediction for the following service(s):
- ServiceNow
- Adding CPU Used Percent and Memory Used Percent.
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 30 minutes into the future for any potential issues at the KPI level for the Confluence service, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- ServiceNow
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v6.1
On Wednesday, September 28, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Onboarding for Service(s):
- Application – Onboarded KPIs: Redshift Database Connections, Redshift Read/Write Latency
- Compute – Onboarded KPIs: Redshift; Health Status, CPU Used Percent, Disk Used Percent, Read/Write Throughput.
- Network – ALB Request Count & Active Connections, API Gateway 5xx Errors.
- Compute – Switched EC2 KPI’s with Fargate.
- Compute – Switched EC2 KPI’s with Fargate.
- Application – Add Synthetic Maker synthetics. Fail Percent.
- Compute – Onboarded KPIs: CPU, Memory, Disk for the service’s Infrastructure.
- EQRS Portal
- QNP
- Mailman
- QPP
- FAS
- Machine Learning Enablement – Anomaly Detection for the following service(s):
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners with investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- ServiceNow and Splunk:
- Machine Learning Enablement – Uptime Prediction additional KPIs for the following service(s):
- Adding CPU Used Percent, Volume Queue Length & Request Count to the existing KPIs of In Queue Count and Out Queue Count
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 30 minutes into the future for any potential issues at the KPI level for the Confluence service, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Barracuda
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v5.5
On Wednesday, September 7, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Monthly Service Availability Dashboard
- A new dashboard that is specific to service availability for all services and provides a single view of service availability for the past 30 days. In addition, the view will include services that reported outages in the past 30 days.
- Machine Learning Enablement – Anomaly Detection for the following service(s):
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners with investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- Jenkins:
- Machine Learning Enablement – 30 Minute Uptime Prediction for the following service(s):
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 30 minutes into the future for any potential issues at the KPI level for the Confluence service, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Ansible Tower:
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v5.4
On Friday, August 26, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Onboarding for Service(s):
- Synthetics – Determines the availability of the QMARS application; Duration and Success Percent are the two KPIs ingesting into QNOD.
- Application – Onboarded KPIs: Web Transaction Time, Read/Write Latency, Transaction Throughput.
- Compute – Onboarded KPIs: CPU, Memory, Disk, Read/Write IOPS for the service’s Infrastructure.
- Application – Onboarded KPIs: Application Web Transaction Time, Error Rate, and Transaction Throughput; RDS Connections and Read/Write Latency.
- Compute – Onboarded KPIs: EC2 CPU, Memory, and Disk; RDS CPU, Memory, Disk, and Read/Write IOPS; ECS CPU and Memory.
- Network – Onboarded KPIs: ALB Requests and Active Connections.
- Synthetics – Add Legacy synthetics. Determines the availability of the QMARS application; Duration and Success Percent are the two KPIs ingesting into QNOD.
- Network – Onboarded KPIs: ALB Active Connections, ALB Processed Bytes, and ALB Request Count.
- Application – Onboarded KPIs: Web Transaction Time, Read/Write Latency, Transaction Throughput, Processes Count, and Application Error Rate.
- Compute – Onboarded KPIs: CPU, Memory, Disk, Read/Write IOPS for the service’s Infrastructure.
- Network – ALB Request Count, ALB Avg. Healthy Host Count, ALB Response Time, Nginx Requests Per Second, Nginx Active Connections, Nginx Connections Waiting.
- EQRS Scoring and Feedback
- MAT
- QMARS
- QSEP
- Machine Learning Enablement – 30 Minute Uptime Prediction for the following service(s):
- This AI/ML capability with deep learning models provides the service owner with a look ahead of 30 minutes into the future for any potential issues at the KPI level for the Confluence service, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Jenkins and Splunk:
Performance Improvements:
- Removed the additional components involved in data transfer between data collection and data storage layers. Also removed the additional components between data processing and data storage layers. This has resulted in significant improvements in performance and will enable quicker restoration compared to the current process. This will also reduce the downtime during deployments to 1–2 minutes compared to current 10–15 minutes.
- Multiple orgs. within the Influxdb data are merged into one for better visibility and testing within the data storage layer.
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v5.3
On Wednesday, August 10, 2022, at 8 p.m. ET, we will be releasing a new version of our QualityNet Operations Dashboard (QNOD). This release will cause QNOD to be unavailable until 10 p.m. ET.
Why is this release happening?
New versions of QNOD are released each sprint to ensure that QNOD users have continued access to the latest product features and enhancements, as well as fixes to any issues that have been resolved.
What are some of the enhancements included in the upgrade?
Users can expect the following new functionality with this release as well as fixes and/or security patching.
New Feature(s):
- Onboarding for Service(s):
- Synthetics – Determines the availability of the QMARS application; Duration and Success Percent are the two KPIs ingesting into QNOD.
- Application – Onboarded KPIs: Web Transaction Time, Read/Write Latency, Transaction Throughput.
- Compute – Onboarded KPIs: CPU, Memory, Disk, Read/Write IOPS, for the service’s Infrastructure.
- Split out KPIs into distinct applications for CASPER, MDS, and PBJ.
- User Experience – Expand synthetic tests to include following KPIs: 'First Byte', 'First Paint', 'First Contentful Paint', and 'Page Load'.
- Application – Update KPIs for APM Response Time, APM Throughput, APM Error Rate.
- QMARS
- QIES
- Machine Learning Enablement – Anomaly Detection for the following service(s):
- This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service.
- This capability will aid service owners with investigating root causes as well as fixing issues with their service that would otherwise lead to a potential future service issue or degradation.
- Confluence and Syslog:
- QNOD Notifications:
- Using the Slack channel #alerts-qnod-prod, the Anomaly Detection process from the Machine Learning capability above will send alerts when an anomalous event is detected in these services. An alert will also be sent when the condition has cleared for the service.
- This capability will aid service owners with identifying and quickly fixing issues with their service that would otherwise lead to a potential service issue or degradation.
- Confluence and Syslog:
- Service Availability:
- The Service Availability percentage refers to the percentage of time service stayed in a “Non-Critical” state out of the total time reported to New Relic.
- Service Availability percentage for the past 24 hours is now displayed in service drilldown dashboards.
Issue(s) Resolved:
- QIES
- User Experience – corrected a failing synthetic test.
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v5.2
New Feature(s):
- Synthetics for the following service(s):
- MedTrak
- Full Decomposition for the following service(s):
- QCOR
- QDIVS
- MedTrak
- Machine Learning Enablement – 30-Minute Uptime Prediction Model for the following service(s):
- Confluence – This AI/ML capability with deep learning models provides the service owner with a look ahead of 30 minutes into the future for any potential issues at the KPI level for the Confluence service, with an 86% confidence level in this prediction. This gives service owners an opportunity to investigate their service and look for potential issues.
- Machine Learning Enablement – Anomaly Detection for the following service(s):
- Barracuda – This AI/ML capability with deep learning models provides service owners with a 24-hour historical view of anomalies that may have occurred with their service. This capability will aid service owners with investigating the root cause and fixing any issues with their service that would otherwise lead to a potential future service issue or degradation.
Issue(s) Resolved:
- None this release
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v5.1.2
New Feature(s):
- Machine Learning Enablement – 30 Minute Uptime Prediction Model for the following service(s):
- Barracuda
- Syslog
QualityNet Operations Dashboard v5.1.1
Roll Back New Feature(s):
- Roll Back Machine Learning Enablement – 30 Minute Uptime Prediction Model for the following service(s):
- Barracuda
- Syslog
QualityNet Operations Dashboard v5.1
New Feature(s):
- Full Decomposition for the following service(s):
- QIES
- QTSO
- Machine Learning Enablement – 30 Minute Uptime Prediction Model for the following service(s):
- Barracuda
- Syslog
Issue(s) Resolved:
- The following issues with service drilldown dashboards are fixed:
- AD – Process Count KPI panels updated to show process names along with host names.
- Certificate Authority – Process Count KPI panels updated to show process names along with host names.
- ClamAV – Process Count KPI panels updated to show process names along with host names.
- PRS – Updated legend name for the KPI panels.
- Hive – Average Healthy Host Count and Process Count KPI panels updated with correct legend names.
- Drilldown dashboards for Office365, DELWeb, and McAfee WG services fixed to show host names in KPI panel labels.
Known Issue(s):
- Time picker for service drilldown dashboards has been temporarily disabled to address possible issues with system stability.
QualityNet Operations Dashboard v4.6
New Feature(s):
- Synthetic Monitoring for the following service(s):
- MedTrax
- Full Decomposition for the following service(s):
- EQRS Portal Service
- HARP/HIDS Automation (Additional KPIs)
- WAN - New devices added into New Relic and reporting the same in QNOD
Issue(s) Resolved:
- Resolved the issue with the Alerter process to be able to send notifications for service state changes
- Fixed the ‘Disk Free Percent’ KPI reported via New Relic for multiple services
- Updated F5 URLs synthetic test scripts to accommodate the F5 network device’s move to new hardware
QualityNet Operations Dashboard v4.5
New Feature(s)
- Synthetic Monitoring for the following service(s):
- EQRS Scoring and Feedback
- QIES
- QTSO
- Full Decomposition for the following service(s):
- iQIES/PASCID
- Additional reports have been added to the Grafana Metrics API:
- The Recovery Rate report calculate the ratio of failed deployments to the total number of deployments, shown on a quarterly bases.
- The Mean Time to Recover report shows, on a quarterly basis, shows the average amount of time it takes for application to recover from a failed deployment
QualityNet Operations Dashboard v4.4
New Feature(s)
- Synthetic Monitoring for the following services
- FAS
- QCOR
- HARP/HIDS Automation
- MFT
- AWS RSS messages
- US-East-1 and global regions are continuously retrieved from AWS and available at #aws-rss-alerts Slack channel
- New dashboards available
- 24-hour Service Issues Summary
- Service Issues Reports
- New component added for New Relic service drilldown to capture New Relic minion (synthetic test monitors) health
Bug Fixes:
- FireEye ETP
- Updated Synthetic Availability to where it no longer reports a constant degraded state.
- FireEye vNX
- Updated Interface KPIs to where they are no longer reporting a constant degraded state
- McAfee GW
- Updated Interface KPIs to where they are no longer reporting a constant degraded state
- Certificate Authority
- Both Disks are now reporting correctly, and the overall service health is more accurate
- Nexus
- Fixed thresholds for Disk, previously reporting “Insufficient Data” when there was indeed data
- Slack
- Broken incident links to https://status.slack.com have been fixed
QualityNet Operations Dashboard v4.3
New Feature(s)
- Synthetic Monitoring for the following services:
- QDIVS
- Bonnie/MAT
- QSEP/ITSP
- CCSQ QuickSight
- Full Decomposition for the following services:
- DEL
- Anomaly detection displayed for the following services:
- Confluence
- Improved reporting
- AWS evaluation now includes reports from the AWS global and us-east-1 public health API
- Landing page Enhancements
- Service panels now include a hover function to display current service health %.
- New service status icon to identify services which have KPI issues yet do not fully affect a service’s state.
- Metric Sources Dashboard
- A dashboard displaying a summary of metric sources is now available (metric summaries dashboard) and is available as a link from the landing page.
- A dashboard drilling into each metric source, detailing degraded, failed, and missing values is available from the metric summaries dashboard.
Bug Fixes:
- Services where KPI’s were appearing as blank on the service decomposition diagram on a service drilldown now appear as No Data (grey color)
- Missing KPI’s for services now contribute to the overall state of a service.
- Most services were not being evaluated in their ‘minutes_threshold’ value via their service decomposition definitions. This caused a single point to affect the overall state of a service. Service KPI’s are now being evaluated correctly.
QualityNet Operations Dashboard v4.2
New Feature(s)
- Full Decomposition for the following services:
- Ambari Infrastructure
- TestRail
- SAS Viya
- Zeppelin
QualityNet Operations Dashboard v4.1
New Feature(s)
- Full Decomposition for the following services:
- Airflow
- Hive
- Ranger
Infrastructure Upgrade
- Grafana upgraded to v8.4.4 from v8.1.8
QualityNet Operations Dashboard v3.5
New Services
- EQRS Portal
- iQIES
New Features/Bug Fixes
- Thresholds for FileCloud and Routing services are adjusted to display the status of the application accurately.
- Fixed the Request API Count KPI to display data and services having Request Count start reporting data.
QualityNet Operations Dashboard v3.4
Issues Resolved
- DNS, Office365, AD, Certificate Authority - resolved issue where KPIs were not reporting in QNOD after a New Relic upgrade
New Features
- HQR - added Application and Network metrics, expanded Compute metrics, and added two more subsystems
- HARP - moved from Collaboration panel to Identity & Access panel
- Modified thresholds from 0 minutes to 3 minutes to reduce noise
QualityNet Operations Dashboard v3.3
New Services
- CDR (Ambari, HIVE, and Ranger subsystems)
- HQR
- DELWeb
New Features
- HARP Service drilldown updated to include the subsystems along with HOMER subsystem.
- Updated service drilldown dashboards to include Jira issues panel.
QualityNet Operations Dashboard v3.2
New Services
- Airflow
- SAS Viya
- Zeppelin
New Features
- FireEye vNX
- Added Device Availability and Response Time KPIs
- Metrics API
- Added new report APIs to communicate deployment recovery metrics and new roles to API keys to enhance security
QualityNet Operations Dashboard v3.1.1
Category Changes in Current Service Status Overview Dashboard:
- Zscaler has moved from Security to Network
- Syslog has moved from Security to Monitoring
Issues Resolved
- Provided a fix to the NewRelic service to reflect the current health status more accurately.
- Implemented update to the F5 service synthetic tests to reflect the current service health status in the dashboard.
QualityNet Operations Dashboard v3.1
New Features:
- New Services
- Network
- AWS
- QMARS Fax (Biscom)
- Network Routing
- Presentation Zone
- WAN Connectivity
- Collaboration
- TestRail
- Network
QualityNet Operations Dashboard v2.7.1
New Features:
- DevSecOps Metrics API MVP
- Usable functionality will be a dashboard presenting the number of deployments per day, per application.
- Please reach out to Tim Regulski for an API Key and instructions on configuration
QualityNet Operations Dashboard v2.7.0
New Features:
- Entity Discovery Automation
- We can catalog all devices on our data sources.
- Provides functionality for us to be much faster in understanding what devices exist and what data gaps we may have for a particular service.
- DAS QNOD Integration Prep
- Worked with the HARP team to create a new DAS entitlement with HARP to provide the teams access to QNOD.
- SaaS Issue Reporting
- Slack service drill down can display RSS feeds for the latest active and resolved incidents.
New Services:
- SNOW
- F5
- FirePower IPS
- Office 365/Exchange
- QNET
- Mailman
- PRS
- FireEye VNX
QualityNet Operations Dashboard v2.6.0
New Features:
- Implemented Grafana upgrade from 8.1.2 to 8.1.8 to address CVE-2021-43798.
- QNOD alert notifications per service can be configured to send to multiple Email distros and Slack channels. Alerts will be sent on service state changes.
New Services:
- McAfee Web Gateway
- Trend Micro DS
QualityNet Operations Dashboard v2.5.0
Architecture Improvements:
- Data collection processes have been decoupled from each other on a per-service basis. This will improve performance and make it less likely for a failure in on service's data pull to affect others.
New Features:
- Enable QNOD notifications to send alerts to Email distribution or Slack channel
- Service Status will now be derived from the weighted system health score. This will improve the accuracy of the system status and make it less subject to a single KPI status
New Services:
- CA Certificates
- Survey Monkey
- FireEye ETP
Issues Resolved:
KPIs that are designated to alarm only after a specified period of time will now alarm only after the specified period as intended
QualityNet Operations Dashboard v2.4.0
New Features:
- Added Service Health Panel to all dashboards to display the weighted score of the service over time
- Added Current Issues Dashboard so that KPI issues are seen on a single dashboard
New Service:
- Active Directory
QualityNet Operations Dashboard v2.3.1
Issues Resolved:
- Updated the view for Current Service Status Overview Dashboard
- Fixed the unittype in the FileCloud and MFT service drilldown panels
- Nexus service divided into subsystems for more visibility into the service
QualityNet Operations Dashboard v2.3
New Features
- Migrated to serverless metric ingestion, increasing reliability and efficiency
- Improved the layout of the current status dashboard to more succinctly show service groups
- Fixed the display of certain metrics for Jira and Confluence
- Implemented Notifications sent to Slack for service status change
New Services:
- DNS
- Slack
QualityNet Operations Dashboard v2.2
New Features
- Added weighted system health score to represent system health in a more dynamic way
- Implemented automatic creation of dashboard drill-downs to ensure consistency and improve velocity
- Produced POC/MVP of automatic discovery and metric ingestion engine
- Redesigned the current status overview dashboard
- Incorporated Unit Tests for Lambda functions
QualityNet Operations Dashboard v2.0
Issues Resolved
- KPIs, where no data is okay, will no longer affect the component or system status
- Tuning of the logging frequency for Flux tasks
QualityNet Operations Dashboard v1.5
New Services:
- MFT
- Tenable Nessus
New Features:
- Enhanced the layout of the drilldown and service dependency tree diagram to improve the viability of the KPIs.
QualityNet Operations Dashboard v1.4.1
New Features:
- Improved Performance -- serverless computing (AWS Lambda) was deployed to increase the efficiency and timing of queries, which lowers the load up to 90% on database tier
- No Data -- the dashboard now displays 'No Data' if there is insufficient data to represent the service's status and KPI
- UI Improvements -- various small improvements to the UI, color uniformity, panel type, etc
Bug Fixes:
- "No Query Returned Results" error has been fix on the Executive Dashboard
QualityNet Operations Dashboard v1.4
New Service(s):
The following service(s) will be integrated with the new release:
- Barracuda (Mailman)
- HARP
New Features:
- FileCloud, Nexus, and Splunk are fully decomposed with metrics provided in their drilldowns
- Syslog synthetic test panels added to drilldowns
- Confluence and Jira data ingest processing and visualization for Network and Database metrics added in drilldowns
Bug Fixes:
- Added info to panels in drilldown dashboard
QualityNet Operations Dashboard v1.3.1
Bug Fixes:
- Infrastructure upgraded to address the intermittent "query not returning results" error on the Dashboard panels
- Updated AMI to address security compliance findings
QualityNet Operations Dashboard v1.3
New Service(s):
The following service(s) will be integrated with the new release:
- GitHub
- NewRelic
New Feature(s):
Service Drill Downs
- Ansible, Jenkins, GitHub and New Relic drill-downs provide fully decomposed metrics.
Bug Fixes:
The following issues will be resolved with the new release:
- QNet Dashboard Logo updated with a new transparent icon.
- Logic to determine component status updated to reflect correct status of component.
- Service Dependency Diagram panel updated for better visibility.
- Updated the Executive Dashboard view to render the service status correctly.
QualityNet Operations Dashboard v1.2
New Service(s):
- ZScaler
What is new:
Service Drill Downs
- Confluence, JIRA, and ZScaler drill-downs provide fully decomposed metrics.
Added scan latency panel to ClamAV User Experience component.
Link to Confluence page with Jira issues fixed to ensure that it opens in a new tab instead of same tab.
QualityNet Operations Dashboard v1.1.1
Bug Fixes:
- Increased the number of containers to 2 for Grafana to fix the "503 service temporarily unavailable"
- Fixed the link to Jira issues page on confluence from dashboard.
- Increased the query timeout and query concurrency in Influxdb to resolve the "query length limit exceeded" error.
- Increased the CPU and memory allocation for Grafana, Influxdb and Telegraf.
QualityNet Operations Dashboard v1.1
What is New:
The following systems have their own drilldown dashboard:
- Confluence
- JIRA
- Service Now
- FileCloud
- Ansible
- Nexus
- Jenkins
- Splunk
- ClamAV
- Syslog
What is included in this release:
- Upgraded to Grafana v8.0.3
- 508 Accessibility
- Added 'alt' attributes to images
- Removed heading <h1,2,3,4> attributes
- Fixed some contrast issues
- Removed 3 semi-hidden panels, reduced code by 180 lines
QualityNet Operations Dashboard v1.0
The following Applications are currently being monitored for availability (up/down):
- Confluence
- JIRA
- Service Now
- FileCloud
- Ansible
- Nexus
- Jenkins
- Splunk
What is included in this release:
- Removed the ability for users to log into the dashboard with local accounts, users are forced to have a HARP account
- Okta/HARP integration for authentication
- 4 hour authentication timeout after no activity
- Automated vulnerability scanning utilizing Netsparker and Nessus
- Implemented Sonar Scanner to validate code in GitHub for vulnerabilities and bugs
- Fixed Overlay UI issues
- Fixed Panel Lengths so they all match and are even
- Updated the queries to fix the service status results in Grafana
- Grafana synthetic testing to validate dashboard availability
- Container and Host based alerts to Slack ie CPU Utilization %, Memory Usage %, Disk Space, Host not responding, and Database storage utilization alert
Known Issues:
- Internet Explorer 11 not currently supported. For more information https://grafana.com/docs/grafana/latest/installation/requirements/#supported-web-browsers
- Application logs are not ingesting into Splunk
- ADO-HIDS-Ventech Solutions is the only available organization in HARP
- CCSQ Support Central: Provides you with multi-program support to submit a new ticket and track the status of an existing case, incident, or request. No login required. CCSQ Service Central
Phone: (866) 288-8914 (TRS: 711)
Slack: #help-service-center-sos
Email: ServiceCenterSOS@cms.hhs.gov
- QNOD Help Slack Channel: #help-qnod
- QNOD Service Desk: For changes or enhancements to your service submit your request to the QNOD Service Desk
- Join our community by clicking here to participate in upcoming user research studies.
- No labels