CCSQ Program Dashboard Testing and Validation

Report Name: Current Program Backlog Composition

Validation Date: 10/6/2022

Validator: Seth Evans/Randolph Anagho

Table Result Key: Dashboard/JQL

NOTE: Updated report filter to include the following statuses to meet the report description (The number of ready and in progress features and enablers in the program backlog today.) Included statuses: "Ready for PI", "PI Ready", Implementing, "In Progress" 

HQR JQL: Project in ("HQR Program Project", "HQR UX-Men") AND issuetype in (10000, Enabler) AND Status in (Implementing, "PI Ready") ORDER BY issuetype ASC

EQRS JQL:

All features and enablers: Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype in (10000, Enabler) AND Status in ("Ready for PI", "PI Ready", Implementing, "In Progress") ORDER BY status DESC

All Enablers: Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype in (10000, Enabler) AND Status in ("Ready for PI", "PI Ready", Implementing, "In Progress") AND labels in (Enabler, Exploration_Enabler, Architectural_Enabler, Infrastructure_Enabler, Compliance_Enabler) ORDER  BY status DESC

IQIES JQL:

All features and enablers: Project in ("iQIES Program Board ", "iQIES Enigma", "iQIES HCD", "iQIES Frontiers", "iQIES Bohemian Rhapsody (DevOps)", "iQIES Dune (CLIA)", "iQIES FACT (Data and Backend)", "iQIES Outlander", "iQIES PA Reports", "iQIES QMIR Data", "iQIES SAS Viya / Dashboards", "iQIES S&C Reports", "IQIES SNOOPY", "iQIES Stargate", "iQIES Immeasurable", "iQIES The Misfits", "iQIES The Shop", "IQIES Woodstock")  AND issuetype in (10000, Enabler) AND Status in ("Ready for PI", "PI Ready", Implementing, "In Progress") ORDER BY status DESC

All Enablers: Project = "iQIES Program Board "  AND issuetype in (10000, Enabler) AND Status in ("Ready for PI", "PI Ready", Implementing, "In Progress") AND labels in (Enabler, Exploration_Enabler, Architectural_Enabler, Infrastructure_Enabler, Compliance_Enabler) ORDER  BY status DESC

Report: Current Program Backlog Health

Validation Date: 10/06/2022

Validator: Marc Santini

Comments: Overall good results. Egregious deviations are seen on low numbers as usual. Did data refresh and ruled out stale data. Investigation into iQIES errata needed

Report: Features in Status on PI start Date

Validation Date: 2/1/2023 (10/12/2022)

Validator: Seth Evans

Validation Notes: I also revalidated iQies and EQRS. The previous results listed below are still valid. (Numbers for most recent PIs did change because they are active)

Legend: EazyBI count/JQL count

JQL - PI23: project in ("HQR UX-Men", "HQR Program Project") AND issuetype = 10000 AND Status was "PI Ready" ON ("2022-10-18 23:59") - *Change status and date as applicable

Report: Features in Status on PI start Date

Validation Date: 8/15/2022

Validator: Marc Santini

Validation Notes:

1. Issues with status changes to implementing were not being caught because the earlier status was being returned by the validation query. Changing the time from (YYYY-M(M)-D(D)) to ('YYYY-M(M)-D(D) HH:MM'), with "HH:MM" being "23:59" catches the status at 11:59 PM. This brought all counts well under the "Accurate" STD category

Validation Date : 8/11/2022

Validator:  Marc Santini

Report Name: Features Started and Closed per PI

Legend: EazyBI count/JQL count

Validation Date: 01/31/2023

Validator: Seth Evans

PI19: Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND Status changed to Implementing during (2021-11-16, 2022-02-07) (change status to Done or Implementing)

PI20: Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND Status changed to Implementing during (2022-02-08, 2022-05-02) (change status to Done or Implementing)

PI21: Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND Status changed to Implementing during (2022-05-03, 2022-07-25) (change status to Done or Implementing)

PI22: Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND Status changed to Implementing during (2022-07-26, 2022-10-17) (change status to Done or Implementing)

PI23: Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND Status changed to Implementing during (2022-10-18, 2023-01-09)

PI4: Project = "iQIES Program Board " AND issuetype = 10000 AND status changed to Done during (2021-10-13, 2022-01-18) (change status to Done or Implementing)

PI5: Project = "iQIES Program Board " AND issuetype = 10000 AND status changed to Done during (2022-01-19, 2022-04-19) (change status to Done or Implementing)

PI6: Project = "iQIES Program Board " AND issuetype = 10000 AND status changed to Done during (2022-04-20, 2022-07-19) (change status to Done or Implementing)

PI7: Project = "iQIES Program Board " AND issuetype = 10000 AND status changed to Done during (2022-07-20, 2022-10-11) (change status to Done or Implementing)

PI8: Project = "iQIES Program Board " AND issuetype = 10000 AND status changed to Done during (2022-10-12, 2022-12-20)

PI16: Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype = 10000 AND status changed to Done during (2021-10-13, 2022-01-18) (change status to Done or Implementing)

PI17: Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype = 10000 AND status changed to Done during (2022-01-19, 2022-04-12) (change status to Done or Implementing)

PI18: Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype = 10000 AND status changed to Done during (2022-04-13, 2022-07-05) (change status to Done or Implementing)

PI19: Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype = 10000 AND status changed to Done during (2022-07-06, 2022-09-27) (change status to Done or Implementing)

***I added a bug LACE-3760 - Getting issue details... STATUS These numbers will need to be revalidated once the bug is resolved.***

10/26/22 While validating LACE-3760 it was found the time included in the date was causing issues to be dropped. Removing the time fixed the issue and resulted in the updated numbers logged above.

Validation Date : 8/17/2022

Validator: Marc Santini

Because the eazybi measure was using dateBetween with the actual PI start and end dates, it was not picking up items changed to the Implementing or Done statuses on the defined PI Start and End dates, because it was only looking for items in between those dates, not including them I updated the query to check After the last minute of the day before the PI, and Before the first minute of the day following the PI. I updated the Eazybi measure and then changed the Jira query to use the BEFORE and AFTER operators rather than the DURING and used matching timestamps in both the measure and the query. I suspect that the DURING operator in Jira is not as high resolution as EazyBi's Datebetween function. 

I will create two stories to investigate the anomalous counts for iQIES PI 6 implementing, and EQRS PI 18 Done

Updated DateBetween function:  [Time].[Day].DateMembersBetween('Jan 18 2022 23:59', 'Apr 13 2022 00:01')

Updated Query:  Project = EQRS AND issuetype = 10000 AND Status changed to Done AFTER  ("2022-01-19 00:01")  AND Status changed to Done BEFORE  ("2022-04-13 00:01")

Validation Date : 8/16/2022

Validator: Seth Evans

Project = "iQIES Program Board "  AND issuetype = 10000 AND Status changed to Done during (2022-01-19, 2022-04-19)  ORDER BY key DESC

Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype = 10000 AND Status changed to Done during (2021-04-04, 2021-07-20) --Update status and dates of PI

Report Name: Work Delivered by Category per PI

Dashboard: CCSQ Program Dashboard

Validation Date: Updated 01/31/23 (11/3/22)

Validator: Seth Evans

Validation Notes: N/A

Report Name: Work Delivered by Category per PI

Dashboard: CCSQ Program Dashboard

Validation Date: 9/7/22

Validator: Marc Santini

Validation Notes: iQIES numbers improved, all numbers within the "Accurate" standard deviation range!

EQRS numbers largely acceptable with a few outliers. I am curious as to why the average StDev jumped from 1% or less to ~5% starting in PI 17

Validation Queries

Validation Date: 8/31/2022

Validator: Seth Evans

JQL Query:

Report Name: Work Delivered by Category per PI

Dashboard : Enterprise Program Dashboard

Legend: EazyBI count/JQL count

iQies numbers improved or stayed the same in all cases!

I call this acceptable, although I am curious why the StDev jumped up to around 5% starting in PI 17. The high diffs are small issue counts and can be researched

Report Name: All ART Teams' Backlog Composition at PI Start

Dashboard: CCSQ Program Dashboard

Validation Date: 11/15/2022

Validator: Seth Evans

*For Enablers we looked into some of these discrepancies where items were showing in JQL results but were not showing in EazyBi. The only deltas we could find in the Enablers were that some of them did not have a Feature linked.

Base JQL: Project in (HQRPP, HTPI) AND issuetype in (Story, "Action Item", Improvement, Insight, "Prb/Enh") AND status was in ("Analyzing (SAFe)", "BA Review", Backlog, Brainstorming, Committed, "Feature Definition", Funnel, "On Hold", New, "New Story", Open, "PI Ready", "Ready for Refinement", "Ready to Estimate", "Refinement Backlog", "Selected for Action", "Story Definition", "To Do", "Ready for Peer Review", "Selected for PI", Sized, "Pending CMS Approval", "Selected for Development", "Ready for PI", Pending, "Feature Refinement", "Feature Deep Dive", "Ready to Work", "In Refinement", Draft, Requested, "Not Started", Unassigned, Intake, Accepted, Analysis, Analyzing, "Analyzing/Planning", "Awaiting Response", Blocked, "Blockers/Issues", "Client Review", "CMS Approving", Deferred, Deployed, "Dev Review", Discovery, Implementing, "In Analysis", "In Development", "In Progress", "In QA Test", "In Review", "In UAT Testing", "Initial Write-Up", "Internal Review", Monitoring, "Peer Review", "Ready for Acceptance", "Ready for Deployment", "Ready for Peer Review", "Ready to Test", Refining, "Test Writing", Testing, "Under Test (SBits)", UX, Development, Planning, Review, "Coding Review", "Ready for Test", Design, Prototyping, Ready, "In Test", "Mitigation in Progress", "User Stories Ready", "Ready for Testing", "Next", Validating, "Quality Review", "Ready for Release", "Realized Risk", "PR Review", "Ready for Validation", Ideation, "CMS Review", "Ready for Dev", Collaborating, Socializing, Recurring, "In Discovery", Evaluation, "Waiting for Decision", "Code Review", "Ready to Start", "DevOps In Progress", "Ready for PROD", "Ready for Delivery", Decided, Barrier, "Create User Stories", "Work in Progress", "Classify Response", "Response Completed", "Testing Complete", "In Definition & Refinement", "Design In Progress", "Ready for PO Design Review", "User Research Prep", Operational, Research, "Pending with Media Team Admins", "Dev Complete/Follow Up", "Dev Complete", "Pending with Division", "Pending with SOG Leadership", "Feature / Stories Needed", "Development Complete", "Response Development", "Ready for UAT", Done) on 2022-10-18 AND resolution is EMPTY ORDER BY status DESC, resolved DESC

Change date:

Change issue type:

Base JQL: Project in (EQRS, EQRSCI, EQRSD, EQRSARCH, EQRSEDSM, EQRSONBRD, EQRSMD, EQRSMS, EQRSPRTL) AND issuetype in (Story, "Action Item", Improvement, Insight, "Prb/Enh") AND status was in ("Analyzing (SAFe)", "BA Review", Backlog, Brainstorming, Committed, "Feature Definition", Funnel, "On Hold", New, "New Story", Open, "PI Ready", "Ready for Refinement", "Ready to Estimate", "Refinement Backlog", "Selected for Action", "Story Definition", "To Do", "Ready for Peer Review", "Selected for PI", Sized, "Pending CMS Approval", "Selected for Development", "Ready for PI", Pending, "Feature Refinement", "Feature Deep Dive", "Ready to Work", "In Refinement", Draft, Requested, "Not Started", Unassigned, Intake, Accepted, Analysis, Analyzing, "Analyzing/Planning", "Awaiting Response", Blocked, "Blockers/Issues", "Client Review", "CMS Approving", Deferred, Deployed, "Dev Review", Discovery, Implementing, "In Analysis", "In Development", "In Progress", "In QA Test", "In Review", "In UAT Testing", "Initial Write-Up", "Internal Review", Monitoring, "Peer Review", "Ready for Acceptance", "Ready for Deployment", "Ready for Peer Review", "Ready to Test", Refining, "Test Writing", Testing, "Under Test (SBits)", UX, Development, Planning, Review, "Coding Review", "Ready for Test", Design, Prototyping, Ready, "In Test", "Mitigation in Progress", "User Stories Ready", "Ready for Testing", "Next", Validating, "Quality Review", "Ready for Release", "Realized Risk", "PR Review", "Ready for Validation", Ideation, "CMS Review", "Ready for Dev", Collaborating, Socializing, Recurring, "In Discovery", Evaluation, "Waiting for Decision", "Code Review", "Ready to Start", "DevOps In Progress", "Ready for PROD", "Ready for Delivery", Decided, Barrier, "Create User Stories", "Work in Progress", "Classify Response", "Response Completed", "Testing Complete", "In Definition & Refinement", "Design In Progress", "Ready for PO Design Review", "User Research Prep", Operational, Research, "Pending with Media Team Admins", "Dev Complete/Follow Up", "Dev Complete", "Pending with Division", "Pending with SOG Leadership", "Feature / Stories Needed", "Development Complete", "Response Development", "Ready for UAT", Done) on 2022-10-18 AND resolution is EMPTY ORDER BY status DESC, resolved DESC

Change date:

Change issue type:

Base JQL: Project in (IQIESRHAP, IQIESDUNE, IQIESPAEN, IQIESFACT, IQIESFRT, IQIESHCD, IQIESMCS, IQIESOUT, IQIESPAR, IQIESPB, IQIESQMIRD, IQIESSCR, IQIESSVD, IQIESNOOP, IQIESSTG, IQIESPATM, IQIESSHOP, IQIESWOOD) AND issuetype in (Story, "Action Item", Improvement, Insight, "Prb/Enh") AND status was in ("Analyzing (SAFe)", "BA Review", Backlog, Brainstorming, Committed, "Feature Definition", Funnel, "On Hold", New, "New Story", Open, "PI Ready", "Ready for Refinement", "Ready to Estimate", "Refinement Backlog", "Selected for Action", "Story Definition", "To Do", "Ready for Peer Review", "Selected for PI", Sized, "Pending CMS Approval", "Selected for Development", "Ready for PI", Pending, "Feature Refinement", "Feature Deep Dive", "Ready to Work", "In Refinement", Draft, Requested, "Not Started", Unassigned, Intake, Accepted, Analysis, Analyzing, "Analyzing/Planning", "Awaiting Response", Blocked, "Blockers/Issues", "Client Review", "CMS Approving", Deferred, Deployed, "Dev Review", Discovery, Implementing, "In Analysis", "In Development", "In Progress", "In QA Test", "In Review", "In UAT Testing", "Initial Write-Up", "Internal Review", Monitoring, "Peer Review", "Ready for Acceptance", "Ready for Deployment", "Ready for Peer Review", "Ready to Test", Refining, "Test Writing", Testing, "Under Test (SBits)", UX, Development, Planning, Review, "Coding Review", "Ready for Test", Design, Prototyping, Ready, "In Test", "Mitigation in Progress", "User Stories Ready", "Ready for Testing", "Next", Validating, "Quality Review", "Ready for Release", "Realized Risk", "PR Review", "Ready for Validation", Ideation, "CMS Review", "Ready for Dev", Collaborating, Socializing, Recurring, "In Discovery", Evaluation, "Waiting for Decision", "Code Review", "Ready to Start", "DevOps In Progress", "Ready for PROD", "Ready for Delivery", Decided, Barrier, "Create User Stories", "Work in Progress", "Classify Response", "Response Completed", "Testing Complete", "In Definition & Refinement", "Design In Progress", "Ready for PO Design Review", "User Research Prep", Operational, Research, "Pending with Media Team Admins", "Dev Complete/Follow Up", "Dev Complete", "Pending with Division", "Pending with SOG Leadership", "Feature / Stories Needed", "Development Complete", "Response Development", "Ready for UAT", Done) on 2022-10-18 AND resolution is EMPTY ORDER BY status DESC, resolved DESC

Change date:

Change issue type:

Report Name: IQies Project data in "All ART Teams"

Dashboard: iQIES All ART Teams dashboard

Legend: JQL/Easy BI

Validation Date: 9/08/2022

Validator: Randolph Anagho

JQL/Dashboard 

Report Link: PI 4 All ART Teams IQIES Test Report - Issues - LACE Metrics Development - eazyBI (cms.gov)

Jira Query

I call this acceptable, although I am curious why the StDev jumped up to around 5% starting in PI 17. The high diffs are small issue counts and can be researched

Report Name: Feature Cumulative Flow

Dashboard: CCSQ Program Dashboard

Validation Date: 10/27/2022

Validator: Seth Evans

Program: HQR

Legend: JQL/Easy BI

JQL (change status as needed)

10/26/22 - Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND Status = "Refinement Backlog"

4/29/22 - Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND status was "PI Ready"  ON endOfDay(-181)

11/10/21 - Project in ("HQR Program Project", "HQR UX-Men") AND issuetype = 10000 AND status was "PI Ready"  ON endOfDay(-351)

**Modified the available statuses to only use the statuses that the 3 programs are using (removed the groupings that were used)