ISG Playbook
APIs
Standard deviations of less than 10 percent can result from the data being one day stale, or in edge cases where an issue was moved in and out of a status multiple times. Greater deviations require investigation
| Standard Deviation | ||
|---|---|---|
| Green | +/- 5% | Accurate |
| Orange | +/- 5% to +/- 10% | Acceptable |
| Red | Above +/- 10% | Requires Triage* |
*triage required if the delta between Jira Validation Query and the report both numerically and proportionally large. Examples of non-triaged counts with greater than than +/-10%
6/5 has a delta of one issue, but is greater than the acceptable standard deviation
Validation Date: 8/15/2022
Validator: Marc Santini
Validation Notes:
- Issues with status changes to implementing were not being caught because the earlier status was being returned by the validation query. Changing the time from (YYYY-M(M)-D(D)) to ('YYYY-M(M)-D(D) HH:MM'), with "HH:MM" being "23:59" catches the status at 11:59 PM. This brought all counts well under the "Accurate" STD category
| iQies | PI3 | PI4 | PI5 | PI6 | PI7 |
|---|---|---|---|---|---|
| Funnel | 160/156 | 179/177 | 175/173 | 199/198 | 299/300 |
| Feature Definition | 1/1 | 3/3 | 5/5 | 6/6 | 4/4 |
| Feature Refinement | 3/3 | 5/5 | 4/4 | 3/3 | 1/1 |
| Feature Deep Dive | 8/8 | 11/11 | 4/4 | 18/18 | 10/10 |
| Story Definition | 84/85 | 92/92 | 54/54 | 68/68 | 52/52 |
| PI Ready | 68/68 | 59/59 | 84/84 | 80/80 | 82/82 |
| Implementing | 57/56 | 68/68 | 69/69 | 50/50 | 50/50 |
| EQRS | PI 14 | PI 15 | PI 16 | PI 17 | PI 18 |
|---|---|---|---|---|---|
| Funnel | 21/21 | 125/125 | 107/107 | 84/84 | 100/100 |
| Analyzing (SAFe) | 0/0 | 0/0 | 3/3 | 3/3 | 3/3 |
| Refining | 16/16 | 13/13 | 14/13* | 8/8 | 6/5* |
| Ready For PI | 0/0 | 9/9 | 17/17 | 21/22 | 25/25 |
| Implementing | 5/5 | 12/12 | 25/25 | 29/29 | 36/39 |
Validation Date : 8/11/2022
Validator: Marc Santini
| iQies | PI3 | PI4 | PI5 | PI6 | PI7 |
|---|---|---|---|---|---|
| Funnel | 160/156 | 179/177 | 175/173 | 199/198 | 299/302 |
| Feature Definition | 1/1 | 3/3 | 5/5 | 6/6 | 4/4 |
| Feature Refinement | 3/3 | 5/5 | 4/4 | 3/3 | 1/1 |
| Feature Deep Dive | 8/8 | 11/11 | 4/4 | 18/18 | 17/10 |
| Story Definition | 84/85 | 92/92 | 54/70 | 68/73 | 52/55* |
| PI Ready | 68/68 | 59/65 | 84/99 | 80/82 | 82/88 |
| Implementing | 57/59 | 68/72 | 69/75 | 50/50 | 50/50 |
| EQRS | PI 14 | PI 15 | PI 16 | PI 17 | PI 18 |
|---|---|---|---|---|---|
| Funnel | 21/21 | 125/125 | 107/107 | 84/84 | 100/102 |
| Analyzing (SAFe) | 0/0 | 0/0 | 3/3 | 3/3 | 3/3 |
| Refining | 16/16 | 13/13 | 14/13* | 8/8 | 6/5* |
| Ready For PI | 0/0 | 9/9 | 17/17 | 21/22 | 25/27* |
| Implementing | 5/5 | 12/12 | 25/25 | 29/33 | 36/38* |
Validation Date:
Validators: Randolph Anago, Marc Santini
Validation notes:
| Types | iQIES | EQRS |
|---|---|---|
| Infrastructure Enabler | 1/1 | 0/0 |
| Architectural Enabler | 1/1 | 0/0 |
| Exploration Enabler | 1/0 | 19/22 |
| Compliance Enabler | 2/2 | 0/0 |
| Enabler | 91/90 | 8/8 |
| Features | 499/426 | 130/62 |
Validation Date: 08/12/2022
Validator: Randolph Anagho
| Types | iQIES | EQRS |
|---|---|---|
| Infrastructure Enabler | 1/1 | 0/0 |
| Architectural Enabler | 1/1 | 0/0 |
| Exploration Enabler | 1/0 | 19/19 |
| Compliance Enabler | 2/2 | 0/0 |
| Enabler | 91/90 | 8/8 |
| Features | 499/426 | 130/62 |
Report Name: Features Started and Closed per PI
Legend: EazyBI count/JQL count
Validation Date : 8/17/2022
Validator: Marc Santini
Because The eazybi measure was using dateBetween with the actual PI start and end dates, it was not picking up items changed to the Implementing or Done statuses on the defined PI Start and End dates, because it was only looking for items in between those dates, not including them I updated the query to check After the the last minute of the day before the PI, and Before the first minute of the day following the PI. I updated the Eazybi measure and then changed the Jira query to use the BEFORE and AFTER operators rather than the DURING and used matching timestamps in both the measure and the query. I suspect that the DURING operator in Jira is not as high resolution as EazyBi's Datebetween function.
I will create two stories to investigate the anomalous counts for iQIES PI 6 implementing, and EQRS PI 18 Done
Updated DateBetween function: [Time].[Day].DateMembersBetween('Jan 18 2022 23:59', 'Apr 13 2022 00:01')
Updated Query: Project = EQRS AND issuetype = 10000 AND Status changed to Done AFTER ("2022-01-19 00:01") AND Status changed to Done BEFORE ("2022-04-13 00:01")
| iQies | PI3 | PI4 | PI5 | PI6 | PI7 |
|---|---|---|---|---|---|
| Implementing | 104/104 | 129/132 | 123/122 | 123/113 | 83/85 |
| Done | 196/196 | 132/135 | 97/97 | 115/115 | 16/17* |
| EQRS | PI 14 | PI 15 | PI 16 | PI 17 | PI 18 | PI 19 |
|---|---|---|---|---|---|---|
| Implementing | 16/16 | 30/30 | 36/36 | 32/32 | 21/21 | 16/16 |
| Done | 10/10 | 25/25 | 32/32 | 33/32 | 33/29 | 19/19 |
Validation Date : 8/16/2022
Validator: Seth Evans
| iQies | PI3 | PI4 | PI5 | PI6 | PI7 |
|---|---|---|---|---|---|
| Implementing | 104/104 | 129/132 | 105/122 | 199/198 | 73/83 |
| Done | 196/196 | 132/135 | 93/96 | 106/113 | 16/16 |
Project = "iQIES Program Board " AND issuetype = 10000 AND Status changed to Done during (2022-01-19, 2022-04-19) ORDER BY key DESC
| EQRS | PI 14 | PI 15 | PI 16 | PI 17 | PI 18 | PI 19 |
|---|---|---|---|---|---|---|
| Implementing | 16/16 | 30/30 | 36/36 | 27/29 | 18/21 | 16/16 |
| Done | 10/10 | 25/25 | 32/32 | 27/31 | 27/29 | 19/19 |
Project = " End-Stage Renal Disease Quality Reporting System" AND issuetype = 10000 AND Status changed to Done during (2021-04-04, 2021-07-20) --Update status and dates of PI
Report Name: Work Delivered by Category per PI
Dashboard: iQIES program dashboard
Legend: EazyBI count/JQL count
Validation Date: 8/31/2022
Validator: Seth Evans
iQIES | PI3 | PI4 | PI5 | PI6 | PI7 | ||||||||||
| Chart | JQL | % Diff | Chart | JQL | % Diff | Chart | JQL | % Diff | Chart | JQL | % Diff | Chart | JQL | % Diff | |
| Total | 1901 | 1880 | 1.10% | 1876 | 1894 | -0.96% | 1820 | 1840 | -1.10% | 1830 | 1832 | -0.11% | 810 | 830 | -2.47% |
| Story | 1262 | 1245 | 1.35% | 1294 | 1315 | -1.62% | 1431 | 1450 | -1.33% | 1320 | 1325 | -0.38% | 621 | 638 | -2.74% |
| Spike | 222 | 222 | 0.00% | 212 | 209 | 1.42% | 155 | 158 | -1.94% | 137 | 134 | 2.19% | 63 | 65 | -3.17% |
| Bug | 417 | 413 | 0.96% | 370 | 370 | 0.00% | 234 | 232 | 0.85% | 376 | 370 | 1.60% | 126 | 127 | -0.79% |
JQL Query:
| PI3 | Project in ("iQIES Enigma", "iQIES HCD", "iQIES Frontiers", "iQIES Bohemian Rhapsody (DevOps)", "iQIES Dune (CLIA)", "iQIES FACT (Data and Backend)", "iQIES Outlander", "iQIES PA Reports", "iQIES QMIR Data", "iQIES SAS Viya / Dashboards", "iQIES S&C Reports", "IQIES SNOOPY", "iQIES Stargate", "iQIES Immeasurable", "iQIES The Misfits", "iQIES The Shop", "IQIES Woodstock") AND issuetype in (Story, Spike, Bug) AND resolutiondate >= 2021-07-07 AND resolutionDate <= 2021-10-12 |
| PI4 | Project in ("iQIES Enigma", "iQIES HCD", "iQIES Frontiers", "iQIES Bohemian Rhapsody (DevOps)", "iQIES Dune (CLIA)", "iQIES FACT (Data and Backend)", "iQIES Outlander", "iQIES PA Reports", "iQIES QMIR Data", "iQIES SAS Viya / Dashboards", "iQIES S&C Reports", "IQIES SNOOPY", "iQIES Stargate", "iQIES Immeasurable", "iQIES The Misfits", "iQIES The Shop", "IQIES Woodstock") AND issuetype in (Story, Spike, Bug) AND resolutiondate >= 2021-10-13 AND resolutionDate <= 2022-01-18 |
| PI5 | Project in ("iQIES Enigma", "iQIES HCD", "iQIES Frontiers", "iQIES Bohemian Rhapsody (DevOps)", "iQIES Dune (CLIA)", "iQIES FACT (Data and Backend)", "iQIES Outlander", "iQIES PA Reports", "iQIES QMIR Data", "iQIES SAS Viya / Dashboards", "iQIES S&C Reports", "IQIES SNOOPY", "iQIES Stargate", "iQIES Immeasurable", "iQIES The Misfits", "iQIES The Shop", "IQIES Woodstock") AND issuetype in (Story, Spike, Bug) AND resolutiondate >= 2022-01-19 AND resolutionDate <= 2022-04-19 |
| PI6 | Project in ("iQIES Enigma", "iQIES HCD", "iQIES Frontiers", "iQIES Bohemian Rhapsody (DevOps)", "iQIES Dune (CLIA)", "iQIES FACT (Data and Backend)", "iQIES Outlander", "iQIES PA Reports", "iQIES QMIR Data", "iQIES SAS Viya / Dashboards", "iQIES S&C Reports", "IQIES SNOOPY", "iQIES Stargate", "iQIES Immeasurable", "iQIES The Misfits", "iQIES The Shop", "IQIES Woodstock") AND issuetype in (Story, Spike, Bug) AND resolutiondate >= 2022-04-20 AND resolutionDate <= 2022-07-19 ORDER BY status DESC |
| PI7 | Project in ("iQIES Enigma", "iQIES HCD", "iQIES Frontiers", "iQIES Bohemian Rhapsody (DevOps)", "iQIES Dune (CLIA)", "iQIES FACT (Data and Backend)", "iQIES Outlander", "iQIES PA Reports", "iQIES QMIR Data", "iQIES SAS Viya / Dashboards", "iQIES S&C Reports", "IQIES SNOOPY", "iQIES Stargate", "iQIES Immeasurable", "iQIES The Misfits", "iQIES The Shop", "IQIES Woodstock") AND issuetype in (Story, Spike, Bug) AND resolutiondate >= 2022-07-20 AND resolutionDate <= 2022-10-11 ORDER BY status DESC |
Report Name: Work Delivered by Category per PI
Dashboard : Enterprise Program Dashboard
Legend: EazyBI count/JQL count
Validation Date: 9/7/22
Validator: Marc Santini
iQIES | PI3 | PI4 | PI5 | PI6 | PI7 | ||||||||||
| Chart | JQL | % Diff | Chart | JQL | % Diff | Chart | JQL | % Diff | Chart | JQL | % Diff | Chart | JQL | % Diff | |
| Total | 1901 | 1880 | 1.10% | 1876 | 1894 | -0.96% | 1820 | 1840 | -1.10% | 1829 | 1832 | -0.11% | 810 | 830 | -2.47% |
| Story | 1262 | 1245 | 1.35% | 1294 | 1315 | -1.62% | 1431 | 1450 | -1.33% | 1320 | 1325 | -0.38% | 621 | 638 | -2.74% |
| Spike | 222 | 222 | 0.00% | 212 | 209 | 1.42% | 155 | 158 | -1.94% | 134 | 134 | 0% | 63 | 65 | -3.17% |
| Bug | 417 | 413 | 0.96% | 370 | 370 | 0.00% | 234 | 232 | 0.85% | 375 | 370 | 1.60% | 126 | 127 | -0.79% |
iQies numbers improved or stayed the same in all cases!
| EQRS | PI 15 | PI 16 | PI 17 | PI 18 | PI 19 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Report | JQL | StDev | Report | JQL | StDev | Report | JQL | StDev | Report | JQL | StDev | Report | JQL | StDev | |
Total | 700 | 699 | .02% | 731 | 733 | .07% | 858 | 817 | 4.8% | 781 | 749 | 4.6% | 478 | 505 | 5.3% |
Story | 567 | 567 | 0% | 523 | 527 | .08% | 693 | 661 | 4.7% | 653 | 629 | 3.8% | 365 | 386 | 5.4% |
Bug | 131 | 130 | 08% | 193 | 191 | 1.1% | 154 | 149 | 3% | 117 | 109 | 6.8% | 107 | 113 | 5.3% |
Spike | 2 | 2 | 0% | 8 | 8 | 0% | 6 | 3 | 50% | 7 | 7 | 0% | 2 | 2 | 0% |
| Enabler | 0 | 0 | 0% | 7 | 7 | 0% | 4 | 4 | 0% | 6 | 4 | 33.3% | 4 | 4 | 0% |
I call this acceptable, although I am curious why the StDev jumped up to around 5% starting in PI 17. The high diffs are small issue counts and can be researched
| PI5 | Project in (EQRS, EQRSCI, EQRSD, EQRSARCH, EQRSEDSM, EQRSONBRD, EQRSMD, EQRSMS, EQRSPRTL) AND issuetype in(Story, Spike, Bug, Enabler) AND resolutiondate >= 2021-07-20 AND resolutionDate <= 2021-10-12 |
| PI6 | Project in (EQRS, EQRSCI, EQRSD, EQRSARCH, EQRSEDSM, EQRSONBRD, EQRSMD, EQRSMS, EQRSPRTL) AND issuetype in(Story, Bug, Spike, Enabler) AND resolutiondate >= 2021-10-13 AND resolutionDate <= 2022-01-18 |
| PI7 | Project in (EQRS, EQRSCI, EQRSD, EQRSARCH, EQRSEDSM, EQRSONBRD, EQRSMD, EQRSMS, EQRSPRTL) AND issuetype in(Story, Spike, Bug, Enabler) AND resolutiondate >= 2022-01-19 AND resolutionDate <= 2022-04-12 |
| PI8 | Project in (EQRS, EQRSCI, EQRSD, EQRSARCH, EQRSEDSM, EQRSONBRD, EQRSMD, EQRSMS, EQRSPRTL) AND issuetype in(Story, Spike, Bug, Enabler) AND resolutiondate >= 2022-04-13 AND resolutionDate<= 2022-07-05 |
| PI9 | Project in (EQRS, EQRSCI, EQRSD, EQRSARCH, EQRSEDSM, EQRSONBRD, EQRSMD, EQRSMS, EQRSPRTL) AND issuetype in(Story, Spike, Bug, Enabler) AND resolutiondate >= 2022-07-06 AND resolutionDate<= 2022-09-27 |