Appraisal Assistant Workshop

1. Appraisal Assistant Workshop Evidence collection (CMMI Appraisals)

2. Overview • Terms and definitions • Evidence related screens • Getting evidences into the database • Importing a list of documents • Using spreadsheet to collect initial PIIs • Using evidence registry to collect PIIs • Characterize PIIs • Using cross-instantiation evidence panel to collect interview notes • Reviewing evidence coverage on model elements • Transfer evidence data between appraisal projects Workshop exercise setup: project : CMMI cont, with RD/PI on target CL5, with instantiation: Org.Process and PSP wizard

3. Using Evidence Registry – Terms and definitions • Single Instantiation Evidence: Evidences that are owned by ONE instantation • Project plan, design spec, test cases etc • Can be entered through Single Instance spreadsheet, or Evidence registry • Multi-Instantiation Evidences:Evidences that are related to more than one instantiations • Appraisal interviews, organization wide standards, policies • Sometimes referred as Cross-instantiation evidences in user manual • Can be entered through Multi-Instance spreadsheet, or Evidence registry, or cross-instantiation evidence panel • Evidence and PII: • Evidence: the individual record of observations, documents from the organization unit • PII: A piece of evidence is mapped to a model practice to indicate the implementation of that model practice.

4. Evidence Related Screens – evidence registry panel • Evidence list • Evidence detail Info • Relate evidence to model practice • Comment and characterized how evidence is mapped to the practice • Evidence coverage based on the 50% affirmation rule

5. Evidence Screen – cross-instantiation evidence panel • Evidence List • Evidence notes can be related to multiple instantiations

6. Evidence Related Screen – practice characterization tab • Evidence grouped by Instantiation • Can not add/remove evidence in the tab • Can modify indicator type/PII characterization and PII comments

7. Importing a list of documents - 1 • Documents are NOT mapped to any practices (unlike the initial PII spreadsheets, discuss later) • Use the included DocumentList.XLS to ensure correct file format • Can be extracted from the OU’s document database • Instantiation name must be the same as defined in appraisal scope • The row will be ignored if the Appraisal Assistant can not find a matched instantiation in the appraisal scope • Save the Evidence list tab as TAB delimited Text before importing

8. Importing a list of documents - 2 • 3 documents of 2 instantiations in this example • Not mapped to any practice • Check error log for rows that are ignored after importing

9. Importing a list of documents - 3 Result of importing the previous document list

10. Using Spreadsheet to Collect Initial PIIs - 1 • Appraisal Assistant generates one spreadsheet for each instantiation in the scope and an additional spreadsheet for multi-instantiation evidences • Each evidence can be mapped to multiple practices • Each evidence can be mapped to the same practice once and only once • One row per PII, insert rows for more PII • Leave the row EMPTY if there is NO PII for that practice • User ensures data accuracy and consistency • XML spreadsheets can be opened by Excel 2002 or newer • Save as ‘Tab Delimited Text’ file before being imported • Program merges evidences only if they have the same name AND the same document link strings

11. Using Spreadsheet to Collect Initial PIIs - 2 • 3 evidences here, associated to 3 practices • Single instantiation spreadsheet, instantiation name in the top most cell • Detail description of process areas and practices in cell comments

12. Using Spreadsheet to Collect Initial PIIs - 3 • 3 evidences imported from the previous spreadsheet • All 3 evidences belong to the same instantiation

13. Using Spreadsheet to Collect Initial PIIs - 4 • 3 multi-instantiation evidences here • MI evd 1 is mapped to: PI/SP1.1 - Org.Processes/Test project 1, PI/SP1.2 - Org.Processes/Project 2 • MI evd 2 is mapped to: PI/SP1.1 - Project 2 , PI/SP1.3 - Test project 1 • MI evd 3 is mapped to: PI/SP1.1 - Test project 1/Project 2

14. 3 evidences imported from the previous spreadsheet All imported evidences are labelled as Multi-Inst Using Spreadsheet to Collect Initial PIIs -5

15. Exercise • Import 2 evidences of Org.Processes from a spreadsheet • Evidence 1: • Name: Project plan • Document Link: www.sqi.gu.edu.au • Associated to: RD-GP 2.1, RD-GP2.2 as Direct artifact • Evidence 2: • Name: Test plan • Document Link: www.cit.gu.edu.au • Associated to: RD-GP 2.2, RD-GP 2.4 as Indirect artifact

16. Using Evidence Registry - Overview

17. Using Evidence Registry – single instantiation evidence 2 3 4 5 • Click Add Button (Ctrl-A) to add a new record • Enter the Name of the evidence • Specify the instantiation • Associate it to the practice(s) • Specify the type of the PII • Comment on why/how the evidence is related to the practice 1 6

18. Exercise • Add 1 evidence to the registry • Name: Design Review Record • Instantiation: Org.Processes • Associated to: • RD – GP2.1 as Direct Artifact, comment: bah! • RD – GP2.2 as Indirect Artifact, comment: doh!

19. Using Evidence Registry – multi-instantiation evidences 2 3 4 5 • Click Add button to add a new a new record • Name the new evidence record • Select Multi-Inst from the instantiation column • Locate the practice and the practice-instantiation, Tick the check box to map the evidence • Specify the Indicator Type • Comments on how/why the evidence is considered a PII 1 6

20. Using Evidence Registry – Additional evidence information • Only evidence name and instantiation are mandatory • Evidence File: Link to a file or web page • Label: short string defined by appraisal team to identify evidence • Source/Affirmation: the owner of the evidence • Last Modified by /Last Modified on: auto fields

21. Using Evidence Registry - Characterize P.I.I. • Click on the ‘Characteristic’ column of the evidence under the practice • Select the characteristic from the dropdown list • Though non-mandatory, useful for characterizing practice implementation

22. Using Cross-Instantiation Evidence Panel • Evidences covers more than one instantiation • Can be launched from evidence registry and from the main U.I. • Only consists of process areas that are within the appraisal scope • Process areas are grouped by their category

23. Using Cross-Instantiation Evidence Panel • Data will be automatic passed into the evidence registry • If the PII note on an instantiation of a practice is cleared, the evidence is disassociated from the practice on that instantiation • If an evidence is deleted, ALL related interview notes will also be deleted • You can also associate/disassociated multi-instance evidence to a practice through the evidence registry (discussed above)

24. Using Cross-Instantiation Evidence Panel 2 3 • Click Add button to add a new record • Name the evidence • Find the interested practice from the tabs and click on its instantiation column • Comment on how evidence is related to the instantiation of the practice • Specify Indicator Type 1 5 4

25. Exercise • Enter notes for “Interview 2” • PI SP1.2-2 : • Instantiation: Org.Processes • Characteristic: Strength • Notes: “Testing interview 2 comment on PI SP1.2-2, we thing it is a strength” • PI SP 1.3-3: • Instantiation: PSP wizard • Characteristic: Strength • Notes: “Testing interview 2 comment on PI SP 1.3-3, it is a strength”

26. Using Evidence Registry – Review evidence coverage • Automatic calculate affirmation coverage based on the 50% rule • Example indicators /work products are shown in the edit boxes below • BLUE triangle: At least ONE direct artifact from that practice-instantiation insertion, RED triangle: No direct artifact from that practice-instantiation insertion.

27. Using Evidence Registry – Shortcut keys • Ctrl – A : Add new evidence record • F1: Online help (under construction) • F2 – F5 : Expand/Collapse the model hierarchy tree nodes (if the tree has the input focus) • F7 : Locate all evidence-practice associations that are related to the selected evidence • Arrow keys/Tab : Navigate the model hierarchy tree (if the tree has the input focus) • Right click anywhere to bring up the context menu

28. Using Evidence Registry - Icons

29. Transfer Evidence Between Appraisal Projects • Evidences that are out of destination scope are NOT imported • Evidences that are not mapped to the practices are NOT exported • Source appraisal model representation can be different from the destination representation • Import routine does NOT check for duplication with the existing evidences • Import the database once and ONLY once. • Import the database before generating any mini teams

30. Workshop revision history