S519: Evaluation of Information Systems

1. S519: Evaluation of Information Systems Meta-evaluation D-Ch11

2. Last week • What are these checkpoints? • How to put your result together? • What are lessons-learned during the exercise?

3. Meta-evaluation • It is evaluation of an evaluation (Scriven, 1991) to determin the quality and/or value of an evaluation

4. Five criteria • Validity • Utility • Conduct • Credibility • Costs

5. Validity • To what extent, the conclusions of the evaluation are justified • Check carefully the facts they use and values they assign • Question everything • Constraints (budget, time, etc.) • Using KEC checkpoints • Using standards (www.wmich.edu/evalctr/jc/)

6. Validity • Covers all relevant sources of value • Comprehensively covers process, outcome and cost • Include no irrelevant or illicit criteria • Data used to directly address the criteria • Include analyses that are appropriate for the data • Clearly states how data are interpreted • Is clear about where evaluative conclusions come from • Include valid recommendation

7. Utility • Are the findings • Relevant to the questions or decisions being faced by the audience • Timely • Clearly communicated • Cost effective

8. Conduct • Legal • Ethical • Professional standards • Cultrual appropriateness • Unobtrusiveness • Minimal disruptive to the evaluand

9. Credibility • Familiar with the context • Independence, impartiality, and/or lack of conflict of interest • Expertise in evaluation and in the evaluation subject field

10. Cost • How reasonable the costs of an evaluation

11. Meta-evaluation rating table • Table 11.1

12. Exercise • D-p217, exercise 2 • Form a pair, do the evaluation • Form a group to select the best result, • Design a poster • Present them in a poster • Using three sentenses to summarize your evaluation • What you learn • What you feel • How to improve