Human Resource Development: Principles and Practice

CHAPTER 13: EVALUATION 13.1 • MISCONCEPTIONS ABOUT EVALUATION • ASSESSMENT OF LEARNING • KIRKPATRICK’S FOUR LEVELS • THE PRESAGE FACTORS • THE SCIENTIFIC MODELS • COST-BENEFIT ANALYSIS • THE EVALUATION PLAN • THE EVALUATION REPORT • THE LEGITIMATE SYSTEM • THE SHADOW SYSTEM • THE NEED FOR DIALECTIC THINKING Human Resource Development: Principles and Practice By Brian Delahaye

MISCONCEPTIONS ABOUT EVALUATION 13.3 While evaluation is seen as professionally and theoretically desirable, some have doubts about its worth. 1. It is too costly. This misconception is based on a very short-term view of costs. The cost of risk is also part of the equation. These risks include not knowing if mistakes were made and not knowing how to avoid these mistakes in the future. 2. The measures are not exact. This misconception does not recognise that any complex decision is based on less than perfect information. 3. It is too difficult. True, most evaluation recommendations apply more obviously to the legitimate system. However, the developmental efforts of the shadow system can be evaluated. Human Resource Development: Principles and Practice By Brian Delahaye

THE ROLE OF EVALUATION 13.4 The role of the evaluation stage is four-fold: 1. to measure what change has occurred 2. to improve the other three stages —investigation, design and implementation 3. to see if the change is attributable to the learning episode 4. to see if the amount of change is worthwhile. Human Resource Development: Principles and Practice By Brian Delahaye

ASSESSMENT OF LEARNING 13.6 • Assessment addresses the question ‘What changes have occurred in the mind of the learner as a result of the learning episode?’. To measure this change, the learning has to be converted into a behaviour that can be observed and this has some weaknesses: • The behaviour is only a sample of the learning. • The behaviour represents explicit knowledge only. • The learning generation processes may take some time. • The test has to tap into the appropriate knowledge. • The test has to tap into the same potential knowledge each time. • There are several issues that need to be considered when planning assessment. Human Resource Development: Principles and Practice By Brian Delahaye

TYPES OF ASSESSMENT 13.7 • Assessment can be divided into six types: • skill tests — are used for procedural skills • objective written tests — e.g. multiple choice • subjective written tests — e.g. essays • performance tests — used to assess complex processes. The examiner may asses the process itself or the product often using a pre-designed observation form (see Fig.13.1 on page 346). • learning diaries — Journals written by the learner; good for assessing externalisation • portfolio assessment — collections of the learner’s work; takes a long term view, focuses on quality and the learner reflects on own work. Human Resource Development: Principles and Practice By Brian Delahaye

THE MEANING OF SCORES 13.8 • When a test has been examined and given a quantitative score the result is called a raw score. • This raw score can be interpreted in a number of ways: • criterion-referenced scoring — the learner’s score is compared to a ‘pass’ mark • norm-referenced scoring — the raw score is compared to the average of a nominated group • formative assessment — the score is used for developmental feedback • summative assessment — the score is used for evidence of learning. Human Resource Development: Principles and Practice By Brian Delahaye

OVERLAP OF THE FOUR HRD PROCESSES 13.9 The assessment of learning plays a dual role — being critical to both the implementation stage and the evaluation stage of HRD. In the implementation stage, assessment provides dynamic feedback. Firstly, this feedback is often immediate. Secondly, it provides irrefutable evidence that the learner finds difficult to ignore. In the evaluation stage, assessment provides initial evidence of the success or otherwise of the learning experience. This feedback may be taken into account immediately (e.g. the HR developer changes the learning processes) or later (e.g. recommendations in the Evaluation Report). Human Resource Development: Principles and Practice By Brian Delahaye

KIRKPATRICK’S FOUR LEVELS 13.11 Kirkpatrick’s model is based on four levels of ascending order: 1. reaction —measures the reactions of the learners to the learning episode 2. learning —measures the knowledge gained by the learners. Has been discussed under ‘Assessment of Learning’. 3. behaviour —examines the change of behaviour of the learner on the job 4. results —measures the impact of the learning on the organisation as a whole Several comments can be made about Kirkpatrick’s model. Human Resource Development: Principles and Practice By Brian Delahaye

COMMENTS ON KIRKPATRICK’S MODEL 13.12 • The strengths of Kirkpatrick’s model include: • simple and seemingly robust model • recognises the impact of a learning episode should extend beyond the individual learner • provides an easily remembered checklist • each level provides a unique examination of HRD. • The criticisms of Kirkpatrick’s model include: • the levels are not co-dependent • the model concentrates on processes and outputs, not inputs • the initial level is easier but measurement of the deeper levels are more susceptible to contamination. Human Resource Development: Principles and Practice By Brian Delahaye

THE PRESAGE FACTORS 13.14 Brinkerhoff’s model proposes six levels: Stage I:Evaluate needs and goals ( this is similar to the HRDNI). Stage II: Evaluate the HR design Stage III: Evaluate implementation Stage IV: Evaluate learning (similar to Kirkpatrick’s level 1) Stage V: Evaluate usage and endurance of learning (similar to Kirkpatrick’s level 3) Stage VI: Evaluate pay-off (similar to Kirkpatrick’s level 4). Brinkerhoff, then, adds two that occur prior to the learning episode —evaluate HR design and evaluate implementation. These are called presage factors. Human Resource Development: Principles and Practice By Brian Delahaye

THE PRESAGE FACTORS (Continued) 13.15 • Stage II, evaluating the HR design focuses on the quality of the HR plan (i.e. the training or workshop program). • A stage II evaluation should occur when: • the design is unique or experimental • the costs of mounting the program are high • the HRD issues are crucial • the participant groups are volatile, influential or demanding. • The design should be reviewed by a variety of stakeholders. Human Resource Development: Principles and Practice By Brian Delahaye

THE PRESAGE FACTORS (Continued) 13.16 Stage III, evaluating during the implementation provides data that will help ‘shepherd’ the learning event to a successful conclusion. This highlights the covert activity of the conducting HR developer where the program is continually monitored to compare reality with the program plan. The HR developer can also be monitored, usually by another HR developer. However, the ‘ big brother’ syndrome has to be avoided. Human Resource Development: Principles and Practice By Brian Delahaye

TIME OUT 13.17 To this stage of the discussion, we have examined assessment of learning, Kirkpatrick’s model and Brinkerhoff’s model. This has answered two of the roles of evaluation: 1. to identify what change has occurred 2. to improve the other three stages. The other two roles will now be addressed: 3. to see if the change is attributable to the learning episode. This will be examined by looking at the SCIENTIFIC MODELS. 4. to see if the amount of change was worthwhile. This will be examined by looking at COST BENEFIT ANALYSIS. Human Resource Development: Principles and Practice By Brian Delahaye

THE SCIENTIFIC MODELS 13.19 • The scientific models are based on the experimental methods used in research laboratories and are used to demonstrate causality. They also assume that changes can be measured. • Post-test = learning experience/evaluation • Pre-test - Post-test = evaluation/learning experience/evaluation • Time series evaluation = pre-test/pre-test/pre-test/learning experience/post-test/post-test/post-test • Control group = • Experimental group —pre-test/learning experience/post-test • Control group — pre-test/usual duty/ post-test • The Solomon four uses three control groups and the experimental group • The HR developer has to compare the costs of the designs compared to the expected benefits gained. Human Resource Development: Principles and Practice By Brian Delahaye

COST-BENEFIT ANALYSIS 13.21 • Cost-benefit analysis (CBA) is used to see if the amount of change was worthwhile. It is based on the premise: • identify the costs, in dollar terms, for the learning experience • identify the the benefits accruing from the learning experience, in dollar terms • the ratio between the costs and benefits should be in favour of the benefits. • There are two problems often encountered: • converting the benefits into monetary values • deciding on the cut-off points. • There are two good reasons for conducting a CBA. Firstly, the survival of the HR section may depend on proving the value of development. Secondly, it helps in the decision of selecting the learning programs that have the most impact. Human Resource Development: Principles and Practice By Brian Delahaye

THE EVALUATION PLAN 13.23 • Planning for evaluation commences during the design stage. This ensures that the appropriate evaluation occurs at the appropriate time. It also allows the coordination of developmental and judgemental evaluation. Planning or evaluation should include: • develop the assessment for developmental purposes first • incorporate this into the evaluation plus any further assessment required • decide what presage variables will be evaluated and when • incorporate appropriate HRDNI investigation instruments • design daily and course/workshop reaction sheets • design pre-test and post-test instruments, if appropriate • identify and plan methods to be used at Stage V and VI • if CBA is to be used, arrange collection of costs and benefits • prepare a budget • send evaluation plan to staff who are affected. Human Resource Development: Principles and Practice By Brian Delahaye

THE EVALUATION REPORT 13.25 • Once the evaluation has been completed a report should be completed for the key stakeholders. The evaluation plan has a communication role, decision making role and becomes an historical document. This report should include: • an executive summary • a findings/recommendations section • a contents list • the main body, which should include: • reasons for the evaluation • list of personnel involved • discussion of the various types of evaluation undertaken • a discussion on the findings, the options and the conclusions • a list of, and discussion of, the recommendations • appendices. Human Resource Development: Principles and Practice By Brian Delahaye

THE LEGITIMATE SYSTEM 13.27 Most of what has been discussed in this chapter applies directly to the legitimate system. The methods mainly rely on negative feedback systems. The methods also emphasise the legitimate systems focus on the efficient use of resources and the immediate survival of the organisation. For the legitimate system, one of the advantages of competency-based learning has been the recognition of the strong link between learning in a planned program and the application of that learning in the workplace. It should also be recognised that, if the assessment process fails to detect a lack of comprehension on the part of the learners, the HR developer could be held liable. Human Resource Development: Principles and Practice By Brian Delahaye

THE SHADOW SYSTEM 13.29 Evaluation in the shadow system is a very delicate affair. The use, by a third party, of the methods described in this chapter results in negative feedback loops — and this kills the creativity needed in the shadow system. The main evaluation method for the shadow system is the use of values systems. The manger instills a particular value in the system and checks for evidence for this value. The actors in the shadow system, however, should use the evaluation methods on themselves and their own endeavours. Human Resource Development: Principles and Practice By Brian Delahaye

THE NEED FOR DIALECTIC THINKING 13.31 • Some of the misperceptions about evaluation can be explained by the concept of dialectic thinking (see Chapter 2). • Evaluation has a number of opposing characteristics: • evaluation can be both developmental and judgemental • the more objective the measure used, the less rich the insights — and vice versa • liberal education vs. vocational education. • Evaluation serves and supports a number of dual roles. Balancing these conflicting roles is the new challenge as knowledge is a valuable but delicate resource. Human Resource Development: Principles and Practice By Brian Delahaye

Human Resource Development: Principles and Practice