Valid Research Methods for Evaluating Recognition Programs

1. Valid Research Methods for Evaluating Recognition Programs David J. Cherrington, SPHR, DBA Recognition Professionals International Conference 27 April 2010 Henderson, Nevada

2. Why Evaluate?Significant questions managers ought to ask: • How do our employees feel about our recognition programs? Which ones, if any, are having the biggest impact? • Are supervisors adequately trained to provide meaningful recognition? • Does the involvement of top management make any difference? • Is our annual recognition banquet worth the time and money we spend on it? • How do the people who are not recognized feel about our recognition program?

3. Three Strategic Research Questions: • 1. Have we chosen the best recognition programs? (Selection and Design issues) • Informal programs • Formal programs • 2. How well have we implemented them? (Training, Communication, and Presentation issues) • How much do employees know about them? • How often are they being used? • How do they make employees feel? • 3. Do our recognition programs impact employee attitudes, productivity, or profitability? (Outcomes or Results issues)

4. Valid Research Requires Two Things: • 1. Reliable Data – also called dependent variables or metrics • Research Question: “How do you assess reliability?” • 2. A Valid Design – also called the research method • When should you collect the data? • After the program is implemented • Before and after • Several times before and after • From whom? • Experimental group • Control group • How much? • Total sample • Random sample • Stratified random sample • Research Question: “How do you assess validity?”

5. Reliable Data

6. Reliability • Definition: Reliability means repeatability or consistency of measurement. Reliable data are not random numbers, subjective estimates, or capricious measures. • Operational Definitions: • Test-retest reliability • Alternate forms reliability • Split-halves reliability • Conspect reliability (inter-rater agreement)

7. Dependent Variables or Metrics Employee Attitudes Organizational Metrics Absence Rate Turnover Rate HR Expense Factor Profit per employee Revenue per employee Cost of recognition programs as a percent of payroll • Job satisfaction • Appreciation • Company satisfaction • Organizational commitment • Intent to leave • Perceived organizational support

8. Absence Rate: The absence rate allows organizations to track employee attendance over time or to compare their number of lost days due to absences with the rates of others in their industry and nationwide. Workdays lost due to absences Absence rate = # of employees x # of days Workdays Lost Due to Absence = full days of employee absence whether paid or unpaid, excused or unexcused and includes unscheduled days for sickness, personal business, emergency, family illness or death, disciplinary suspension, and unexcused absences. For long-term absences, count only the first four days. Do not count scheduled time off for holidays, vacations, and other leave.

9. Illustration What is the Absence Rate for a company that has 60 full-time employees and during the past year they had 410 total days of absence. All employees have two weeks of paid vacation during the year. 0.027 or 2.70% 410 days of absence = 60 Employees x 250 days* *250 days = 50 weeks per year times 5 days per week

10. Absence Rates by Size of Company and for selected Industries Data provided by the Utah Employers Council, 2009

11. HR Functions that can be costedWayne Cascio: Costing Human Resources: The Financial Impact of Behavior in Organizations, 3rd ed., Boston: Kent Pub. Co. 1991. • Turnover • Absenteeism and sick leave • Smoking • Employee Assistance programs • Programs to improve Attitudes • Labor Contract Costs: wages, benefits, vacations, overtime, shift differentials, holidays, pensions • Employee Selection Tests • Assessment Centers for Selection • Dollar Value of Job performance • Utility of Training Programs.

12. Criteria for Evaluating Training[From: Ralph F. Catalanello and Donald L. Kirkpatrick, “Evaluating Training Programs: The State of the Art,” Training and Development Journal, 22, 1968, 2-9. ] • Reactionsof participants, e.g. job satisfaction, appreciation, company satisfaction, organizational commitment, engagement. 2. Learningnew knowledge and skills, e.g. test scores using multiple choice, fill in the blank, matching, and essay questions. 3. Behavior changes, e.g. quantity and quality of performance, attendance, job performance. 4. Measurable Results, e.g. organizational metrics.

13. Sources of Data • Questionnaires and surveys • Interviews • Observations • Performance appraisals • Unobtrusive Measures • Archive data • Number of hits on the awards and catalogue websites • Number of supervisors who complete online training • Email responses from family members to say thank-you.

14. Employee attitude Measures

15. Semantic Differential Scales Me at Work – How I feel Most of the Time The average of these semantic differential scales measures a “General Affective Tone.” See, William E. Scott, Jr. “The Development of Semantic Differential Scales as Measures of ‘Morale’,” Personnel Psychology, 20 (1967): 179-198.

16. AppreciationLikert Scales Scales created by David J. Cherrington, Brigham Young University, Provo, UT, 84602

17. Job Satisfaction * The scoring of this item is reversed. From: Brayfield, A. H., & Rothe, H. F. (1951). “An index of job satisfaction.” Journal of Applied Psychology, 35, 307-311.

18. Engagement From May, D., Gilson, R., & Harter, L. (2004). “The psychological conditions of meaningfulness, safety and availability and the engagement of the human spirit at work.” Journal of Occupational and Organizational Psychology. 77, 11-37.

19. Organizational Commitment: Affective Commitment From: Allen, N. J., & Meyer, J. P. (1990). “The measurement and antecedents of affective, continuance and normative commitment to the organization.” Journal of Occupational Psychology, 63, 1-18.

20. Recognition Budget per Employee • The recognition budgets for four companies that have been recognized for outstanding recognition programs are: • A telecommunications company with 36,000 employees budgets 1.0 % of payroll • An international bank with 60,000 employees budgets 0.7% of payroll • A healthcare company with 3,400 employees budgets 0.2 % of payroll • A bank with 23,000 employees budgets $8 mil (which equals $348 per employee).

21. Valid Research Designs

22. Research Designs~ Key X = Experimental treatment or training program O = Observation or measurement R = Random assignment of participants to group/training condition

23. Validity • Definition: Do the results measure what you think they are measuring? Can you interpret the results accurately? • Two types of validity: • Internal validity: is the experimental design sound so that the data can be properly interpreted? • External validity: can the results of this study be generalized (or applied) to other situations?

24. 1. Case Study Design • X O You implement the program and then measure it. • X = independent variable, e.g. • New recognition program • Training program • New incentive program • O = dependent variable • “smile sheets” after a training program • Participation in the peer-to-peer nominations • Measures of job satisfaction or engagement • A very weak but highly popular research design

25. 2. Pretest-Posttest Comparison • O X O • Illustration: Changes in turnover or job satisfaction before and after a new recognition program is initiated. ??? • Problems with this design: • Mortality – some people drop out, usually the dissatisfied. • Maturity – people constantly change during any period of time. • History – unique external events can influence the measures. • Instrument decay – measurement errors and unreliable data. • Sensitizing effect of the pretest – measuring is not a neutral event; people often behave differently after they are measured. • This is a very weak research design.

26. 3. Pretest-Posttest Control Group Design • R O X O • R O -- O • R = random assignment to groups • X = new recognition program, such as a new kit for supervisors to use. • O = number of thank you notes sent to subordinates • This is an excellent research design when it can be implemented.

27. 4. Posttest Only Control Group Design • R X O • R -- O • This design assumes that the two groups are equivalent due to random assignment. • This is an excellent research design that is especially appropriate when pretesting is not possible. • Illustration: testing the effects of a new employee orientation training on participation in the peer-to-peer recognition program.

28. 5. Solomon 4-group Design • R O X O R O – OR - X OR - - O • This is the ideal research design because it controls for all of the challenges to the internal validity of a research design. • Illustration: testing the impact of a training program on “Developing a Culture of Recognition” on employee attitudes, such as empowerment, satisfaction, or appreciation.

29. 6. Time Series Design • O OOOOO X O OOOOO • This design fails to account for the effects of history. • Otherwise, this research design is ideal for examining systematic changes in an ongoing program. Pretest measures serve as a contrast for posttest measures. • Illustrations: • awarding points to nominators, • increasing the points for supervisors, • increasing the percentage of employees recognized, • adding new merchandise to the catalogue, • Creating a “wall of fame” • Including a recognition column in the company’s newsletter

30. 7. Quasi-Experimental Design • R O X O O – OR O X O O – O This design uses groups that are already formed and members cannot be randomly assigned. Illustration: A new recognition program cannot be implemented to randomly selected members of a group. Therefore, randomly select some groups to start the program first, while other groups have to wait for six months before starting.

31. 8. Path Analysis • This design utilizes a regression analysis of multiple variables to test the relationships between them. • The unit of analysis may be groups, such as testing whether the average satisfaction scores of employees in various divisions or departments predict levels of customer satisfaction in those divisions or departments. • Path analysis tests the relationships in a hypothetical model, also known as structural equation modeling.

32. Hypothesized Recognition Model • Do recognition programs influence employee attitudes? • Do improved employee attitudes lead to improved employee performance? • Does better employee performance impact the company’s bottom line?

33. Do employee recognition programs contribute to the bottom line? Improved Employee Attitudes Higher Employee performance Employee Recognition Increased Profits Possible Metrics: Absence Rate Turnover Rate HR Expense Factor Profit per employee Revenue per employee

34. “Applause” Program: ROI Linkage Analysis Applause Participation Employee Satisfaction Customer Loyalty Financial Return Higher Applause participation is linked to higher employee satisfaction Growth in Applause = 3-4% growth in employee satisfaction 6-9% difference in employee satisfaction = 3-4% difference in customer satisfaction 1% change in customer satisfaction = 3.4% change in revenue