Evaluating Technology Programs: Strategies for Success

1. The Ultimate Question: Does Your Technology Program Work? Elizabeth Byrom, Principal Investigator Anna Li, Evaluator

2. Objectives • Think about the context for evaluating technology programs • Identify the key elements of an evaluation model • Walk through steps for developing an evaluation plan • Identify Evaluation Resources

3. Why is evaluating technology Programs a challenge? • Differences among adopters • Scale effects • Geography • Media as systems • Rapid change • Trail of use (B. Bruce, 2001)

4. Why is evaluation a challenge? • Re-creation of technology • New roles for teachers and students • Technical characteristics • Access (B. Bruce, 2001)

5. Observations from SEIR*TEC • Evaluation is often the weakest part of a technology program. • Competing priorities • Expertise • Policymakers often have unrealistic expectations. • Traditional measures do not always apply.

6. Some Things to Consider • It takes four or five years for most teachers to become highly proficient in teaching with technology • Effective use of technology usually requires changes in teaching strategies.

7. Some Things to Consider • It’s the combined effect of good teaching, appropriate technologies, and conducive environment that makes a difference in student achievement. • Good technology does not make up for poor teaching.

9. Professional Development Map Inputs Plans Needs Assessments Mandates & Policies Research & Best Practices

10. Evaluation Questions • At least one question per objective • Questions on • Accountability • Quality • Impact • Sustainability • Lessons learned

11. Indicators Outcomes Questions Methods Criteria

12. Kinds of Questions • Accountability: Is the program doing what it is supposed to do? • Quality: How well are we implementing program activities and strategies? How good (useful, effective, well received) are products and services?

13. Kinds of Questions • Impact: Is the program making a difference? What effects are services and products having on target populations/ • Proximal effects • Distal effects

14. Kinds of Questions • Sustainability – What elements are, or need to be, in place for sustained level of improvement in teaching and learning with technology to occur? • Lessons learned – What lessons are we learning about the processes and factors that support or inhibit the accomplishment of objectives?

15. Sample Questions • To what extent are teachers using technology to increase the depth of student understanding and engagement? • How have students been impacted by technology integration? • How effective has our professional development been in helping teachers attain basic technology proficiency? In helping them learn effective teaching practices?

16. Indicators • Definition: a statement of what you would expect to find out or see that demonstrates particular attributes. • Focus on: • Quality, effectiveness, efficacy, usefulness, client satisfaction, impact

17. Information Sources • Self-reports • Questionnaires • Interviews • Journals and anecdotal accounts • Products from participants • Sample of work, tests, and portfolios • Observations • Media: videotape, audiotape, photographs • Archives

18. Sources of Information – Tracking Tools • Milken Exchange Framework • CEO Forum STaR Chart • Learning with Technology Profile Tool (NCRTEC) • SEIR*TEC – Technology Integration Gauge for Success • Profiler (HPRTEC)

19. profiler.hprtec.org

20. Methods/strategies for collecting data • Questionnaire • Survey • Interview • Focus group • Observation • Archival records

21. Criteria and Benchmark • Stick a stake in the ground and say “we are here today” • Likert-type Scale or rubrics • STaR Chart • SEIR*TEC Progress Gauge • Percentages: e.g. 75% passing rate

23. Outcomes • Decisions are made about maintaining, changing, or eliminating aspects of the program • Convincing evidence is gathered for proposals and plans • Products developed and distributed • Reports • Plans

24. Data Analysis • Quantitative Data • Use Excel Spreadsheet • SPSS for Windows or Mac • Qualitative Data • Content analysis • Look for Emerging Themes and Summarize

25. Using Evaluation Results • Make data-informed decisions • Make a case for continued funding • Inform research and the public

26. Evaluation Data on Impact • SEIR*TEC Professional Development Models – listed by greatest impact: High Quality Met Needs Timely Important Resource Institutes 100% 100% 100% 100% Academies 100% 97.4% 97.4% 96.5% Core Groups 94.7% 86.2% 92.9% 96.3% Workshops 89% 84% 87.5% 85.9% Presentations 89.3% 78.8% 83.7% 79%

27. Hints for Successful Evaluation • Think positively. Evaluation is an opportunity to learn • Try to be objective. • Make evaluation an integral part of the program • Involve stakeholders in the evaluation process • Brag on your successes, however small. • Ask for help when you need it.

28. Recommended Books • King, Morris, & Fitz-Gibbon, How to assess program Implementation. Sage, 1987 • Patton, Utilization Focused Evaluation, Sage, 3d edition, 1996. • Patton, How to use Qualitative Methods in Evaluation, Sage, 1987. • Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards, 2nd Edition. Sage, 1994. • Campbell, & Stanley, Experimental and Quasi-Experimental Designs for Research, Houghton-Miflin, 1963. (CS)

29. Evaluation Resources • SEIR*TEC Web Site: http://www.seirtec.org • US Department of Education: • http://www.ed.gov/pubs/EdTechGuide/ • http://www.ed.gov/Technology/TechConf/1999/whitepapers/paper8.html • Education Evaluation Primer: http://www.ed.gov/offices/OUS/eval/primer1.html

30. Evaluation Resources • Muraskin, Understanding Evaluation: The way to Better Prevention Programs.http://www.ed.gov/PDFDocs/handbook.pdf • National Science Foundation, User-Friendly Handbook for Program Evaluation, www.ehr.nsf.gov/EHR/RED/EVAL/handbook/handbook.htm • W.K. Kellogg Foundation Evaluation Handbook, http://www.wkkf.org/Publications/evalhdbk/default.htm

31. For Further Information, Contact • Elizabeth Byrom, Ed.D. Ebyrom@serve.org • Anna Li Ali@serve.org SouthEast Initiatives Regional Technology in Education Consortium 1-800-755-3277 www.seirtec.org