Assessment , Evaluation, & Statistics: A Brief Overview

1. Assessment, Evaluation, & Statistics:A Brief Overview WISTPC 2014 Florida International University Miami, FL Steven J. Condly, PhD United States Military Academy at West Point scondly@gmail.com

2. Definitions • Measurement (What do you know?) • Assigning numbers to things, events, people, actions, etc. • Assessment (How do you know?) • Measurements, actions, processes, data that answer the question. • Evaluation (How are we doing?) • Comparing results and observations with goals and objectives (implied or otherwise).

3. Implications • Measurement (assigning #s) • Need for consistency, accuracy, and precision. • Assessment (collected data) • Should relate to what it is purportedly describing. • Evaluation (comparing results to standards) • Not entirely objective, but should be reasonable and logical.

4. Types of Evaluation • Formative • Serves to strengthen or improve the program being evaluated • In-process • Summative • Examines the effect(s) program • Examines how well program goals and objectives were met • Contains implications for corrective action

5. Formative Evaluation • How well identified and defined is the problem? • How well does the program deal with the problem? • How well does the program progress? [feedback loop]

6. Summative Evaluation • Similar to Formative, but looks back over the entire life of the program. • Reaches conclusions regarding effectiveness, cost, overall success, and likelihood of generalizability (or moving on to the next level). • Easier to perform if formative evaluations are being performed and data/results collected.

7. Types of Assessment • Assessment is data collection with a purpose • Really only two ways to do it: • Question • Observe • Raw data have to be processed (statistics)

8. Statistics • Select a good comparison criterion or group • Standard statistical techniques are alright for Likert-scaled survey data • Don’t use p-values (NHST) • Strongly influenced by sample size • Small p does not necessarily indicate a stronger relationship or effect, or practical significance • What people think it is: P(H0=0|sample) • What it actually is: P(sample|H0=0) • How much there is there?

9. Effect Size Statistics • For Likert or interval-level data, when comparing two groups, use Cohen’s d • M1 - M2 / [(s1 + s2) / 2] • For ordinal data, when comparing two groups, use Probability of Superiority • MWU / (n1n2) • For correlations between two groups, use r2 • (r) (r) x 100 gives % of variance explained

10. Websites • http://oerl.sri.com/ccli_resources.html • www.socialresearchmethods.net/kb/contents.php • http://www.uccs.edu/~lbecker/ • Grissom, R. J. (1994). The probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79(2), 314-316.