Evaluating Nutrition Education Programs

1. Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RD Professor Division of Epidemiology and Community Health University of Minnesota

2. l • Lytle@epi.umn.edu • Cyhp@epi.umn.edu (Center for Youth Health Promotion website at UMN- tools, materials available for free)

3. The WINNERS!!! • How to assess behavior • Evaluating young children • Assessing knowledge and attitudes • Validity and reliability • Sample size (unit of assignment) • Strong study design

4. Other questions • How to evaluate “one-shot” nutrition ed?/minumum exposure for nutr.ed to stick? • Evaluating parental response to interventions targeted at the child (TEENS, CATCH examples) • Home influence on kids’ eating choices • Teacher evaluating change in kids…

5. Types of evaluation • Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): “Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.”

6. Types of evaluation • Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): • Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007) • Assumes that a causal relationship is being tested

7. Assessing behaviors, knowledge and attitudes What behaviors to focus on? How do knowledge and attitudes fit in? What tools are out there? Which behaviors are easier/more difficult to assess? How does age influence the evaluation tools that we use?

8. Recommended Behavioral Outcomes from FNS • Eat fruits and vegetables • Eat whole grains • Consume fat-free or low-fat milk products every day • Be physically active every day • Balance caloric intake from food and beverages with calories expended

9. Recommended Behavioral Outcomes from FNS • Evaluation tools should reflect knowledge, attitudes, behaviors related to these behavioral outcomes…

10. Designing an intervention and evaluation plan • START with a conceptual model or logic model

11. Intention Behavior Theory of Planned Behavior Attitude toward the behavior Subjective norm Perceived behavioral control

12. Examining nutrition education impact/outcomes KNOWLEDGE  ATTITUDES  BEHAVIORS

13. Example of how KAB should line up in evaluation KNOWLEDGE: Does the student know how long is needed for adequate handwashing? ATTITUDES: Does the student believe that it is important to wash their hands? BEHAVIORS: Does the student report frequent handwashing or is observed consistently washing hands?

14. Examining nutrition education impact/outcomes KNOWLEDGE: “Why” versus “How to knowledge” “How to” KNOWLEDGE: Knowledge that will help one make healthier choices rather than knowledge for the sake of knowledge

15. Examining nutrition education impact/outcomes WHY KNOWLEDGE: • Which of the following organs is responsible for secreting insulin? a. Kidneys b. Lungs c. Liver d. Pancreas

16. Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Nebraska): • Which food is a lower fat snack? a. Pretzels b. Potato chips c. Doughnuts

17. Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Wyoming) • Write the name of one food found in each food group from MyPyramid. Milk group Grain group Meat group Fruit group Vegetable group

18. More “how to” knowledge questions…(Kansas State) • What is the healthiest snack choice? • Soda pop and chips • Milkshake and fries • Fruit juice and pretzels • Which foods would always be safe to pack in a sack lunch? • Sliced ham • Peanut Butter • Sliced cheese • How long should I wash my hands before I touch or eat food? • As long as it takes to say my name • As long as it takes to sing “Happy Birthday” • As long as it takes to eat an apple

19. Assessing attitudes • Motivation • Beliefs, opinions • Perceived benefits/barriers • Subjective norm (how do others feel about this behavior?) • Self efficacy (how confident do I feel about performing some behavior?

20. Attitude examples (Iowa State) • I like to eat fruits and vegetables for snacks • My friends like to eat fruits and vegetables for snacks • How sure are you that you can fix fruits and vegetables to eat as snacks at home?

22. Measuring behavior • Preference • Self-report behavior • Observed behavior

24. Other preference type questions • See CATCH HBS: Section A: What food would you pick? • Fruit and veggies lists • 5-a-day Card Sorting Task for younger kids

25. Behavioral measures to assess • Eat fruits and vegetables • Eat whole grains • Consume fat-free or low-fat milk products every day • Be physically active every day • Balance caloric intake from food and beverages with calories expended

26. Eat Fruits and Vegetables: Self report measures • 24 hour recalls (valid in youth as young as 8 years old) • Youth Adolescent Questionnaire-YAQ food frequency (valid in youth from about 6th grade and up) • Validated 6 item fruit and veggie screener from BRFSS (valid in youth from about 6th grade and up)

27. Assessing fruit and veggie intake: 6th graders and olderBRFSS tool • These questions are about the foods you usually eat or drink. Please tell us how often, over the past year, you ate or drank each one. Mark the box for the response that best describes how often you eat or drink the food. Be sure to include foods you ate or drank at home, at school, at restaurants or anywhere else. 1. Over the past year, how many times did you drink fruit juices such as orange, grapefruit, or tomato? (Mark only ONE box.)

28. 2. Over the past year, how many times did you eat fruit (not counting juice)? Mark only ONE box.)

29. Over the past year, how many times did you eat green salad? (Mark only one box.)

30. Over the past year, how many times did you eat potatoes (not including french fries, fried potatoes, or potato chips)?

31. Over the past year, how many times did you eat carrots? (Mark only ONE box.)

32. Over the past year, how many times did you eat vegetables (not counting carrots, potatoes, or salad)? (Mark only ONE box)

33. Eat Fruits and Vegetables: Self report measures • How about for kids younger than 8? • Observation • Parental reporting

34. Behavioral outcomes to assess • Fruits and vegetables • Whole grain consumption: Are there valid and reliable tools? Can youth identify whole grain foods? • Fat free or low fat dairy • Be physically active every day • Balance caloric intake from food and beverages with calories expended

35. Behavioral outcomes to assess • Fruits and vegetables • Whole grain consumption • Fat free or low fat dairy: EAT Survey 10-item scale • Be physically active every day • Balance caloric intake from food and beverages with calories expended

36. Assessing Calcium intakes in middle school age youth • Project EAT (Lisa Harnack, P.I.) • 10 item scale • Reliability and validity assessed • Reliability: test:retest one week apart • Validity: 3 24-hour recalls as criterion measure Harnack, Lytle, Story et al, JADA 2006;106: 1790-1795

37. Reliability and Validity • Reliability: will the instrument produce the same result if applied twice? • Do respondents understand the question in a consistent manner? (test-retest) • Are observers consistent in documenting what they see? (inter-rater reliability) • Do the items that I have included to assess a construct relate to each other as I would expect them to? (cronbach’s alpha)

38. Reliability and Validity • Criterion Validity: How well does a measure compare with a “gold standard” or criterion measure? • Face Validity: Would others reading the item understand the concept that I am trying to assess? Does the measure “appear”to measure the desired construct? • Content validity: Does my instrument capture all the elements that I think are important • Construct validity: Is the measure related to other measures as one would expect?

39. Assessing Calcium intakes in middle school age youth Test-retest correlations: Reliability Sample ICC Total sample 0.74 Females 0.79 Males 0.66 11-12 y.o 0.74 13-14 y.o. 0.75

40. Assessing Calcium intakes in middle school age youth Assessing Validity: Comparing calcium intake using 3 methods (n=248) • 24 hour recalls: 993mg (499) • 10 item CA FFQ: 856mg (570) • 1 YRBS question: 423mg (344) • Correlations • Recalls and FFQ= 0.43 • Recalls and YRBS= 0.37

41. Behavioral outcomes to assess • Fruits and vegetables • Whole grain consumption • Fat free or low fat dairy • Be physically active every day • Balance caloric intake from food and beverages with calories expended

42. TEENS Survey Questions • Do you get some regular physical activity outside of school? By regular we mean at least three times a week for at least 20 minutes at a time. • Most of the time • Usually • Once in a while • Hardly ever • Never • When you think about how hard you work out when you are physically active, do you find that you are • Breathing much harder than usual • Breathing somewhat harder than usual • Breathing only a little harder than usual • Breathing the same as usual

43. How many hours per day do you usually watch TV during the weekdays? • I don’t watch TV during the weekdays • Less than 1 hour per day • 1-2 hours per day • 3-4 hours per day • More than 4 hours per day • How many hours per day do you usually watch TV during the weekend? • I don’t watch TV during the weekend • Less than 1 hour per day • 1-2 hours per day • 3-4 hours per day • More than 4 hours per day

44. How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekdays? • I don’t play video games during the weekdays • Less than 1 hour per day • 1-2 hours per day • 3-4 hours per day • More than 4 hours per day • How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekend? • I don’t play video games during the weekend • Less than 1 hour per day • 1-2 hours per day • 3-4 hours per day • More than 4 hours per day

45. Behavioral outcomes to assess • Fruits and vegetables • Whole grain consumption • Fat free or low fat dairy • Be physically active every day • Balance caloric intake from food and beverages with calories expended

46. Balance caloric intake from food and beverages with calories expended • Assessing caloric intake and expenditure is very difficult • Related behavioral messages and outcomes to assess on the energy intake side may include: • Consumption of a healthy portion size • Eating meals and few snacks • Conscious eating • Reduction of empty calories • Reduction of soft drinks

47. Balance caloric intake from food and beverages with calories expended • Assessing caloric intake and expenditure is very difficult • Related behavioral messages and outcomes to assess on the energy expenditure side include: • Reducing television time • Increased walking or biking for transportation • Small increases in activity every day • More leisure time activity

48. Balance caloric intake from food and beverages with calories expended: Examples • On a typical day, how many hours do you watch TV, view videos or work/play on the computer? (Iowa) • I don’t watch tv, etc.. • Less than one hour • 1 hour/day • 2 hours/day • I eat breakfast in the morning (UMN) • Never Sometimes Almost Always Always

49. Energy balance knowledge: Examples from IDEA Question % correct Youth Parent If someone sits all day, they do Not need to eat any calories (F) 78% 98% Alcohol contains calories (T) 77% 99% The sweetner used in Gatorade and Other soft drinks is healthier than the Sweetener used in sodas (F) 47% 69% Most youth who are active need to Consume sports drinks to replace Electrolytes and minerals (F) 40% 72%

50. How do assess young children (less than 7 years old or 2nd grade and younger) • Very challenging • Cannot think abstractly, limited reading and writing skills • Will be able to perform simple knowledge tests • Use observation assessment tools as possible, conducted by objective observers, to assess behavior