EVALUATION OF A FAMILY LITERACY PROGRAM (MOCEP): The Turkish Example

1. EVALUATION OF A FAMILY LITERACY PROGRAM (MOCEP): The Turkish Example AYHAN AKSU-KOÇ Yeditepe University & Boğaziçi University koc@boun.edu.tr in collaboration withSEVİM CESUR & SABİHA ÖRÜNĞ Istanbul University 5th Meeting of the “QualiFLY” Project Hamburg, 19-22 February 2007

2. Overview of talk • My experiences in evaluation: • Two family literacy interventions launched by MOCEF • Mother Child Education Program (MOCEP): home-based • Evaluation study only • Summer School Intervention Program: center based • Program development and evaluation study • Focus on issues of sample selection, measures, analyses, and outcomes

3. FAMILY LITERACY: a multi-dimensional concept • Beliefs about and practices of literacy among family members and its intergenerational transfer to children. (Wasik & Herrmann, 2004). • Research in this domin covers (Morrow and Paratore, 1993): • Naturally occuring literacy and language practices within families, • Family and parent influences on children’s literacy, language and reading. • Intergenerational literacy intervention programs, • Home-school partnerships.

4. LITERACY • Literacy is a set of complex multi-dimensional skills that improve over an individual’s lifetime such as • Reading, writing, use of analytical skills and knowledge in a given subject matter (Anderson & Pearson,1984; Snow, Burns , & Griffin, 1998) • Learning how to read and write calls for • grammatical and discourse competence achieved at home • Literacy: a social practice that varies with • the family’s socio-economic level, • social and political relations, • cultural beliefs and values, (Berman & Slobin, 1994; Dickinson & Tabors, 2001; Gee, 1989; Heath, 1983; Wigglesworth & Stavans, 2001; Minami, 2002, Snow & Dickinson, 1990)

5. NATURALLY OCCURRING LITERACY PRACTICES in the FAMILY • Importance of social and cultural contexts (Gadsden, 1995, 1996; Hart & Risley, 1995; Heath, 1983; Taylor, 1983) • MC regard literacy as entertainment; promote play with print, more independent reading by child • LC: regard literacy as work, emphasize reading practice • Risk factors for low-income and migrant children: • low literate parents, • poorer educational opportunities, • home language other than school language (Baker, Serpell, & Sonnenschein, 1995; Bus, van IJzendoorn, & Pellegrini, 1995; Lonigan, 1994 (Snow, 1994; Snow et al., 1998; Teale, 1986;

6. FAMILY LITERACY INTERVENTIONS • Provide services to parents and children related to language and literacy skills. • Include: (1) direct service to children in center-based programs (2) adult education, (3) intergenerational programs including early childhood, parenting and majority language education,

7. Mother Child Education Program(MOCEP) an Intergenerational Literacy Intervention • A home-based early childhood education and parent involvement program for parents with limited formal education (Kağıtçıbaşı, Bekman, Kuşçul, Özkök & Sucuka, 1995). • Holistic approach focusing on mother-child interaction to stimulate healthy development by: • sensitizing the mother to developmental needs of the child, • promoting the development of cognitive and linguistic skills through the mediation of mother, • directing the mother to interact and communicate with her child effectively

8. MOCEP • Duration of program: 6 months • weekly meetings to train mothers to do educational activities with their children • weekly group discussions for effective parenting • monthly home visits with feedback to ensure fidelity of implementation • Training aimed to develop: • cognitive skills: (through exercises in classification, seriation, discrimination, etc.) • text-level abstract language skills: (through book-reading, personal narratives, dinner-time conversations) • print-related skills:(through exercises in phoneme recognition and segmentation, letter recognition and writing) (Kağıtçıbaşı, Sunar, & Bekman, 1988).

9. What were mothers taught and what were children exposed to? • Mothers were trained to co-work with their children on: • bookreading with questions-answers, • letter or word recognition by sound, • isolating first sound of words represented pictorially, • copying numbers, letters or shapes, • making one-to-one correspondences, • identifying similarities and differences, • classification and seriation. • Mothers received handouts, books, crayons etc., to implement the activities at home. • Mothers expected to work with child for 15-20 mins/day for 5 days.

10. AIMS OF THE EVALUATION STUDY • To investigate the effects of home enrichment as an intergenerational intervention program, • immediately after its completion, • at the end of the first grade, and • at the end of the fourth grades. • To explore the effect of different socio-economic backgrounds on children’s • cognitive and preliteracy skills, • literacy acquisition, • later academic achievements. • Therefore, it is also a study of naturally occurring literacy practices. • To evaluate the outcome, not the process.

11. DESIGN • A quasi-experimental design • One experimental and twocontrol groups: • Working class training (WCT) • Working class control (WCC) • Middle class (MC) • Children were assessed at: • Baseline (pre-test) • End of the program (post-test 1) • Follow-up I (end of 1st grade) • Follow-up II (end of 4th grade)

12. SAMPLE • Program implemented in low-income areas of Istanbul. • Participation on voluntary basis by mothers with a low-level of formal education and a 5-6- years-old • WCT group -- selected among those attending the program. • WCC group -- selected among families matched for child’s age and area of residence • MC group -- selected among families matched for child’s age, living in high-income areas; child should NOT have attended preschool.

13. QUESTION of REPRESENTATIVENESS • Cautionin drawing conclusions since random selection is seldom possible: • Problem of self-selection: • of familieseligible for intervention, only those who chose to do so participate • of participantsselected for the evaluation, some may not agree to participate. • To ensurerepresentativeness: • recruit a group of volunteers representative of the population and randomly assign into the treatment and control groups, • determine how many families in a given area were eligible and how many participated, • determine the level of attrition in the treatment and control groups and compare their pretest score, parent education, and marital status with those who remained to see if similar.

14. SAMPLE Prefer larger sample size as caution for drop-outs in successive follow-ups.

15. VARIABLES OF ASSESMENT • For Home-Environment Comparisons: • Mother interview: • Demographic characteristics of family • Parents’ age, education, occupation, income-level • Literacy activities and resources in the home • Parental literacy practices • Daily newspaper, mother book-reading, mother’s favourite book, frequency of book-reading • Parent-child literacy activities • Length of time of bookreading to child, frequency of bookreading • Availability of literacy materials • Number of books of child, of parents, etc. . • Child-rearing practices • Observations: • Availability of literacy materials • Parent-Child bookreading behaviours in the home

16. Content of Intervention and Outcomes

17. 1st grade and 4th grade outcome variables

18. Measures: STANDARDIZED vs. NONSTANDARDIZED • Prefer standardized tests if available: • for language /culture • TOLD available for English but not Turkish, • PPVT for Turkish and English • For age of child • PPVT can be used at age 5, age 9 or later • For domain to be assessed • Mother-child bookreading behaviors • Consider the time required for administration relative to information to be obtained • Narrative production – coding is time consuming • Picture sequencing – easy to code but may not reflect the skill you want • Use informal measures otherwise: 2 examples

19. Measures: STANDARDIZED vs. NONSTANDARDIZED • Nonstandardized measures: • are also effective predictors, may be more suitable for some contents and contexts • Narrative retelling – for production • Story-picture elicitation – for production • Child bookreading behaviors, • Tests for classification, discrimination • Elicited Imitation – for grammar

20. An example: NARRATIVE COMPREHENSION & PRODUCTION • Narratives (story-telling) entail units beyond the sentence, i.e., discourse level competence. • Narrativesrequire two types of mental schemes in addition to language schemes. • scripts (Nelson, 1996) • Abstract knowledge about familiar events that occur in particular contexts (shopping, bath). • story grammars(Stein & Glenn, 1979, 1982): • Schemas used to structure the actual telling of the story • a beginning, a complication or a problem, and an outcome • information about characters’ motives, plans and feelings.

21. Importance of Narrative for Literacy Acquisition • Scripts and story grammarsalso serve text comprehension in classroom discourse. • Narratives: • serve as a developmental bridge to more advanced literary forms such as expository texts because events are related temporal and causally rather than logically. • provide the context for learning new syntactic structures and vocabulary items. • Scripts and story grammars develop during the preschool years through: • adult-child conversations (e.g. dinner-time talk) • experience with oral stories, story-books and TV • telling and acting out stories (Nicolopoulou, 2004, use in Head Start)

22. Mesures to assess Narrative Competence • Elicitation of stories with pictures: • Single picture stories • Sequenced picture stories • Retelling after an orally presented story with or without pictures • Stories produced using toys as props • Spontaneously produced stories

23. CAT STORY:Picture descriptionvs. Story telling • MC (7;0. YB.1a.EY )….. Once there was, once there wasn’t. Once upon a time a bird had three ba… {there were} birds. Their mother had gone [MCL] to bring them worms [COMP]. They look and (see) a cat! {Good, a bit louder.} … Their mother was going. The cat, a cat was climbing the tree. The dog held its tail, the cat fell down. Their mother brought the worm. And the dog ran and went away. • WC (7;2. EK.1a.SG) Cat, bird, dog {But tell nicely} Dog, bird, fish, sea, tree {What’s happening in this story?} There is cat, bird. And the tree also has grass. …And the tree has its grass, I said.{What else? Why not look at the pictures?} There is cat, there is dog, there is bird, there is the grass of the tree, the ground is yellow, there is dog, that is all. • 49.8% of MC compared to 23.6% low-income children displayed‘story telling’. Aksu-Koç, Taylan & Bekman (2002).Need assessment in early childhood education and an evaluation of children’s levelof linguistic competence in three provinces in Turkey. Project Report 00 R101. Report presented to MOCEF and Bogazici University Research Fund. June 2002.

24. CAT STORYMean Vocabulary and Narrative scoresfor Istanbul children by Home-context and Age MC and Low-income children differ in terms of the organizing schemas (p<.05) rather than linguistic structures used for their expression.

25. Another example: Nonstandardized measure for SYNTACTIC COMPETENCE • Elicited Imitation Test: • 16 sentences incorporating specificgrammatical structures of Turkish, • controlled forlength in words (5-6), in morphemes (10-14), and for memory constraints. • Each child is asked to repeat the model sentence right after the experimenter reads it. • Coding in terms of three types of repetition: • (i) correct (verbatim or with minor modification), • (ii) modified (resulting in structural and semantic change), • (iii) no or ungrammatical repetition

26. % of Ungrammatical Repetitions by Experimental and Control groups to Stimulus Sentences ordered in decreasing difficulty

27. ANALYSES and RESULTS MOCEP EVALUATION STUDY

28. HOME ENVIRONMENT MEASURES on LITERACY ACTIVITIES

29. CHILD MEASURES: Pre-test Comparisons • WCT and WCC were matched for parental education, income level, mother’s age, rate of mothers’ employment. • WCT = 64 mos WCC = 68 mos, MC = 67 (p<.05); age effects were partialled out in the subsequent analyses. • MC children scoredhigher than WCT & WCC on:(p>.001) • PPVT • Classification • Discrimination • Phonological awareness • Listening comprehension • Word definitions • MC children scoredhigher than WCT on:(p>.001) • Part-whole relations • Rhyming • Letter naming • Letter recognition • Syllabification • WCC childrenscored higher than WCT on (p>.001) • Phonemic awareness • Letter naming

30. IMMEDIATE EFFECTS:Posttest I- Pretest differences byEnvironment

31. IMMEDIATE EFFECTS of INTERVENTIONSummary • Not apparent for all variables when compared to control groups, • Therefore should look at within group growth curves for positive change • Some skills may need more time to benefit from intervention

32. FOLLOW-UP I: Impact on Literacy Skills at Grade 1

33. FOLLOW-UP I: Posttraining Predictors of Literacy Skills at Grade 1

34. FOLLOW-UP I: Posttraining Predictors of Literacy Skills at Grade 1

35. FOLLOW-UP ISummary • Significant effects on mechanical print skills, but not onreading comprehension, a higher-level conceptual skill. • More powerful assessment if age-appropriate measures had been re-administered. • e.g. PPVT, WISC-R, Listening Comprehension • Should control for the quality of school children go. • Should cooperate with teachers to capture other differences in performance.

36. FOLLOW–UP IIImpact of Intervention at Grade 4 Measures: • Encoding:writing the description of a children’s game told by experimenter • Reading comprehension:reading a story and answering questions • Narrative production:sequencing a set of story pictures, telling the story • Word definition: WISC-R definitions task • Achievement Tests in Math, Science, Turkish, & Soc. Sci.

37. FOLLOW–UP IIImpact of Intervention at Grade 4Academic Achievement at Grade 4 by Environment

38. Predictors of Literacy Skills and Achievement at 4th Grade Predictors Outcomes

39. FOLLOW–UP IIPosttraining Predictors of Literacy Skills at Grade 4

40. FOLLOW–UP IIPosttraining Predictors of Literacy Skills at Grade 4

41. FOLLOW-UP IISummary • Effective in developing skills necessary for assimilation of verbal material. • Exposing the child to narrative discourse by book-reading and story telling promotes long term competence for text-level language use. • An educated mother who engages in literacy interactions with the child has most positive effects. • Early competence in basic concepts and analytic skills contributes to science achievement.

42. FOLLOW-UP IISummary • Need to use more powerful analyses. • Use of statististical methods that allow us to see the contribution of different variables measured at different points in development. • e.g. Path analysis, as in evaluation of the Summer School Intervention.

43. Path analysis for Summer School Intervention

44. CONCLUSIONS, IMPLICATIONSand RECOMMENDATIONS • Effects onmechanical skills(encoding and decoding) are observed earlier than effects onhigher level cognitive functioning (listening comprehension). These develop in home contexts with high literacy activities, over longer periods of time. • Enrichment programmes should start earlier and be designed for longer periods, for example, start at 4;0 years and spread over two years • Effects of home enrichment programmes need time for the sedimentation of the skills gained, and the right context for their expression. • Long-term follow-ups are also necessary • Preplanning for follow-ups is important (sample size, measures, monitoring, etc.) • For a full evaluation of the intervention, need to assess the parent education component of the program.

45. CONCLUSIONS, IMPLICATIONSand RECOMMENDATIONS • For a full evaluation of the intervention, need to assess the process of implementation of intervention. • records of rates of attendance and compliance • the degree to which mothers deliver the key parts of the curriculum at home • Summer School Intervention • a challenge for home based programs ! • For long term effects, try to ensure continuity between intervention, home and school. • teachers can be trained to support children’s development, observe and do in progress-assessments. • Portfolios kept at home and in school. • Assessment should involve multiple perspectives and sources of data.

46. CONCLUSIONS, IMPLICATIONSand RECOMMENDATIONS • Meaningful to assess general intelligence for matching purposes (for both child and mother) • Evaluation study done by an independent researcher not involved in program development • Important to take into account values, beliefs, social practices and other learning experiences of the group reached.

47. THANK YOU