1. IEA INTERNATIONAL CIVIC AND CITIZENSHIP EDUCATION STUDY (ICCS) A new comparative study of civic and citizenship education AERA Annual Meeting New York, 2008

2. Symposium Outline • Introduction • Concept and Design • Assessing Knowledge, Background & Perceptions • Collecting School and Teacher Data • Studying CCE in the European Context • Citizenship Competencies in the Latin American Region • Discussant • General Discussion

3. Purposes of study • International comparison of outcomes of civic and citizenship education at Grade 8 • Responses to challenges in civic and citizenship education in changed contexts since CIVED in 1999

4. Instrumentation • International cognitive test • International student questionnaire • Background • Perceptions • Teacher questionnaire • School questionnaire • National Contexts Survey • Regional student instruments • European • Latin American • Asian (under development)

5. Participating countries

6. ICCS Current Status • 38 countries participating • Field trial • October 07 to January 08 • Field trial data • 19,800 students (30 per school) • 10,500 teachers (16 per school) • 668 schools • 29 countries

7. Next steps • Analysis of Field Trial data • Revision of instruments • Final forms • Main Survey • Southern hemisphere (end 2008) • Northern hemisphere (early 2009) • Data compilation and analysis (2009) • Reporting (2010)

9. Concept and Design of the International Civic and Citizenship Study Julian Fraillon and Wolfram Schulz (ACER) AERA Annual Meeting New York, 2008

10. Contents • ICCS Overview • The ICCS Assessment Framework Summary Introduction • ICCS Assessment Framework links to CIVED • ICCS Assessment Framework Structure • ICCS Assessment Framework Domains - Detail

11. ICCS Overview • The purpose of the International Civic and Citizenship Education Study (ICCS) is to investigate, in a range of countries, the ways in which young people are prepared and consequently ready and able to undertake their roles as citizens. • The study will report on: • student achievement in a test of knowledge and conceptual understandings in civics and citizenship. • data about student activities, dispositions, and attitudes related to civic and citizenship education. • contextual data that will help to explain variation in the outcome variables.

12. ICCS Overview • The study builds on the previous IEA studies of civics and citizenship education. • The study is underpinned by six research questions that address the degree to which Grade 8 (or equivalent)* students are ready and able to undertake their roles as citizens AND the contextual factors that play a role in this readiness. *Mean age of student not less than 13.5 years

13. ICCS Assessment Framework • The Assessment Framework has been developed in consultation with experts and iterative review with ICCS country representatives. • The Assessment Framework comprises two main components • The civics and citizenship framework outlines the aspects to be addressed when collecting the outcome measures through the cognitive test and the student perceptions questionnaire. • The contextual framework provides a mapping of context factors that might influence outcome variables and explain their variation.

14. ICCS AF – Links to CIVED • Key conceptualisation of student learning: • The student as the central agent in their civic world, with both an influence on and being influenced by their multiple connections with their civic communities. • Young people learn about civics and citizenship through their interactions with their multiple civic communities and not only through formal classroom instruction.

15. ICCS AF – Links to CIVED • Conceptually similar survey design matrices that link question types to civics and citizenship content. • Some secure trend items from CIVED as a concrete scaling link between the two studies.

16. ICCS AF - Structure • The civics and citizenship framework consists of: • Four content domains • Four affective/behavioural domains • Two cognitive domains

17. ICCS AF – Content Domains • The four content domains in the ICCS Civics and Citizenship Framework are: • Civic society and systems • Civic principles • Civic participation • Civic identities.

18. ICCS AF – Content Domains

19. ICCS AF – Affective Behavioural Domains • The four affective-behavioural domains in the ICCS Civics and Citizenship Framework are: • Value beliefs • Attitudes • Behavioural intentions • Behaviours.

20. ICCS AF – Cognitive Domains • The two cognitive domains in the ICCS framework are: • Knowing • Reasoning and analyzing.

21. ICCS AF Design Matrix

22. Antecedents Processes Outcomes Wider community Educational system History and culture Wider community Educational policies Political events School/classroom: Characteristics Composition Resources School/classroom: Instruction Governance Indicators related to: Civic society and systems Civic principles Civic participation Civic identities Student: Socialization& learning Student: Characteristics Home environment: Family background Social group Home environment: Communication Activities ICCS AF Contextual Domains

23. ICCS AF Contextual Domains

24. Additional information • The paper: fraillon@acer.edu.au OR schulz@acer.edu.au • The paper and/or the Assessment Framework: http://iccs.acer.edu.au

26. Assessing Student Knowledge, Background and Perceptions in the International Civic and Citizenship Study Wolfram Schulz (ACER) Falk Brese (IEA DPC) AERA Annual Meeting New York, March 2008

27. Contents of presentation • Overview of areas assessed with student instruments in ICCS • General principle for scaling and analysis • Review of measurement invariance • Some preliminary results

28. Student instruments • Student cognitive test • One hour testing time • Six rotated booklets • Multiple-choice, true/false and open-ended items • Student questionnaire • Questions on student background and items measuring student perceptions • Different formats (with and without “don’t know” category) • Three forms with different combinations of item batteries

29. Cognitive Test • Four content domains • Civic society and systems • Civic principles • Civic participation • Civic identities • Two cognitive domains • Knowing • Reasoning and analysing • Cluster with CIVED link items • Multiple-choice items

30. Student questionnaire(perceptions) • Four content domains (as above) • Civic society and systems • Civic principles • Civic participation • Civic identities • Affective-behavioural domains: • Value beliefs • Attitudes • Behavioural intentions • Behaviours

31. Value beliefs • Democratic values • Set of modified CIVED items • Citizenship values • Items already used in CIVED • Students’ acceptance of socially undesirable behaviour

32. Attitudes (1) • Students' self-cognitions related to Civics and Citizenship • Interest in political events and social issues • Self-concept regarding political participation (internal efficacy) • Citizenship self-efficacy • Perceptions of distinctiveness compared to others in the country • Sense of belonging to communities

33. Attitudes (2) • Students' attitudes towards rights and responsibilities • Attitudes towards gender rights • Attitudes toward rights of ethnic/racial groups • Attitudes toward immigrants • Attitudes towards policies to avert threats to democracy

34. Attitudes (3) • Students' attitudes towards institutions • Trust in institutions • Responsiveness of the political system (external efficacy) • Confidence in student participation at school • Attitudes towards one’s nation • Satisfaction with country’s achievements

35. Behavioural intentions • Preparedness to participate in forms of civic protest • Legal • Illegal • Behavioural intentions regarding future political participation as adult • Electoral participation • Active political participation • Behavioural intentions regarding future participation as a young person

36. Behaviours • Involvement in civic-related participation in the community • Only minority of students is active in the community! • Involvement in civic-related activities at school

37. Student questionnaire: background (1) • Context of schools and classrooms • Classroom climate for civic and citizenship education at school • Perceptions about students’ influence on decision-making at school • Student perceptions of school climate

38. Student questionnaire: background (2) • Context of the home environment • Parental socio-economic status • Parental occupation • Parental education • Household possessions • Cultural/ethnic background • Country of birth (students and parents) • Language use at home • Self-reports on ethnicity (country-specific)

39. Student questionnaire: background (3) • Context of the home environment • Family composition • Indicators of social interaction • Discussion of social/political issues • Media information • Social activities

40. Student questionnaire: background (4) • Context of the individual student • Age • Gender • Expected educational level • with reference to ISCED classification • Out-of-school activities

41. Field trial data analysis • All ICCS instruments were tested in international field trial (October to December 2007) • Samples of about 600 students (from about 25 schools) per country • Data from 29 countries in current (preliminary analysis)

42. Purposes of field trial data analysis • Inform on item selection for the main study • Inform on most appropriate formats for questionnaire items • Inform on selection of constructs • Predictive validity • Cross-country validity

43. Test data analysis • Item Response Theory • One-parameter Rasch model • Review of item characteristics • Scalability of items • Dimensionality of items • Review of differential item difficulty • Gender DIF • Item-by-country interaction • Review of link items • Relative item difficulty • Review of test length

44. Analysis of Coder reliability • 10 percent of cognitive items are open-ended and need to be coded • Necessary to • inform the development of scoring rubrics • inform the translation verification of scoring rubrics • examine the extent of between country differences in coders

45. Questionnaire data analysis • Review percentages in each category and missing responses • Analyse the dimensional structure of item batteries • Exploratory and confirmatory factor analyses • Determine scaling properties of items and constructs • IRT scaling (Partial Credit Model) • Review relationships between constructs and between constructs and test results

46. Ways of Construct Validation in International Studies • Assessing item dimensionality through confirmatory factor analysis and compare fit across countries • Testing parameter invariance • by constraining item loadings (SEM) • by reviewing item-by-country interaction (IRT) • by constraining item parameters (IRT) • Analysing relationships with reference variables

47. Some first results • Field trial results are encouraging and indicate a high quality of most of the item material • Cognitive test items have generally good scaling properties • Most of the questionnaire items have good measurement properties

48. Match of test and abilities

49. Questionnaire scale reliabilities (1)

50. Questionnaire scale reliabilities (2)