Topic 8 Reflections on the foundations of the MOI-Policy Evaluation Studies

1. Topic 8Reflections on the foundations of the MOI-Policy Evaluation Studies PEDU 7209 Foundations of EAP

2. The framework of practical logical of policy evaluation: Post-positivist methodological approach to policy evaluation: By post-positivism, it refers to "a contemporary school of social science that attempts to combine the discourse of social and political theory with the rigor of modern science. It calls for a marriage of scientific knowledge with interpretive and philosophical knowledge about norms and values." (Fischer, 1995, p. 243) According, Fischer advocates that policy evaluation research should extend beyond the dominance of empirical-positivism and incorporate practical discourse about the relevance and rightness of normative claims in policy evaluation research.. Frank Fischer’s Framework of Policy Evaluation

3. The framework of practical logical of policy evaluation: Fischer names such a post-positivist approach in policy evaluation the framework of the "Practical Logic of Policy Evaluation". "It is design to illuminate the basic discursive components of a full or complete evaluation, one which incorporates the full range of both the empirical and normative concerns that can be brought to bear on an evaluation. Structured around four interrelated discourses, the approaches extends from concrete empirical questions pertinent to a particularsituation up to the abstract normative issues concerning a way of life." (Fischer, 1995, p. 18) Frank Fischer’s Conception of Four Levels of Policy Evaluation

4. The framework of practical logical of policy evaluation: Accordingly, the framework of practical logic of policy evaluation is made up of First-order evaluation level: Technical-analytical discourse Contextual discourse Second-order evaluation level Systems discourse Ideological discourse Frank Fischer’s Conception of Four Levels of Policy Evaluation Cognitive-Technical perspective Situational-practical perspective Systemic-practical perspective Critical perspective

5. Technical-analytical discourse: Program verification By program verification, it refers to concerns "with measurement of the efficiency of program outcome. …The basic questions of verification are Does the program empirically fulfill its stated objective(s)? Does the empirical analysis uncover secondary or unanticipated effects that offset the program objectives? Does the program fulfill the objectives more efficientlythan alternative means available?" (p. 20) Frank Fischer’s Conception of Four Levels of Policy Evaluation

6. Technical-analytical discourse: Program verification "Program verification employs such methodologies as experimental research and cost-benefit analysis. The goal is to produce a quantitative assessment of the degree to which a program fulfills a particular objective (standard or rule) and a determination (in terms of a comparison of input and output) of how efficiently the objective is fulfill (typically measured as a ratio of costs to benefits) compared to other possible means." (p. 20) Frank Fischer’s Conception of Four Levels of Policy Evaluation

7. Contextual discourse: Situational validation Validation focuses on whether or not the particular program objectives are relevant to the situation. …Instead of measuring program objectives per se, validation examines the conceptualizations and assumptions underlying the problem situation which the program is designed to influence. Validation centers around the following questions: Is the program objective(s) relevant to the problem situation? Are there circumstances in the situation that require an exception to be made to the objective(s)? Are two or more criteria equally relevant to the problem situation?" (Pp. 20-21) Frank Fischer’s Conception of Four Levels of Policy Evaluation

8. Contextual discourse: Situational validation "Validation is an interpretive process of reasoning that takes place within the framework of the normative belief systems brought to bear on the problem situation. Validation draw in particular on qualitative methods, such as those developed for sociological anthropological research." (p.21) Frank Fischer’s Conception of Four Levels of Policy Evaluation

9. Systems discourse: Societal vindication "In vindication… the basic task is to show that a policy goal (from which specific program objectives were drawn) addresses a valuable function for the existing societal arrangement. Vindication is organized around the following questions: Does the policy goal have instrumental or contributive value for the society as a whole? Does the policy goal result in unanticipated problems with important societal consequences? Does a commitment to the policy goal lead to consequence (e.g. benefits and costs) that are judged to be equitably distributed?" (p. 21) Frank Fischer’s Conception of Four Levels of Policy Evaluation

10. Systems discourse: Societal vindication As second-order vindication, this type of policy evaluation "steps outside of the situational action context in which program criteria are applied and implemented in order to assess empirically the instrumental consequences of a policy goal in terms of the system as a whole." (p. 21) Hence, the methodological approach adopted by societal vindication is macroscopic-institutional analysis which usually takes the forms of comparative and/or historical-sociological methods. Frank Fischer’s Conception of Four Levels of Policy Evaluation

11. Ideological discourse: Social choice "Social choice seeks to establish and examine the selection of a critical basis for marking rationally informed choices about societal systems and their respective ways of life. …Social choice raises the following types of questions: Does the fundamental ideal (or ideology) that organize the accepted social order provide a basis for a legitimate resolution of conflicting judgments? If the social order is unable to resolve basic values conflicts, do other social orders equitably accommodate the relevant interests and needs that the conflicts reflect?” Do normative reflection and empirical support the justification and adoption of an alternative ideology and the social order it prescribes?” (p. 22) Frank Fischer’s Conception of Four Levels of Policy Evaluation

12. Ideological discourse: Social choice "Social choice involves the interpretive tasks of social and political critique, particularly as practiced in political theory and philosophy. Most fundamental are the concepts of a 'rational way of life' and 'good society'. Based on the identification and organization of a specific configuration or values ― such as equality, freedom, or community ― models of the good society serve as a basis for the adoption of higher level evaluative criteria." (p. 22) The typical methodological approach adopted by this type of evaluative studies is critical approach to political, social and philosophical inquiry. Frank Fischer’s Conception of Four Levels of Policy Evaluation

13. Ideological Choice Because of Critical Discursive Perspective Systems Vindication Because of Interpretive Political Perspective Situational Validation Because of Warrant Since Analytic Technical Perspective Data Technical Verification So, (Qualifier), Conclusion

14. Evaluation Study of MOI Policy for Secondary Schools in HKSAR Topic 9Policy Process Studies: Policy Evaluation Studies

15. A policy measure in search of an objective: Summative or even judgmental evaluation of the efficiency of CMI in comparison with EMI Informative evaluation of the efficiency of preparing secondary-school leavers to become trilingual and bilateral Upholding mother-tongue instruction Summative evaluation of the effectiveness of eliminating mixed-coded teaching in secondary schools in HKSAR Technical-analytical Discourse: Program verification

16. “The Objectives of the Longitudinal Study (1999-2002) are • to trace the academic & personal development of students in schools adopting either Chinese or English as the MOI; • to compare the degree of improvement of students’ academic and personal development in schools adopting either Chinese or English as the MOI; • to compare the language ability (in both Chinese & English) of students in schools adopting either Chinese or English as the MOI; and • to identify facilitating & hindering factors affecting students learning in school adopting Chinese as the MOI.” (p.2 of Tender ref. ED/PR/EMICM/99)

17. The Objectives of the Longitudinal Study (2002-2004) • Study are to find out • The effects of different MOI arrangements at senior levels on students’ learning, e.g. their academic performance, personal development, language ability and high-order thinking skills; • the major factors that enhance/hinder students’ learning in Chinese or English MOI at senior secondary levels; and • the major factors that enhance/hinder the introduction of English as MOI or in part only at senior secondary levels. • based on the findings, to design/recommend measures that support students’ learning under different MOI arrangement for their whole secondary schooling. (Tender ref. ED/P&R/EMICMI/02)

18. To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome-based and endogenous evaluation research Instructional paradox in MOI policy outcomes Socio-psychological paradox in MOI policy outcomes School-organizational paradox in MOI policy outcomes Technical-analytical Discourse: Program verification

19. Subjects • Cohorts of Students under Study

20. 00-01 99-00 01-02 02-03 03-04 04-05 05-06 98 Cohort HK CEE 98 AAI F. 2 Tests F. 3 Tests F. 4 Tests HK ALE 99 Cohort F. 1 Tests F. 2 Tests F. 4 Tests HK CEE 99 AAI HK ALE F. 3 Tests

21. Teachers’ Perception of Learning Environment A. Sense of efficacy B. Sense of administrative and collegial support C. Sense of reward & recognition from work D. Sense of autonomy & empowerment E. Feeling of stress in work Students’ Performance & Perception of Learning Environment A. Academic Development a. Language Subject - Chinese Language - English Language b. Content Subject - Mathematics - Science - Social Subjects B. Personal Development a. Self-esteem b. Citizenship Development c. Social Efficacy d. Language & learning attitudes C. Perception of Learning Environment a. Learning process & motivation b. Teacher’s efficacy c. Classroom climate d. Quality of school life EMI HKCEE Results EMI CHIG Students’ Academic & Socio-economic Background Implementation of the MOI Guidance CMI EMI CMID CMI EMI CLOW CMI School Administration & Learning Environment A. Principal’s Leadership B. Organizations of Academic & Disciplinary Orders C. Organizational Climate & Collaborative Culture. • Conceptual Framework of the Project

22. Instructional paradox in MOI policy outcomes (Junior Forms) Technical-analytical Discourse: Program verification

29. Instructional paradox in MOI policy outcomes (Senior Forms; HKCEE) Technical-analytical Discourse: Program verification

40. Instructional paradox in MOI policy outcomes (Access to University Education & HKALE) Technical-analytical Discourse: Program verification

41. Educational Attainment Measure: JUPAS Minimum entry requirement:1. Grade E or above in either 2 AL subjects  or 1 AL subject + 2 AS subjects (other than UE and Chi Lang & Culture)2. Grade E or above in AS UE3. Grade E or above in AS Chi Lang & Culture

44. Educational Attainment Measure: JUPAS Maximum entry requirement:1. Grade E or above in either 2 AL subjects  or 1 AL subject + 2 AS subjects (other than UE and Chi Lang & Culture)2. Grade D or above in AS UE3. Grade E or above in AS Chi Lang & Culture

49. To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome and endogenous evaluation research Instructional paradox in MOI policy outcomes Paradox in socio-psychological outcomes School-organizational paradox in MOI policy outcomes Technical-analytical Discourse: Program verification

50. Paradox in Socio-psychological Outcomes • Academic Self-concept • Attitudes towards English & Motives & Strategies in English Learning • Quality of School Life • School Identity and Education Aspiration