Designing and Utilizing Evaluation for Gifted Program ImprovementElissa F. Brown, Ph.D.Director,Center for Gifted EducationCollege of William and Maryefbrow@wm.edu(757) 221-2362S.C. Consortium for Gifted EducationCharleston, SCNovember 29, 2006
A Dynamic Model for Gifted Program Improvement PLAN (Program / Curriculum) DO (Implementation / Instruction) ACT (Plan of Action / Improvement) STUDY (Evaluation / Assessment) Center for Gifted Education – School of Education – The College of William and Mary
Key Beliefs • The fundamental role of evaluation is to provide information that can be used to improve and advance the state of the art of gifted programs. • Evaluation research is a collaborative process among stakeholders including the state legislature, the state department of education, local school districts, and the contractor. • The use of multiple data sources helps to illuminate the complexity and salience of issues needing to be considered.
Guiding Questions Who has a vested interest in the gifted program? Who needs the information for program improvement? Are there others that may be impacted by the program evaluation? Matching stakeholder groups Parents, teachers, students, administrators Teachers, administrators, school board Support staff, Dept. of Education, regular education teachers Planning a program evaluation: Identifying Stakeholders
Evaluation Questions Question 1: To what extent are the stated mission and goals of the gifted program fulfilled in their actual operation? Question 2: To what extent is the gifted program meeting the needs of identified students as perceived by relevant groups? Question 3: What evidence exists to document positive student performance trends for students participating in the gifted program?
Evaluation Questions (cont.) Question 4: What are the program strengths and weaknesses in relation to the state of the art or best practices in gifted education? Question 5: What are recommendations for program improvement or revision?
Data Sources • Survey of students, parents, teachers, and administrators • Focus groups of students, parents, teachers, administrators • Interviews with key administrators • Classroom observations • Student impact data • Review of written documents
Process of Constructing Stakeholder Surveys in Gifted Program Evaluation • Identify stakeholder groups • Develop questions for cross-group validation • Develop reliable and valid stakeholder surveys
Process of Constructing Stakeholder Survey • Write brief questions • Write understandable questions • Ask relevant people relevant questions • Write uni-dimensional questions • Write uni-dimensional categories • Write mutually exclusive categories • Avoid writing loaded questions
Content of Questions • Align with different components of NAGC standards • Align with program goals and objectives • Derive information from preliminary visits • Review the literature
NAGC Program Standards • Program design • Student identification • Curriculum and instruction • Socio-emotional guidance and counseling • Professional development • Program administration and management • Program evaluation
Focus Groups Definition: • “an interview style designed for small groups” (Berg, 1998) • “explicit group interaction to produce data and insights that would be less accessible without the interaction found in the group.” (Morgan, 1988) Rationale for use in gifted evaluations: • To deepen understanding of key gifted program issues, using a multiple perspectives lens
Focus Groups:Literature Review • Uses in gifted program evaluation have been predominately for confirmatory and triangulation purposes (Fern,2001; Knodel, 1993; Morgan, 1988; Wolff, Knodel, & Sittitrai, 1993). • The number of focus groups needed in an evaluation depends on the purpose, issues, and group characteristics (Fern, 2001; Knodel, 1993). • Groups should have within group homogeneity and cross-group heterogeneity (Morgan, 1998; Morgan & Kruger, 1993; Stewart & Shamdasani, 1990).
Process for Developing Focus Group Protocol • Size of group ranges from 6-12 participants. • Design questions in a way to allow for cross group comparison in the analysis process • Tailor questions for different stakeholder groups • Make questions relevant to stakeholders • Use open-ended probes
Process for Administering Focus Groups • Moderator describes the purpose of the focus group • to gain an overall picture of the gifted program from different perspectives • to provide an opportunity for small group sharing of participants’ experiences and ideas • Moderator reads each question one at a time and elicits responses from participants • each participant records their responses on an index card • participants talk about their responses • a note taker records key points on a flip chart • Moderator debriefs session
An Example of Triangulation of Focus Group Data *Stakeholder codes: S=Student; RT=Regular Classroom Teacher; A=Administrator; GT=Teachers of Gifted Students; P=Parents **In a focus group design with five stakeholder groups, the consensus of at least three groups is needed for extracting a theme, following a minimum of 50% consensus rule.
An Example of Stakeholder Focus Group Responses
Assessing Classroom Practice:Purposes • Conduct classroom observations in multiple instructional contexts. • Examine differences in instructional behaviors in different organizational patterns, different teacher groups, and different subject areas. • Shape an emerging consensus around acceptable levels of practice in gifted education programs.
Assessing Classroom Practice:Literature Review • No documented differentiation practices for gifted in heterogeneous classrooms (84%); (Westberg, Archambault, Dobyns, & Salvin,1993) • Ineffective teachers over 3 years resulting in depressed effects on student achievement in math regardless of ability level (Sanders & Rivers, 1996) • Positive effects of employing key practices (e.g. critical thinking or metacognition) on student learning in math & science for elementary and middle school levels (Wenglinsky, 2000)
Assessing Classroom Practice:Literature Review • Higher-level reform behavior takes a minimum of two years of intensive training to demonstrate results (Borko, 1993) • Content-based curriculum intervention for gifted coupled with staff development results in significant & important growth gains (Little, Feng, VanTassel-Baska, Rogers, & Avery, 2003; VanTassel-Baska, Zuo, Avery & Little, 2002; VanTassel-Baska, Bass, Ries, Poland & Avery, 1998)
Assessing Classroom Practice:Instrument Construction (COS-R) • Categories are consonant with research on effective teaching practices, teacher reform literature, and teaching high-ability learners • Curriculum Planning and Delivery • Accommodations for Individual Differences • Problem Solving • Critical Thinking Strategies • Creative Thinking Strategies • Research Strategies
Assessing Classroom Practice:Discussion Teachers of the Gifted: • Strong in general teaching strategies (e.g. lesson planning and clarity in directions) but fewer differentiation practices employed than desired • Little emphasis placed on accelerative practices • Problem-solving behaviors were not routinely observed • “Compare/contrast” activities most prevalent critical thinking approach employed • Little emphasis placed on metacognition or extension
Sample Findings Across Data Sources:Outcome and Process of Triangulation *DR=Document Review, I=Interview, FG=Focus Group, CO=Classroom Observation, S=Survey data
A Stage Model for Evaluating Gifted Programs to Enhance Utilization -VanTassel-Baska, 2003
Dissemination and Utilization Map -VanTassel-Baska, 2003
Evaluation has moved from being primarily a reservoir of methods for evaluation to now also becoming and being a reservoir for knowledge about generic patterns of program effectiveness. -- Patton
