CGS Annual Meeting December 8, 2006 Revised GRE General Test: A Prelaunch Update David G. Payne, PhD Educational Testing Service
The Revised GRE General Test • Brief Background • Reasons for Change • Value of the Revised Test • Key Aspects of Changes to the General Test • Contingency Planning for Launch • Communications Efforts • Additional Slides for Deans to Use on Campus
What Is the Value of the Revised GRE General Test? • Improved security – Administration plan that directly addresses current and potential future security challenges over the long term • Improved predictive validity – The revised test reduces the possible effects of memorization. • Improved construct validity - Measures skills more directly related to graduate study • Linear Testing Format is more test taker friendly than current Computer Adaptive Test (CAT)
Why Are We Changing the General Test? • Test delivery is changing from continuous testing to fixed administrations in order to address security issues • Eliminate the potential security risk that exists with continuous testing regarding the exposure of test questions • Test content is changing to increase the validity of the test by reducing the effects of memorization • Eliminate single-word verbal questions • Introduce new question types and tasks • Reduce possibility of non-original essay materials
Additional Goals for the Revision of the GRE General Test • Provide faculty with better information on applicants’ performance • Essay distribution • New Verbal and Quantitative Score Scales • Better measurement of skills • New question formats other than traditional multiple choice • Allow calculator use in order to reduce emphasis on computation in Quantitative section • Improved test access for examinees worldwide
Emphasis on skills related to graduate work such as complex reasoning Increased emphasis on inferential reasoning Increased emphasis on verbal reasoning in context Increased number of reading comprehension questions based on a greater variety of reading passages Reduced emphasis on vocabulary out of context No Antonyms or Analogies Inclusion of new question formats other than traditional multiple choice (e.g., highlighting a sentence in a passage that serves the function described in the question) Value of the New Verbal Reasoning Section
Quantitative reasoning skills that are similar to skills typically used in graduate school Increased emphasis on questions involving “real-life” scenarios Increased emphasis on data interpretation questions Reduced emphasis on Geometry On-screen 4-function calculator with square root Reduced emphasis on computation Inclusion of new question formats other than traditional multiple choice (such as entering a numeric answer via the keyboard) Value of the New Quantitative Reasoning Section
Why Are We Changing the GRE Verbal and Quantitative Score Scales? • Sound measurement practice and professional standards (developed by the AERA, APA, and NCME) recommend a change in the score scale when significant changes to test specifications are made. • The ways in which we are measuring the verbal and quantitative reasoning skills (i.e., question types) and the relative emphasis among the component skills in the revised Verbal and Quantitative sections are sufficiently different from those on the current test, so that it is appropriate to use a new scale for the revised test.
New Verbal and Quantitative Score Scales • Change will allow us to address two situations that have developed over the life of the current test: • Mean Verbal scores (470) and Quantitative scores (593) have drifted apart over time • Quantitative scores cluster at the top of the scale
Ranked Preference for New Score Ranges Preferred Most Preferred Least
Comparison of Old and New Verbal and Quantitative Score Scales NOTE: The Critical Thinking and Analytical Writing score scale will not be changed.
Advantages of the New Score Scale • The new scale will make more apparent the differences between candidates and will facilitate more appropriate comparisons between candidates. • Score users will be less likely to interpret small score differences as meaningful differences between candidates. • The Verbal and Quantitative means will be aligned which will allow score users to see relative strengths for a particular candidate.
Information to Assist GRE Score Recipients • Percentiles (based on first 3 administrations) will be available on the GRE Web site (www.ets.org/gre) and printed on score reports to assist score users in interpreting scores • Aconcordance table to assist score users in determining the relationship between old and new Verbal and Quantitative scores will be available on the GRE Web site. • For Verbal and Quantitative scores earned prior to September 2007 and reported on score reports, approximate score equivalents on the new scale will be included on score reports to assist Score Recipients in comparing old and new scores. • Broad major field score distributions will be available on the GRE Web site.
Sample Revised General Test Score Report *For tests taken beginning in September 2007, scores will be printed in the “New Score” column and dashes will be printed in the “Original Score” column. For tests taken before September 2007, scores will be printed in the “Original Score” column and approximate equivalents on the new scale will be printed in the “New Score” column. These equivalents are based on a concordance study; the full table showing the relationship between the two scales will be posted on the GRE Web site at www.ets.org/gre.
Familiarity with Skills Measured in Sections of the GRE General Test Very Familiar Not at all Familiar
Appropriateness of Alternative Names Very appropriate Not at all appropriate
A performance-based measure that integrates the assessment of critical thinking and analytical writing Consists of two complementary analytical writing tasks: 30-minute “Analyze an Issue" task 30-minute “Analyze an Argument" task New, more focused prompts Requires a more specific response from the test taker Will reduce possibility of reliance on memorized materials Limited or no reuse of prompts Value of the New Critical Thinking and Analytical Writing Section
The New Critical Thinking and Analytical Writing Section Measures the ability to: • Articulate complex ideas clearly and effectively • Examine claims and accompanying evidence • Support ideas with relevant reasons and examples • Sustain a well-focused, coherent discussion • Control the elements of standard written English Note: Essay responses are typed on computer, with a basic word processor, and no spell checker or grammar checker.
Essay Distribution • Essay responses will be made available electronically to designated score recipients. • The details of operation of this new service will be announced in early 2007. • It is critical that faculty understand that test takers construct their essay responses, which are essentially first drafts, under these conditions: • Each essay must be completed in 30 minutes • It is a high stakes testing situation • Test takers use an elementary word processing system with no spell check or grammar check
rGRE Contingency Planning • Issues • Need to set the new Score Scales for Verbal and Quantitative • Requires 30,000 representative test takers in first 3 administrations • Only possible to do this in the Fall • Not possible to do a phased launch like the one used with TOEFL iBT • Constraints on Score Reporting Timelines • Concerns about capacity in new IBT Network • Primarily in large domestic urban areas
rGRE Contingency Planning • Strategies • Actively Recruiting Test Centers – Apply now! • Exploring options for alternative testing sites • Large Flexible Testing Centers (e.g., Commercial sites) • Mobile Test Centers • Exploring options for increasing the number of administration dates • Goal is to allow every test taker to be able to schedule a test within 30 days in their preferred area for testing.
Key Communications Activities • Print Materials • Web Seminars • rGRE eNewsletter • Annotated PowerPoint Presentations • Outreach to Academic Discipline Societies • Conference Presentations • Academic Discipline Societies • CGS Regional Meetings, CHBGS, NAGAP • DVD/Video Presentations • General Overview • Specific topics (e.g., Using the Revised GRE Scores for Verbal and Quantitative)
Key Communications Activities • “Just-in-time” communications to Score Recipients, or ”What happened to the 200-800 scores and what the heck do these new scores mean?” • Reason for Score Scale Changes • Nature of New Score Scales • How to Use Score Scales
Need More Information? To get the most up-to-date information about the changes to the GRE General Test: • Visit the GRE Web site at www.ets.org/gre/revgentest.html • Subscribe to the quarterly GRE eNewsletter (see link on www.ets.org/gre) If you have questions: • E-mail David Payne at firstname.lastname@example.org
Benefits of Using GRE Test Scores • Measure skills faculty and graduate deans have identified as essential to graduate school success, including verbal reasoning, quantitative reasoning, critical thinking and analytical writing • Provide the only common measures of these skills for comparing the qualifications of applicants from different educational backgrounds and countries of origin
Benefits of Using GRE Test Scores (cont’d) • Furnish independent information to supplement the evaluation of grades and recommendations • Offer a valid predictor of graduate school performance, as confirmed by a recent independent study using 82,000 graduate students and over 1,700 studies containing validity data for GRE tests • Ensure that essay responses are the original work of the examinee
The New Verbal Reasoning Section Measures the ability to: • Understand the meanings of words, sentences, and entire texts; understand relationships among words and among concepts • Select important points; distinguish major from minor or irrelevant points; summarize text; understand the structure of a text • Analyze and draw conclusions from discourse; reason from incomplete data; identify author’s/speaker’s assumptions and/or perspective; understand multiple levels of meaning (such as literal, figurative, text’s intent, etc.)
Sample Verbal Question While chocolate was highly esteemed in Mesoamerica, where it originated, its adoption in Europe was initially slow. There is a common belief that Europeans needed to “transform” chocolate to make it appetizing. However, while Spaniards did put sugar, which was unknown to indigenous Americans, into chocolate beverages, this additive was not completely innovative. Mesoamericans were already sweetening chocolate with honey, and the step from honey to sugar– increasingly more available than honey because of expanding sugar plantations in the Americas– is a small one. Likewise, although Spaniards adjusted Mesoamerican recipes by using European spices, the spices chosen suggest an attempt to replicate harder-to-find native flowers. There is no indication the Spaniards deliberately tried to change the original flavor of chocolate. Click on the sentence that presents a misconception that the passage challenges.
The New Quantitative Reasoning Section Measures the ability to: • Understand quantitative information • Interpret and analyze quantitative information • Solve problems using mathematical models • Apply basic mathematical skills and elementary mathematical concepts of arithmetic, algebra, geometry, probability, and statistics
Sample Quantitative Question The table shows the distribution of prices of 45 houses for sale in a certain region. Select two of the following choices and place them in the blanks below so that the resulting statement is true. $175,000 $185,000 $190,000 at most at least $42,000 $57,000 If the highest price of the 45 houses is , then the range of the prices of the 45 houses is . Click on a choice, then click on a blank.
Best Ways to Use Revised General Test Scores (V and Q) • The new scaled scores (and new score equivalents) are a good way to evaluate all applicants. • Percentile information provides information about the relative standing of the applicant compared to the fall 2007 test-taking population. • Broad intended graduate major field score distributions provide a good way to evaluate applicants within a particular group of disciplines.
Best Ways to Use Revised General Test Scores (continued) • For applicants with multiple scores (both old and new), use the new score scales (for old General Test scores, use the approximate score equivalent new scores); whichever method used (e.g., average, most recent, or highest) should be used consistently • Programs should use multiple pieces of information in making admissions decisions (e.g., GPA, GRE scores, letters of recommendation).
Best Ways to Use Revised General Test Scores (continued) • When there is a mismatch between the level of skill in application writing samples (e.g., personal statement) and the Critical Thinking and Analytical Writing scores, it may be helpful to view the applicant’s essay responses on the Critical Thinking and Analytical Writing section.
If You Have Score Use or Score Interpretation Questions • The GRE Board has a long history of encouraging appropriate score use • See the GRE Board Score Use Guidelines on the GRE Web site at www.ets.org/gre/edupubs.html for a fuller description • Visit the URL above to view score interpretative information for the General Test and Subject Tests • The GRE Program is happy to answer score interpretation questions from GRE score users. Individuals should contact the GRE Program: • By telephone at 609-683-2002 • Via email at email@example.com
Sample Argument Topic “Hospital statistics regarding people who go to the emergency room after roller-skating accidents indicate the need for more protective equipment. Within that group of people, 75 percent of those who had accidents in streets or parking lots had not been wearing any protective clothing (helmets, knee pads, etc.) or any light-reflecting material (clip-on lights, glow-in-the-dark wrist pads, etc.). Clearly, the statistics indicate that by investing in high-quality protective gear and reflective equipment, roller skaters will greatly reduce their risk of being severely injured in an accident.” Write a response in which you examine the argument’s unstated assumptions making sure to explain how the argument depends on the assumptions and what the implications are if the assumptions prove unwarranted.
Sample Issue Task:Reliance on Technology "As people rely more and more on technology to solve problems, the ability of humans to think for themselves will surely deteriorate.“ Discuss the extent to which you agree or disagree with the statement above and explain your reasoning for the position you take. In developing and supporting your position, you should consider ways in which the statement might or might not hold true and explain how those considerations shape your position.
How Essay Responses Are Evaluated GRE readers, who are college and university faculty, read each essay response and evaluate its overall quality according to how well the test taker: • Responds to the specific instructions on the issue • Considers the complexities of the issue • Organizes, develops, and expresses his/her ideas • Supports his/her position with relevant reasons and/or examples • Controls the elements of standard written English
Understanding How the Critical Thinking and Analytical Writing Section is Scored • Each essay is scored by two trained readers, using a 6-point holistic scale • Two scores that differ by more than one point are adjudicated by a third reader • Scores from the two readings of an essay are averaged • The final scores on the two essays are then averaged and rounded up to the nearest half-point interval • A single score is reported for the section • Scoring guides and score level descriptions are available on the GRE Web site at www.ets.org/gre
Selected Score Level Descriptions 5.5 and 6 – Sustains insightful, in-depth analyses of complex ideas; develops and supports main points with logically compelling reasons and/or highly persuasive examples; is well focused and well organized; skillfully uses sentence variety and precise vocabulary to convey meaning effectively; demonstrates superior facility with sentence structure and language usage but may have minor errors that do not interfere with meaning 3.5 and 4 – Provides competent analysis of complex ideas; develops and supports main points with relevant reasons and/or examples; is adequately organized; conveys meaning with reasonable clarity; demonstrates satisfactory control of sentence structure and language usage but may have some errors that affect clarity. 1.5 and 2 – Displays serious weaknesses in analytical writing. The writing is seriously flawed in at least one of the following ways: serious lack of analysis or development; lack of organization; serious and frequent problems in sentence structure or language usage, with errors that obscure meaning.
Test Structure *Questions in this section are either being tried out for possible use in future tests or are being used to ensure that scores on new editions of the test are comparable to scores on earlier test editions.
Test Administration Changes *For statistical equating purposes, some test questions need to be reused across test editions.