TEST SLIDE Align Display
The Concept and Development of an NCTA Professional Journal NCTA National Conference San Diego California September 23, 2011
The Concept and Development of an NCTA Professional Journal The Journal Concept Project Status
Journal Publication • Internet-based publishing • Easier Access • Lower costs • Publication frequency • Annually at first • Later, semi-annually
Issue Composition • Articles • Featured pieces written by subject experts • Subject to peer review before publication • Ideally, three to six each issue • “Short Items” • Shorter articles • Used to address subject briefly • Between 0 and 2 per issue
Issue Composition • Book Reviews • Review of recent publications related to Testing • Not peer reviewed, but edited • Varied articles per issue, about 1500 words in length • Notes on Contributors • Announcements • About the Journal • About NCTA
Interface with the Research Community
Sample of Psychometric Topics Placement Test Development/Validation
Test Security Test Development and Classroom Intervention Strategies to Reduce Cheating
Voluntary System for Accountability • Psychometric Considerations: • sample size • random vs. systematic samples • longitudinal vs. cross-sectional designs • floor & ceiling effects • validity of measures
The effect of high school socioeconomic status on the predictive validity of SAT scores and high school grade-point average Zwick, R. & Himelfarb, I. (2011). Journal of Educational Measurement, 48, 101-121.
Attempts to predict College FY GPA from HS GPA have been shown to result in over-prediction for African-Americans and Latinos. • They are predicted to do better than they actually do • Including SAT (or presumably ACT) scores into formula helps, but there is still noticeable bias. • FY GPA is slightly under-predicted for other ethnic groups Hypothesis is that over-prediction stems from HS's having different resources, and correlation between the availability of those resources and ethnic composition of HS.
Created a measure of HS SES = sum of three variables • Aggregated father education • Aggregated mother education • Aggregated family income Many other variables they wanted to consider, but access to the data was difficult. These HS SES variables were all captured as part of the SAT student survey. They are all self-report measures.
Predicting FY GPA from • Model 1: HS GPA • Model 2: HS GPA + SAT subscores • Model 3: HS GPA + SAT subscores + HS SES
Journal Leadership • Editor / Co-Editors • Point of contact for Journal • Selects reviewers • Decides whether to publish/not publish • Editorial Advisory/Review Committee • Helps set direction for the Journal • Edits manuscripts • Participates in decisions to publish/not publish
Role of Outside Experts • Non-NCTA members • As authors • As peer reviewers and subject matter experts • As contributors to Journal leadership • Why they are valuable • Build on our knowledge base • Bring insights based on varied expertise • Contribute to the Journal’s credibility
How do Articles get Published? • Manuscript is submitted to the editor • Reviewers Evaluate the Manuscript • Comment on content • Make recommendation to publish/not publish • Editor Assigns Peer Reviewers • Double Blind Review • Each manuscript gets at least two reviewers
How do Articles get Published? • Editor Decides on Publication • Publish “as-is” • Publish subject to changes • Revise and Re-Submit • Not to publish • Article is assigned to an issue of the Journal • Article is formatted and copy-edited
The Concept and Development of an NCTA Professional Journal Discussion Q & A
Dr. Jack Turner University of South Carolina firstname.lastname@example.org Dr. James Wollack University of Wisconsin - Madison email@example.com
Presentation Blurb: As NCTA has grown there have been increased opportunities to benefit the membership and influence the testing field. A professional journal hosted by NCTA is an opportunity to highlight testing-related research, technology advances, and business trends in a way that is relevant to our membership and beneficial to their professional development. This workshop will describe the partnership formed between NCTA and the Measurement Services Group within the American Educational Research Association (AERA) to develop a formal proposal for "The Journal of the National College Testing Association" to be submitted to the NCTA Board. Workshop attendees will be invited to comment on the concepts and plans being discussed. Opportunities for ongoing involvement in the project will be available.