A standardization and validity study of a speech and language screening tool

1. Michèle Minor-Corriveau, M.Sc.S. Speech-LanguagePathologist AssociateProfessor LaurentianUniversity Sudbury, Ontario Canada Baccalauréat (B.Sc.S.) et maîtrise (M.Sc.S.) ès sciences de la santé (orthophonie) Abstract :The purpose of this longitudinal and cross-sectional study is to standardize and validate the Speech and Language Profile, a speech and language screening tool developed by and for educational Speech-Language Pathologists. Reliability and validity measures obtained will draw upon data taken from administering the screening tool, which will be measured against each individual child’s academic performance through teacher report and standardized testing, including numeracy and literacy scores obtained by the Education Quality and Accountability Office’s (EQAO) provincial mandatory standardized testing. This study hopes to demonstrate predictive, concurrent and criterion-referenced validity of this tool, as well as interrater agreement and test-retest reliability. Favorable results could support the use of this tool by educational Speech-Language Pathologists to help them determine, at the earliest possible moment in the child’s academic trajectory, which services would be required in order to maximize his/her learning potential. A sufficiently high level of reliability and validity could also support the importance for standardizing this tool with other populations in minority and majority language settings at an international level. Statistical analysis will speak to the measures of internal consistency, content validity and face validity. Michèle Minor-Corriveau, M.Sc.S. Speech-LanguagePathologist AssociateProfessor LaurentianUniversity Sudbury, Ontario Canada Context • This is an observational and cross-sectionalstudyaimedatdemonstrating • reliability and validity of the PLLP . • OBSERVATIONAL STUDY • Normative sample (n=2251) : Data obtainedthrough administration of the PLLP in 2004, 2005, 2008, 2009 and 2010. Analysis of data willdetermine if thereis a difference in resultsobtainedduringtheseyears, and whetherthisdifferencecanbeattributed to sex or language of administration (French or English). • Test-retestreliability (n=24) isused to measure the correlationbetween the • scores obtained by the samepersons on 2 administrations of the test. • The PLLP wasadministered once in October 2008 and again in January 2009. • The goal is to establish test stability over time. • Interraterreliability (n=75)refers to the degree to which the sameindividual • wouldreceive the same score, even if the test wasadministered by different • examiners. The PLLP wasadministered as part of the interraterreliability • study. All sessions werevideotaped. 5 SLPswhowereblind to the original • test resultsrated the children’s test scores using the same test form as used • by the original examiner, whilewatching the videotaped session. The goal is • to establish the transparency of the test. • Criterion-relatedvalidity (n=26)refers to the effectivenesswithwhich the • test is able to correlate to an individual’s performance in specifiedactivities • or on other tests administeredat the same time or within a short interval. • The PLLPwasadministeredalongsideotherassessment batteries normed in • Québec. This data collection is part of another doctoral study (see • Mayer-Crittenden, PhD candidate, HumanStudies, LaurentianUniversity). • CROSS-SECTIONAL STUDY • Predictivevalidityrefers to the effectivenesswithwhich a test is able to • predict future performance in specifiedactivities or on like tests administered • after a longer interval: Test scores obtained on the PLLPwillbemeasured • against the students’ academicachievements (report cardresults and IEPs) in • grades 1 – 6, as well as provincial test results (EQAO) in reading, writing and • mathematics. • TEST ANALYSIS • Item analysiswillmeasuresuccess and failure rates of each item to determinedegree of difficulty of items and whether items shouldbeadded, substituted or deleted. • Internalconsistencywillmeasurewhether test items belonging to the samesubset of scores are correlated and whethertheybelong to the sameconstruct. • Face validitywilldeterminewhether the test items are appropriate and to whatdegreetheyreliablyrepresent the targeted items. • Content validitywilllink the theory in support of the test items used, thoseadded, substituted or deleted. Methods • 2001 : SLPsfrom CSCNO schoolboard design the Profile de la langue, du langage et de la • parole (PLLP) a toolthatwould help themidentifywhichchildrenshouldreceive • community-based speech and language services upon entry into Junior Kindergarten • (from 46 to 58 months) • 2007 : PhDresearchaimedatvalidating and standardizing the PLLP begins • 2008 : an internet pollaimedat French-speaking S-LPsconfirms : no gold standard isused to • identify speech or languagedelays in francophone childrenattending French-language • schools • Of 74 respondants (49% response rate) : • 39% = no protocol in place for a universal screening in JK in theirschoolboards • 61% = a protocolis in place for screening in JK theirschoolboards • IS THERE A UNIVERSAL SYSTEMATIC PROTOCOL IN PLACE USED TO HELP DETECT SPEECH • AND LANGUAGE DELAY IN YOUR SCHOOL BOARD ? IF SO, WHAT ARE S-LPs USING ? Oct 2008 Jan 2009 A standardization and validity study ofa speech and language screening tool 11% - observationalchecklist 14% - parent questionnaire 4% - non-wordrepetition test 4% - FluhartyPreschool Screening Test* 19% OEW (French equivalent of WBTT) 7% PLLP 4% TPLS* 4% TSRI* * unstandardized, unadaptedmeasures X5 NO 33% YES 67% Carrow-Woolfolk Data and results Normative sample and Item analysis Chi-square analysis (p<0,05) indicatethat on the whole, test items were consistent withexistingtheory. in the results for 2004, 2005, 2009 and 2010 ≠significantdifferencelinked to sex linked to language (French ; English). Test-retestreliabilityresults show universal agreement among 19/48 items ≠significantdifference on 25/48items significantdifference for 4/48 items (English /l/, s, cup, why). Interraterreliabilityresults show highinterrater agreement, rangingfrom 0.90 to 0.98. IEP A+ EQAO Alook at a Canadian linguistic minority outside Québec INTERRATER RELIABILITY • There issufficientevidence to suggestthat the PLLPshouldbeused by S-LPs • to help identifywhich JK studentswouldbenefitfromfurther speech and languageassessment. • Validitymeasuresdemonstrate the PLLP’sstabilityacross time (test-retestreliability), and acrossexaminers (interrater agreement). • Preliminaryresultsindicatea highcorrelationbetweenPLLP and somesubtestsbelonging to the Leiter-R, weakto moderatecorrelationbetween the PLLP test results and academicachievement. (grades obtained on report cards in grades 1 to 4, and IEPs up to grade 6). By the sametoken, data obtained for the samesampleindicate a weakcorrelationbetweenacademicachievement in grades 1 to 4 and EQAO provincial test results. Correlation to IEP ispending. Conclusions % interrater agreement • Criterion-referencedvalidity • Alongside the PLLP, the following tests wereadministered. Preliminaryresultsyieldhighcorrelationbetween the PLLPvocabularysubtest and the Leiter-R: • Carrow-Woolfolk (1999) • Concepts and directions (CELF-CDN-FR) • Échelle de vocabulaire en images – Peabody (ÉVIP) • Edmonton Narrative Norms Instrument (ENNI) • Leiter International Performance Scale • Non-wordrepetition test (Courcy, 2000) • Numberrepetition (CELF-CDN-FR) • Rapidautomaticnaming • Sentence repetition (CELF-P) • Predictivevalidity (preliminaryresults) • The results of 2251 PLLP scores weremeasuredagainsttheir report card grades and IEPsfrom grade 1 to grade 6 as well as on EQAO provincial testing. Preliminarypredictivevalidityresultsindicate a weak to moderatecorrelation, depending on the area. Correlationwasalsoweakwhenmeasuringacademicachievementagainst EQAO provincial testing, evenwithin the sameschoolyear. Correlation of PLLP scores and academicachievementis comparable to thatbetweenacademicachievement and EQAO provincial testing. • Internalconsistencywillbemeasuredusing the split-halfreliabilitymethod and Cronbach’s Alpha. Data analysisisongoing. Preliminaryresultsyieldpromisingfindings. ATTRIBUTES • PLLPAvailable in French and English • Average administration time = 12 minutes • Excellent platform for S-LPs to identifystudents in need of service or follow-up • See Direct Benefits (below) • PLLP data entered online • Accompanying report providesrecommendationsbased on results (cut-off scores) BENEFITS To Parents • Goals are targetedearly on to help remediateminor speech or language issues in the short term. To Children • All childrenentering JK at CSCNO between 2001 and 2010 have been seen and/or flaged as requiringfollow-up. To S-LPs and SchoolBoards • Listscompiled help prioritize service delivery and help manage waitlistsefficiently. Future directions • Resultsfromongoingvalidity and reliabilitymeasureswillbepublished as part of the author’s doctoral thesis. • Development of an iPad application currently in the works to enable S-LPs to administer • the PLLP electronicallyand enter data online. References • American EducationalResearch Association, American Psychological Association & National Council on Measurement in Education. (1999). Standards for Educational and PsychologicalTesting. • USA : American EducationalResearch Association. • Anastasi A., & Urbina, S. (I1997). Psychologicaltesting(7thed). New Jersey : Prentice Hall. • Brown, L., & Bryant, b. (1984). The why and how of specialnorms. Remedial and SpecialEducation, 5(4), 52-61. This initiative has been made possible thanks to fundingfrom LaurentianUniversity/Université Laurentienne and Health Canada (CNFS).