Mapping

1. Mapping Workshop on Universal Reporting Parameters for the Speech of Individuals with Cleft Palate Washington DC, April/May 2004 Tara Whitehill, Hong Kong

2. Proposed Three-Stage Plan • Evaluating: The usual speech evaluation by the local speech specialist in cleft palate. • Mapping: Conversion from the local speech evaluation to the universal reporting measures. • Reporting:Universal parameters that describe the cardinal features of speech in individuals with cleft lip/palate and related craniofacial anomalies regardless of the individual’s primary language.

3. Already covered • Evaluation (range of protocols and procedures; key issues) • Parameters (proposal; key issues) • Focus: mapping

4. Fanny LAW Sze Kuen International Parameters for Cleft Palate Speech: A Pilot Study Honours dissertation Bachelor of Science (Speech & Hearing Science) University of Hong Kong May 2004 Supervisor: Dr. T. L. Whitehill DRAFT - in progress

5. 6 sets of client forms from 5 centres • 5 clients each set = 30 client forms • Original forms only (no recordings, no additional data such as transcriptions, unless provided on original form) • Two groups of “mappers” • 5 “experts”- members of WG who supplied forms • 5 students - little/no previous experience cleft

6. Used “current” version of parameters and guidelines • Students given session to familiarize/ clarify terminology • Mapped from client forms onto parameters form • Identifying information masked (client, examiner, centre) • Provided feedback on ease/difficulty of mapping each parameter • Either for client form or by set (centre) • Weighted scores - not included in this study

7. “Outcome measures” • Agreement • Within expert group • Within student group • Across all mappers • Feedback (ease/difficulty)

8. Why use students? • In future, unlikely mappers will be naïve • However, “expert” group was restricted to individuals who had developed and debated the parameters

9. Agreement • A big problem was “missing data” • OK if client form stated MD (or WNL or NAD), but what if didn’t? - assume OK? tick MD? • Calculated (a) absolute agreement (Condition 1) (b) disregarding “missing data” (Condition 2)

10. Table 1. Examples to illustrate how agreement was defined

11. Primary ParametersHypernasality Agreement • Experts Condition 1 - 60% Condition 2 - 63% • Students Condition 1 - 73% Condition 2 - 73% • All Condition 1 - 57% Condition 2 - 60% Not much missing data

12. Primary ParametersHypernasality Feedback/Issues to consider • Some difficulties when no. scale points differed (e.g., client form - 7 pt. scale; parameters form - 4 pt. scale) • In such cases, mappers reported difficulty, and agreement decreased • Implication: explicit instructions? • Another difficulty: “mild to moderate” or “moderate to severe” (students selected less severe and experts more severe!) • “Set B” - definition of “mild” = “evident but acceptable”. One expert rater mapped onto WNL.

13. Primary ParametersHyponasality Agreement • Experts Condition 1 - 53% Condition 2 - 96% • Students Condition 1 - 87% Condition 2 - 100% • All Condition 1 - 50% Condition 2 - 97% Excellent, when missing data is accounted for

14. Primary ParametersHyponasality Feedback/Issues to consider • Missing data • Otherwise, excellent • (binary scale)

15. Level One (absent, present, MD) Experts Condition 1 - 63% Condition 2 - 87% Students Condition 1 - 77% Condition 2 - 80% All Condition 1 - 57% Condition 2 - 70% Level Two (subcategories) Experts Condition 1 - 23% Condition 2 - 60% Students Condition 1 - 30% Condition 2 - 38% All Condition 1 - 17% Condition 2 - 27% Primary ParametersAudible Nasal Emissionwith/without Nasal Turbulence

16. Primary ParametersAudible Nasal Emissionwith/without Nasal Turbulence Feedback/Issues to consider • Low agreement for subcategories • “phoneme specific” - e.g. “audible for plosive and /s/” • Confusion about pervasiveness e.g. “mild but consistent”, “phoneme specific and frequent” • Results of mirror test (if not audible in speech) • (NE w/wout nasal turbulence - not raised here)

17. Primary ParametersWeak Oral Pressures Agreement • Experts Condition 1 - 17% Condition 2 - 75% • Students Condition 1 - 13% Condition 2 - 62% • All Condition 1 - 0% Condition 2 - 46%

18. Primary Parameters Weak Oral Pressures Feedback/Issues to consider • All 9 mappers found this parameter difficult to map • Primarily because of lack of data on client forms - no form had specific category for this • Mappers needed to derive the information from other information; particularly difficult for students • Also, students unclear about definition

19. Level One (absent, present, MD) Experts Condition 1 - 57% Condition 2 - 73% Students Condition 1 - 73% Condition 2 - 85% All Condition 1 - 57% Condition 2 - 70% Level Two (subcategories) Experts Condition 1 - 10% Condition 2 - 30% Students Condition 1 - 10% Condition 2 - 15% All Condition 1 - 10% Condition 2 - 20% Primary Parameters Substitution Errors • Agreement not bad for level one, but very poor for subcategories; missing data was not the main problem

20. Primary Parameters Substitution Errors Feedback/Issues to consider • All forms included level one (present/absent/MD) • Subcategories: most forms had patterns, but few included frequency information • Some patterns which expert listeners considered ‘related to cleft palate’ not included here (e.g. nasal substitution/realization’, ‘double articulation’) • Confusion between this parameter and ‘Developmental Delay or Other Articulation/Phonological Errors’ (even for experts) e.g. omission, backing /t/ -> [k].

21. Secondary ParametersErrors related to dentition, occlusion, palatal vault configuration Agreement • Experts Condition 1 - 13% Condition 2 - 69% • Students Condition 1 - 30% Condition 2 - 59% • All Condition 1 - 7% Condition 2 - 53% Better when missing data taken into consideration, but still pretty poor

22. Secondary ParametersErrors related to dentition, occlusion, palatal vault configuration Feedback/Issues to consider • No specific category for this on client forms; information had to be extracted from other information • Some forms (Sets B, E, F) included categories such as ‘lateralization’, ‘palatalization’, ‘dentalization’ - but still not clear whether these due to organic factors or “Other Articulation/Phonological Errors” - even for experts • Errors related to hearing problem? fistula?

23. Secondary ParametersDevelopmental Delay or Other Articulation/Phonological Errors Agreement • Experts Condition 1 - 16% Condition 2 - 72% • Students Condition 1 - 30% Condition 2 - 50% • All Condition 1 - 10% Condition 2 - 43% Again, better when missing data taken into account, but still pretty poor

24. Secondary ParametersDevelopmental Delay or Other Articulation/Phonological Errors Feedback/Issues to consider • Mappers needed to make judgement regarding whether errors were related to cleft or not • Confusion with both “Substitution Errors” and “Errors related to dentition..” • Client forms rarely specified cause or explanation of error patterns - mappers needed to judge • Special cases e.g. “no oral consonants”

25. Secondary ParametersVoice/Laryngeal Disorder Agreement • Experts Condition 1 - 56% Condition 2 - 90% • Students Condition 1 - 63% Condition 2 - 97% • All Condition 1 - 60% Condition 2 - 96% Agreement was good, especially when missing data was taken into account

26. Secondary ParametersVoice/Laryngeal Disorder Feedback/Issues to consider • Similar to hyponasality - mappers were familiar with term • When data was provided on client form, it was clear • Binary choice; no subcategories

27. Global RatingSeverity/Intelligibility • Not decided at time of this study • Only 2 of the 6 sets included such data

28. Conclusions • Reliability lower among students than experts • Experts were those who developed form • Who will be mapping in the future?