Department of Dermatology, Zealand University Hospital , Roskilde, Denmark.

1. Interrater agreement and reliability of outcome measurement instruments and staging systems used in hidradenitis suppurativa Department of Dermatology, Zealand University Hospital , Roskilde, Denmark. Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, The Capital Region of Denmark. Copenhagen, Denmark. Department of Dermatology, Hofstra North Shore-LIJ School of Medicine, Hempstead, NY,USA Dermatology and Venereology, University of Ferrara, Italy Institute of Infection & Immunity, University Hospital of Wales, Heath Park, Cardiff, U.K. Université Libre de Bruxelles, Department of Dermatology and Venereology, Brussels, Belgium Department of Dermatology, Venereology and Allergology, Medical University, Wrocław, Poland. Department of Dermatology, Hospital General Universitario de Alicante, Alicante, Spain Department of Dermato-Venereology, Henri Mondor Hospital, France KarolinskaInstitutet, Department of Clinical Science and Education, Södersjukhuset, Sweden University Hospital of North Norway, Thomsø, Norway Eramus University Rotterdam, Rotterdam, the Netherlands Departments of Dermatology, Venereology, Allergology and Immunology, Dessau Medical Center, Dessau, Germany Department of Dermatology, ICahn School of Medicine at Mt Sinai, NY, NY, USA Department of Rheumatology, Odense University Hospital, Denmark. Thorlacius L1,2, Garg A3, Theut Riis P1, Bettoli V4, Ingram JR5, del Marmol V6, Matusiak L7, Pascual JC8, Revuz J9, Sartorius K10 , Tzellos T11,van der Zee HH12, Zouboulis CC13, Saunte DM1 Gottlieb AB14, Christensen R2,15, Jemec GBE1 British Journal of Dermatology. DOI: 10.111/bjd.17716

2. Thorlacius L1,2 Jemec GBE1 Department of Dermatology, Zealand University Hospital , Roskilde, Denmark. Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Denmark

3. Introduction What’s already known? • Without valid and reliable instruments to measure outcomes, researchers and clinicians lack the necessary benchmarks to assess primary and secondary endpoints of interventional trials properly • Hidradenitis suppurativa (HS) is a chronic, inflammatory skin disease • Several outcome measure instruments exist for HS, but their validation is generally incomplete or of relatively low methodological quality

4. Objective • To investigate interrater reliability and agreement of both the most frequently used outcome measurement instruments (OMIs) and staging systems in HS

5. Methods (1) • The study was designed in accordance with the ‘Guidelines for Reporting Reliability and Agreement Studies’ (GRRAS) (Kottner et al., 2011) • Raters were dermatologists from 12 different countries, with a clinical background with a least 10 years of experience with HS • 24 patients with HS severity from mild to severe were recruited at random

6. Methods (2)

7. Methods (3) • All patients were examined and scored with nine different instruments by 12 different raters on the same day • Raters were also asked to count the number of typical HS lesions in each anatomical area • Raters were not allowed any communication with each other • The order in which each rater met each patient was based on a pre-specified patient-by-rater visit algorithm

8. Methods (4): Statistical analysis • For the quantification of the interrater reliability the intraclass correlation coefficient (ICC) type ICC (2,1) • For quantification of the interrater agreement for variables with >4 categories, a modified Bland-Altman method was applied • For variables with ≤4 categories, the observed agreement was calculated as the number of patients for which all the raters agreed

9. Results (1)

10. Results (2)

11. Discussion (1) • For the staging systems, we found good interrater reliability for Hurley staging in the axillae and gluteal region and moderate results in the groin • Accordingly, Hurley staging seems to be an acceptable instrument for classification in HS • Experts in clinimetrics emphasise that agreementparameters are preferred when evaluating instruments designed to measure changes in health status • Our observed limits of agreement intervals were very wide compared to the usual ranges, of all the included OMI scales

12. Discussion (2) • None of the included OMIs can be recommended under the given conditions in terms of interraterreliability • Our results illustrate how difficult it is, even for experts, to measure HS physical signs of HS by a simple rater counting • Consequently, other assessment methods of physicals signs, such as ultrasound evaluation, do need consideration. • More research is needed to examine intrarater agreement and to examine if interrater agreement can be improved e.g. by training

13. ConclusionsWhat does this study add? • In a prospective completely balanced design the study examined interrater reliability with HS-experienced dermatologists as the rater population of interest • The study did not find very good reliability for any included instrument or lesion counts • The study illustrates how difficult it is even for experts to agree on the type and number of lesions • The results question whether physical signs are best measured by a traditional physician lesion count instrument

15. Call for correspondence • Why not join the debate on this article through our correspondence section? • Rapid responses should not exceed 350 words, four references and one figure • Further details can be found here