Author Classification of ARVO Conference Abstracts as Controlled Clinical Trials

1. 2004 2004 2005 2005 Gold Standard By hand searcher Yes No Total Yes 269 0 269 No 26 5315 5341 Total 295 5315 5610 Gold Standard By Author Yes No Total Yes 229 405 634 No 66 4910 4976 Total 295 5315 5610 Gold Standard By hand searcher Yes No Total Yes 307 0 307 No 22 5403 5403 Total 329 5403 5732 Gold Standard By Author Yes No Total Yes 232 253 485 No 97 5150 5247 Total 329 5403 5732 Author Classification of ARVO Conference Abstracts as Controlled Clinical Trials Roberta W. Scherer1, Pamela C. Sieving2, and Kay Dickersin1, Johns Hopkins Bloomberg School of Public Health1, NIH Library2 RESULTS OBJECTIVE Author Identification of Clinical Trials by Design Sub-classification • 295 of 5,610 abstracts from the 2004 ARVO meeting, and 329 of 5,732 abstracts from the 2005 ARVO meeting comprised the gold standard of abstracts describing CTs. • Abstract authors checked the box indicating that they considered the abstract to describe a human CT in 634 abstracts in 2004, and 485 in 2005. To test whether abstract authors could contribute to efficient identification of controlled clinical trial (CT) reports published in ARVO proceedings. BACKGROUND 2004 2005 Only about 60% of controlled clinical trial (CTs) presented at scientific meetings and published as abstracts in conference proceedings are published in full (Scherer, 2005). Accordingly, it is important to handsearch conference proceedings to ensure comprehensive trial identification for systematic reviews. The annual meeting of the Association for Research in Vision and Ophthalmology (ARVO) is an important source of reports of clinical trials in the field of vision yet only a small proportion of abstracts describe CTs (e.g., 214/5,061 (4%) 2001 ARVO abstracts were identified as CTs by a handsearcher.) Handsearching ARVO abstracts is thus not an efficient method of identifying CTs. Classification as Clinical Trial by Author Most of the differences in classification between authors and handsearchers are among abstracts categorized as controlled clinical trials (CCTs). Authors identified 57.5% abstracts categorized as CCTs in 2004 while the handsearcher identified 86.8% (92/108). In 2005, authors identified 46.2% (58/119), while the handsearcher identified 84.0% (100/119) of CCTs. Authors may more readily understand that randomized studies (RCTs) are trials compared with CCTs. A study is a CCT if it could possibly be a randomized or quasi-randomized study, so another explanation is that some trials are incorrectly classified as trials when they are not by the handsearcher. In 2004, authors correctly classified CTs for 229/295 CT abstracts, (sensitivity = 77.6%). However, authors failed to recognize 66/295 CTs (22.4%) and mis-classified 405 studies as CTs that were not (positive predictive value = 229/634, 36.1%). In 2005, sensitivity was lower than for 2004 (70.5%, 232/329 CT abstracts) and there were more false negatives (97/329; 29.5%) but the positive predictive value was higher (47.8% (232/485 abstracts)). COMMENT AND FUTURE DIRECTIONS METHODS Classification as Clinical Trial by Handsearcher • Comment: Earlier studies showed handsearching of the printed ARVO meeting abstracts requires a considerable time investment (about 45 hours per annum). Our project to develop a valid and reliable system for author classification of CTs and electronic searching has the potential to shorten this time. Because we found a large proportion of abstracts misclassifed (false positives) in 2004, the 2005 ARVO abstract submission information was modified by changing the study design question to ask “Is the research presented in your abstract a human controlled clinical trial?” • Failure to identify abstracts as human CTs (false negatives) is of greater concern than false positives, as omission is less easily addressed than overselection in the handsearching process. The proportion of false negatives increased, if anything, between 2004 and 2005. • Asking authors to classify abstracts identifies a small number of abstracts (26 and 22, respectively) that would be missed by a handsearcher. • Some abstracts describing CTs may have been missed by both the author and the handsearcher; this is a limitation of the study. • Future directions • We plan to explore: • How best to ensure capture of CTs not so identified by authors (false negatives). • Identification of a concise set of text words and terms with high sensitivity for identification of CTs in electronic-only meeting abstracts. These will be validated using proceedings from years other than 2004 or 2005. • Keywords and abstract classification codes provided with the ARVO abstracts, to determine their usefulness in searching.  At our request, the ARVO conference organizers included a question and accompanying definition on the electronic abstract submission form for 2004: “Is the research presented in your abstract a human clinical trial? [Yes] [No].” In 2005, the question was modified to “Is the research presented in your abstract a humancontrolled clinical trial? [Yes] [No]. For both years, the definition, obtainable via hypertext of the blue underlined text, was “A planned study in humans designed to assess the efficacy and/or safety of one or more test interventions by comparing outcomes in individuals assigned the test intervention(s) with those receiving no intervention or a comparison intervention, and where individuals in all groups are enrolled, treated, and followed concurrently.” One of us (RS), masked to authors’ CT classification, handsearched 2004 and 2005 ARVO abstracts online. A gold standard of abstracts classified as CTs was determined by examination of the two sets of abstracts: 1) abstracts classified as CTs by the author and 2) those classified as CTs by the hand-searcher as CTs. Each abstract in the gold standard set was further categorized as a randomized controlled trial (RCT) or controlled clinical trial (CCT), using Cochrane Collaboration definitions. We evaluated the author and handsearcher classifications against the gold standard, and explored factors associated with author misclassification. Of the 295 abstracts in the gold standard, the handsearcher classified 91.1% (269/295) as a CT in 2004 and 93.3% (307/329) in 2005. Reasons* for Misclassification of Abstracts as a CT by the Abstract Author 2004 2005 CONCLUSIONS • More than one reason applied in some abstracts. The most common reason for misclassification (false positives) by abstract authors was that there was no comparison group, ie., the study was a prospective series with everyone receiving the same treatment. The second most common reason was participants were assigned to a group using some participant characteristic (such as having a disease or not). Typically, these were case control studies. • Neither the author checkox nor handsearching identified all CT abstracts. • Including the checkbox on the abstract submission form identifies a small groups of abstracts missed by handsearching. • RCTs were more readily identified as clinical trials by both authors or hand- searchers compared to CCTs (quasi-randomized or possible randomized trials).