Clayton W. Naeve , Ph.D. Endowed Chair in Bioinformatics SVP & CIO

1. The St. Jude Children’s Research Hospital/Washington University Pediatric Cancer Genome Project: A CIO’s Perspective Clayton W. Naeve, Ph.D. Endowed Chair in Bioinformatics SVP & CIO St. Jude Children’s Research Hospital

2. St. Jude Data: The First 50 Years PCGP Data: 917 TB, 148 million files 48 Years (800 TB) 2 1/2 Years (1000 TB) The Data Deluge

3. St. Jude/WashU Pediatric Cancer Genome Project • Launched Feb. 2010 • St. Jude/WashU collaboration • WGS on 600 patients (leukemia, brain tumors, solid tumors) • Matched germline and tumor samples • 1200 genomes (~90 billion bp/genome) in 36 months • ~2 Petabytes of data The PCGP Project

4. Challenges to Information Sciences • Moving data • Data workflow • Data analysis • Computational horsepower • Data storage • Data sharing PCGP Challenges

5. Multi-Terabyte data transit across networks is not trivial • DNA sequence raw data reads, contig assembly, alignment to reference, variants, etc. shipped to SJCRH as binary BAM files: ~100 GB • 24 hrs to infinity to send via commodity internet • Internet2 connectivity (10 Gbs via MRC) to transfer files from WashU to SJCRH • Evaluated 5 different fast data transfer algorithms….selected FDT (developed at CalTech to transfer LHC data at Cern) • Developed a pipeline to facilitate transfer • Today: ~5 hour transit time/file Moving Data

6. Moving Data

7. Moving Data

8. Began work on PCGP 9 months prior to launch • Developed a LIMS system for Validation Lab • Developed a PCGP SharePoint site to facilitate collaboration internally • Developed a bioinformatics workflow engine: PALLAS • Security management • Data provenance management • Intermediate and final result tracking • Flexible workflow design • Rapid new analytical algorithms/tools configuration • Web-based LSF job submission and monitoring • Support a range of protocols to connect to other web application systems, databases, file systems, and etc. • Integrated with applications, such as SRM, Genome Browser and etc. • Data integration with tissue sample, clinical, and research data • Vision: parse each algorithm to the appropriate computing environment Data Workflow

9. Jinghui Zhang and CompBio Team • BAM Quality Assurance: • Tumor Purity Algorithm (SJCRH) • Not Disease/Genomic Swap (SNP checks) • XenograftFilter (Remove Contaminating Mouse Reads) • Gene Exon and Genome Coverage algorithms (Gang Wu) • BAM file work: • Bam file extraction and visualization • Samtoolsand C++/bioperlapi’s • Bambino • IGV • Single Nucleotide Variation: • Freebayes • In-house PCGP • Copy Number Variation: • Stan’s Copy Number Algorithm • Regression Tree Algorithm • Structural Variation: • One End Anchored Inference: • CREST • ViralTopology • Fusion Detection: • In-house (Michael Rusch) • RNAseq: • RNAseqmysql/Cufflinks   • ChipSeq: • ChiPseqmysql/in house (John Obenauer) • viralScan • in-house (McGoldrick) • Integration: • GFF intersect • Gff2fasta • gffBuilders • Cancer warehouse • Visualization: • Circos maker • BED GFF Tracks maker Data Analyses

10. IBM BladeCenter (810 cores/3TB RAM) • IBM iDataplex (1,008 cores/4TB RAM) – April 2010 • SGI Altix UV1000 (640 cores/5TB RAM/60TB storage using Lustre v2.2) – December 2011 • IBM SoNAS(780 TB) – March 2011 • Data Transfer Node (10 Gbps I2 connection) – April 2011 • Internal Data Transfer Node (10 Gbps x2) – June 2011 • QDR Infiniband (40 Gbps for all HPC equipment) – January 2012 • Software (Platform LSF, Intel Parallel Studio) • Total: 2,366 cores, 13TB RAM (estimated 11.6 Tflops) • 2010: 365,000 cpu hours • 2011: 712,000 cpu hours Computational Horsepower (HPCF)

11. IBM SoNAS(780 TB) – March 2011 • Scales to 21PB; 1 billion files/filesystem; 7,200 drives • Current total on campus: 3.8 Petabytes (3,800,000 Gb) • PCGP uses 917 TB (<- +500TB on tape), 148 million data files • IBM TSM systems for backup/archive (Tiered) • 240 SAS (15k) drives • 480 SAS-NL (7.2k) drives • Current 7,900 tape capacity, up to 1.6TB/tape; 12.6+ PB total • 734 TB usable under one file system • High speed/low latency backend interconnect (QDR InfiniBand20Gb per port and 100ns latency) Data Storage

12. >356 Patients/712 Complete Genomes Gene sequencing project identifies potential drug targets in common childhood brain tumorNatureJune 20, 2012Researchers studying the genetic roots of the most common malignant childhood brain tumor have discovered missteps in three of the four subtypes of the cancer that involve genes already targeted for drug development. The most significant gene alterations are linked to subtypes of medulloblastoma that currently have the best and worst prognosis. They were among 41 genes associated for the first time to medulloblastoma by the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project. World's largest release of comprehensive human cancer genome data helps researchers everywhere speed discoveriesNature GeneticsMay 29, 2012To speed progress against cancer and other diseases, the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project today announced the largest-ever release of comprehensive human cancer genome data for free access by the global scientific community. The amount of information released more than doubles the volume of high-coverage, whole genome data currently available from all human genome sources combined. This information is valuable not just to cancer researchers, but also to scientists studying almost any disease. Genome sequencing initiative links altered gene to age-related neuroblastoma riskJournal of the American Medical AssociationMarch 13, 2012St. Jude Children’s Research Hospital – Washington University Pediatric Cancer Genome Project and Memorial Sloan-Kettering Cancer Center discover the first gene alteration associated with patient age and neuroblastoma outcome. Researchers have identified the first gene mutation associated with a chronic and often fatal form of neuroblastoma that typically strikes adolescents and young adults. The finding provides the first clue about the genetic basis of the long-recognized but poorly understood link between treatment outcome and age at diagnosis. Cancer sequencing initiative discovers mutations tied to aggressive childhood brain tumorsNature GeneticsJanuary 29, 2012Findings from the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project (PCGP) offer important insight into a poorly understood tumor that kills more than 90 percent of patients within two years. The tumor, diffuse intrinsic pontineglioma (DIPG), is found almost exclusively in children and accounts for 10 to 15 percent of pediatric tumors of the brain and central nervous system. Cancer sequencing project identifies potential approaches to combat aggressive leukemiaNature January 11, 2012Researchers with the St. Jude Children's Research Hospital - Washington University Pediatric Cancer Genome Project (PCGP) have discovered that a subtype of leukemia characterized by a poor prognosis is fueled by mutations in pathways distinctly different from a seemingly similar leukemia associated with a much better outcome. The work provides the first details of the genetic alterations fueling a subtype of acute lymphoblastic leukemia (ALL) known as early T-cell precursor ALL (ETP-ALL). The results suggest ETP-ALL has more in common with acute myeloid leukemia (AML) than with other subtypes of ALL. Gene identified as a new target for treatment of aggressive childhood eye tumorNature January 11, 2012New findings from the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project (PCGP) have helped identify the mechanism that makes the childhood eye tumor retinoblastoma so aggressive. The discovery explains why the tumor develops so rapidly while other cancers can take years or even decades to form. The finding also led investigators to a new treatment target and possible therapy for the rare childhood tumor of the retina, the light-sensing tissue at the back of the eye. Progress

13. http://www.pediatriccancergenomeproject.org http://explore.pediatriccancergenomeproject.org Data Sharing

14. Data Integration is critical: platform data (expression, WGS, methylation, etc.) and processed data (“genomics” data with phenotype data (clinical care, clinical research)) • Data Sharing

15. 19 Academic Departments 8-10 Faculty 50-60 Support Staff Information Sciences Shared Resources Computational Biology 2 PhD 2 Support Enterprise Informatics PCGP 5 PhD 1 Dev. 10 PhD Bioinformatics 2 developers 127 FTEs Research Informatics Clinical Informatics Offshore Developers 56 FTEs 81 FTEs HPC 15 FTEs Total=>150 FTEs with “research informatics” skills • Key: Staff

16. $ummary • Project total cost: $65M (11 Illuminas @ WashUand 4 @ SJCRH, sequencing costs, staffing, IT, etc.) • New “IT” staff @ SJCRH: 10 FTEs in CompBiol, 0 FTEs in IS • Capital IT investment: ~$7.2 M at SJCRH, $9M at WashU • IT is ~25% of overall project costs (doesn’t include costs of other participating SJ FTEs)

17. AshishPagare • David Zhao • Dan Alford • Stephen Espy • Kiran Chand Bobba • Scott Malone • Dr. Antonio Ferreira • Bill Pappas • James McMurry • Dr. Jianmin Wang • Dr. John Obenauer • Jared Becksfort • Pankaj Gupta • Dr. SurajMukatira • Simon Hagstrom • Sundeep Shakya • Asmita Vaidya • Swetha Mandava • Bhagavathy Krishna • Manohar Gorthi • Sandhya Rani Kolli • Sivaram Chintalapudi • Roshan Shrestha • Irina McGuire • PJ Stevens • Thanh Le • John Penrod • Pat Eddy • Dr. Dan McGoldrick Information Sciences PCGP Team • Key: Staff

18. Questions?

19. cluster large memory GPU SV CIRCOS Contig assembly CNV SNV INDELS PALLAS Data Workflow