1 / 30

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure. Anthony Kolasny Johns Hopkins University. Center for Imaging Science. Institute for Computational Medicine. Computational Anatomy.

ankti
Télécharger la présentation

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure Anthony Kolasny Johns Hopkins University

  2. Center for Imaging Science

  3. Institute for Computational Medicine

  4. Computational Anatomy • Computational Anatomy's goal is to define methods for the quantization of shape within biological structures. • Origins of Computational Anatomy (CA) may be found in the central thesis of Sir D'Arcy Wentworth Thompson's 1917 book entitled On Growth and Form. D'Arcy believed that biologists of his day over emphasized the role of evolution, and under emphasized the roles of physical laws and mechanics, as determinants of the form and structure of living organisms.

  5. Deformable Templates Image from D'arcy Thompson "On Growth and Form" "Computational anatomy: an emerging discipline," Ulf Grenander and Michael I. Miller (Quart. Appl. Math. 56[4]: 617-94, December 1998)‏

  6. Large Deformation Diffeomorphic Metric Mapping • The Large Deformation Diffeomorphic Metric Mapping (LDDMM) is an application which computes a metric distance on the space of anatomical images in Computational Anatomy thereby allowing for the direct comparison and quantization of morphometric changes in shapes.

  7. Metric Distance a metric or distance function is a function which defines a distance between elements of a set. finalMetricDistance.txt 6.4898851

  8. Multidimensional Scaling (MDS)‏ Metric multidimensional scalinng (MDS)  A superset of classical MDS that generalizes the optimization procedure to a variety of loss functions and input matrices of known distances with weights and so on. Linear discriminant analysis (LDA)‏ Used in statistics to find the linear combination of features which best separate two or more classes of object or event.

  9. Biomedical Informatics Research Network (BIRN)‏ BIRN is a National Center for Research Resources (NCRR) initiative aimed at creating a testbed to address biomedical researchers

  10. 4 3 5 TeraGrid Supercomputing Shape Analysis - A Morphometry BIRN Project Data Donor Sites 1 Storage De-identification And upload 2 JHU CIS-KKI Shape Analysis of Segmented Structures MGH Segmentation BWH Visualization Goal: comparison and quantification of structures’ shape and volumetric differences across patient populations

  11. Identifying Shape Analysis Requirements • 2002 – Collecting Requirements • Identify the testbed process - WashU/MGH/JHU/BWH • Identifying the population – Normal/Alzheimer's/Sematic Dementia • LDDMM only runs on an IBM SP (96 cpus, we use 8 cpus per job) • Each job takes about 8 hrs. • 101 datasets • We need to compare right/left unscaled/scaled (40,804 jobs)‏ • 40TB (cost for storage $300K)‏ • 326,432 cpu/hrs (37.2 cpu/years or 3.1 years on the IBM SP)‏ HELP!

  12. Leverage BIRN SRB Storage BIRN + TeraGrid would write to a common area Port LDDMM to utilize TeraGrid Work with Intel to optimize code. (30% speedup)‏ Provide a portal interface to LDDMM Allow other people access to the program Utilize VTK for visualization Utilize Wikis for documentation Leveraging the BIRN Infrastructure

  13. TeraGrid

  14. First & Second Iteration of Shape Analysis Project First Iteration Second Iteration • Processed 18 data sets • 1xN comparisons • Utilized IBM SP • Local Storage • Output VTK • Non-Conclusive • Processed 45 data sets • NxN comparisons • Utilized TeraGRid • Utilized SRB • Visualized VTK • Emerging Classifier

  15. Classification (45 Train only)‏ Training 1: Control 2: Alzheimer’s 3: SD {

  16. Preparing for the 101 Data Set • Already did 45x45 • 32,704 jobs to do. • 200,000 cpu/hrs – Utilize SDSC, NCSA, BIRN, JHU • 32 TB Storage – Utilize GPFS-WAN • Glue clusters and storage using sshfs • Implement a simple queue to submit to the various queueing systems. • Visualize with Paraview, Mayvi and 3D Slicer

  17. Classification (45 Train + 56 Test)‏ Training 1: Control 2: Alzheimer’s 3: SD Testing •: Control ▲: Alzheimer’s * : mean(1,•)‏ * : mean(2,3,▲)‏ {

  18. Statistical Inference 2 class 56 Test points 0.0042 ± 0.001 • Estimated p-values of a permutation test • Inference: The original LDDMM operation captures shape information in the MR images that is correlated with clinical diagnoses. Publications: M. Miller, C. Priebe, B. Fischl, A. Kolasny, Y. Park, E. Busa, J. Jovivich, P. Yu, B. Dickerson, R. Buckner, Morphometry BIRN , Collaborative Computational Anatomy:The Perfect Storm for MRI Morphometric Study of the Human Brain via Diffeomorphic Metric Mapping, Multidimensional Scaling and Linear Disriminant Analysis, Proceedings of the National Academy of Sciences - Submitted for review

  19. Secure Shell FileSystem (SSHFS)‏ a file system for Linux (and other operating systems with a FUSE implementation, such as Mac OS X) capable of operating on files on a remote computer using just a secure shell login on the remote computer. On the local computer where the SSHFS is mounted, the implementation makes use of the FUSE (Filesystem in Userspace) kernel module. The practical effect of this is that the end user can seamlessly interact with remote files being securely served over SSH just as if they were local files on his/her computer. The current implementation of SSHFS using FUSE is a rewrite of an earlier version. The rewrite was done by Miklos Szeredi, who also wrote FUSE. sshfs remoteuser@remotehost:/path/to/remote_dir local_mountpoint

  20. Autofs and SSHFS Autofs allows automatic mounting of remote sshfs filesytems. CIS users may access remote data as local directories. Common local directory structure allows for more effective scripting and analysis of data.

  21. Gluing Clusters and Storage with SSHFS Utilizing samba and smbwebclient, we were able create a web interface to clustered data.

  22. Common Data Namespace Improved Throughput /gpfs-wan is shared at SDSC and NCSA in a common location. Using sshfs, /gpfs-wan was mounted on BIRN SDSC Cluster and the BIRN JHU cluster adding more processing power. • run – Contains the list of jobs for the queues at SDSC, NCSA, BIRN, JHU • submit_q_script_ncsa – monitors qstatus and adds more jobs on an hourly basis • ncsa1 – Contains current jobs running in the NCSA queue • done – when job is finished, send it to the done directory

  23. Monitoring the Queues Monitoring the SDSC, NCSA, JHU and BIRN clusters is as simple as reading email. Monitoring what's left to run is as easy as 'cd run; ls | wc'.

  24. Wikis Wikis are extremely helpful in keeping track of projects and monitor progress

  25. Visulalization Utilizing Paraview we are able to visualize structures and velocity data.

  26. BIRN LDDMM Portal • Provides access to lddmm • Utilized BIRN Infrastructure

  27. $300K - $40K = Saved $260K in storage costs. Invested $70k in development cluster. Current cumulative TeraGrid time 640K cpu/hrs. TeraGrid help desk an extremely valuable asset. Savings

  28. Timeline 2002.09 - BIRN All Hands 2002.10 - Supercomputing Baltimore emerging TeraGrid 2003.02 - Morph BIRN - define SASHA project 2003.05 - BIRN Rack installed 2003.08 - lddmm writes vtk output 2003.09 - Installed itanium2 cluster 2003.09 - lddmm processing on IBM SP (18 subjects)‏ 2003.10 - BIRN All Hands – Shape Analysis Pipeline (not conclusive from 18 subjects)‏ 2004.02 - Morph BIRN All Hands 2004.02 - 40K cpu/hrs award 45x45 processing used SRB 2004.06 - Human Brain Mapping - Emerging Classifier 2004.08 - Intel HPC workshop 30% performance improvement 2005.09 - 300K cpu/hrs award 2006.01 - Started processing 101x101 processing using GPFS-WAN 2006.06 - Human Brain Mapping Conference – Present lddmm, MDS, LDA 2006.09 - Submitted PNAS paper 2006.10 - SDSC Calendar highlights BIRN Shape Analysis Project 2006.09 - 300K cpu/hrs other lddmm projects Mouse BIRN, OASIS, VETSA, ADNI

  29. http://en.wikipedia.org/wiki/D'Arcy_Thompson http://en.wikipedia.org/wiki/Pattern_theory http://www.cis.jhu.edu/software http://www.cis.jhu.edu/portal/birn/ http://www.cis.jhu.edu/software/lddmm/clinical.html http://en.wikipedia.org/wiki/Multidimensional_scaling http://www.analytictech.com/borgatti/papers/Visualizing_proximities.pdf http://www.nbirn.net/press/archive/ahm_2006/ppts/sshfs_ahm2006.ppt http://en.wikipedia.org/wiki/SSHFS http://smbwebclient.sourceforge.net/ http://paraview.org/ http://www.slicer.org/ http://www.sdsc.edu/News%20Items/PR100606.html References

  30. Contributors CIS Members Michael I. Miller Carey Priebe Can Ceritoglu Timothy Brown Youngser Park CIS Alumni Faisal Beg BIRN Collaborators Mark Ellisman Steve Pieper Bruce Fischl Randy Buckner TeraGrid Support Amitava Majumdar Nancy Wilkins-Diehr Thank You

More Related