Automatic Processing Pipelines with XNAT and REDCap Vanderbilt University

1. Automatic Processing Pipelines with XNATand REDCapVanderbilt University 2013-08-20 Benjamin Yvernault, Bennett Landman, Brian Boyd,

2. Requirements At this point, you should know how to : • Create project / sharing data • Browse data and view image previews • Search for projects, subjects, and data • Download imaging data, processed, and stored meta-data • Delete scans / sessions / projects • Use the Pre-Archive • Upload scans DICOMs • Quality Assurance

3. After this short course, you will be able to: • Create a project on REDCap • Set up the design of your REDCap project • Request API access for your project on REDCap • What is a spider: • source code • Which spiders already exist? • Run spiders from your computer • Run a spider on your local machine • ACCRE: run spiders (status / failed / ….)

4. Setting REDCap project for XNAT processing

5. REDCap: Setup the main instrument Variable Name: project_xnat-> type: Text Box Variable Name: subject_xnat -> type: Text Box Variable Name: experiment_xnat -> type: Text Box Variable Name: scan_xnat -> type: Text Box Variable Name: scan_type_xnat-> type: Text Box Variable Name: scan_sd_type-> type: Text Box Variable Name: script_name -> type: Text Box Variable Name: script_version -> type: Text Box Variable Name: pdf_report -> type: File Upload

6. REDCap: Upload main instrument Take time to create all this fields each time you have a new project. Variable Name: project_xnat-> type: Text Box Variable Name: subject_xnat -> type: Text Box Variable Name: experiment_xnat -> type: Text Box Variable Name: scan_xnat -> type: Text Box Variable Name: scan_type_xnat-> type: Text Box Variable Name: scan_sd_type-> type: Text Box Variable Name: script_name -> type: Text Box Variable Name: script_version -> type: Text Box Variable Name: pdf_report -> type: File Upload Upload the main instrument dictionary as a first step. File location: http://xnat.vanderbilt.edu -> Everything about spiders -> How to upload to REDCap -> link with the csv file.

7. REDCap: Download Shared Libraries

8. REDCap: API Key • To send data from XNAT to REDCap, you need to have an API key for your project on REDCap

9. REDCap Project Online Designer

10. Spiders … • Script executing a process on XNAT • Download images / data from XNAT • Execute a process on them • Copy the data to a directory • A specific spider to upload results on XNAT • All the spiders on http://xnat.vanderbilt.edu • Source code in the project Masimatlab on NITRC • Three kind of Spiders : • Spider that creates results for a Scan • Spider that corresponds to an assessor • Infrastructure spiders

11. Scan Spiders • Converter Spider : • Spider_dcm2nii_all.py (using dcm2nii) • Spider_dcm2nii_all_v2.py (using r2agui) • Changing attributes: • Spider_set_scan_type.py • Spider_Series_Description.py • Extracting/creating files: • Spider_physlog.py • Spider_Make_Preview_from_nifti.py

12. Process Spiders • This spiders can work on a scan level (one assessor per scan) or an experiment level (one assessor per session) • Spider that generates an assessor: • Spider_fMRQA.py • Spider_VBMQA.py • Spider_dtiQA_v2.py • Spider_labelFusion.py • Spider_intra_sess_reg.py • Spider_nonrigi_reg_to_ATLAS.py • Spider_FreesurferReconAllWithQC.py • Spider_ON_segmentation.py

13. Infrastructure Spider • Spider_report_for_projects.py to create a report on projects (same than Xnatreport) • Spider_send_data_to_redcap.py to send data from complet processes on XNAT to REDCap • Spider_Process_Upload to upload the processes that finished. Any process Spiders will copy the results needed to a specific directory. You should run it when you need to upload processes.

14. Spider flow Spider_process_upload.py / Xnatprocessupload Python script Download data Spiders Any language script: - Matlab/SPM - Perl - shell script Folder Run process UPLOAD_SPIDER_DIR

15. How to run a spider? • Spiders can run on your local machine if you have the requirements. • E.G: Most of the spiders use xvfb-run • python Spider_fMRIQA.py -h to see the options • Your .bashrc or .profile set up with your XNAT logs • Python packages install • Specific python package to run spiders on ACCRE develop by our lab.

16. ACCRE and Spiders • Advanced Computing Center for Research & Education • ACCRE is a Cluster for Vanderbilt University • Python script running every night on the cluster • Submitting one job per assessor • Status on the assessor related to the job status: • NEED_INPUTS-> the inputs files are not present on XNAT • NEED_TO_RUN-> ready to be submitted on ACCRE • JOB_RUNNING -> job submitted and running on ACCRE • JOB_FAILED -> job stop with an error • READY_TO_UPLOAD-> job ended and copy the results to the Upload directory • UPLOADING-> Spider_Process_Upload running and uploading results • READY_TO_COMPLETE-> a script will check and set the walltime and memory used by the job on ACCRE. • COMPLETE-> jobs done (processes finished and uploaded) with results on XNAT

17. ACCRE and CCI package Spider_process_upload.py / Xnatprocessupload Crontab job running every night a python script Get assessors that need to run CCI Package Folder UPLOAD_SPIDER_DIR Run jobs for each assessors (Spider flow on ACCRE)

18. Status sync between XNAT and REDCap

19. Interactive Assignment • Add to your .bashrcthe line (or .profile on Mac OS): export UPLOAD_SPIDER_DIR=path • Create a project on REDCap • Upload the dictionary from http://xnat.vanderbilt.edu • Everything about Spiders -> How to upload on REDCap • Download the shared libraries for fMRIQA / VBMQA / dtiQA • Run the Spider VBMQA on one T1 in the project you created in the last courses (it’s going to take 10mins) • Look at the results and upload them using Xnatprocessupload • Delete the project