iPRG 2012: A Study on Detecting Modified Peptides in a Complex Mixture

1. iPRG 2012:A Study on Detecting Modified Peptides in a Complex Mixture ABRF 2012, Orlando, FL 3/17-20/2012

2. iPRG2012 Study:DESIGN

3. Study Goals Primary: Evaluate the ability of participants to identify modified peptides present in a complex mixture Secondary: Find out why result sets might differ between participants Tertiary: Produce a benchmark dataset, along with an analysis resource

4. Study Design Use a common, rich dataset Use a common sequence database Allow participants to use the bioinformatic tools and methods of their choosing Use a common reporting template Report results at an estimated 1% FDR (at the spectrum level) Ignore protein inference

5. Sample • Tryptic digest of yeast (RM8323 – NIST), spiked with 69 synthetic modified peptides (tryptic peptides from 6 different proteins – sPRG) • Phospho (STY) • Sulfo (Y) • Mono-, di-, trimethyl (K) • Mono-, dimethyl (R) • Acetyl (K) • Nitro (Y)

6. Supplied Study Materials • 5600 TripleTOF dataset (i.e. WIFF file) • WIFF, mzML, dta, MGF (de-isotoped);– conversions by MS Data Converter 1.1.0 • MGF (not de-isotoped – conversion by Mascot Distiller 2.4) • 1 fasta file (UniProtKB/SwissProtS. cerevisiae, human, + 1 bovine protein + trypsin from Dec. 2011) • 1 template (Excel) • 1 on-line survey (Survey Monkey)

7. Instructions to Participants Retrieve and analyze the data file in the format of your choosing, with the method(s) of your choosing Report the peptide to spectrum matches in the provided template Report measures of reliability for PTM site assignments (optional) Fill out the survey Attach a 1-2 page description of the methodology employed

8. iPRG 2012 STUDY:PARTICIPATION

9. Soliciting Participants and Logistics Study advertised on the ABRF website and listserv and by direct invitation from iPRG members 1. Email participation request to ‘iPRGxxxx@gmail.com’ Participant 2. Send official study letter with instructions iPRG members Questions / Answers 3. All further communication (e.g., questions, submission) through ‘iPRGxxx.anonymous@gmail.com’ “Anonymizer”

10. Participants (i) – overall numbers • 24 submissions • One participant submitted two result sets • 9 initialed iPRG member submissions (with appended ‘i’) • 2 vendor submissions (identifiable by appended ‘v’)

11. About the Participant 11

12. About the Participant’s Lab 12

13. Participation in sPRG Study • Only one participant indicated he used sPRG information to aid his analysis. • This person was one of the least successful in identifying the spiked-in peptides!

14. Search Engine Used 14

15. Site Localization Software • 4 participants did not list using software for site localization.

16. Summary of Submitted Results Only reported modified peptides

17. Summary of IDs and Localizations Peptide Identification in all Spectra Site Localization in Spectra With Interesting Modifications

18. Overlap of spectrum identifications 7840 agreed on by 3 or more participants

19. Room for improvement in thresholding?

20. ESR and FDR Extraordinary Skill Rate or High False Discovery Rate?ESR + FDR = 100* (Y<3P+YD)/total ids Y 24 participants 3 for consensus

21. Characteristics of consensus spectra 7840 spectra >=3 participants agreeing on sequence Consensus requires agreement on Sequence, but not modification localization

22. Peak lists • Two types of peak lists were supplied • Deisotoped and non deisotoped • Can only tell fragment charge state from non-deisotoped • Requires search engine to be able to de-isotope spectrum

23. Peaklists • Number of spectra with undefined precursor charge state Deisotoped 1031 (304 in consensus results) Non-deisotoped 6094 (1140 in consensus results) • For 1013 out of 7840 consensus spectra the precursor m/z differ by greater than 0.02 Da between deisotoped and non-deisotoped peak list. • For 238 consensus spectra the peak lists had different specified charge state • 193 consensus results only possible with deisotoped peak list • 45 consensus results only possible with non-deisotoped peak list • For 19 consensus results multiple people who searched the nd peak list agreed on a confident different answer • For 4 consensus results multiple people who searched the deisotoped peak list agreed on a confident different answer

24. Mixed Spectra 465.19 2+ 464.59 3+ 465.19 2+ Deisotoped peaklist 464.59 3+ Non-deisotoped peaklist

25. Synthetic Peptide ID by Peptide Sulfo Trimethyl Methyl (K) Methyl (R) Phospho Dimethyl (K) Dimethyl (R) Acetyl Nitro # participants # participants

26. Synthetic Peptide ID by Participant

27. Correct Localization of Modified Synthetic Peptides 70 synthetic modified peptides were spiked into sample. 7 of these were confidently found by no participant Correct localization & name of modification reported

28. FLR of Modified Synthetic Peptides Ignored PSMs contain mods of residues other than s,t,y,k,r . Sample handling mods (n,q,d,e, etc). FLR = 100% * # PSMs wrong localization of s,t,y,k,r # PSMs wrong + right localization of s,t,y,k,r

29. Incorrect Localization by Peptide • Number of PSM’s with Incorrect Site Localization – Mod Loc Confidence Y • Present as sulfo-Tyr • Present as phospho S-10 often mislocalized as S-12 or Y-14 • Present as mono, di, tri methyl K often mislocalized at R

30. PhosphovsSulfo DISLSDY(Phospho)K Observe modified fragment ions. DISLSDY(Sulfo)K Observe ‘unmodified’ fragment ions. Spectrum looks essentially identical to unmodified peptide spectrum

31. Conclusions • Reasonable number of participants from around the globe, mainly experienced users but a few first-timers • Large spread in number of spectra identified • False negatives (NS) are generally much higher than false positives, so there is generally room for improvement • Peak list was a significant factor on performance • Varied performance in detecting PTMs • Most participants struggled with sulfation • Multiply phosphorylated harder to find than singly • Most common errors in site assignment were: • Reporting sulfo(Y) as phospho(ST) • Mis-assignment of site/s in multiply phosphorylated peptides

32. What did the participants think? “The spiked proteins made it possible to game the study - look for the uncommon modifications only on the spikes. Of course we didn't do this. Overall I'd say this was a flawed but very interesting ABRF study.” 22 out of 24 participants found the study useful “Too many modifications at the same time. Manual validation is necessary and the right time necessary for this study is too demanding for this challenge.”

33. Participant’s Confidence in Analyzing PTM Data Before After 33

34. How difficult do you think this study was? What was your total analysis time for the entire project?

35. Based on this study, would you consider participating in future ABRF studies?

36. Thank you! Questions? THANK YOU TO ALL STUDY PARTICIPANTS! iPRG Nuno Bandeira Robert Chalkley(chair) Matt Chambers Karl Clauser John Cottrell Eric Deutsch Eugene Kapp Henry Lam Hayes McDonald Tom Neubert (EB liaison) Ruixiang Sun Dataset Creation Chris Colangelo Anonymizer: Jeremy Carver, UCSD