Statistical Significance for Peptide Identification by Tandem Mass Spectrometry

1. Statistical Significance for Peptide Identification by Tandem Mass Spectrometry Nathan Edwards Center for Bioinformatics and Computational Biology University of Maryland, College Park

2. High Quality Peptide Identification: E-value < 10-8

3. Moderate quality peptide identification: E-value < 10-3

4. Peptide Identification • Peptide fragmentation by CID is poorly understood • MS/MS spectra represent incomplete information about amino-acid sequence • I/L, K/Q, GG/N, … • Correct identifications don’t come with a certificate!

5. Peptide Identification • High-throughput workflows demand we analyze all spectra, all the time. • Spectra may not contain enough information to be interpreted correctly • …bad static on a cell phone • Peptides may not match our assumptions • …its all Greek to me • “Don’t know”is an acceptable answer!

6. Peptide Identification • Rank the best peptide identifications • Is the top ranked peptide correct?

7. Peptide Identification • Rank the best peptide identifications • Is the top ranked peptide correct?

8. Peptide Identification • Rank the best peptide identifications • Is the top ranked peptide correct?

9. Peptide Identification • Incorrect peptide has best score • Correct peptide is missing? • Potential for incorrect conclusion • What score ensures no incorrect peptides? • Correct peptide has weak score • Insufficient fragmentation, poor score • Potential for weakened conclusion • What score ensures we find all correct peptides?

10. Statistical Significance • Can’t prove particular identifications are right or wrong... • ...need to know fragmentation in advance! • A minimal standard for identification scores... • ...better than guessing. • p-value, E-value, statistical significance

11. Pin the tail on the donkey…

12. Throwing darts One at a time Blindfolded Uniform distribution? Independent? Identically distributed? Pr [ Dart hits 20 ] = 0.05 Probability Concepts

13. Probability Concepts Throwing darts • One at a time • Blindfolded • Three darts Pr [Hitting 20 3 times] = 0.05 * 0.05 * 0.05 Pr [Hit 20 at least twice] = 0.007125 + 0.000125

14. Probability Concepts

15. Probability Concepts Throwing darts • One at a time • Blindfolded • 100 darts Pr [Hitting 20 3 times] = 0.139575 Pr [Hit 20 at least twice] = 0.9629188

16. Probability Concepts

17. Match Score • Dartboard represents the mass range of the spectrum • Peaks of a spectrum are “slices” • Width of slice corresponds to mass tolerance • Darts represent • random masses • masses of fragments of a random peptide • masses of peptides of a random protein • masses of biomarkers from a random class • How many darts do we get to throw?

18. 100 % Intensity 0 m/z 250 500 750 1000 Match Score What is the probability that we match at least 5 peaks? 270 330 870 550 755 580

19. Match Score • Pr [ Match ≥ s peaks ] = Binomial( p , n ) ≈ Poisson( p n ), for small p and large n p is prob. of random mass / peak match, n is number of darts (fragments in our answer)

20. Match Score Theoretical distribution • Used by OMSSA • Proposed, in various forms, by many. • Probability of random mass / peak match • IID (independent, identically distributed) • Based on match tolerance

21. Match Score Theoretical distribution assumptions • Each dart is independent • Peaks are not “related” • Each dart is identically distributed • Chance of random mass / peak match is the same for all peaks

22. Tournament Size 100 people 1000 people 100 Darts, # 20’s 100000 people 10000 people

23. Tournament Size 100 people 1000 people 100 Darts, # 20’s 100000 people 10000 people

24. Number of Trials • Tournament size == number of trials • Number of peptides tried • Related to sequence database size • Probability that a random match score is ≥ s • 1 – Pr [ all match scores < s ] • 1 – Pr [ match score < s ] Trials (*) • Assumes IID! • Expect value • E = Trials * Pr [ match ≥ s ] • Corresponds to Bonferroni bound on (*)

25. Better Dart Throwers

26. Better Random Models • Comparison with completely random model isn’t really fair • Match scores for real spectra with real peptides obey rules • Even incorrect peptides match with non-random structure!

27. Better Random Models • Want to generate random fragment masses (darts) that behave more like the real thing: • Some fragments are more likely than others • Some fragments depend on others • Theoretical models can only incorporate this structure to a limited extent.

28. Better Random Models • Generate random peptides • Real looking fragment masses • No theoretical model! • Must use empirical distribution • Usually require they have the correct precursor mass • Score function can model anything we like!

29. Better Random Models Fenyo & Beavis, Anal. Chem., 2003

30. Better Random Models Fenyo & Beavis, Anal. Chem., 2003

31. Better Random Models • Truly random peptides don’t look much like real peptides • Just use peptides from the sequence database! • Caveats: • Correct peptide (non-random) may be included • Peptides are not independent • Reverse sequence avoids only the first problem

32. Extrapolating from the Empirical Distribution • Often, the empirical shape is consistent with a theoretical model Geer et al., J. Proteome Research, 2004 Fenyo & Beavis, Anal. Chem., 2003

33. False Positive Rate Estimation • Each spectrum is a chance to be right, wrong, or inconclusive. • How many decisions are wrong? • Given identification criteria: • SEQUEST Xcorr, E-value, Score, etc., plus... • ...threshold • Use “decoy” sequences • random, reverse, cross-species • Identifications must be incorrect!

34. False Positive Rate Estimation • # FP in real search = # hits in decoy search • Need same size database, or rate conversion • FP Rate: # decoy hits # real hits • FP Rate: 2 x # decoy hits . (# real hits + # decoy hits)

35. False Positive Rate Estimation • A form of statistical significance • In “theory”, E-value and a FP rate are the same. • Search engine independent • Easy to implement • Assumes a single threshold for all spectra • Spectrum/Peptide Identification scores are not iid!... • ...but E-values, in principle, are.

36. Peptide Prophet • From the Institute for Systems Biology • Keller et al., Anal. Chem. 2002 • Re-analysis of SEQUEST results • Spectra are trials • Assumes that many of the spectra are not correctly identified

37. Peptide Prophet Keller et al., Anal. Chem. 2002 Distribution of spectral scores in the results

38. Peptide Prophet • Assumes a bimodal distribution of scores, with a particular shape • Ignores database size • …but it is included implicitly • Like empirical distribution for peptide sampling, can be applied to any score function • Can be applied to any search engines’ results

39. Peptide Prophet • Caveats • Are spectra scores sampled from the same distribution? • Is there enough correct identifications for second peak? • Are spectra independent observations? • Are distributions appropriately shaped? • Huge improvement over raw SEQUEST results

40. Peptides to Proteins Nesvizhskii et al., Anal. Chem. 2003

41. Peptides to Proteins

42. Peptides to Proteins • A peptide sequence may occur in many different protein sequences • Variants, paralogues, protein families • Separation, digestion and ionization is not well understood • Proteins in sequence database are extremely non-random, and very dependent

43. Publication Guidelines

44. Publication Guidelines • Computational parameters • Spectral processing • Sequence database • Search program • Statistical analysis • Number of peptides per protein • Each peptide sequence counts once! • Multiple forms of the same peptide count once!

45. Publication Guidelines • Single-peptide proteins must be explicitly justified by • Peptide sequence • N and C terminal amino-acids • Precursor mass and charge • Peptide Scores • Multiple forms of the peptide counted once! • Biological conclusions based on single-peptide proteins must show the spectrum

46. Publication Guidelines • More stringent requirements for PMF data analysis • Similar to that for tandem mass spectra • Management of protein redundancy • Peptides identified from a different species? • Spectra submission encouraged

47. Summary • Could guessing be as effective as a search? • More guesses improves the best guess • Better guessers help us be more discriminating • Peptide to proteins is not as simple as it seems • Publication guidelines reflect sound statistical principles.