1. Conditional Graphical Models forProtein Structure Prediction Yan Liu Language Technologies Institute School of Computer Science Carnegie Mellon University Oct 24, 2006

2. Nobelprize.org DSCTFTTAAAAKAGKAKAG Protein sequence + Protein function Protein structure Snapshot of Cell Biology

3. Protein Structures and Functions Example: triple beta-spiral fold Adenovirus Fibre Shaft Virus Capsid Courtesy of Nobelprize.org

4. Protein Structure Determination • Lab experiments: time and labor- consuming • X-ray crystallography Nobel Prize, Kendrew & Perutz, 1962 • NMR spectroscopy Nobel Prize, Kurt Wuthrich, 2002 • The gap between sequence and structure necessitates computational methods of protein structure determination • 3,023,461 sequences v.s. 36,247 resolved structures (1.2%) 1MBN 1BUS

5. Protein Structure Hierarchy We focus on predicting the topology of the structures from sequences APAFSVSPASGACGPECA

6. Major Challenges • Protein structures are non-linear • Long-range dependencies • Structural similarity often does not indicate sequence similarity • Sequence alignment reaches twilight zone (under 25% similarity) β-α-β motif Ubiquitin (blue) Ubx-Faf1 (gold)

7. Previous Work • Sequence similarity perspective • Sequence similarity searches, e.g. PSI-BLAST [Altschul et al, 1997] • Profile HMM, .e.g. HMMER [Durbin et al, 1998] and SAM [Karplus et al, 1998] • Window-based methods, e.g. PSI_pred [Jones, 2001] • Physical forces perspective • Homology modeling or threading, e.g. Threader [Jones, 1998] • Structural biology perspective • Methods of careful design for specific structures, e.g.αα- and ββ- hairpins, β-turn and β-helix [Efimov, 1991; Wilmot and Thornton, 1990; Bradley at al, 2001] Fail to capture the structure properties Generative models based on physical free-energy Hard to generalize due to the various informative features

8. Structured Prediction • Many prediction tasks involve outputs with correlations or constraints Structure Sequence • Tree Grid Input John ate the cat . SEQUENCEXS…WGIKQLQAR Output HHHCCCEEE…EECCCCEEE • Fundamental importance in many areas • Potential for significant theoretical and practical advances

9. Graphical Models • A graphical model is a graph representation of probability dependencies [Pearl 1993; Jordan 1999] • Node: random variables • Edges: dependency relations • Directed graphical model (Bayesian networks) • Undirected graphical model (Markov random fields)

10. Conditional Random Fields • Hidden Markov model (HMM)[Rabiner, 1989] • Conditional random fields (CRFs)[Lafferty et al, 2001] • Model conditional probability directly • Allow arbitrary dependencies in observation • Adaptive to different loss functions and regularizers • Promising results in multiple applications

11. Protein Structure Prediction • Dependency between residues (single observation) • Dependency between components (subsequences of observations)

12. Outline • Brief introduction to protein structures • Graphical models for structured-prediction • Conditional graphical models for protein structure prediction • General framework • Specific models • Experiment results • Conclusion and discussion

13. Our Solution: Conditional Graphical Models Local dependency Long-range dependency • Outputs Y = {M, {Wi} }, where Wi = {pi, qi, si} • Feature definition • Node feature • Local interaction feature • Long-range interaction feature

14. Conditional Graphical Models (II) • Conditional probability given observed sequences x is defined as • Prediction: • Training phase : learn the model parameters λ • Minimizing regularized negative log loss • Iterative search algorithms by seeking the direction whose empirical values agree with the expectation

15. Major Components • Graph topology • Secondary structure prediction: CRF, kernel CRF • Tertiary fold recognition: Segmentation CRF, Chain graph model • Quaternary fold recognition: Linked segmentation CRF • Efficient inference • Prefer exact inference with O(nd) complexity • Resort to approximate inference • Features • Allows flexible and rich feature definition

16. Protein Secondary Structure Prediction • Given a protein sequence, predict its secondary structure assignments • Three classes: helix (H), sheets (E) and coil (C) • Input: APAFSVSPASGACGPECA • Output: CCEEEEECCCCCHHHCCC

17. CRF on Secondary Structure Prediction [Liu et al, Bioinformatics 2004] C C E E …. ... C • Node semantics –secondary structure assignment • Graphical model - conditional random fields (CRFs) or kernel CRF • Inference algorithm - efficient inferences exists, such as forward-backward or Viterbi algorithm

18. Training Phase Testing Phase • Input: ..APAFSVSPASGACGPECA.. • Output 1: Does the target fold exist? • Output 2: ..NNEEEEECCCCCHHHCCC.. Yes Protein Fold Recognition and Alignment • Protein fold: identifiable regular arrangement of secondary structural elements • Different from previous simple fold classification • Provide important information and novel biological insights

19. Conditional Graphical Model for Fixed Template Fold[Liu et al, RECOMB 2005] • Node semantics - secondary structure elements of variable lengths • Graphical model - segmentation conditional random fields (SCRFs) • Inference - forward-backward and Viterbi-like algorithm can be derived given some assumptions β-α-β motif

20. Conditional Graphical Model for Repetitive Fold Recognition [Liu et al, ICML 2005] • Node semantics - two layer segmentation Y = {M, {Ξi}, T} • Level 1: envelop, or one repeat, level 2: components of one repeat • Graphical model - Chain graph model • A graph consisting of directed and undirected graphs • Inference - forward-backward algorithm and Viterbi-like algorithm

21. Conditional Graphical Model for for Quaternary Fold Recognition[Liu et al, IJCAI 2007] • Node semantics – secondary structure elements and/or simple fold • Graphical model - linked segmentation CRF (L-SCRF) • Fix template and/or repetitive subunits • Inter-chain and intra-chain interactions

22. Approximate Inference • Varying dimensionality requires reversible jump MCMC sampling [Greens, 1995, Schmidler et al, 2001] • Four types of Metropolis proposals • State switching • Position switching • Segment split • Segment merge • Simulated annealing reversible jump MCMC [Andireu et al, 2000] • Replace the sample with RJ MCMC • Theoretically converge on the global optimum

23. Conditional Graphical Models for Protein Structure Prediction

24. Kernelization Segment Correlations Local and Global Tradeoff Inter-chain Segment Correlations Model Roadmap Generalized as conditional graphical models Conditional random fields Kernel CRFs Segmentation CRFs Chain graph model Linked segmentation CRFs

25. Outline • Brief introduction to protein structures • Graphical models for structured prediction • Conditional graphical models for protein structure prediction • Experiment results • Fold recognition • Fold alignment prediction • Discovery of potential membership proteins • Conclusion and discussion

26. Experiments: Target Fold • Right-handedβ-helix fold [Yoder et al, 1993] • Bacterial infection of plants, binding the O-antigen and so on • Leucine-rich repeats (LLR) [Kobe & Deisenhofer, 1994] • Structural framework for protein-protein interaction

27. Experiments: Target Quaternary Fold • Triple beta-spirals [van Raaij et al. Nature 1999] • Virus fibers in adenovirus, reovirus and PRD1 • Double barrel trimer [Benson et al, 2004] • Coat protein of adenovirus, PRD1, STIV, PBCV

28. Tertiary Fold Recognition: β-Helix fold • Histogram and ranks for known β-helices against PDB-minus dataset 5 Chain graph model reduces the real running time of SCRFs model by around 50 times

29. Quaternary Fold Recognition: Triple β-Spirals • Histogram and ranks for known triple β-spirals against PDB-minus dataset

30. Quaternary Fold Recognition: Double Barrel-Trimer • Histogram and ranks for known double barrel-trimer against PDB-minus dataset

31. Fold Alignment Prediction:β-Helix • Predicted alignment for known β-helices on cross-family validation

32. Fold Alignment Prediction:LLR and Triple β-Spirals • Predicted alignment for known LLRs using chain graph model (left) and triple β-spirals using L-SCRFs

33. Discovery of Potential β-helices • Hypothesize potential β-helices from Uniprot reference databases • Full list can be accessed at www.cs.cmu.edu/~yanliu/SCRF.html • Verification on proteins with later resolved structures from different organisms • 1YP2: potato tuber ADP-glucose pyrophosphorylase • 1PXZ: major allergen from Cedar Pollen • GP14 of Shigella bacteriophage as a β-helix protein

34. Conclusion • Thesis Statement • Conditional graphical models are effective for protein structure prediction • Strong claims • Effective representation for protein structural properties • Flexibility to incorporate different kinds of informative features • Efficient inference algorithms for large-scale applications • Weak claims • Ability to handle long-range interactions • Best performance bounded by prior knowledge

35. Contribution and Limitation • Contribution to machine learning • Enrichment of graphical models • Formulation to incorporate domain knowledge • Contribution to computational biology • Effective for protein structure prediction and fold recognition • Solutions for the long-range interactions (inter-chain and intra-chain) • Limitation • Manual feature extraction • Difficulty in verification • High complexity

36. Protein structure prediction Protein function and protein-protein interaction prediction Drug target design Graph-based semi-supervised learning Active learning for structured data Graph topology learning + Future Work • Computational biology • Machine Learning

37. Acknowledgement • Jaime Carbonell, Eric Xing, John Lafferty, Vanathi Gopalakrishnan • Chris Langmead, Yiming Yang, Roni Rosenfeld, Peter Weigele , Jonathan King, Judith Klein-Seetharaman, , Ivet Bahar, James Conway and many more • And fellow graduate students …

39. Features for Tertiary Fold Recognition • Node features • Regular expression template, HMM profiles • Secondary structure prediction scores • Segment length • Inter-node features • β-strand Side-chain alignment scores • Preferences for parallel alignment scores • Distance between adjacent B23 segments • Features are general and easy to extend

40. Features for Protein Fold Recognition

41. Discovery of Potential Double Barrel-Trimer • Potential proteins suggested in [Benson, 2005]

42. Inference Algorithm for SCRF • Backward-forward algorithm* • Viterbi algorithm* p(state yr ends at r |xl+1 xl+2… xr-1xrand state yl ends at l) =

43. Contrastive Divergence

44. Reversible jump MCMC Algorithm • Three types of proposals • Position switching: randomly select a segment j and a new position assignment dj(i+1) ~U(dj-1(i),dj+1(i)) • Segment split: randomly select a segment j and split it into two segments where (dj(i+1) , dj+1(i+1) ) = G(dj-1(i) ,u(i) ) where u(i) ~ U • Segment merge: randomly select a segment j and merge segment j and j+1 • Simulated annealing reversible jump MCMC for computing y = argmax P(y|x) [Andireu et al, 2000]

45. Simulated annealing reversible jump MCMC

46. Protein Structural Graph for Beta-helix

47. Protein Structure Determination • Lab experiments: time and labor- consuming • X-ray crystallography • NMR spectroscopy • Electron microscopy and many more • Computational methods: • Homology modeling: ≥ 30% sequence similarity • Fold recognition: < 30% sequence similarity • Ab inito methods: no template structure needed • Active research area in multiple scientific fields

48. Q3 (accuracy) Precision, Recall Segment Overlap quantity (SOV) Matthew’s Correlation coefficients Evaluation Measure

49. Outline • Brief introduction to protein structures • Discriminative graphical models • Generalized discriminative graphical models for protein fold recognition • Experiment results • Conclusion and discussion

50. Graphical Models for Structured Prediction • Conditional Random Fields • Model conditional probability directly, not joint probability • Allow arbitrary dependencies in observation (e.g. long range, overlapping) • Adaptive to different loss functions and regularizers • Promising results in multiple applications • Recent developments • Alternative estimation algorithms (Collins, 2002, Dietterich et al, 2004) • Alternative loss functions, use of kernels (Taskar et al., 2003, Altun et al, 2003, Tsochantaridis et al, 2004) • Baysian formulation (Qi and Minka, 2005) and semi-markov version (Sarawagi and cohen, 2004)