Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks

1. Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks Alexis Battle Gal Chechik Daphne Koller Department of Computer Science Stanford University

2. PBAI Competition • Provided rich data set • Interesting interactions across time, subjects, and stimuli • Challenged us to come up with reliable techniques • Thanks to the organizers!

3. Key Points • Predictive voxels selected from whole brain • Probabilistic model makes use of additional correlations • Subjects’ ratings across time steps • Ratings between subjects • Learn strength of each relationship

4. Modeling the fMRI Domain User Ratings BOLD signal funny tim body language Voxels across time Ratings across time A joint distributionin high dimension

5. Modeling the fMRI Domain User Ratings BOLD signal funny tim body language Voxels across time Ratings across time A joint distributionin high dimension Training:Use two movies to learn the relations between voxels and ratings

6. Modeling the fMRI Domain User Ratings BOLD signal funny tim body language Voxels across time Ratings across time A joint distributionin high dimension Testing:Use the learned relations to predict ratings from fMRImeasurements Training:Use two movies to learn the relations between voxels and ratings

7. Probabilistic Model • Each voxel measurement • Each rating to predict from Vox1 Vox2 Vox3 Language

8. Probabilistic Model • Each voxel measurement • Each rating to predict • Rating predicted from voxel measurements • Linear regression model (Gaussian distribution) from Vox1 Vox2 Vox3 Language

9. Probabilistic Model • Each voxel measurement • Each rating to predict • Rating predicted from voxel measurements • Linear regression model (Gaussian distribution) • Selected predictive voxels from whole brain • Regularize (Ridge, Lasso) to handle noise from Vox1 Vox2 Vox3 Language

10. Probabilistic Model Vox1 Vox2 Vox1 Vox2 … Language Language T =1 T =2

11. Probabilistic Model • Ratings correlated across time • Language at time 1 makes language at time 2 likely Vox1 Vox2 Vox1 Vox2 … Language Language T =1 T =2

12. Probabilistic Model • Ratings correlated across time • Language at time 1 makes language at time 2 likely Vox1 Vox2 Vox1 Vox2 … Language Language T =1 T =2

13. Probabilistic Model A*Lang (1)*Lang(2) • Ratings correlated across time • Language at time 1 makes language at time 2 likely • Weight A – how correlated? Vox1 Vox2 Vox1 Vox2 A … Language Language T =1 T =2

14. Probabilistic Model Vox1 Vox2 Vox1 Vox2 Language Language Subject 1 Vox1 Vox2 Vox1 Vox2 Language Language Subject2 … T =2 T =1

15. Probabilistic Model Vox1 Vox2 Vox1 Vox2 • Ratings likely to be correlated between subjects Language Language Subject 1 Vox1 Vox2 Vox1 Vox2 Language Language Subject2 … T =2 T =1

16. Probabilistic Model Vox1 Vox2 Vox1 Vox2 • Ratings likely to be correlated between subjects • Weighted correlation, NOT equality Language Language Subject 1 B Vox1 Vox2 Vox1 Vox2 B Language Language Subject2 … T =2 T =1

17. Probabilistic Model Joint model over all time points: Sub1 … Sub2 Time Gaussian Markov Random Field – joint Gaussian over all rating nodes conditioned on voxel data

18. Voxel Parameters Vox1 Vox2 Vox3 • Regularized linear regression for voxel parameters Language

19. Voxel Parameters Vox1 Vox2 Vox3 • Regularized linear regression for voxel parameters Language

20. Voxel Parameters Vox1 Vox2 Vox3 • Regularized linear regression for voxel parameters Language β1= 0.45 β2 = 0.55

21. Inter-Rating parameters • Other weights also learned from data • Example: cross-subject weights Vox1 Vox2 L(1) B Vox1 Vox2 L(2) C = 0.6

22. Inter-Rating parameters • Other weights also learned from data • Example: cross-subject weights Vox1 Vox2 Faces Attention L(1) B Vox1 Vox2 L(2) C = 0.6

23. Inter-Rating parameters • Other weights also learned from data • Example: cross-subject weights Vox1 Vox2 Faces Attention L(1) B Vox1 Vox2 L(2) B = 0.3 B = 0.7 C = 0.6

24. Prediction Results • Use full learned model, including all weights • Predict ratings for a new movie given fMRI data

25. Prediction Results • Use full learned model, including all weights • Predict ratings for a new movie given fMRI data

26. Prediction Results • Comparison to models without time or subject interactions

27. Voxel Selection • Voxels selected by correlation with rating • Number of voxels determined by cross-validation

28. Voxel Selection • Voxels selected by correlation with rating • Number of voxels determined by cross-validation

29. Selected Voxels L L Faces Language

30. Selected Voxels L L Motion Arousal

31. Voxel Selection • Voxels selected for Language included some in ‘Face’ regions: L

32. Voxel Selection • Voxels selected for Language included some in ‘Face’ regions: L • Language and face stimuli correlated in videos • Complex, interwoven stimuli affect voxel specificity

33. Voxel Selection • Voxel selection extension – “spatial bias” • Prefer grouped voxels 0.33 0.38 * after competition submission

34. Voxel Selection • Voxel selection extension – “spatial bias” • Prefer grouped voxels 0.33 0.38 * after competition submission

35. Voxel Selection • Voxel selection extension – “spatial bias” • Prefer grouped voxels • Additional terms in linear regression objective: • |β1| |β2| D(Vox1, Vox2) 0.33 0.38 D || Vox1 –Vox2||2 * after competition submission

36. Adding Spatial Bias L L Faces

37. Conclusions • Reliable prediction of subjective ratings from fMRI data • Time step correlations aid in prediction reliability • Cross-subject correlations also beneficial • Individual voxels selected from whole brain • Reliability from regularization • Some found in expected regions • Some “cross-over” for correlated prediction tasks

38. Comments? • Poster #675 • ajbattle@cs.stanford.edu