Feature-Based Alignment of Volumetric Multi-modal Images

1. Feature-Based Alignmentof Volumetric Multi-modal Images Matthew Toews, LillaZöllei, William Wells III June 29, 2013 IPMI

2. Robust Image Alignment • Challenges: Arbitrary subjects, anatomies (brain, body, …), modalities (MR, CT, …), pathology, lack of one-to-one homology, unknown initialization, DICOM errors, … Thoracic CT Infant Brain MR TBI, MR Brain MR, CT (tumor)

3. Applications • Robust clinical usage • Initialize registration, segmentation routines • Large-scale data mining, e.g. Google style Thoracic CT Infant Brain MR TBI, MR Brain MR, CT (tumor)

4. Feature-based Alignment Method • Alignment via 3D scale-invariant feature correspondences Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

5. Feature-based Alignment Method • Strengths • Robust: lack of one-to-one homology, disease, resection, • Globally optimal: no ‘capture radius’, initialization • Efficient: Memory, computation • Useful: Alignment, disease classification, prediction • Weaknesses • Does not align different modalities • Requires pre-aligned training data Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

6. Contributions • Inverted local feature correspondence • Extend scale-invariant feature representation to multi-modal alignment • Group-wise feature-based alignment • Remove requirement for pre-aligned training data • Multiple modalities

7. Scale-invariant feature representation Inverted feature correspondence Group-wise feature-based alignment Overview

8. Scale-invariant feature representation Inverted feature correspondence Group-wise feature-based alignment Overview

9. Scale-Invariant Features • Distinctive image patches • Image-to-image correspondence • SIFT method: computer vision • Invariant to scaling, rotation, translation, illumination changes • Fast, efficient, robust • Large-scale image search Distinctive Image Features from Scale-Invariant Keypoints D. G. Lowe, IJCV, 2004.

10. Scale-Invariant Features in 3D • 3D Geometry S = {X, σ, Θ} • Location X = (x, y, z) • Scale σ • Orientation (axis: 3 unit vectors ) • Appearance I • Intensity descriptor Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

11. Identifying Location X, Scale σ • Difference-of-Gaussian scale-space extrema. Feature locations, scales: blob-like image patterns

12. Assigning Orientation Θ • Dominant local image gradient directions 3D gradient orientation histogram Vote location (upon unit sphere) Vote magnitude Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

13. Feature Descriptor I • Encode local image content • For image correspondence / matching • Gradient orientation histogram (GoH) • Quantization: 8 location x 8 orientation bins • 113 voxels → 64 bins (small size) Normalized Image Patch GoH Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

14. Scale-invariant feature representation Inverted feature correspondence Group-wise feature-based alignment Overview

15. Matching Across Modalities • Joint Intensity relationship • Globally multi-modal • Locally linear negative localcorrelation positive local correlation T2 MP-RAGE Non-rigid registration of multi-modal images using both mutual information and cross-correlation. Medical Image Analysis, 2008. A. Andronache, M.V. Siebenthal, G. Szekely, P. Cattin.

16. Matching Across Modalities • Positive local correlation • Conventional correspondence methods, descriptor matching positive correlation T2 MP-RAGE

17. Matching Across Modalities • Negative local correlation • Conventional correspondence fails • Inverted local gradient, orientations , descriptor negativecorrelation T2 MP-RAGE

18. Matching Across Modalities • Inverted correspondence • Rotate orientation, descriptor elements by -π about • Correspondence successful negativecorrelation T2 MP-RAGE

19. Matching Across Modalities • MP-RAGE, T2, intra-subject • No conventional correspondences in GM / WM • 22 inverted correspondences within GM / WM T2 MP-RAGE

20. Matching Across Modalities • Infant T1 MR: newborn ↔ 2 years old • GM / WM contrast inversion due to mylenation • No conventional correspondences in GM / WM • 4 inverted correspondences within GM / WM Unbiased average age-appropriate atlases for pediatric studies NeuroImage 2011. V.S. Fonov, A.C. Evans, D.L. Collins et al.

21. Scale-invariant feature representation Inverted feature correspondence Group-wise feature-based alignment Overview

22. Group-wise Alignment • Automatically align a set of subject images • Arbitrary initialization, modalities

23. Group-wise Alignment: Model Transform set: image i to atlas (similarity transform) Feature descriptor, geometry set: image i, feature j Latent model feature set, feature k, conventional & inverted modes l={0,1} Bayes rule Conditional feature independence Marginalization over F Input features (Iij,Sij) are - Conditionally independent - Identically distributed according to a Gaussian mixture model

24. Gaussian Mixture Model (GMM) Background

25. GMM: Unrecognized Input Feature Background

26. GMM: Conventional Input Feature Background

27. GMM: Inverted Input Feature Background

28. Likelihood: Appearance Descriptor Conditional independence Descriptor: Isotropic Gaussian density over descriptor elements. Note: descriptor conditionally independent of alignment Ti, - Fast Matching, no search over Ti -

29. Likelihood: Geometry Location: Isotropic Gaussian Scale: Gaussian in log σij Orientation: Isotropic Gaussian Approximate Von Mises

30. Prior Latent feature probability: Discrete Conventional appearance mode Inverted appearance mode

31. Group-wise Alignment: Algorithm Inputs: Volumetric images Outputs: Alignment solutions: {Ti} Feature-based model: {fk,l}, p(Iij, Sij|fk,l,Ti), p(fk,l)

32. Group-wise Alignment: Algorithm • Feature extraction • Initialization Approximate {Ti} • Model Learning Fixed {Ti}, vary {fk,l}, p(Iij, Sij|fk,l,Ti), p(fk,l) Mixture model; density, probability parameter estimation. • Alignment / Model Fitting Fixed {fk,l}, p(Iij, Sij|fk,l,Ti), p(fk,l), vary {Ti} Subject-to-model alignment. Iterate between 3) & 4)until convergence, i.e. {Ti} no longer changes.

33. 1) Feature Extraction • Features extracted once from individual images • Note: algorithms use feature data only • ~100X data reduction compared to original image volumes. • ~25 seconds for 2563-voxel image, standard PC. • ~2K features per brain image

34. 2) Initialization • Approximate image alignment • Nearest neighbor descriptor matching • Hough transform, similarity • Note: some initial misalignment OK • A small subset of image should be aligned • Misaligned features sets have negligible impact

35. 3) Model Estimation • Mixture modeling • Estimate set {fk,l}, parameters of p(Iij, Sij|fk,l,Ti), p(fk,l) • Robust feature clustering across subjects • Similar to mean-shift algorithm • Note • Model conventional appearance • Same structure, two distinct latent features, e.g.: Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

36. 4) Model Fitting • Maximum a-posteriori estimation • Maximize Ti individually Conditional independence

37. 4) Model Fitting • Maximum a-posteriori estimation • Maximize Ti individually • Two approaches • Conventional • Multi-Modal (conventional & inverted modes) • Assume Conditional independence

38. Experiments • Group-wise Alignment: RIRE data set • Modalities: T1, T2, PD, MP-RAGE, CT • Brain: 9 subjects, 39 images • All subjects exhibit brain tumors • Difficult problem: subject abnormality, no prior information used regarding modalities, initialization. • Compare conventional vs. multi-modal fitting Comparison and evaluation of retrospective intermodality brain image registration techniques Journal of Computer Assisted Tomography, 1997. J.B. West, J.M. Fitzpatrick et al.

39. Results: Alignment

40. Results • Conventional alignment: 3 failure cases (all CT) • Multi-Modal alignment: success Failure Success Failure case

41. Results • Model feature examples

42. Discussion: Inverted Correspondence • Useful for fitting/matching between modalities • Less useful once model has been learned • May be more prone to false correspondences • Analogous to mutual information • Useful when prior information is weak A marginalized MAP approach and EM optimization for pair-wise registration IPMI 2007. L. Zollei, M. Jenkinson, S. Timoner, W.M. Wells III

43. Discussion: Group-wise Alignment • Alignment of difficult multi-modal data • Unknown initialization • Also effective for infant MR, torso CT, lung CT • Fast • 22 minutes (vs. 10-20 hours for group-wise registration) • Deformable alignment? • Global similarity Ti + local linear deformations about correspondences.

44. Acknowledgements • NIH grants: • P41-EB-015902P41-RR-013218R00 HD061485-03P41-EB-015898P41-RR-019703