Indexing and Mining Biological Images

1. Indexing and Mining Biological Images Christos Faloutsos CMU

2. THANKS

3. Outline • PART1: ViVo: Visual Vocabulary for cat retina images • [PART2: other related work • FALCON: relevance feedback for image by content • Drosophila embryo image mining ]

4. PART1: ViVo with Ambuj Singh, Mark Verardo, Vebjorn Ljosa, Arnab Bhattacharya (UCSB) Jia-Yu Tim Pan, HJ Yang (CMU)

5. Detachment Development 1 day after detachment 3 days after detachment Normal 3 months after detachment 7 days after detachment 28 days after detachment

6. Data and Problem • (Data) Retinal images taken from cats • (Problem) What happens in retina after detachment? • What tissues (regions) are involved? • How do they change over time? • How will a program convey this info? • More than classification“we want to learn what classifier learned”

7. Main idea • extract characteristic visual ‘words’ • Equivalent to characteristic keywords, in a collection of text documents

8. V1 V2 Visual Vocabulary (ViVo) generation Visualvocabulary Step 3: ViVo generation Step 1: Tile image 8x12 tiles Step 2: Extract tile features Feature 2 Feature 1

9. skip ViVos

10. Evaluation of ViVo method • how meaningful are the discovered ViVos? • can they help in classification? • generality? • how else can they help biologists create hypotheses?

11. Biological interpretation

12. Goals: • how meaningful are the discovered ViVos? • can they help in classification? • generality? • how else can they help biologists create hypotheses?

13. Quality of ViVo – by classification Predicted Truth 86% accuracy 46 ViVos (90% energy)

14. Goals: • how meaningful are the discovered ViVos? • can they help in classification? • generality? • how else can they help biologists create hypotheses?

15. ViVos for protein images

16. Protein images – MPEG7 CS Predicted Truth 84% accuracy 4 ViVos (93% energy) 1-NN classifier

17. Evaluation of ViVo method • how meaningful are the discovered ViVos? • can they help in classification? • generality? • how else can they help biologists create hypotheses? ‘ViVo-annotation’!

18. Automatic ViVo-annotation of images • A tile represents a ViVo vk if the largest coefficient of the tile is along the kth basis vector • A ViVo vk represents a class ci if the majority of its tiles are in that class • For each image, the representative ViVos for the class are automatically highlighted

19. Which tissue is significant on 7-day?

20. 6 days after O2 treatment

21. 28 days after surgery

22. Conclusions: • how meaningful are the discovered ViVos? • can they help in classification? • generality? • how else can they help biologists create hypotheses?

23. Outcome/status • ViVos: Automatic Visual Vocabulary generation for biomedical image mining, Bhattacharya, Ljosa, Pan, Verardo, Yang, Faloutsos, Singh; ICDM’05 (one of 5 best student paper award) • Software – MATLAB code • Tutorial in SIGMOD’05 (Murphy+Faloutsos)

24. Outline • PART1: ViVo: Visual Vocabulary for cat retina images • PART2: FALCON: relevance feedback for image by content: SEE DEMO, later • Ongoing work: Drosophila Fly Embryos

25. FALCON - Example query: Vs Inverted Vs Trader wants only ‘unstable’ stocks

26. Outline • PART1: ViVo: Visual Vocabulary for cat retina images • PART2: FALCON: relevance feedback for image by content: SEE DEMO, later • Ongoing work: Drosophila Fly Embryos

27. FEMine: Mining Fly Embryos

28. FEMine: Mining Fly Embryos

29. FEMine: Drosophila embryos • Feature extraction • ICA • query by image content, mining, clustering with Andre Balan, Eric Xing, Tim Pan

30. Conclusions Machine vision + Data mining + Data bases + Biology: => necessary partners