KMeD: A Knowledge-Based Multimedia Medical Database System

1. KMeD: A Knowledge-Based Multimedia Medical Database System Wesley W. Chu Computer Science Department University of California, Los Angeles http://www.cobase.cs.ucla.edu

2. KMeD October 1, 1991 to September 30, 1993 A Knowledge-Based Multimedia Medical Distributed Database System A Cooperative, Spatial, Evolutionary Medical Database System Knowledge-Based Image Retrieval with Spatial and Temporal Constructs July 1, 1993 to June 30, 1997 May 1, 1997 toApril 30, 2001 Wesley W. Chu Computer Science Department Alfonso F. Cardenas Computer Science Department Ricky K. Taira Department of Radiological Sciences

3. Students John David N. Dionisio Chih-Cheng Hsu David Johnson Christine Chih Collaborators Computer Science Department Alfonso F. Cardenas UCLA Medical School Denise Aberle, MD Robert Lufkin, MD Ricky K. Taira, MD Research Team

4. Wesley W. Chu, PhD Hooshang Kangarloo, MD Usha Sinha, PhD David B. Johnson, PhD Bernard Churchill, MD A NIH Grant at UCLA (2001-2005) A Medical Digital library---A Digital File Room for Patient Care, Education, and Research

5. Significance • Query multimedia data based on image content and spatial predicates • Use domain knowledge to relax and interpret medical queries • Present integrated view of multiple temporal and evolutionary data in a timeline metaphor • Retrieve Scenario Specific Free-text documents in a Medical Digital Library

6. Overview • Image retrieval by feature and content • Query relaxation • Spatial query answering • Similarity query answering • Visual query interface • Timeline interface • Retrieval of scenario specific free text medical documents

7. Image Retrieval by Content • Features • size, shape, texture, density, histology • Spatial Relations • angle of coverage, shortest distance, overlapping ratio, contact ratio, relative direction • Evolution of Object Growth • fusion, fission

12. Characteristics of Medical Queries • Multimedia • Temporal • Evolutionary • Spatial • Imprecise

13. 01 01 O O’ O O Om On Evolution: Object O evolves into a new object O’ Fusion: Object 01, …, Om fuse into a new object Fission: Object O splits into object 01, …, On Representing of Temporal and Evolution Objects

14. Case a: Case b: The object exists with its supertype or aggregated type. The life span of the object starts after and ends with its supertype or aggregated type. Case c: Case d: The life span of the object starts with and ends before its supertype or aggregated type. The life span of the object starts after and ends before its supertype or aggregated type. Representing of Temporal and Evolution Objects (cont)

15. Lesion Micro- Lesion Micro- Lesion An Example of Temporal and Evolution Object

18. Spatial Distance and Angle of Coverage of Two Objects

20. Query Modification Techniques • Relaxation • Generalization • Specialization • Association

21. More Conceptual Query Specialization Generalization Conceptual Query Conceptual Query Specialization Generalization Specific Query Specific Query Generalization and Specialization

22. Type Abstraction Hierarchy • Presents abstract view of • Types • Attribute values • Image features • Temporal and evolutionary behavior • Spatial relationships among objects • Provides multi-level knowledge representation

23. TAH Generation for Numerical Attribute Values • Relaxation Error • Difference between the exact value and the returned approximate value • The expected error is weighted by the probability of occurrence of each value • DISC (Distribution Sensitive Clustering) is based on the attribute values and frequency distribution of the data

24. TAH Generation for Numerical Attribute Values (cont.) • Computation Complexity: O(n2), where n is the number of distinct value in a cluster • DISC performs better than Biggest Cap(value only) or Max Entropy(frequency only) methods • MDISC is developed for multiple attribute TAHs. Computation Complexity: O(mn2), where m is the number of attributes

25. Query Display Yes Relax Attribute Answers Database No Query Modification TAHs Query Relaxation

26. An Cooperative Query Answering Example • Query • Find the treatment used for the tumor similar-to(loc, size) X1 on 12 year-oldKorean males. • Relaxed Query • Find the treatment used for the tumor Class Xon preteenAsians. • Association • The success rate, side effects, and cost of the treatment.

27. Tumor (location, size) Age Ethnic Group Class X [loc1loc3] [s1 s3] Class Y [locY sY] Preteens Teen Adult Asian African European 11 12 10 9 Japanese Filipino Korean Chinese X3 [loc3 s3] X1 [loc1 s1] X2 [loc2 s2] Type Abstraction Hierarchies for Medical Domain

28. TAH Lateral Ventricle TAH SR(t,b) TAH Tumor Size TAH SR(t,l) Knowledge Level SR(t,l) SR(t,b) Schema Level Lateral Ventricle Tumor Brain Representation Level (features and contents) SR: Spatial Relation b: Brain t: Tumor l: Lateral Ventricle Knowledge-Based Image Model

29. Queries Query Analysis and Feature Selection Knowledge-Based Content Matching Via TAHs Query Relaxation Query Answers Knowledge-based Query Processing

30. User Model To customize query conditions and knowledge-based query processing • User type • Default Parameter Values • Feature and Content Matching Policies • Complete Match • Partial Match

31. User Model (cont.) • Relaxation Control Policies • Relaxation Order • Unrelaxable Object • Preference List • Measure for Ranking

33. Query Preprocessing • Segment and label contours for objects of interest • Determine relevant features and spatial relationships (e.g., location, containment, intersection) of the selected objects • Organize the features and spatial relationships of objects into a feature database • Classify the feature database into a Type Abstraction Hierarchy (TAH)

34. Similarity Query Answering • Determine relevant features based on query input • Select TAH based on these features • Traverse through the TAH nodes to match all the images with similar features in the database • Present the images and rank their similarity (e.g., by mean square error)

36. Visual Query Language and Interface • Point-click-drag interface • Objects may be represented iconically • Spatial relationships among objects are represented graphically

37. Visual Query Example Retrieve brain tumor cases where a tumor is located in the region as indicated in the picture

44. A Visual Query Example

45. A Visual Temporal Query Example

46. Implementation • Sun Sparc 20 workstations (128 MB RAM, 24-bit frame buffer) • Oracle Database Management System • X/Motif Development Environment, C++ • Mass Storage of Images (9 GB)

50. Summary I • Image retrieval by feature and content • Matching and relaxation images based on features • Processing of queries based on spatial relationships among objects • Answering of imprecise queries • Expression of queries via visual query language • Integrated view of temporal multimedia data in a timeline metaphor