A Genetic Algorithm-Based Approach to Content-Based Image Retrieval

1. A Genetic Algorithm-Based Approach to Content-Based Image Retrieval Bo-Yen Wang(王博彥) 894410003

2. Outline • An Introduction to Content-Based Image Retrieval • Genetic Algorithm: Primary • Example of Applying Genetic Algorithm to CBIR • Conclusion

3. What is Content-Based Image Retrieval • Inspired from CSI (Crime Scene Investigation)

4. Conventional Retrieval Method • Text-based search: Search images with keywords • Manual tagging images • Limitations • Maintain huge image databases • Lack of languages independent • Inconsistent between keyword and human perception

5. Content Based Approach • Versatile content within images • Various content feature • Color • Texture • Local Shape

6. Content Based Image Retrieval Query Processor outputs the list of similar images F(Iq) • Typical System Attribute Comparison using distance function f(x) Similar Images I4, I8, … Ik Color histogram Texture Description Shape details Iq - Query Image I1, I2, .. –Image Files Feature Extraction

7. Dataflow of Content-Based Retrieval Data Store In I-File Active Picture Set Store In F-File Annotate And Interact Compute Local Property Compute Object Feature Display And Feedback Compute Similarity Query Annotate And Interact Store In Session Compute Object Feature Compute Local Property Selection Query Picture Reference source: Smeulders et al.“Content-Based Image Retrieval at the End of the Early Years”

8. Sample CBIR architecture User System Administrator Feature Extraction Insert/Delete Images Search For Images Indexing Mechanism Database Handler Database

9. Challenges of CBIR • Sensory Gap • The sensory gap is the gap between the object in the world and the information in a (computational) description derived from a recording of that scene. • Semantic Gap • The semantic gap is the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation. Reference source: Smeulders et al.“Content-Based Image Retrieval at the End of the Early Years”

10. What visual properties do these images of tomatoes all have in common?

11. What is this? tomato?, setting sun?, clown’s nose?….

12. Semantic Gap “democracy” SEMANTIC GAP

13. Semantic Gap or Sensory Gap

14. Bridging Semantic Gap • A critical point in the advancement of content-based retrieval is the semantic gap, where the meaning of an image is rarely self-evident. • The aim of content-based retrieval systems must be to provide maximum support in bridging the semantic gap between the simplicity of available visual features and the richness of the user semantics. Reference source: Smeulders et al.“Content-Based Image Retrieval at the End of the Early Years”

15. Bridging Semantic Gap • Imagery Features and Similarity Measure • Select effective imagery features • Subjective experiments • Image categorization

16. Another approach: Relevance Feedback • Limitations of the computer-centric approach, recent research focus in CBIR has moved to an interactive mechanism that involves a human as part of the retrieval process • Human and computer system interacted to refine high-level queries to representations based on low-level features Reference source: Yong Rui et al.“Relevance Feedback: A Power Tool for Interactive Content-Based Image Retrieval” ”

17. Feature Extraction Query Feedback Feedback Feedback Dataflow of Relevance Feedback Images DatabaseIndex and Storage Query Image Result Refined Result Refined Result ………

18. Display UserFeedback Example of Relevance Feedback 1st Round Feedbackto system Similarity Measuring 2nd Round Display UserFeedback

19. Genetic Algorithm: Primary • Charles Robert Darwin • Diversity drives changes • Survival from fitness • Population based Optimization

20. Generation cycle of Genetic Algorithm Population (chromosomes) parents offsprings Evaluation (fitness) Genetic operators Crossover Mutation Reproduction (selection) Mating pool Mates Natural Selection (Evaluated by Fitness Values) individual (chromosomes)

21. A Simple Genetic Algorithm Simple Genetic Algorithm() { Initialize population; evaluate population; while(termination criterion not reached) { select solutions for next population(reproduction); perform crossover and mutation; evaluate population; } }

22. Major parts of GA • Representation • Genetic operators • Selection method • Objective function

23. Representation of GA • Binary encoding • Real-coded encoding • real-valued vector • Order-based encoding • Permutation problem

24. Selection Methods • Proportionate selection operators • roulette wheel • Tournament selection operators • Ranking selection operators

25.     Example of roulette wheel Maximize f(s)  01101 169 14.4%  11000 576 49.2%  01000 64 5.5%  10011 361 30.9%

26. Crossover Methods • Real valued • Binary valued • Uniform crossover

27. crossover child1 parent1 parent2 child2 Example of Binary valued Crossover • Single-point(one-point) crossover

28. child1 parent1 parent2 child2 Example of Binary valued Crossover • Multi-point crossover

29. individual 1 0 1 1 1 0 0 1 1 0 1 0 individual 2 1 0 1 0 1 1 0 0 1 0 1 parent sample 1 0 1 1 0 0 0 1 1 0 1 0 sample 2 1 0 0 1 1 1 0 0 1 0 1 Random offspring 1 11 10 1 11 1111 offspring 2 00 11 0 00 0000 children Example of Uniform Crossover

30. Mutation • Real valued mutation • Binary valued mutation

31. Why GA is robust than traditional search methods • Working with a coding of parameter set • Searching from a population of points, not a single point • using fitness information, not derivatives • probabilistic transition (guided random search)

32. Example of Applying Genetic Algorithm to CBIR • GA-based approach to relevance feedback using LSP • Genetic algorithm (GA) based method used for finding an optimal assignment of similarity criteria to image regions, and incorporated in the relevance feedback mechanism Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

33. Definition of Local Similarity Patterns (LSP) • The assignment of image features to image regions, which are obtained by uniform partitioning of the image area. Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

34. Definition of Local Similarity Patterns (LSP) Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

35. LSP-based image queries • LSP is proposed for computing similarity between the query image and the database images. • Well-defined query consists of the query image together with the intended interpretation (corresponding similarity criteria ) • An LSP-based query of the query image that the user selects, and the LSP system automatically infers based on the user’s relevance feedback. Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

36. Adapting GA for Image Retrieval • Given a user’s query image, a GA is used to automatically generate a LSP, and complete a LSP based query • GA is to realize the feature assignment function to assign image features to the image regions obtained by the uniform partitioning of the image area

37. Operation of GA in LSP • Chromosomes coding • uniformly partitioned into N*N regions • Each gene of a chromosome corresponds to one image region, and stores the type of image feature assigned to that region

38. Chromosomes coding

39. Operation of GA in LSP • Fitness measure • fitness of a chromosome is expressed as the retrieval precision of the corresponding LSP-based query • GA parameters • Selection: roulette-wheel • Crossover: uniform with probability 0.6 • Mutation: standard with probability 0.1 • Population size : 50 • Generations: 250

40. Methods for comparison(1/3) • Conventional method without relevance feedback • Similarity computation with common feature • Color • Shape • Texture • Color-Texture • Color-Shape • Shape-Texture

41. Methods for comparison(2/3) • Relevance feedback with Standard deviation-based method • Variance is computed over all the relevant images, and the weight of a feature is set to be the inverse of the corresponding variance • The smaller the variance, the bigger the weight

42. Methods for comparison(3/3) • Relevance feedback with Genetic Algorithms-based method • Find an optimal assignment of the weights to the image features • Maximize the average rank of the relevant images (Maximizing the retrieval precision)

43. Experiments Results

44. Experiments Results

45. Experiments Results

46. Experiments Results

47. Conclusion • Genetic algorithm-based approach introduced a slow process but acceptable solution in content-based image retrieval • Population-based characteristic of content-based image retrieval drives feasibility of using evolutionary computation. • Limited stochastic process causes complex problem reduced.