Object Recognition Using Attention

1. Projects for implementing a systemfor: Object Recognition Using Attention Tammy Avraham and Assoc.Prof Michael Lindenbaum Computer Science Dept., Technion Haifa, Israel

2. Do you see faces ? How many & where ? The system we are going to build • Fast! • How ? By directing attention to the important parts of the image. • Faces is an example. We want to be able to recognize also some other types of objects Our system Yes, I see 10 faces in locations…

3. Object recognition Project: build object recognizer • Computer Vision : Giving computers the ability to understand what a scene (=a matrix of pixels) represents. • A subtask – object recognition: an expert system that detects a certain type of objects. For instance: faces, cars, chairs… • A very difficult task • variations due to illumination conditions, viewing position, posture. . • Categorical (e.g. – many types of chairs) • Usually object recognition systems scan all the image for the desired objects. • The candidates are, for instance, all possible rectangles in the image in a few sizes … …

4. Segmentation Project: implement a segmentation algorithm • Instead of exhaustive scanning perform segmentation as a pre-process. • Segmentation/Grouping=dividing the image into regions, each hopefully representing an object.

5. High-level Object Recognizer Candidate Selection Image candidate identity Ordering\Attention algorithm Adding Attention Objective: minimize those calls

6. Human Attention • Yarbus 67 – eye movement study

7. Human Attention – Psychology theory #1 Triesman & Gelade 80 - Feature integration Theory • Pop-outvs. sequential scan Pop-out: when the target is different in on feature from all others Sequential: when the target is distinct by a combination of features • Theory of the visual search process: • Pre-attentive stage: parallel feature extraction • Combining the feature maps to one saliency map • Attention spotlight (Posner)

8. Saliency Map A computer model based on Feature integration Theory Project: build saliency maps • Itti,Koch&Niebur 1998: • Bottom-up saliency • Top-down information

9. Human Attention – Psychology theory #2- based on inner-scene similarity • Duncan & Humphreys 89 – Similarity Theory • Search Difficulty depends on: • Target vs. Non-target similarity • Non-target vs. Non-target similarity • Theory: • Stage 1 – hierarchical segmentation to structural units including similarity grouping/linkage. • Stage 2 – attention, spreading suppression.

10. Using Inner Scene similarity for computer vision • Mutual Similarity of candidates as useful information • Dynamic search – use previous recognizer answers to change the priorities of the non-yet-attended candidates. • Example: if the middle green creature turned out to be not interesting, mark also the other similar creatures as not interesting, and try a micki-mouse. Project: measuring similarity Project: dynamic attention

11. Coming back to the system we want to build • A computer-vision system for object recognition using ideas from human attention to direct us faster to the important objects in the scene. • Candidate selection: • Simple selection of rectangles, or: • Segmentation • The attention will rely on: • Saliency • Prior knowledge on desired objects • Similarity between objects in the scene. • We should be able to test all combinations of above options.

12. The system’s scheme 5. Object recognition candidate index recognition results locations of targets & search order 2. Candidates selection 3. Similarity measures 4. Attention order control 1. Compute saliency Image + user parameters candidates description distance matrix saliency map candidates priorities features maps a few target and nontarget models (optional) Real-Time presentation

13. More details on: • 1. Computing saliency image Combine to one saliency map Calculate saliency per feature Calculate feature maps Saliency map model images Params: Saliency map per feature Feature maps If model images are used, they can effect the importance of each Feature when combining them to one map of saliency.

14. More details on: • 2. Candidate selection Segmentation algorithm Image: Colored/gray Compute priority of each candidate Description of indexed candidates. +Priority of each • Params: • -segmentation or simple rectangular candidates • -which algorithm to use for segmentation • Remove regions beyond a certain size ? • Provide only most salient candidates. How many ? • … Trivial candidate selection Saliency map If possible – consider only the salient regions during segmentation to save time

15. More details on: • 3. Similarity measures : option 1 image Feature extraction Distance function Models images Extract some partial description of each candidate: A feature vector or a histogram Similarity matrix (distance matrix) Itti Use some distance between the features of the candidates to measure dissimilarity Candidates description Params: -which features to extract -which metric to use as the distance function - … Feature vectors/histograms If model images are used, treat them as candidates and get also a similarity measure between them and the other candidates And between themselves.

16. More details on: • 3. Similarity measures : option 2 : use the feature maps calculated in the saliency stage instead of stating from the image pixels. Calculate feature vectors Distance function Similarity matrix (distance matrix) Feature maps From the feature maps that describes each pixel individually create a feature vector\histogram per candidate Itti Use some distance between the features of the candidates to measure dissimilarity Candidates description Params: -calc a feature vector or histogram -which metric to use as the distance function - … Feature vectors/histograms

17. More details on: • 4. Attention order control Index of candidate to attend to Recognition results Iteratively decide on next candidate to attend Distance matrix Priorities vector The target candidates The scan order Params: -consider priorities -consider similarity -when to stop - …

18. More details on: • 5.Recognition training set (target and non-target examples) Learning\training offline stage (?) Yes/no Image Recognition (classification) Candidates description Amount of certainty Candidate index Params All projects implementing object recognition should test their software to recognize a few categories of objects, and supply interface for testing more categories.

19. Implementation & Integration • The system will combine C and Matlab code. In some cases we will force one of them. • Each part should be able to work alone with its own graphical user Interface, and as a part of the system. How? For instance - Each project will implement a main function : OutParams MainFunction( InParams ) When this part is tested alone: The User Interface callback function will call this function. When it is used as a part of the system: The main program that will run all the system will call it. • Integrators: • Provide UI for all the system for feeding parameters needed for all its parts. • Design the way parameters are passed form one part to another. • Write the main programs for running the system. • Responsible that the system will work ! Meaning – meeting with the other groups all through the semester. • Responsible that the system will work in all possible paths – • Using saliency or skipping • Use inner-scene similarity consideration for attention • Use top-down knowledge for attention. If yes, use it when computing the saliency map or when computing the distance matrix. • The type of candidates selection. • Use a high resolution image only for recognition and a lower resolution image for the rest. • All above (and much more) are parameters the user can set using the UI. • Responsible for testing the system with a variety of images. • Responsible for presenting the results both in real time and the results statistics.

20. Important (also for your grade) • DOCUMENTATION – both in the code and in the report. Each class and each method/function should be explained in details • RUNNING TIME – important to save time when possible. Add time measures and report them. • OPEN TO CHANGES – your code will be submitted as well and it is possible that someone else will want to use it and change it in the future.

21. Time Table & Grading Remark: where ‘grade’ is filled, it means you have to submit something until that date

22. A little more on each stage • Theoretic stage: 1. Read the paper/papers you are given, understand it, summarize it. 2.Explain how it fits in the system. What the input and output parameters are. 3.How do you plan to test it. • Software design: Design of your code. Includes all functions and parameters you are going to implement. Description of your user Interface and your software interface for the system. (Already using the datatypes designed for the input and output parameters by the integrators). • Software implementation and testing: implement and perform many tests. Pass the code to the integrators, even before it is perfect. At the end date of this stage it should be perfect. • Integration stage: Putting the system together. The integrators are the most busy, but the other must be available to correct bugs if needed. At this time you should be working on your presentation. • Presentation: Prepare a short power-point presentation (about 20 minutes) describing the important theory involved and show some results. • Final Report: A wide description of the theory, the software documentation, and the results. Submit the code well documented (Electronically).

23. The integrators at each stage • The integrators must be always one step ahead. • During the ‘theoretic stage’ they do ‘software design’: Design the datatypes that will be used as input and output parameters for each stage. Plan the software they have to write (for instance – simple candidate selection). Describe the user interface and the way they are going to present the system’s results. • During the ‘software design’ stage the integrators should implement their software with dummy functions where the models should be. • During the ‘implementation stage’ they should find and film many images to be used for the full system’s testing and bring them to the other people. Gradually integrate parts to the systems when people are done (or almost done). • Integration stage: Here they are the most busy people. • Presentation: describe all the system and show some results.

24. Remarks • If we correct mistakes in your theoretic report and the software design, you must correct them before going to the next stage. • This slides don’t include all the details of the systems. Each group, when assigned with the project will get much more details concerning their part. • Set a meeting with Tammy A.S.A.P. • Some of the projects will be guided by Tammy, some by Micha. • Meeting with us will be held each one-two weeks. • You can set a meeting each time you have questions. • Start working ASAP ! For instance: read the paper and come to ask questions before Passover