Efficient Multiple Frame Motion Inference with Belief Propagation

1. Multiple Frame Motion Inference Using Belief Propagation Jiang GaoJianbo Shi Presented By: Gilad Kapelushnik Visual Recognition, Spring 2005, Technion IIT.

2. Abstract S4(X,Y) • Find “best fit” upper body joint configuration. • Input is a 2D video • Each joint is described by its location on a 2D grid. S5(X,Y) S1(X,Y) S2(X,Y) S6(X,Y) S3(X,Y) Let J be a joint configuration – {S1,S2,S3,S4,S5,S6} We would like to find:

3. Motion Energy Image • Step 1: Subtract two sequential frames. • Step 2: Apply threshold.

4. From #NrgPixels To Probability • Sum the Energy Pixels in the Patch. • Calculate probability using the following: S5(10,60) S6(40,30)

5. 0.12 0.84 0.19 0.68 0.02 Main Idea • Find configuration J with the highest probability. • Computing all possible probabilities is inefficient. • a-Priori data give better and faster results. • removing impossible configurations reduce inference time.

6. a-Priori Data • A probability table for Each P(Sx,Sy). • Compute probability at grid crossing. • Use nearest neighbor for the rest of the image. • Example: • For right arm - P(S2,S3) • Red – Low probability • Green – High probability

7. Detect Candidate states (1) • Face is detected using face detection algorithm. • Initial assumption of Shoulders from face and pose. • Even using BP there are too many possible states to go through. • Candidates for elbows from shoulders & Energy Map. • Candidates for Wrists from skin color model.

8. Red for left wrist Pink for right wrist Detect Candidate states (2) • Many states can be discarded. • Remove close candidate states. • Pros: Much faster inference. • Cons: Less accurate. • Note: This is only an option. Fits skin color and wrist location Blue for elbow

9. The Markov Model • Empty Circles - States - 2D positions of joints • Full Circles - Observations - Computed from energy map. • Each state correspond to an observation.

10. Belief Propagation (1) • Solve inference problem using an algorithm with Linear complexity. • Each joint has a vector with probabilities for each candidate. Shoulder Elbow Wrist

11. Belief Propagation (2) Sum over all candidates Message from k to i (all messages from the neighbors). This is actually a vector with a probability for each state. • For each iteration: • Each node sends a message to its neighbor nodes containing the “wanted” probability (for each state). • Messages are computed according to: m41 1 Normalize variable. m14 m12 A-priori Data for each state. m21 2 Observation (# of Energy pixels in patch) for each state converted to a probability. Message from i to j. m32 m23 3

12. 1 2 Message from 1 to 2 Belief Propagation (3) - Example 4 states 2 states

13. Belief Propagation (4) • BP converge after 2-4 iterations (giving the right a-Priori data). • For every joint there is a probability vector for each candidate state.

14. Multiple Frame Probability • Multiple frame (8) is proposed for smoother transition between configurations. • Prevents joints changing their state to a different which is “far away” (Euclidian distance). • Though BP was designed to work with loopy-free models, the author stated that it worked fine. And for those who really want to know:

15. 2D to 3D • 2D -> 3D by Taylor (2000). • Assuming (u1,v1) and (u2,v2) are projections then depth can be retrieved using the following:

16. Results(1)

17. Results(2)

18. Results(3) Errors accrue when 2 joints intersect each other. On some occasions, even when limbs intersect, it was possible to infer correctly.

19. Q?