620 likes | 872 Vues
An Object Tacking Paradigm with Active Appearance Models for Augmented Reality. Presented by Pat Chan Pik Wah 28/04/2005 Qualifying Examination. Outline. Research Objective Introduction Augmented Reality Object Tracking Active Appearance Models (AAMs) Proposed Object Tracking Paradigm
E N D
An Object Tacking Paradigm with Active Appearance Models for Augmented Reality Presented by Pat Chan Pik Wah 28/04/2005 Qualifying Examination
Outline • Research Objective • Introduction • Augmented Reality • Object Tracking • Active Appearance Models (AAMs) • Proposed Object Tracking Paradigm • Paradigm Architecture • Experiments • Research Issues • Conclusion
Research Objective • Object tracking is an essential component for Augmented Reality. • There is a lack of good object tracking paradigm. • Active Appearance Models is promising. • Propose a new object tracking paradigm with AAMs in order to provide a real-time and accurate registration for Augmented Reality. • Nature of the paradigm: • Effective • Accurate • Robust
Augmented Reality • An Augmented Reality system supplements the real world with virtual objects that appear to coexist in the same space as the real world • Properties : • Combine real and virtual objects in a real environment • Runs interactively, and in real time • Registers(aligns) real and virtual objects with each other
Augmented Reality • Projects related to AR
Augmented Reality • Display • Presenting virtual objects on real environment • Tracking • Following user’s and virtual object’s movements by means of a special device or techniques • 3D Modeling • Forming virtual object • Registration • Blending real and virtual objects
edges, corners, lines, curves, and color regions Pixels Object Tracking • Visual content can be modeled as a hierarchy of abstractions. • At the first level are the raw pixels with color or brightness information. • Further processing yields features such as edges, corners, lines, curves, and color regions. • A higher abstraction layer may combine and interpret these features as objects and their attributes. Object
Object Tracking • Accurately tracking the user’s position is crucial for AR registration • The objective is to obtain an accurate estimate of the position (x,y) of the object tracked • Tracking = correspondence + constraints + estimation • Based on reference image of the object, or properties of the objects. • Two main stages for tracking object in video: • Isolation of objects from background in each frames • Association of objects in successive frames in order to trace them
Object Tracking • Object Tracking can be briefly divides into following stages: • Input (object and camera) • Detecting the Objects • Motion Estimation • Corrective Feedback • Occlusion Detection
Object Tracking • Expectation Maximization • Find the local maximum likelihood solution • Some variables are hidden or incomplete • Kalman Filter • Optimal linear predict the state of a model • Condensation • Combines factored sampling with learned dynamical models • propagate an entire probability of object position and shape
Object Tracking • Pervious Work : • Marker-based Tracking • Feature-based Tracking • Template-based object tracking • Correlation-based tracking • Change-based tracking • 2D layer tracking • tracking of articulated objects
Pervious Work • Marker-based Tracking • Marker-less based Tracking • Feature-based Tracking • Shape-based approaches • Color-based approaches
Pervious Work • Template-based object tracking • Fixed template matching • Image subtraction • Correlation • Deformable template matching
Pervious Work • Object tracking using motion information • Motion-based approaches • Model-based approaches • Boundary-based approaches • Snakes • Geodesic active contour models • Region-based approaches
Active Appearance Models • The Active Appearance Model (AAM) algorithm is a powerful tool for modeling images of deformable objects. • AAM combines a subspace-based deformable model of an object’s appearance • Fit the model to a previously unseen image.
Active Appearance Models (AAMs) • 2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh. • Shape scan be expressed as a base shape s0. • pi are the shape parameter. • s0 is the mean shape and the matrices si are the eigenvectors corresponding to the m largest eigenvalues
A0(u) A1(u) A2(u) A3(u) Active Appearance Models (AAMs) • The appearanceof an independent AAM is defined within the base mesh s0. A(u) defined over the pixels u∈ s0 • A(u) can be expressed as a base appearance A0(u) plus a linear combination of l appearance • Coefficients λi are the appearance parameters.
Active Appearance Models (AAMs) • The AAM model instance with shape parameters pand appearance parameters λ is then created by warping the appearance Afrom the base mesh s0 to the model shape s. Piecewise affine warp W(u; p): (1) for any pixel u in s0 find out which triangle it lies in, (2) warp u with the affine warp for that triangle. M(W(u;p))
u u u u Fitting AAMs • Minimize the error between I (u) and M(W(u; p)) = A(u). • If u is a pixel in s0, then the corresponding pixel in the input image I is W(u; p). • At pixel u the AAM has the appearance • At pixel W(u; p), the input image has the intensity I (W(u; p)). • Minimize the sum of squares of the difference between these two quantities:
Recent Work for Improving AAMs • Combine 2D+3D AAMs
Combined 2D + 3D AAMs • At time t, we have • 2D AAM shape vector in all N images into a matrix: • Represent as a 3D linear shape modesW = MB =
Compute the 3D Model AAM shapes AAM appearance First three 3D shapes modes
Constraining an AAM with 3D Shape • Constraints on the 2D AAM shape parameters p = (p1, … , pm) that force the AAM to only move in a way that is consistent with the 3D shape modes: • and the 2D shape variation of the 3D shape modes over all imaging condition is: • Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:
Training Images Proposed Object Tracking Paradigm Paradigm Architecture Occlusion Detection Training Active Appearance Model Video • Shape Model • Appearance Model Motion Modeling Initialization Kalman Filter
Steps in Object Tracking Paradigm • Preporcessing • Training the Active Appearance Model. • Get the shape model and the appearance model for the object to be tracked. • Initialization • Locating the object position in the video. • In our scheme, we make use of AAMs. • Motion Modeling • Estimate the motion of the object • Modeling the AAMs as a problem in the Kalman filter to perform the prediction. • Occlusion Detection • Preventing the lost of position of the object by occluding of other objects.
Enhancing Active Appearance Models • Shape • Appearance • Combine the shape and the appearance parameters for optimization • In video, shape and appearance may not enough, there are many characteristics and features, such as lightering, brightness, etc… L=[L1, L2, ……, Lm]T
Iterative Search for Fitting Active Appearance Model • Can be improved by: • Prediction matrix • Searching space
Motion Modeling • Initial estimate in a frame should be better predicted than just the adaptation from the previous frame. • Can be achieved by motion estimation • AAMs can do the modeling part • Kalman filter can do the prediction part
Kalman Filter • Adaptive filter • Model the state of a discrete dynamic system. • Originally developed in 1960 • Filter out noise in electronic signals.
Kalman Filter • Formally, we have the model • For our tracking system,
Occlusion Detection • WHY? • Positioning of objects • To perform cropping • When a real object overlays a virtual one, the virtual object should be cropped before the overlay • HOW? • High resolution and sharp object boundaries • Right occluding boundaries of objects • Camera matrix for video capturing
Training Images Proposed Object Tracking Paradigm Paradigm Architecture Occlusion Detection Training Active Appearance Model Video • Shape Model • Appearance Model Active Appearance Model Fitting Initialization Kalman Filter
Experimental Setup • AAM-api from DTU • OpenCV • Pentium 4 CPU 2.00GHz and 512MB RAM
Experiment on AAMs (1) • Training Image
Experiment on AAMs (1) Texture Shape
Experiment on AAMs (1) After optimized Initialization
Experiment on AAMs (2) • Training Images
Experiment on AAMs Texture Shape
Experiment on AAMs • Trapped in local minimum After optimized Initialization