1 / 25

5. Video Object Tracking and Processing

5. Video Object Tracking and Processing. To achieve augmented reality, computer generated graphics should be shown together with the live video In addition, the generated graphics should be changed according to the live video contents

snow
Télécharger la présentation

5. Video Object Tracking and Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. 5. Video Object Tracking and Processing • To achieve augmented reality, computer generated graphics should be shown together with the live video • In addition, the generated graphics should be changed according to the live video contents • Hence we need to be able to understand the contents of the live video • References: • Gonzalez, Rafael C., Digital image processing, Prentice-Hall, 2002. • The Mandelbrot DAZIBAO, “Lesson 5 – The HSV Colorspace”, http://www.mandelbrot-dazibao.com/HSV/HSV.htm. • 3. Ogre Tutorials, http://www.ogre3d.org/wiki/index.php/Ogre_Tutorials

  2. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Project Requirements • In this project, a few color balls are randomly placed on the racing track • You are required to develop computer programs that can detect the color balls in the live video obtained from the camera installed on the robot car • Computer graphics should be generated to cover the areas in the video where the balls are found • The size and position of the graphics should be adjusted according to the distance and relative direction between the balls and the robot car (i.e. the camera)

  3. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Procedure for Detecting the Color Balls • Grab a frame of the live video • Convert the video data from RGB color space to HSV color space • HSV color space is more close to human description to colors • Select all pixels in the image that matches with the color of the ball • Assume there are no other objects in the video that have the same color and similar shape • Remove the isolated pixels (optional) • Assume noise in the images may have similar color as the ball but exists in the form of isolated pixels • Locate the center of the remaining pixels • Assume the remaining pixels all belong to the ball of interested

  4. RGB Color Space • For the current digital displays, each pixel is made of a Red, a Green and a Blue sub-pixel • Each sub-pixel is activated by a byte of data, i.e. a number between 0 and 255 • With these three bytes, the computer can generate 256x256x256 = 16,777,216 combinations • If R = x, G = y and B = z, we can plot an RGB cube as follows: • While we have a huge amount of possibilities, it is also tricky to define a precise color by adjusting the three components: • If R=G=B=0, we have black • If R=G=B=255, we have white • If R=G=255 and B=0, we have yellow • But how about the others???

  5. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun HSV Color Space • HSV codification provides an intuitive method for color selection. Closer to human perception: • The blend of the three components is defined by a single parameter called "Hue" • The "Saturation" parameters selects how grey or pure the color will be • The "Value" parameter defines the brightness of the color • The HSV Color Space can be visualized as a cone

  6. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun RGB to HSV Conversion r – value of Red g – value of Green b – value of Blue max – the greatest of (r, g, b) min – the smallest of (r, g, b) h ∈ [0, 360) is the hue angle in degrees, and s, v ∈ [0, 100] are the saturation and value

  7. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Why HSV? • The color of an object as shown in the video can differ quite significantly from what is seen using naked eyes • Many environmental factors can affect the color of an object in the video • Exposure of the camera • White balance setting • Ambient lighting • Reflection of the spot lights around • Theoretically, an object of red color should have the RGB value of (255,0,0) • In practice, one can never find such value although the object is really red in color when seeing it with naked eyes • Some tolerance should be allowed if we want to detect an object of red color in the video. But the problem is how to set this tolerance

  8. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Why HSV (cont)? • We should consider an object in the video as red in color if it is “rather red” to “very red” • Besides, a red object should be considered as red no matter it is pictured in a dark environment or a bright environment • How to set such a tolerance? • By using the HSV codification • Hue (H) – Define a set of Red color • Saturation (S) – Define how Red it is • Value (V) – Define how bright it is

  9. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Why HSV (cont)? • In RGB color space, for every R value, there can be a set of G and B values that will give you a red color • Difficult to adjust if a set of Red colors are to be defined • In HSV color space, H defines a set of Red colors • S defines the saturation • V defines the brightness • A possible set of values for Red can be • H < 25 & H > 335 • S > 45 • V > 60 Result of using the Paint.NET

  10. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Original Video Frame • A blue ball and a red ball are put in front of the camera

  11. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Select the color Noise • We set a tolerance as follows: ?? < H < ??; S > ??; V > ?? • We compare the HSV value of every pixel of the video frame to see if it falls within the tolerance • Yes – mark it as green • No – no action • No matter how good is your estimation, there must be some pixels selected not belong to the object of interest • They are considered as noise

  12. Denoising • For every 5x5 block, if most pixels have color falls within the tolerance, consider them as a part of the object • For this block, only a few pixels are green. Hence they are noise • Noise on the image is often considered as isolated dots or islands • It can be reduced by using a simple denoising algorithm as follows • For this 5x5 block, all of them have been marked as green. They are a part of the object • Should carry out the above test for the whole image

  13. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Denoising Result Noise are removed • The above algorithm should be carried out pixel-by-pixel for the whole image • As a result, most of the noise are removed • Depends on the actual environment, the denoising performance may not be as good as that • Need long time fine-tuning

  14. Locate the object For each row, if a marked pixel is found, add 1 • The location of the ball can be easily estimated by summing up all marked pixels row-by-row and column-by-column and look for their peaks … … The peak For each column, if a marked pixel is found, add 1

  15. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Restriction of the algorithm • Certainly, the algorithm can only detect one blue ball • If there are multiple blue balls, multiple peaks will result • Become very difficult to determine which peak corresponds to which blue ball • The difficulty also applies to the situation that there are other large objects that are also blue in color • Multiple balls with different color is not a problem • One can detect balls one color at a time

  16. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Replace the detected object with 3-D graphics • To achieve augmented reality, we need to combine graphics with live video • If the position of the object is detected, we can replace it with 3-D graphics, such as fire generated by Ogre particle system

  17. Create a fire in Ogre // The particle "Examples/Smoke" is in the file // Example.particle // Assume mParticleSys and mParticleNode are two private // variables of EIE330projectApp mParticleSys = mSceneMgr->createParticleSystem("Smoke", "Examples/Smoke"); mParticleNode = mSceneMgr->getRootSceneNode() ->createChildSceneNode(); mParticleNode->setVisible(false); // Default - invisible mParticleNode->setPosition(0, 0, -140); // Set the position of the fire mParticleNode->attachObject(mParticleSys); : mParticleSys->setDefaultDimensions(220, 220); // Set the size of the fire

  18. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Resize the fire • Big fire!

  19. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Changing the size and orientation of graphics • For a real object on the screen, its size and orientation will change with the change of the camera position • If a 3-D graphic is used to replace a real object on the screen, its size and orientation should also change with the change of the camera position Camera is closer to the object

  20. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Changing the size and orientation of graphics Camera is moved to the right hand side of the screen Camera is moved to the left hand side of the screen

  21. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun How do we know the relative direction and distance of camera? • By using the markers • Each ball and the robot car will have a marker associated with it • In Lab 6, you are required to use the markers’ data to estimate the relative distance and direction between the blue ball and the camera • Need to replace the blue ball with a 3-D object that will change in size and orientation according to the position of the camera marker PC marker Camera

  22. Task for Lab 5 • Modify the function TextureSystem::UpdateTexture() to implement the following: • Change the color space • Select the pixels with red and blue color • Denoise the selected pixels • Detect the position of the red ball • Create a fire at the position of the red ball on the screen

  23. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun Task for Lab 6 (1st part) • Change the size and orientation of an Ogre head with respect to the relative position of the camera, which is given in the first part of Lab 6 • Procedure • Create a Ogre head in EIE330ProjectApp::createScene() • Implement the function EIE330ProjectApp::placingObject(float distance, float direction); // change the size and orientation of the Ogre head // based on the input parameters

  24. Task for Lab 6 (2nd part) • The blue ball will be replaced by a 3-D object of your choice on the screen • By making use of the markers’ data, estimate the relative distance and angle between the blue ball and the camera • Modify the function placingObject() such that it will change the size and orientation of the 3-D object based on the estimated relative distance and angle marker PC marker Camera

  25. Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun More instructions • For Lab 5 and Lab 6, each team submits one lab report and does one demo for each lab • No more lecture for Lab 6 • Once finishing Lab 5, students can start Lab 6 immediately. The lab sheets for Lab 5 and Lab 6 are available in WebCT • Both Lab 5 and Lab 6 do not need robot car and wireless camera • Use that time to fine-tune your robot car. Don’t wait until the last week!

More Related