Video Classification

1. Video Classification By: Maryam S. Mirian For: Multimedia & Pattern Recognition Joint Courses Project

2. Outline • What is Video Classification? • Straightforward or Difficult? • What is its Applications? • What are its methods? • Review of Video Classification Methods • What is my own Project, exactly?

3. What is Video Classification? Classify a Video (Shot) into one of Nc predefined Classes: • Indoor / outdoor • News / Sports • …

4. Is Video Classification Difficult? Why? • YES, Because: • Data Stream is a Multi-dimensional signal. • It has a subjective nature.

5. Classification

6. Required Steps for Classification Object Observations Feature Extraction Feature Reduction Classification Class Labels Using Methods like: PCA, LDA The most Important and the most difficult part

7. Methods of Classification • Bayesian Classification • kNN Classification • Neural Classification • MLP • RBF • Classification based on Support Vector Machines • Rule-based Classification

8. Bayesian Decision Making So, x belongs to w2

9. Methods of Classification • Bayesian Classification • kNN Classification • Neural Classification • MLP • RBF • Classification based on Support Vector Machines • Rule-based Classification

10. kNN Decision Making k = 5, 2 Red Neighbor While 3 Black Neighbor, so X should be Black!

11. Methods of Classification • Bayesian Classification • kNN Classification • Neural Classification • MLP • RBF • Classification based on Support Vector Machines • Rule-based Classification

12. MLP Classifier

13. Video Content Analysis

14. Applications ofAutomatic video classification • Automatic Video segmentation • content based retrieval • browsing and retrieving digitized video • identifying close-up video frames before running a computationally expensive face recognizer. • effective management of ever-increasing amount of broadcast news video: personalization of news video.

15. Classify Shot or Video? • One effective way to organize the video is to segment the video into small, single-story units and classify these units according to their semantics. • A shot represents a contiguous sequence of visually similar frames. It is a syntactical representation and does not usually convey any coherent semantics to the users.

16. Looking @Video Classification

17. Ide et al. [1998] • Problem Domain: News video • Features: • Videotext • motion • face • segmented the video into shots • used clustering techniques • classify each shot into 1 of 5 classes: Speech/report, Anchor, Walking, Gathering, and Computer graphics shots. • Quite simple but seems effective for this restricted class of problems.

18. Huang et al. [1999] • Problem Domain: TV Programs • news report • weather forecast • Commercials • basketball games • football games • Features: • Audio • Color • motion

19. Chen and Wong [2001] • Problem Domain: • news video: • News • Weather • Reporting • Commercials • Basketball • Football • Features: • Motion • Color • text caption • cut rate • used a rule-based approach

20. Looking @ Lekha Chaisorn et.al [2002] in More Details

21. Basic Ideas • Proposes a two-level, multi-modal framework. • The video is analyzed at the shot and story unit (or scene) levels. • At the shot level, a Decision Tree to classify the shot into one of 13 pre-defined categories is employed. • At the scene level, the HMM (Hidden Markov Models) analysis is used to eliminate shot classification errors • Results indicate that a high accuracy of over 95 % for shot classification can be achieved. • The use of HMM analysis helps to improve the accuracy of the shot classification and achieve over 89% accuracy on story segmentation.

22. Predefined Classes

23. Features in Shot Level • Low-level Visual Content Feature • Color Histogram • Temporal Features • Background scene change • Speaker change • Audio • Motion activity • Shot duration • High-level Object-based features • Face • Shot type • Videotext • Centralized Videotext

24. Feature vector of a shot • Si = (a, m, d, f, s, t, c) • a the class of audio, a ∈{ t=speech, m=music, s=silence, n =noise, tn = speech + noise, tm= speech + music, mn=music+noise} • m the motion activity, m ∈{l=low, m=medium, h=high} • d the shot duration, d ∈{s=short, m=medium, l=long} • f the number of faces, Ν ∈ f • s the shot type, s ∈{c= closed-up, m=medium, l=long, u=unknown} • t the number of lines of text in the scene, Ν ∈ t • c set to “true” if the videotexts present are centralized, c ∈{t=true, f=false}

25. Decision Tree for Shot Classification

26. Reading these papers, I decided about My own Project….

27. About Problem Domain… • Sport Classification seems OK • Interesting Enough • It is helpful for Sports-Lovers

28. About Extracting features…. • Features used in video analysis: color,texture,shape,motion vector… • Criteria of choosing features : they should have similar statistical behavior across time • Color histogram: simple and robust • Motion vectors:invariance to color and light

29. So, My Own Project is • Sports Video Classifications : Football, Basketball, ….(Those Well-defined sports, I can find Video On!) • Steps I should take: • Finding or Gathering a Video Collection • Shot Detection • Feature Extraction : • Key Frame (s) Extraction: • Selecting Middle Shot I-Frame • Use of Clustering • … • Motion Vector–based Features • Straight Lines Detection • Design a Classifier • Test the Approach

30. Looking @Ekin,Tekalp[2003] one Research on Football Video Classification

31. Features • Cinematic • result from common video composition and production rules. • shot types, camera motions and replays. • Object-based • Described by their spatial, e.g., color, texture, and shape, and spatio-temporal features, such as object motions and interactions

32. Robust Dominant Color Region Detection • A soccer field has one distinct dominant color (a tone of green) that may vary from stadium to stadium, and also due to weather and lighting conditions within the same stadium. • The statistics of this dominant color, in the HSI space, are learned by the system at start-up, and then automatically updated to adapt to temporal variations.

33. Shot classification • Long Shot • A long shot displays the global view of the field. • In-Field Medium Shot • a whole human body is usually visible. • Close-Up Shot • shows the above-waist view of one person • Out of Field Shot • The audience, coach, and other shots

35. How Extend to Shot from a Frame? • Due to the computational simplicity they find the class of every frame in a shot and assign the shot class to the label of the majority of frames.

36. Decision Schema based on G • The first stage uses G value and two thresholds,TcloseUpand Tmedium to determine the frame view label.

37. Soccer Eevent Detection • Goal Detection • Referee Detection • Controversial calls, such as red-yellow cards and penalties • Penalty Box Detection

38. Goal Detection • Occurrence of a goal is generally followed by a special pattern of cinematic features. • A goal event leads to a break in the game. • one or more close-up views of the actors of the goal event. • show one or more replay(s) • the restart of the game is usually captured by a long shot.

40. Referee Detection • Assumed that there is, a single referee in a: • medium • out of field • close-up shot • So no search for a referee in a long shot

41. Penalty Box Detection • Field lines in a long view can be used to localize the view and/or register the current frame on the standard field model

42. Interesting Summaries • Goal summaries • summaries with Referee and Penalty box objects

43. Adaptation of Parameters • Parameters • Tcolor in dominant color region detection • TcloseUp and Tmedium in shot classification • referee color statistics • The training stage can be performed in a very short time to find Mean and Variance of a Normal pdf.

44. Results for High-Level Analysis and Summarization • Goal detection results

45. Results for High-Level Analysis and Summarization(2) • Referee detection results

46. Results for High-Level Analysis and Summarization(3) • Penalty box detection results

47. References • Automatic soccer video analysis and summarization, in Symp. Electronic Imaging: Science and Technology: Storage and Retrieval for Image and Video Databases IV, IS&T/SPI03, Jan. 2003, CA. • “The Segmentation and Classification of Story Boundaries In News Video”, Proceeding of 6th IFIP working conference on Visual Database Systems- VDB6 2002, Australia 2002 • Pattern Classification, by Duda, Hart, and Stork, 2000

48. Thanks for Your Attention Any Question or Comment?