Multimedia-Systems:

1. Multimedia-Systems: Synchronization (Chapter 15) http://bunuyazayim.blogspot.com/

2. Scope

3. Contents http://www.youtube.com/watch?v=cDfQunJ12Mc • Motivation - What is (Multimedia) Synchronization? • Synchronization and Multimedia • Reference Model for Multimedia - Synchronization • Synchronization in a Distributed Environment • Synchronization Techniques • Case Studies

4. Run time support Synchronization between time dependent and time independent data together with user interactions. Run time support handle multiple media streams to maintain the temporal relations between various media streams. Operating System Layer handle single media streams to avoid jitter 1. Motivation - What is (Multimedia) Synchronization? Synchronization is addressed and supported by many system components including the operating system, databases, documents and applications. Therefore, Synchronization must be considered at several layers in MM systems.

5. Temporal Relations • 1. level: OS and lower communication layers bundle single streams • Objective is to avoid jitter at the presentation time of one stream • 2. level: on top of this level is the RUN-TIME supportfor synchronization of multimedia streams (schedulers) • Objective is bounded skews between various streams • 3. level: next level holds the run-time support for synchronization between time-dependent and time-independent media together with handling of user interaction • Objective is bounded skews between time-dependent and time-independent media

6. Criteria for MM systems • the number of media • the types of supported media • degree of media integration Multimedia system : a system or application that supports the integrated processing of several media types with at least one time dependent medium.

7. Relations between objects of MM We concentrate on: • Temporal relationship • New flavor in the context of multimedia • Essential for communications

8. Content Relations Dependence of media objects on data Examples: • A graphic that visualizes data from a spreadsheet • Two graphics showing different views of the same data Explicit definition of dependences for automated update • Only the data are edited • All views of the data are • generated automatically • cannot be edited directly • An update of the data triggers an update of the related views. Implementation of content relations is based on the use of • common data structures or • common object interfaces

9. Spatial Relations Usually known as layout relationships • Define the space which is used for the presentation of a media object • on an output device • at a certain point of time in a multimedia presentation Typically expressed in layout frames: • A layout frame is placed and a content is assigned to this frame • The positioning of a layout frame in a document may be • fixed to a position in a document • fixed to a position on a page • relative to the positioning of other frames Spatial relations for time-dependent media objects: • A frame or a group of frames may be presented in a window • Note: an audio presentation can be positioned on a stereo output device(imagine, e.g., a virtual meeting room where avatars are positioned ina VRML space: audio should be positioned with talking avatar!)

10. Temporal Relations Importance for time-dependent media objects • new in context of Multimedia (content, spatial existed before) • essential for communications, scheduling • non-trivial yet "general" issue --> subject of standardization • --> to be emphasized in this chapter Example for temporal relations: • Temporal relation between • video object • audio object recorded during a concert At presentation time: • temporal relation of the two media objects must correspond to the temporalrelation at the recording moment Relations: • at the same time • independently • in sequel

11. Intra-object Synchronization Intra-object Synchronization Intraobject synchronization defines the time relation between various presentation units of one time-dependent media object Example: • Time relation between the single frames of a video sequence. For a video with a rate of 25 frames per second each of the frames has tobe displayed for 40 ms Frames of a video sequence that shows ajumping ball

12. Inter-object Synchronization Inter-object Synchronization Inter-object synchronization defines the synchronization betweenmedia objects Example: • Time relations of a multimedia synchronizationthatstarts with an audio/video sequence,followed by several picturesand an animation that is commented by an audio sequence

13. Logical Data Units Logical Data Unit An information unit - usually part of a sequence of information units - of a time-dependent media object Granularity levels of LDUs • Application dependent • Imply a hierarchical decomposition of media objects • Example: The LDU hierarchy of a Symphony

14. Classification of LDUs Closed LDUs • have a predictable duration Example: • LDUs that are parts of stored media objects ofcontinuous media like • audio and video or • stored media objects with a fixed duration. Open LDUs • duration is not predictable before the presentation execution Example: • Open LDUs typically represent • input from a live source, for example a camera or a microphone, or • media objects that include a user interaction

15. Classification of LDUs (cont.) Physical units as LDUs: Physical units of a video sequence as LDUs Blocks of samples as LDUs: (all hierarchylevels except lowest) Blocked physical units of an audio sequence as LDUs

16. Classification of LDUs (cont.) User-defined vs. capturing-defined LDU duration Fixed vs. varying LDU duration • e.g., fixed&user-defined: typical for computer generated media objects. LDUs of an animation sequence of user selected size • e.g., fixed&capturing-defined: typical for recorded user interactions. LDUs of an user interaction in a window system with varying durations

17. Classification of LDUs (cont.) Note: open LDUs always with unpredictable duration • Typically a user interaction with in advance unknown duration An open LDU representing a user interaction Note: Timers represented as stream of empty LDUs LDUs of a timer

18. Classification of LDUs - Overview • open vs. closed • open: always unknown duration • closed: fixed or variable duration • fixed vs. variable duration • user-defined vs. capturing-related • physical  blocked

19. Synchronization Examples Lip synchronization • demands for a tight coupling of audio and video streams with • a limited skew between the two media streams LDU view of lip synchronization Slide show with audio comment: LDU view of a slide show

20. Application Example Example used further in the course: A lip synchronized audio video sequence (Audio1 and Video) isfollowedby a replay of a recorded user interaction (RI), a slide sequence (P1 - P3) and an animation (Animation) which is partially commented using an audio sequence (Audio2). Starting the animation presentation, a multiple choice question is presented to the user (Interaction). If the user has made a selection, a final picture (P4) is shown. Synchronization example

21. 2. Synchronization and Multimedia Typical Requirements: • hard vs. soft Strong Interaction with QoS • acceptable, too bad ... • Variety of Mechanisms • Live Synchronisation • Live Synchronisation with Storage • Synthetical Synchronisation

22. Why Multimedia-Synchronization?

23. Multimedia Synchronization - Experiments Example: • at IBM ENC (European Networking Center) Heidelberg to quantify synchronization requirements for: • Audio/video synchronization • Audio/pointer synchronization Selection of material: • Duration • 30s in experiments • 5s would have been sufficient • Reuse of same material for all tests Introduction of artificial skew: • By media composition with professional video equipment • With frame based granularity Test conditions: • Huge set of test candidates • Professional: cutter at TV studios • Casual: every day “user” • Awareness of the synchronization issues • Set of tests with different skews lasted 45 min

24. Feedback - Questionnaire

25. Lip Synchronization: Major Influencing Factors Video: • Content • Continuous (talking head) vs. discrete events (hammer and nails) • Background (no distraction) • Resolution and quality • View mode (head view, shoulder view, body view) Audio: • Content • Background noise or music • Language and articulation

26. Lip Synchronization: Major Influencing Factors Analysis of the phenomenon is difficult • Sample size is small • Responses depend on human mind and mood • Media content to be synchronized

27. Lip Synchronization: Level of Detection Audio ahead of video Audio behind video Areas: • In sync QoS: +/- 80 ms • Transient • Out of sync

28. Lip Synch.: Level of Accuracy/Annoyance Some observations: • Asymmetry • Additional tests with long movie • +/- 80 ms: no distraction • -240 ms, +160 ms: disturbing

29. Pointer Synchronization • Fundamental CSCW (Computer Supported Cooperative Work) shared workspace issue Analysis of CSCW scenarios: • Discrete pointer movement (e.g. “technical sketch”) • Continuous pointer movements (e.g. “route on map”) Most challenging probes: • Short audio (fast speaker using short words) • Fast pointer movement

30. Pointer Synchronization: Level of Detection Observations: • Difficult to detect “out of sync”: • i.e., other magnitude than lip sync • Asymmetry: • According to every day experience

31. Pointer Synchronization: Level of Annoyance Areas: • In sync: QoS -500 ms, +750 ms • Transient • Out of sync

32. Synchronization QoS: Skew Values

33. 3. Reference Model for Multimedia - Synchronization A reference model is needed • to understand the various requirements for • multimedia synchronization • to identify and structure runtime mechanisms • to identify interfaces between runtime mechanisms • to compare system solutions for multimedia synchronization systems

34. Four Layer Model Each layer implements synchronization mechanisms by an appropriate interface. These interfaces can be used to specify and/or enforce the temporal relations. As we go upwards higher QoS and programming abstraction

35. SPECIFICATION LEVEL (is an open layer, includes applications and tools Which allow to create synchronization specifications, e.g. sync editors); editing And formatting, mapping of user QoS to abstractions at object level OBJECT/SERVICE LEVEL (operates on all types of media and hides differences between discrete and continuous media), plan and coordinate presentations, initiate presentations STREAM LEVEL (operates on multiple media streams, provides inter-stream synchronization), resource reservation and scheduling MEDIA LEVEL (operates on single stream; treaded as a sequence of LDUs, provides intra-stream synchronization), file and device access Reference Model for Synchronization • Synchronization of multimedia objects are classified with respect to four-level system

36. Example • Display subtitles at predefined times during the playout of a digital movie

37. Media Layer • Application operated on a single continuous media stream, sequence of LDUs • Interface provides device independent operations like read, write, etc.

38. Stream Layer • Abstraction for QoS for intrastream synchronization in a stream and interstream synchronization between streams of a group • LDUs are not visible. • Application: start(stream), stop(stream),start(group), stop(group), create_group(list_of_streams), • set_cue_point(stream/group, at, event): for interactions with time independent media and user interactions is done by the attachment of events to continuous media.

39. Object Layer • Operates on all kinds of media and hides the difference between discrete and continuous media • Provides complete synchronized presentation to the application • Responsible for the correct scheduling of the overall presentation • Uses the services of lower layer for interstream and intrastream synchronization.

40. Specification Layer • Open layer. Does not offer an explicit interface • Contains applications and tools • synchronization editors • MM document editors • Authoring systems • Responsible for mapping QoS parameters at the user level to the object layer interface • Synchronization specification methods can be classified as: • Interval based • Axis base • Control flow based • Event based

41. Specification Layer Macromedia Director

42. Specification Layer

43. Layered Model - Summary

44. Synchronization Methods • Interval based • Axes based • Control flow based • Event based • Scripts

45. Interval Based Specification • An interval is the presentation duration of an object. Two time intervals can be synchronized in 13 modes Types of temporal relations between two objects

46. 10 operators have been defined to handle these interval relations Operations with one delay parameter A A A A 1 1 1 1 B B B B Before(1) Beforeendof(1) Cobegin(1) Coend(1)

47. Operations with two delay parameters A A A A A 2 2 1 1 2 1 2 1 2 1 B B B B B while(1,2) delayed(1,2) startin(1,2) endin(1,2) cross(1,2)

48. A 1 3 2 B overlaps(1,2,3) Operations with three delay parameters

49. Example • A slide show with slides Slidei (1 £ i £ n) and an audio object can be specified as: Slide1 cobegin(0) Audio Slidei before(0) Slidei+1 (1 £ i £ n-1) Lip synchronization can be specified as  Audio while(0,0) video

50. R1 Interaction Audio1 P1 P2 P3 Audio2 P4 Video Animation t The application example can be sketched as follows Audio1 while(0,0) video Audio1 before(0) RecordedInteraction RecordedInteraction before(0) P1 P1 before(0) P2 P2 before(0) P3 P3 before(0) Interaction P3 before(0) Animation Animation while(2,5) Audio2 Interaction before(0) P4