Multimedia Presentation and Delivery

1. Multimedia Presentation and Delivery Ch.13 Principles of Multimedia Database Systems. V.S. Subrahmanian, 1998

2. Presentation & Retrieval • When creating a multimedia presentation, three basic questions must be answered • What objects should be included in the presentation ? • When should these objects to be presented to the user ? • Where should the objects appear on the screen ? • These questions must be answered by the individual who creates the presentation (i.e., author) • Once these have been specified, the Presentation Engine must generate a Retrieval Plan like: • when the objects need to be presented • bandwidth limitation on the network • resource (load, buffer) limitation on the server & client • mismatches between delivery rate and client consumption rate. MM Presentation & Delivery

3. Interaction Between Presentation & Retrieval Schedules MM Presentation & Delivery

4. Creating a Presentation • Suppose objects o1,...,on. are to be presented. • Temporal constraints specify how the objects should be laid out in time. Example: • The presentation of objects 1 and 2 must start at the same time. • The termination of objects 2 and 3 must occur at the same time. • Object 3 must start at the time object 1 ends. • Spatial constraints specify how the objects should be laid out in space. Example: • Object 1 must be to the left of object 2. • Object 1 must be above object 3. MM Presentation & Delivery

5. Seven Possible Temporal Relationships MM Presentation & Delivery

6. Constraint Language • Constants: Every integer (positive and negative) is a constant. • Variables: Associated with each oi are two integer variables, si (denoting the "start" of oi) and ei (denoting the "end" of oi) • Elementary Terms: Elementary terms are defined inductively as follows: • Every constant is an elementary term • Every variable is an elementary term • Difference Constraint: If tl, t2 are elementary terms, and c is a Constant, then • tl - t2 < C is a difference constraint MM Presentation & Delivery

7. Examples of Difference Constraints • e1 - s1  10. This constraint says that object o1 must end within 10 time units of the time its presentation starts. • s2 - e1  0; e1 - s2  0 These two constraints jointly state that object o1's presentation starts as soon as object o1's presentation ends, i.e., s2 - e1 = 0 • s2 - e1  3 This constraint says that object o2's presentation starts within 3 time units of the end of the presentation of object o1. MM Presentation & Delivery

8. A Definition of Temporal Presentation • A temporal presentation is a pair TP = (O, DC) where O is a finite set of objects, and DC is a finite set of difference constraints in the constraint language generated by O. • Suppose DC is a set of difference constraints over O of n virtual objects. A solution of DC is an assignment of integers to each of the variables, s1, ... sn, e1, ... en, which makes all the constraints true. • E.g., • Sets of constraints can have no, one, multiple solutions. MM Presentation & Delivery

9. Possible solutions: • A temporal presentation TP = (O, DC) is feasible iff the set, DC, of difference constraints, has a solution , I.e., a feasible schedule for TP. MM Presentation & Delivery

10. Definitions about the schedule • The start and end of , denoted start() and end(), are defined to be: • start() = min({(si) I1  i  n}). • end() = max({(ei) I1  i  n}). • We are assuming that the constraints {si - ei  0|1  i  n} are included in DC, i.e., a default DC. • For our example: MM Presentation & Delivery

11. How to create a presentation schedule? • We are facing a constraint satisfaction problem which can be solved by linear programming algorithms like simplex method. • There is another classic algorithm called the Bellman Ford Algorithm which • takes as input, a set S of difference constraints. • Convert S to a graph. • S has a solution iff the graph has no negative cycles. • A negative cycle is a cycle whose edges sum up to a negative number. MM Presentation & Delivery

12. An example: Given a set S of DC’s: Form a graph for S: MM Presentation & Delivery

13. An example: (cont.) • Compute the shortest/cheapest paths • If no negative cycles (i.e., costs are not negative), we have MM Presentation & Delivery

14. More about Bellman-Ford Algorithm • Let the cost of a path be the sum of the costs of edges along the path. • A cycle in graph G is a sequence of nodes v1 , … , vk such that v1 = vk and  1  j < k, (vj , vj+1 ) is an edge in the graph. • v1 , … , vkis said to be a negative cycle in G iff it (total) cost is a negative number • A set of difference constraints DC has no solution iff GDC has a negative cycle. • Suppose a graph has no negative cycles. Then the cost of the shortest/cheapest path from the start node to that node (si or ei) directly allows us to get a solution to the original set of difference constraints. MM Presentation & Delivery

15. Algorithm Implementation The Bellman Ford algorithm associates with each node N in GDC, the following two fields: • Bestval: This specifies the cheapest path from the start node to the node N, that has been discovered thus far • Bestpar: Bestpar(N) specifies the immediate predecessor of node N along the best path from the start node to node N, that we have found thus far • Then, • Execute Algorithm 13.3 (which includes Algorithm 13.1, 13.2) in the text. MM Presentation & Delivery

16. A Broader Issue of MM Presentation: Multimedia Synchronization • Attempts to ensure the desired temporal ordering among a set of MM objects in a multimedia scenario, where “MM scenario” denotes the temporal semantics of a MM session • The goals of MM synchronization mechanism are to preserve the playback continuity of media frames within single continuous media and the desired temporal dependencies among multiple related data objects, given that user interactions are allowed. • Temporal dependency can be either implicit (lip synchronization) or explicit (a slide show) • MM synchronization is an important set of QoS parameters • MM synch. differs from traditional synch. mechanisms such as monitor and semaphore in that it imposes real-time constraints (deadlines) on events, and differs from real-time systems in that such deadlines are predetermined, periodic, and soft MM Presentation & Delivery

17. Three Levels of MM Synchronization Intramedia synchronization • also known as intrastream or serial synchronization • preserves the temporal relationship between consecutive frames and the continuity of playback within a single media stream • the basic idea is to smooth the delay jitters experienced in the storage system, network, and host system Intermedia synchronization • also known as interstream or parallel synchronization • coordinates the different media streams to achieve the desired temporal relationships among them MM Presentation & Delivery

18. Intra and Intermedia Synchronization Requirements MM Presentation & Delivery

19. Three Levels of MM Synchronization (cont.) Interparty synchronization • also known as event synchronization • maintains the intramedia and/or intermedia synch. among different participants at distributed locations • e.g., remote learning and video conferencing MM Presentation & Delivery

20. Temporal Synchronization Mechanisms(in a distributed environment) • Synchronization mechanism renders the synchronization specification into a set of playback schedules and retrieval schedule and enforces the playback deadlines by taking into considerations of the network characteristics • e.g. to display a MM object at time s (i.e., the playback deadline), the presentation site sends a request to the corresponding media server at request time r, which is R time units prior to s. R(=s-r)is the response time to account for possible experienced end-to-end delays and packet losses. MM Presentation & Delivery

21. Temporal Synchronization Mechanisms: delay • For discrete object, the playback deadline can be met when R the response time is larger than the total delay (server dser + network dnet + host delay dhost) • e.g. to retrieve an image of size 1.3MB from a remote site with dser=50ms connected by a link of 100Mbps bandwidth, we should have the response time R³ 50ms+1.3x8Mbits/100Mbits+ dhost • dhost is normally negligible. MM Presentation & Delivery

22. Temporal Synchronization Mechanisms: delay jitter • For a continuous media object, each media frame/packet may experience different delay and such intrastream synchronization can be achieved by delaying the the playback deadline of the first media frame by dsyn, i.e., which is the difference between the maximum and minimum network delays of stream i. • Therefore, the response time should be at least (server dser + network dnet + intramedia synchronization dsyn + host delay dhost) • e.g., to retrieve a video clip of fixed frame size 40kB from a remote site with dser=50ms connected by a link of 100Mbps bandwidth and 10ms intramedia, we should have response time R ³ 50ms + 40x8kbits/100Mbits + 10ms + dhost MM Presentation & Delivery

23. More on Synchronization Requirements • Synchronization can be measured by four major parameters: delay, delay jitter (intramedia skew), intermedia skew, and tolerable error • Delay measures end-to-end delay or response time • Delay jitter measures delay variations of continuous media • Intermedia skew measures the time shift between related media from the desired temporal relationship, e.g., lip synchronization requires intermedia skew within ±80ms, stereo audio (audio-audio synchronization) requires ±11ms, video-text synchronization requires ±240ms • Tolerable error measures the allowable bit error rate and packet error rate for a particular stream in a particular application, e.g., bit error rate should be smaller than 0.1 (for uncompressed voice), 0.01 (for uncompressed TV-quality video), 0.0001 (for uncompressed image) MM Synchronization

24. Synchronization Specifications & Mechanisms • Synchronization can be refer to time (our focus here), space, and content. • General MM synchronization can be divided into two levels: • (temporal) synchronization specification (upper level) • (temporal) synchronization mechanism (lower level) ! • Synchronization specification models the temporal relationships among the set of related abstract MM objects (e.g., video clip, text) of a presentation scenario • Synchronization mechanism translates the temporal synchronization specification into the desired presentation scheduling sequences and enforces the playback deadlines despite the indeterministic delays due to the server, the network and the host system MM Synchronization

25. Specification by Scripts (Language-Based Approach) • A script uses a structural language (concurrent programming language) to specify the temporal relationships, e.g., Communication Sequential Processing (CSP) scheme • Scripts are powerful and can be used to specify all types of relationships and user interactions Script slide-show { Parallel Display picture1.jpg for 5 seconds Display audio1.au for 5 seconds Parallel Display picture2.jpg for 6 seconds Display audio2.au for 7 seconds Parallel Display picture3.jpg for 10 seconds Display audio3.au for 10 seconds } MM Synchronization

26. Specification by Scripts (cont.) • The first line specifies the name of the script • Parallel, Display, For, and Seconds are keywords of the script • A parallel structure completes when all statements within it complete; so in the second parallel structure, the audio continues to play for 1 sec after the display of picture 2 has ended (A starts B). • Advantage of using scripts: they can be directly lead to implementation • Disadvantage: they are hard to conceptually visualize and are difficult to verify MM Synchronization

27. Specification by Time Lines • In this specification, the presentation time of each stream is specified relative to a common clock. • Streams are treated independently. • Synchronization is achieved by ensuring that each stream is presented at its specified time relative to the common clock • Advantages: simple and easy to use and implement because streams are treated independently during execution • Disadvantages: cannot specify user interaction or streams with unpredictable duration refer to Fig.9.2, p.279 of reference material MM Synchronization

28. Specification by Graphs • Synchronizationcan be specified or modeled by graph, in particular, object composition petri nets (OCPN) • The basic idea is to represent various components of MM objects as places and describe their inter-relations in the form of transitions • It is capable to explicitly capture all the necessary temporal relations, and to provide simulation of presentation in both the forward and reverse directions (better user interaction) • Each place in the model represents the play-out of a MM object while transitions represent synchronization points • It specifies exact presentation time play-out semantics, which is useful in real-time scheduling MM Synchronization

29. Specification by OCPN • The slide show example: • Another example: MM Synchronization