Technical University of Crete Department of Electronic and Computer Engineering

1. Technical University of CreteDepartment of Electronic and Computer Engineering Multimedia Data Management Euripides G.M. Petrakis http://www.intelligence.tuc.gr/~petrakis http://courses.ced.tuc.gr Chania 2010 Introduction

2. Definition • Multimedia: composite entities combining text, audio, images, video (bit-stream objects), graphics • Multimedia Information Systems: database systems that support all multimedia data types and handle very large volumes of information Introduction

3. TEXT • The most common type of information • The least space intensive data type • The form in which text is stored varies (plain ascii, word files, spreadsheets, annotations, database fields etc.) • Text fonts are becoming complex allowing special effects (color, shade, fill etc.) Introduction

4. AUDIO • Space intensive (one minute can take up Mbytes), presented as analog, digital or MIDI • Analog waveform: electrical signal, • amplitude specifies the loudness of the sound • in microphones, tapes, records, amplifiers, speakers • Digital waveform audio: digital, • less sensitivity to noise and distortion • involves larger processing and storage capacities • Digital Audio Tape (DAT), Compact discs (CD) • WAV (Microsoft’s wave file format) Introduction

5. MIDI (Musical Instrument Digital Interface) Furht et.al.96 • Commands that describe how the music should be played are stored (instead of sound) • A music synthesizer generates sound • Provides high data compression, • Widely accepted Introduction

6. IMAGES • Digital images: sequences of pixels • Pixels: numbers interpreted to display intensity, color, contrast etc • Binary (0-1 values), gray-scale (8 bits/pixel), colour (3x8 values for RGB) • Space overhead depends on image type, resolution, compression scheme • Image formats: tiff, bmb, jpeg etc. Introduction

7. Image Concepts and Structures • Binary images: 1 bit/pixel • black & white photos, facsimile images • Computer Graphics: 4 bits/pixel • Grayscale images: 8 bits/sample • Color images: 16, 24 bits/pixel Introduction

8. RGB Representation • A color is produced by adding • red, green and blue • The straight line R=G=B specifies gray values ranging from black to white Introduction

9. YUV Representation • YUV describes the luminance and chrominance components of an image • 1 luminance: gray scale version of an image • Y = 0.299R + 0.587G + 0.114B • 2 chrominance components: • U = 0.564(B - Y) • V = 0.713(R - Y) Introduction

10. Conversions • Conversion between RGB and YUV requires multiplication operations • an approximation: • Y = R/4 + G/2 +B/2, U=(B-Y)/2, V=(R-Y)/2 • R = Y + 2V, G = Y – (U + V), B = Y + 2U • YCbCr is another color format • for compression • Cb = U/2 + 0.5, Cr = V/1.6 + 0.5 Introduction

11. VIDEO • The most space intensive data type • A sequence of frames • Realistic video playback, transmission, compression/decompression require transfer rates about 30frames/sec • Microsoft’s AVI and Apple’s Quicktime file formats integrate video and audio in the same presentation Introduction

12. Sequence of Frames Khoshafian Baker 96 Introduction

13. Audio-video Modes of Operation Furht et.al.96 • Can be either stored or used / transmitted live in real-time • Can be used interactively or non-interactively Introduction

14. GRAPHICS • Objects described through their basic elements (e.g., 2D, 3D shapes) • these elements can have different sizes, position, orientation, surface, fill etc. • compact representations • generated and can be manipulated by design tools (e.g., CAD tools) • Their descriptions are stored in files Introduction

15. 2D and 3D graphics objects Khoshafian Baker 96 Introduction

16. MULTIMEDIA objects • Text, audio, images, video, graphics are elements of complex multimedia objects • Various tools or applications integrate, process and combine multimedia • Applications: multimedia authoring applications that output documents and databases and end-user applications (e.g., video on demand) • Tools: for viewing, updating, querying (presentation viewers, browsers etc.) Introduction

17. Multimedia Databases (MDB) • Means stored information or database management systems (dbms) • Multimedia dbms (mdbms) integrate conventional database capabilities together with different technologies such as Hierarchical storage management (HSM) and Information retrieval (IR) Introduction

18. Multimedia Technologies • Technologies integrated within a mdbms • HSM support • IR support (exact and approximate) • Spatial data types and queries • Interactive querying, relevance feedback, refining • Automatic feature extraction • Automatic content retrieval and indexing • Query optimization Introduction

19. Database capabilities • Persistence: object persist through invocations • Transactions: content is inserted, deleted, updated • Concurrency control: transactions run concurrently • Recovery: failed transactions are not propagated to the db • Querying: content can be retrieved • Versioning: access previous states of objects • Integrity: transactions guarantee consistency of content • Security: constraints for accessing/updating objects • Performance: optimal data structures and programs Introduction

20. Hierarchical Storage Management (HSM) • Support storage of multimedia objects • On-line: on RAM, magnetic disk • Near-on-line: on optical storage • Off-line: on tapes, shelves • Each level has different • Performance: decreases from top to bottom • Capacity: increases from top to bottom • Cost: decreases from top to bottom Introduction

21. Information Retrieval (IR) Capabilities • Retrieval is the most common operation • Deletions and updates are less common • Exact match: search based on exact information • Inexact: search based on inexact information e.g.,partial, neighborhood search, can be fuzzy or probabilistic • The results are ranked by order of relevance to the query • Query refinement • Iterate over query results • Adjust weights of query terms or features • And finally resubmit queries Introduction

22. MDBMS architecture Khoshafian Baker 96 Introduction

23. MDBMS Implementation • Relies on 3rd party vendors for each component • Relational dbms for typical records • separate optical storage module for text/audio/graphics/images/video • Text retrieval system (e.g. Lucene) • Audio/image/video retrieval system • Feature extraction system • Multimedia object interface system Introduction

24. Object-Oriented Multimedia Databases • Better design, better suited for multimedia applications • Uniform handling of data and operations • Data types are objects with internal structures and operations that capture the behavior of objects (e.g., audio playback, video browsing) • OO dbms does not satisfy all MM requirements • Provides primitives for object handling • Multimedia components need to be implemented or integrated Introduction

25. Object orientation in MMDBMS’s Khoshafian Baker 96 Introduction

26. Client-Server Architectures • Multimedia databases operate in client-server architectures • A number of interconnected server nodes provide I/O capabilities for multimedia objects • Basic DBMS module • IR module • HSM module • More services may include scanner, fax services etc. Introduction

27. Client-Server in mdbms Khoshafian Baker 96 Introduction

28. Various modules and servers in an MMDBMs architecture Khoshafian Baker 96 Introduction

29. Multimedia Applications • Multimedia Systems suggest a variety of applications • Multimedia conferencing • Multimedia on demand (interactive TV, news on demand) • See next page for more … Introduction

30. Multimedia Applications Furht et.al.96 Introduction

31. Multimedia Conferencing • Multimedia conferencing enable a number of participants to exchange multimedia information • Skype, PoWWoWNow, Webex • Each participant has a workstation linked to other workstations over high-speed networks • Each participant can send or receive mm data and perform certain collaborative activities Introduction

32. A video conference system Furht et.al.96 • The biggest performance challenge occurs when the participants transmit voice and video • These are mixed together to form a composite stream consisting of video and voice streams Introduction

33. Software Architecture Furht et.al.96 Introduction

34. System Functions • Multipoint connection set-up: the system negotiates for network resources • Dynamic session control: add/delete participants • Conference directory service: supports registration, announcement, query etc. • Automatic scheduling and recording: planning of network resources. • Conference teardown: release reserved resources Introduction

35. Architectures • Fully distributed: direct connections between the participants • Processing and mixing of media at every location • Shortest delay • The connections increase rapidly • Centralized (star) network: a central is connected to every participant • Processing and mixing at central node • The central node waits until all media is received before mixing and broadcasting Introduction

36. Architectures (cont.) • Double star network: a central node from one star network is connected to another central node of another star network • Hierarchical network: intermediate nodes, root and leaves (participants) connected hierarchically • intermediate nodes perform mixing and processing • the completely mixed data is sent to root who broadcasts directly to the leaves • reduces network traffic significantly Introduction

37. Multimedia conferencing network architectures Furht et.al.96 Introduction

38. Video on Demand • Fast networks coupled with powerful computers and compression techniques will be capable of delivering stream data in real-time • On-demand multimedia services • interactive entertainment • video news distribution • video rental services • digital multimedia libraries Introduction

39. Interactive Television (ITV) • An ITV system must be capable of providing • basic TV • subscription TV • pay per view • video on demand • shopping • education • electronic newspaper • financial transactions • single-user and multi-user games Introduction

40. ITV network Architectures • Distributed architectures, components • information (content) servers: entertainment companies, TV stations connected to stb’s offering different types of services • network: wide area network (ATM) connects the head-ends which are connected with subscribers • set top boxes (stb) : terminal devices that allow users interact with a network providing personalized, on-demand, interactive services Introduction

41. A General ITV Architecture Furht et.al.96 Introduction

42. Hierarchical ITV Architecture Furht et.al.96 Introduction

43. News on-Demand • Delivers news on demand to various corporate and financial services • The Nynex and Dow Jones systems: • The Dow Jones production center produces news which are send through a wideband network to the Nynex media service center • Customer sites can request specific news from the Media Service Center which are delivered over the network and stored at customer sites Introduction

44. Architecture of Nynex and Dow Jones system Furht et.al.96 Introduction