The Representation of Content

1. Professional Content Management Systems4th Lecture: Content Representation and MetadataDr. Andreas MautheSCC – Lancaster University

2. The Representation of Content • Proxy Representation • Correspond to specific views • Highelight particular characteristics • Visualise specific properties

3. Proxy Types • Textual Proxies • Written documentation in plain text • Documents • Abstracts, scripts, production sheets, etc. • Contracts, legal documents • Structured descriptions • Edit Decision List (EDL) • Log sheets (timecoded textual descriptions) • Database Proxies • Structured content representation • Data model • Elements & attributes • Persistent storage within a DBMS • Multimedia Proxies • Browse copies • Low-resolution audio & video copies • Audiovisual abstracts • Keyframes, skims • Automatically extracted and compiled • Combinations of media types • Storyboards (text, images, time-codes) • Transcripts linked to brows copies Metadata

4. Metadata • Definition • “Data about data“ • Describtion of content • Representation of content • Description Aspects • Viewpoints • User view • Application view • System view • Workflow steps • Covers entire workflow from content creation to archiving • Content exchange • Inner-organisational • Inter-organisational • Metadata Types • Content related metadata • Material related metadata • Location related metadata

5. Object related Metadata • Concept • Describes content object as a single entity • Content identifiers • Content description • Title, creators, directors, etc. • Involved instances • Programmes, editorial offices, etc. • Parties, etc. • Representation in the System • Database • Data model capturing all relevant aspects • Database Management System for persistent storage and search • Structured files • Index by fulltext engines • Stored in file systems

6. Workflows and Content Description • Acquisitionand Ingest • Creation information • Automaticly retrieved • GPS positioning, date & time, recording equipment, format, etc. • Mannually entered • Cameraman, sound engineer, etc. • Acquisition • Video & audio source & compression format • Recording parameters, production information • Ingest • Production ID, title, date & time, abstract, shot quality • Production, Broadcast and Archiving • Post-production and editing • Timecode, subtitles, comissioning, picture editor • Programme as Broadcast (PasB) • Date & time of broadcast, channel, statistical information, re-broadcasts • Cataloguing • Cataloguer, Catalguing date, topical classification, programme affiliation, abstract

7. Rights “Content without Rights is a Liability and not an Asset” • Turning Content into Assets • Content is a liability • You have to store it • You have to care for it • You have to migrate it to new formats • Only if you have the right to use it, it can be considered to be an asset. • Clarifying the Rights Situation • Who owns the rights? • What rights have been acquired? • What personality rights are involved? • What are my options?

8. Rights Management • DRM has two Components: • Rights Description stating the rights attached to a piece of content such as: • Rights Ownership, e.g. IPR (authors, composers, directors, photographers, painters, etc), performers rights (actors, musicians, etc.) and personality rights • Usage Restrictions allow a rights owner the use of content considering restrictions such as territory, media (i.e. TV broadcast, radio, cinema, etc.), time period, and number of performances/ broadcasts Related to metadata • Rights Protection to prevent and discover a breach of rights by: • Restricting access • Marking content (e.g. Watermarking) • Scanning content use (e.g. Fingerprinting) Related to essence processing • Problem Area: Internet • Not properly covered by broadcast rights or any other IPR juristiction • Breach of rights difficult to discover • Rights difficult to enforce

9. Object Related Metadata: Data Model Example: BBC SMEF* Data Model: Programme Instances * has not been deployed • Has to represents • all content objects • all instances (e.g. programmes, editorial offices, etc.) • all involved parties (i.e. persons and organisations with roles & rights) • Has to cover • all business processes • all workflow aspects • all technical metadata • Has to consider other systems such as • Enterprise Resource Planning Systems (ERP) • Rights Management Systems (DRM)

10. Metadata Data Models • Basics • Mathematical formalisation • Notion for describing data • Set of operations to manipulate data • Kinds of Data Models • Entity-relationship models • Conceptual description of organisation using ER diagrams • Network models • Binary many to one relationships • Hierarchical models • Relational models • Operations are relations resulting in relations • Object-oriented models • Records with unique address • Object-relational models • Issues to Consider • Kind of data and operations • Classification according to Stonebraker • Legacy Databases • Existing expertise • Start with use cases, processes and workflows, subsequently chose data model

11. Stages of Content Creation and Metadata Sets

12. Segment Related Metadata • Segments in Content Objects • Segmentation can be • Temporal (time-codes) • Spatial (region co-ordinates) • Advantages • Cataloguing flexibility • Describe segments at appropriate level of detail • Target search operations • Temporal Segmentation • Segments associated with time-line • Logical or physical time-line • Text associated with segments • Discrete media elements are associated with time concepts • Log Sheets

13. Segment related Metadata: Stratification

14. Content Hierarchies • Collection of Programmes • Grouping of programmes that are part of the same products • E.g. TV series, news programme, current affairs magazine • Programme • Logically related chunks of programme items • E.g. single broadcast of news programme, current affairs magazine • Programme Item • Single entity part of a larger unit • E.g. news story, interview, performance • Shot • Sequence of frames between transition

15. Serach and Query for Metadata • Requirements • Unified search interface • User groups • experienced vs. non-experienced • Search over multiple databases and information systems • Query Types • Fulltext Query • Natural language search • Performed on indexed files • Query for Labels • Structured queries for specific concepts • Names, places, dates • Mapped onto different information systems • Query for Segment • Segment search only • E.g. “ person = X present at place=Y and keyword= A, B, C“ • Native Query • Database search • No federated search possible

16. references references Inter-Organisational Metadata Exchange ProprietaryData Model P/META, SMEF MPEG-7 K-L-V XML Serialize/Pack Encode Transmit SDTIMOS CORBA . . . SMPTE Metadata Dictionary P/META, SMEF MPEG-7 ProprietaryData Model Unpack/Deserialize Decode K-L-V XML

17. SMEF and P/META • Standard Media Exchange Framework (SMEF) • Developed by the BBC Media Data Group • Exchange data model for the BBC • No BBC wide applicable data model • Existing systems keep their data models • New systems have to be SMEF compliant • Covers complete broadcast production chain • Plan, draft and commissioning • Content creation • Promos and programme scheduling • Playout and transmission • Media management and archive • Probably the most extensive data model today • P/META • Working group within the European Broadcasting Union (EBU) • Goal: EBU standard exchange data model („EURO-SMEF“) • SMEF is one contribution

18. SMEF Basics • Core Concept • Entities: “Some thing an organisation Maintains“ • Entities are characterised by attributes • Dynamic attributes • Reference data (e.g. ISO country codes) • Relationships represent associations between entities • Defined by business rules derived from business processes • Entity-relationships diagrams to illustrate the data model • Media object at the centre of the data model • Entity Structure • Editorial Objects • EDITRORIAL_OBJECT_VERSION ( title, sub-title, creation date & time) • Subtypes: • PROGRAMME_OBJECT_VERSION • MUSIC_SPEECH_SOUND_ITEM_OBJECT • OTHER_ITEM_OBJECT • EDITORIAL_OBJECT_VERSION_WORK • Record of creation and development process • EDITORIAL_OBJECT_VERSION_INSTANCE • Temporary or permanent instance of work • Editorial Object Groups • EDITORIAL_OBJECT_GROUP (e.g. series, programmes strands) • ACQUISTION_BLOCK (e.g. CD) • EDITORIAL_OBJECT_ASSOCIATION (e.g. playlists)

19. Entity Structure (cont.) • Media Object Entity • MEDIA_OBJECT (action start date & time, capture start date & time, etc.) • Single media type (e.g. Audio clip, video segment, stills, etc.) • General and editorial metadata • Subtypes: • AUDIO_CLIP, SHOT • GRAPHIC, STILL,DATA (e.g. captions, web sites, text, etc.), • MEDIA_OBJECT_GROUP • Non-hierarchical grouping of events (e.g. different shots of one event on the football pitch) • MEDIA_OBJECT_GROUP_TYPE • Action, sensor perspective, sound • UNIQUE_MATERIAL_ENTITY (compression ratio, creation date and time) • Specifies media related subtypes and coding standards • Other Entities • Documentation and Management • STORAGE, STORAGE_TYPE • EDITORIAL_DESCRIPTION_SCHEME,EDITORIAL_DESCRIPTION_TERM • Individuals • PERSON, ORGANISATION • ROLE • Production related entities • Transmission related entities

20. SMEF Data Model: Contract Negotiation

21. SMEF Data Model: Editorial Object Example

22. MPEG-7: Multimedia Content Description Interface • Objectives and Scope • Defines a Tool Set for the description of multimedia content • For human users and automatic systems processing media/ essence • Defines metadata elements and their relationships • Description application context dependent • Low-level features for media description (e.g. region motion tracking, colour, encoding) • High-level features for content description (e.g. title, usage history, summaries) • MPEG-7 Elements • Descriptor • Define syntax and semantics of feature representation • Descriptions Scheme • Define syntax and semantics of relationships between components • Description Definition Language (DDL) • Defines the syntax of the description tools for Descriptors and Description Schemes (using XML schema) • Links with other Standards • EBU P/Meta • SMTPE Metadata dictionary • Dublin Core

23. MPEG-7 Parts MPEG-7 Systems: binary format for encoding MPEG-7 description and terminal architecture MPEG-7 DDL: language for Descriptior and Description Scheme definition MPEG-7 Visual: description tools dealing with visual descritpions only (technical & structural aspects) MPEG-7 Audio: description tool dealing with audio descritpions only (technical & structural aspects) MPEG-7 Multimedia Description Schems: description for generic features and multimedia MPEG-7 Reference Software: software implementations of parts with normative character MPEG-7 Conformence: guidlines and procedures for testing standard conformence of MPEG-7 implementations MPEG-7 Extraction and use of description: information material about extraction and use of Description Tools

24. MPEG-7 Systems and DDL • MPEG-7 Systems Part • Coding of metadata • Tools for coding and processing • Considering transport and storage requirements • Terminal architecture and normative references • Language for Description Schemes • Including binary and dynamic descriptions • Specifies incremental delivery using so-called Access Units (AU); parts of the description tree • Binary Format (BiM) for compress and streaming of descriptions • MPEG-7 DDL • Language for the definition of syntax, structure and content of an MPEG-7 document • Descriptors and DS must conform to the DDL • Based on XML Schema Language • DDL structural components • XML Schema structural language components • Namespaces, schema wrappers, element declarations, attribute declarations, type definitions • XML Schema data type language components • Built-in primitive (e.g. string, Boolean) and derived data types • MPEG-7 specific extensions • Array & matrix data types, time concepts

25. MPEG-7 Visual and Audio Parts • Objectives • Description tools for technical and structural description of visual and audio descriptors • Specifying low-level descriptiors • Audio also high-level descriptors • MPEG-7 Visual Descriptors • Colour Descriptors • Describing colour distribution, spatial layout, structure of colour • Texture Descriptors • Describing visual patterns • Shape Descriptors • Describing shape of visual object • Motion Descriptors • Describing motion of video objects • Camera motion and object motion • Face Descriptors • Describing facial characteristics

26. MPEG-7 Audio Descriptors • High-Level Audio Descriptors • Tools for specific applications • Sound recognition and indexing • Taxonomy of sound classes • Ontology of sound recognisers • Spoken Content tools • For indexing and retrieval of spoken content • Combine word and phone lattice • Musical feature description tools • Instrument timbre • Melody description

27. Content Collections Models Content Organisation User Interaction Navigation & Access Creation & Production Usage History Content Management Summaries Usage Media Views User Preferences Structural Aspects Content Description Semantic Apsects Variations Basic Elements MPEG-7 Multimedia Description Schemes Tools (MDS) Basic Datatypes Link & Media Localisation Basic Tools Schema Tools

28. MPEG–7 Conclusion • MPEG–7 Reference Tools & Relationships • Reference Software • Simulation platform XM • Normative character for • Descriptors, Description Schemes, Coding Schemes, DDL • Under development • MPEG-7 Conformance • Testing the conformance of implementations • MPEG-7 Use of Descriptions • Extraction and use of MPEG-7 Description Tools • 25 frames/sec • Summary • Extensive set of low- and high-level descriptors • 183 relevant concepts • Provides standard framework • Does not prescribe model or scheme • Does not include database model or dictionary description • Open issues • Semantic mappings schemata • General applicability B. Manjuhath, P. Salembier, T. Sikora (Eds.): “MPEG-7: Multimedia Content Description Interface”, John Wiley & Sons Ltd, Chichster, UK, 2002