1 / 45

XG Multimedia Semantic News Use Case

This use case involves the analysis of metadata associated with news videos from broadcasting companies, including textual metadata with manually annotated scenes descriptions. The goal is to build a training corpus for aligning video, audio, and text data. The analysis includes semantic analysis, entity extraction, and linguistic knowledge structures.

baby
Télécharger la présentation

XG Multimedia Semantic News Use Case

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. XG Multimedia SemanticNews Use Case Thierry Declerck, DFKI GmbH Language Technology Lab

  2. Automatic Semantic Analysis of Metada associated with News Videos of Broacasting companies On-going work in the projects K-Space and MESH

  3. Metadata of News Broadcasters • We analysed the metadata available from various Broadcasters • Their data consists of audio/video material and textual metadata. This is a very valuable data set, since the textual metadata consists also in manually annotated scenes descriptions. • This dataset can be used for building a training corpus for automated alignment of video, audio and text data. • In the next slides we see some abstraction over the various types of metadata provided.

  4. The Metadata Labels • <DOC filename=„0324000-3_Journal_ ENG_F4001C_26122003_2000“> • <TYPE>Earthquake Iran</TYPE> • <SERIES>Journal F: 4001 C</SERIES> • <SEG sid=“integer”> • <TITLE></TITLE> • <DESCRIPTION></DESCRIPTION> • <SCENES> </SCENES> • <KEYWORDS></KEYWORDS> • </SEG> • </DOC>

  5. The Title Tag <TITLE> TdT: Erdbeben /Iran/Zerstörungen in Bam/Trümmer/Ruinen/Opfer </TITLE> Extract: “Erdbeben” (keyword for disaster ontology) ; location “Iran” (with NE detection). Other terms, but yet still unclear about their role

  6. The Description Tag <DESCRIPTION> Ein schweres Erdbeben hat im Iran die Stadt Bam fast völlig zerstört. </DESCRIPTION> Linguistic and semantic analysis: [Subj-NP Ein schweres <noun-disaster>Erbeben</noun-disaster>][Vhat][LOC-PPin <ne-country>Iran</ne-country>][OBJ-NP die Stadt <ne-city>Bam</ne-city>] [ADVfast völlig] [Vzerstört]. Extraction: Who (causation):Erbeben (Earthquake) What_action: zertören (destroy) What: Stadt Bam (city of Bam). Here the system can infer that Bam is located in Iran. Where: Iran

  7. The Scenes Tag <SCENES> <SCENE sid="1">Bam: Menschen sitzen zwischen Trümmern auf Boden</SCENE> <SCENE sid="2">verzweifelte Menschen sitzen am Strassenrand</SCENE> <SCENE sid="3">Schuttberge</SCENE> <SCENE sid="4">zerstörte Häuser</SCENE> <SCENE sid="5">rauchende Trümmer</SCENE> </SCENES> Descriptons of sequences of images displayed. Extracting related entities:People within ruins, desperate people, destroyed houses, smoking ruins etc. All those terms can be seen as “consequences of the earthquake”. Important also: they provide for a description of what is to be seen in the video.

  8. The Keywords Tag <KEYWORDS> Naher Osten: Iran; Erdbeben </KEYWORDS> The pattern of the content of this tag allows us to infer that Iran is located in “near-east”.

  9. Linguistic Knowledge Structures • Multiple layers and levels • Low-level linguistic features (tokenization, morphology, …) • Semantic properties of terms and phrases • Named Entities • Relation Extraction (incl. Grammatical Relations) • Semantic linking to domain ontologies • Can involve several abstraction layers connected through reasoning/mapping processes • Semantic linking to other media analysis • Associated to the domain ontology of MESH (natural disasters in the news)

  10. Association of Ontologies

  11. Semantic annotation of Text extracted from Images (Thierry Declerck, DFKI & Andreas Cobet, TUB)

  12. Background • The data: The German Broadcast news programme „Tagesthemen“ • Extract Text from key frames of shots. Annotate those terms semantically • Analyse of the position of the text and the kind of text extracted. 6 cases detected so far:

  13. Case1: Above the picture, just a normal phrase, mostly a nominal phrase (NP)

  14. Case 2: Below the picture: Name of a person and of the function of this person

  15. Case 3: Below the picture: Name of a person and of a city/country

  16. Case 4: Above the picture, just a normal nominal phrase, and below the picture, name of a person,

  17. Case 5: Below the picture the word „Bericht“ (or similar) and name of Person (=> Journalist)

  18. Case 6: A location name. No picture of a specific human

  19. Cross-Media Ontologies The next slides by courtesy ofPaul Buitelaar, Michael Sintek, Malte Kiesel (DFKI GmbH) from the Project SmartWeb. Paper „Feature Representation for Cross-Lingual, Cross-Media Semantic Web Applications“, presented at ESWC 2006.

  20. Semiotic Triangle • See (Ogden & Richards, 1923) - based on • Structural Linguistics (de Saussure, 1916) • philosophical work by Peirce (mostly 19th century)

  21. Semiotic Triangle – the real world ... actual goalkeepers in the real world ...

  22. Semiotic Triangle – concepts ... actual goalkeepers in the real world ...

  23. Semiotic Triangle – words goalkeeper (EN) Torwart (DE) doelman (NL) ... ... actual goalkeepers in the real world ...

  24. Semiotic Triangle – images ... actual goalkeepers in the real world ...

  25. Features • Multilingual Features • Terms with Linguistic Info and Context Models • Example: goalkeeper • part-of-speech: noun • morphology: goal-keeper • context (Google hits stats.): [ gets:420000, holds:212000, shoots:55900, … ] • Multimedia Features • Images with Feature Models • Example: goalkeeper • color: #111111 • shape: human • texture: “keypatch-set 223”

  26. Representation – Proposal • Attach multilingual and multimedia features to classes and properties (and also instances) • use of meta-classes ClassWithFeats and PropertyWithFeats with properties lingFeat and imgFeat (with ranges LingFeat and ImgFeat) • The classes LingFeat and ImgFeat are used for complex feature descriptions rdfs:Class rdfs:Property rdfs:subClassOf rdfs:subClassOf feat:ClassWithFeatsfeat:lingFeatfeat:imgFeat feat:PropertyWithFeatsfeat:lingFeatfeat:imgFeat meta-classes lf:LingFeatlf:termlf:lang… if:ImgFeatif:color if:texture… classes

  27. Representation – Simplified Example

  28. Representation – LingInfo Ontology is-a is-a is-a is-a is-a is-a ...

  29. Example Instance: “Fußballspielers” (“of the football player”)

  30. Features – Interacting Layers

  31. Translating XBRL Into Description Logic Thierry Declerck and Hans-Ulrich Krieger DFKI GmbH

  32. Motivation • Toward a large intelligent web-based financial information and decision support systems in the MUSING project • Till now a prototype based on XBRL (eXtensible Business Reporting Language), as developed within the eTen project WINS • There we experienced the limitations of the XBRL schema, due to the lack of reasoning support over XML-based data and information extracted from documents. • Need to translate XBRL into an ontology

  33. XBRL Example: Header & Metadata <?xml version="1.0" encoding="iso-8859-1" standalone="no"?> <group xmlns="http://www.xbrl.org/2001/instance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:t="http://www.xbrl.org/german/ap/ci/2002-02-15" xmlns:ISO4217="http://www.iso.org/4217" xsi:schemaLocation="http://www.xbrl.org/german/ap/ci/2002-02-15 german_ap.xsd"> <numericContext id="cn0" precision="8" cwa="false"> <entity> <identifier scheme="http://www.xbrl.de/xbrl/sg2">#001</identifier> </entity> <period> <startDate>2001-01-01</startDate> <endDate>2001-12-31</endDate> </period> <unit> <measure>ISO4217:EUR</measure> </unit> ….. </nonNumericContext> …. <t:genInfo.doc.author nonNumericContext="c2">XBRL Deutschland e.V.</t:genInfo.doc.author> <t:genInfo.doc.author.city nonNumericContext="c2">Düsseldorf</t:genInfo.doc.author.city> <t:genInfo.doc.author.compName nonNumericContext="c2"/>

  34. XBRL:Financial Data …. <t:bs.ass numericContext="cn0">1338066</t:bs.ass> <t:bs.ass.accountingConvenience numericContext="cn0">0</t:bs.ass.accountingConvenience> <t:bs.ass.accountingConvenience.changeDem2Eur numericContext="cn0">0</t:bs.ass.accountingConvenience.changeDem2Eur> <t:bs.ass.accountingConvenience.startUpCost numericContext="cn0">0</t:bs.ass.accountingConvenience.startUpCost> <t:bs.ass.currAss numericContext="cn0">749385</t:bs.ass.currAss> <t:bs.ass.currAss.cashEquiv numericContext="cn0">259760</t:bs.ass.currAss.cashEquiv> <t:bs.ass.currAss.inventory numericContext="cn0">209343</t:bs.ass.currAss.inventory> <t:bs.ass.currAss.inventory.advPaymPaid numericContext="cn0">0</t:bs.ass.currAss.inventory.advPaymPaid> …

  35. XBRL to OWL-XBRL • XBRL taxonomies make use of XML in order to describe the structure of an XBRL document as well as to define new datatypes and properties relevant to XBRL. • Allows to check whether a concrete (business) document conforms to the syntactic structure, defined in the schema. • But a need for languages and tools that go beyond the expressive syntactic power of XML Schema. • OWL, the Web Ontology Language is the new emerging language for the Semantic Web that originates from the DAML+OIL standardization. OWL still makes use of constructs from RDF and RDFS, such as rdf:resource, rdfs:subClassOf, or rdfs:domain • Two important variants OWL Lite and OWL DL restrict the expressive power of RDFS, thereby ensuring decidability. • What makes OWL unique (as compared to RDFS or even XML Schema) is the fact that it can describe resources in more detail and that it comes with a well-defined model-theoretical semantics, inherited from description logic

  36. Actual Experiment with the Sesame DB • The basic idea during our (manual) effort was that even though we are developing an XBRL taxonomy in OWL using Protégé, the information that is stored on disk is still RDF at the syntactic level. We were thus interested in RDF data base systems, wich make sense of the semantics of OWL and RDFS constructs such as rdfs:subClassOf orowl:equivalentClass • Current experiment with the Sesame open-source middleware framework for storing and retrieving RDF data. Sesame partially supports the semantics of RDFS and OWL constructs via entailment rules that compute “missing" RDF triples (the deductive closure) • From an RDF point of view, additional 62,598 triples were generated through Sesame's deductive closure.

  37. Example of Entailment Rule:hasPart relation <rule name="owl-transitiveProp"> <!-- note: ?p, ?x, ?y, and ?z are variables --> <premise> <subject var="?p"/> <predicate uri="&rdf;type"/> <object uri="&owl;TransitiveProperty"/> </premise> <premise> <subject var="?x"/> <predicate var="?p"/> <object var="?y"/> </premise> <premise> <subject var="?y"/> predicate var="?p"/> <object var="?z"/> </premise> <consequent> <subject var="?x"/> <predicate var="?p"/> <object var="?z"/> </consequent> </rule>

  38. A concrete Example of deduced Relation • Since we have classied hasPart (as well as partOf) as a transitive OWL property, the rule in the former slide will fiere, making implicit knowledge explicit and produces new triples such as <t_bs, hasPart, t_bs.ass.defTax> although only <t_bs, hasPart, t_bs.ass> <t_bs.ass, hasPart, t_bs.ass.defTax> can be found in the original XBRL specification.

  39. Translating the Base Taxonomy • In the GermanAP Commercial and Industrial (German Accounting Principles) taxonomy (http://www.xbrl-deutschland.de/xe news2.htm), the file xbrl-instance.xsd specifies the XBRL base taxonomy using XML Schema. It makes use of XML schema datatypes, such as xsd:string or xsd:date, but also defines simple types (simpleType), complex types (complexType), elements (element), and attributes (attribute). Element and attribute declarations are used to restrict the usage of elements and attributes in XBRL XML documents. • Since OWL only knows the distinction between classes and properties, the correspondences between XBRL and OWL description primitives is not a one-to-one mapping:

  40. Business Intelligence in MUSING • Next generation Business Intelligence: The MUSING European R&D Project (MUlti-industry, Semantic-based next generation business INtelliGence). Towards a new generation of Business Intelligence (BI) tools and modules founded on semantic-based knowledge and content systems, enhancing the technological foundations of knowledge acquisition and reasoning in BI applications.

  41. Application Domains in MUSING • The breakthrough impact of MUSING on semantic-based BI will be measured in three strategic, vertical domains: • Finance, through development and validation of next generation (Basel II and beyond) semantic-based BI solutions, with particular reference to Credit Risk Management • Internationalisation, (i.e., the process that allows an enterprise to evolve its business from a local to an international dimension, hereby expressly focusing on the information acquisition work concerning international partnerships, contracts, investments) through development and validation of next-generation semantic based internationalisation platforms; • Operational Risk Management, through development and validation of semantic-driven knowledge systems for measurement and mitigation tools, with particular reference to IT operational risks faced by IT-intensive organisations.

  42. Processing of Quantitative Data • Typical Input: Finance reports in PDF

  43. PDF to XBRL (OWL-XBRL) • Mapping from PDF to HTML/XML • Detection in the HTML/XML of relevant layout information that helps in reconstructing the logical units of the original PDF documents (title,header/footer, footnote,tables, free text) • Mapping of terms found in the XML version of the document to XBRL labels. Disambiguating where needed. • Checking if all the lines of the PDF documents are XBRL compliant. Non-compliant information to be saved in a log file. Towards a XBRL checker of balance sheets delivered in proprietary formats. • Generation of the results of the PDFtoXBRL procedure in a multilingual setting

  44. Processing of Qualitative Data • TURNOVER, INCOME, GROWTH: “State of revenues, if depurated from sales related to Consip contract award, which remarkably affected the turnover in 2003, would have, on the contrary, recorded an increase of 3,23% against that microinformatics market which recorded an increase of 3,2% (Sirmi, january 2005).” • Task of identifying relevant expressions and to classify them

  45. Integration of Data • The Challenge: Merging data and information extracted from various types of documents. Also in various languages. And in the XG use case, especially integrated information from news wires.

More Related