Lots of bits in imagery Not enough pipe to push imagery to the field

1. Using Data Compression to Increase the Bandwith of Existing Tactical Control SystemMalcolm VantDeputy Director General at DREOChairman of NATO IST • Lots of bits in imagery • Not enough pipe to push imagery to the field • Visualisation quality is tied to compression Problems

2. Margaret Varga Sensors and Electronic Sectors Defence Evaluation & Research Agency Malvern Worcestershire WR14 3PS UK Click to add sub-title Telephone: +1684 895712 Facsimile: +1684 894952 Email: varga@signal.dera.gov.uk Information Management

3. Background • Huge volumes of diverse data to be analysed • Existing systems are outpaced • Manual examination is still the main approach • Need for fast, cost effective and reliable semi-automatic or automatic approaches • To retrieve, screen, evaluate, correlate, disseminate, archive and present information at the right time, place and format

4. Information Deluge

5. An Overview of an Information Management Process Analysis Interpretation / Evaluation Data Information Intelligence Data Mining Transformation Pre-processing Data Selection DataCollection Knowledge Patterns Transformed Data Pre-processed Data Target Data Raw Data Visualisation

6. Information Management Process(MoD) • Data Collection - Reports (digital/hardcopy), formatted, application oriented • Data Selection - Information Retrieval/Extraction/Routing • Pre-Processing - Elimination/reduction of outliers • Transformation - Conversion into appropriate format for processing • Pattern Finding - Data mining • Interpretation/Evaluation of Patterns - Knowledge discovery • Analysis of Knowledge - Intelligence • Summarisation • Information visualisation

7. Information Management Principles • Relevance • Effectiveness and Efficiency • Responsiveness • Accessibility • Custody • Preservation • Accountability

8. Open Source Textual Information Exploitation(Military Relevance) • Timely access to relevant, accurate, precise, comprehensible and credible information (e.g. for decision making) • Ability to track what information is available (covert and overt) • Ability to define rapidly changing topics of interest from diverse users • Zoom or expand over levels of specialised knowledge (i.e. summary or detailed analysis)

9. Limitations of Existing Information Retrieval Approaches • Commercial Off The Shelf (COTS) products are only effective in a few domain specific applications • COTS based on user-specified ad hoc keywords, phrases or topics: • subject scope can become too general and unfamiliar • costly (time & money), depends on predefined search definition • irrelevant reports are retrieved and relevant reports may remain hidden • Full reports are presented • Duplicate information from same and different sources is presented • Retrieved reports are ranked by existence and frequency of keywords- not necessary equivalent to relevancy

10. Relevance Ranking AccurateKey Generation & Topic spotting Summarisation Redundancy Elimination Open Source Textual Information Exploitation Research Programmes Delos - Textual (and speech) information sources Indexation, Profiling, Archival & intelligence Analysis DERA-OKAPI Demonstrator Visualisation Plumber, TextScape, Delos

11. DERA-OKAPI Information Retrieval Demonstrator

12. DERA-OKAPI Information Retrieval Demonstrator

13. DERA-OKAPI Information Retrieval Demonstrator

14. User Interest Oriented Text Summarisation • Accurate ‘keys’ are used as salient summary indicators of the news feeds • Extracted sentences with the occurrence of the accurate keys • Extract the leading paragraph of the news item as a ‘rough’ summary of the reporting event • Extract the sentence before and after the ‘key’ sentence - reduce dangling problems • Paragraphs with more than 3 ‘keys’ or more than 70% relevant sentences

15. Text Summarization Amount of text News Items 100% 1 - 3% Keywords /phrases 5 - 10 % First paragraph Sentences with ‘keys’ 15 - 20% Sentences before and after the sentence with ‘keys’ 25 - 30 % Paragraphs with more than 70% of sentences deemed relevant or > 3 keys 30 - 35%

16. Salient Keys

19. Salient key-based summary

21. Information Visualisation • Represent both the structure & content of the retrieved information • Readily comprehensive manners • Facilitate high level tasks • Identification of similar/duplicate information • Identification of topic trends • Identification of anomalies • Provides access to different levels of summary

22. DERA TextScape

23. DERA TextScape

24. DERA TextScape

25. DERA TextScape

26. DERA TextScape

27. DERA TextScape

28. Evaluation Issues Retrieval effectiveness: • Dynamic data sources • User’s oriented/task oriented • Key word saliency Summarisation • Summaries first • Followed by full report Visualisation • Effective in showing the • Identification of similar/duplicate information • Identification of topic trends • Identification of anomalies • Inter-relationship? Temporal ?

29. Event Stream Analysis • Analysis of large datasets generated during intensive man/machine interaction • The analysis of the large log file generated by these interaction is assisted by DERA’s Plumber • Plumber - a visual programming tool which enables a new visual environment for data processing • Part of a commercial package of DP+MS

30. Plumber

31. Java version