Summarization and Personal Information Management

1. Summarization and Personal Information Management Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute

2. Announcements • Questions? • Homework 3? • SIDE issues --- you will need to redownload • Plan for Today • McDonald & Chen

3. Quotes from Last Time

4. Student Quote • The www is indeed structural in the sense as discussed in the chapter. It has 'information patches' and 'hubs' and 'authorities' which are exploited by search engines to refine their search results. This structure of the web can help in summarization tasks since foraging is one of the most important aspects of summarization. The more relevant results one can get, the more relevant will be the summarization.

5. Student Quote • The correlation between the link structure and the topical similarity of the web pages have also been discussed which is quite fascinating.

6. Student Quote • The results from information scent …I think thats an actual problem with user face in using search engines where a person just drills down into a website and is presented with information patches with low information scent. This problem can be averted by giving the user a glimpse of information patches (summary snippet) along with the main page result which is presented to the user. This would help alleviate the problem of going through all the low information bearing pages in the same website.

7. McDonald & Chen

8. Why is this paper important? • Good “big picture” view of summarization • Summarization to support information seeking on the web • Nice overview of summarization • Different types of summaries (generic vs. query based) • Different types of evaluations (intrinsic vs. extrinsic) • Nice process – Hypothesis driven • Theory • Design Rationale • Implementation • Evaluation

9. Types of Summaries • Generic Summaries – 2 sentences, created by the AZ generic summarizer • Also included top 5 keywords and number of sentences • Hybrid summaries – 2 sentences, created by the AZ full sentence summarizer • Query based but uses full sentences • Also mentioned which query terms matched and how many times they matched • Query based snippet summary – length equal to two sentences • Original summary – what was posted on the news sites where the articles came from • created using different techniques

10. Student Question • Regarding the text-tiling based approach in the paper, I am not sure as to why the sentences have been picked in a round-robin fashion from a sorted sentence list prepared for every topic discovered by text tiling algorithm.

11. Take Home Message * Is this surprising? What questions were you left with?

12. Good Point! • Snippets have to be provided in a limited space. There can be at the maximum 3 or 4 sentences which should be the most important sentences as relevant to the query term. • What about generic snippet summaries? How would they work? Would you expect them to be successful?

13. Good idea! • A better approach can be to evaluate all the occurrences of the query terms in the document and rank them in the order of the their importance with respect to the query. The author's criteria of ranking the snippets seems to be just the similarity of the query terms (as a bag of word) to parts of sentences. The order of the query terms and matching the stop words occurring between query terms can improve the performance of such systems.

14. Tasks Caveats: To what extent do you think it’s possible to really identify 3 documents that are more relevant than others for the open ended tasks? What about tasks in the middle? Are the open ended tasks both open ended in the same way?

15. Generic versus Hybrid

16. Hybrid versus Snipet

17. Original Summaries

18. Technical Innovations • What’s new? • TF-IDF is related to match with query terms • Proper nouns • Sentence length • Topic segmentation to encourage broad coverage

19. Brain Storming • Assigning weights to heuristics • What could you do instead to combine multiple ranking criteria? • Topic representativeness • TextTiling looks for dips in lexical cohesion • What else could you do to ensure representativeness of topics? • Student Quote: “Its true that generic summaries should extract sentences from all comprehensible topics present in a document however it should ideally extract more from a topic which represents or is more co-related with the central theme of a paper.”

20. Student Question • Also with respect to tf-idf based sentence retrieval, I wonder how models like Okapi would fare. Ideally they should work better on sentential data than plain tf-idf.

21. Types of Evaluations • Intrinsic • Comparison with a gold standard • But there is no perfect summary • Have several humans create a summary and combine • Mainly used to show that the generic summarizer was state-of-the-art • Extrinsic • How useful is the summary for doing a task

22. Experimental Design • 4X4 within subject factorial design • 4 tasks • 4 types of summaries • Random combinations to factor out order effects • Users picked out 3 relevant documents from 12 based on the summaries

23. Experimental Design • Users picked out 3 relevant documents from 12 based on the summaries • Caveats • Experimenters picked out 2 query terms per task • We don’t know what the query oriented summaries would have looked like if the users could have entered their own terms • That would affect which documents they were selecting from and what the summaries would look like

24. Types of Tasks • Browsing/non-specific • Context more important • Generic summaries might be better • Searching/specific • Context less important • Specific user need more important • Query based should be better

25. Findings New finding!

26. Why is this important? • Search engines tend to provide query based summaries • Assumes that user is doing a search task rather than a browsing task

27. Questions?