Experiments on Fact Checking & Literature Review on Deep Learning Research

1. More experiments on FND & Literature Review on Fact Checking Deep Learning Research & Application Center 25 September 2017 Claire Li

2. New experiments with fake.csv dataset using content only and eliminating the sentences less than 20 words (average length: 458) New experiments with LIAR dataset (average length: 17.9) Review on fact checking

3. Fake.csv with content only (diffFN-0920), stacked LSTM For 4,000 validating data FN: 44; FP: 127; TN: 1,852; TP: 1,976 Precision: 0.94; Recall: 0.98; F1: 0.96; Accuracy: 0.96

4. Fake.csv with content only, eliminating sentences less than 20 words, stacked LSTM For 4,000 validating data FN: 49; FP: 44; TN: 1,996; TP: 1,911 Precision: 0.98; Recall: 0.97; F1: 0.97; Accuracy: 0.98

5. Fake.csv with content only, eliminating sentences less than 20 words, Bi-directional LSTM For 4,000 validating data

6. Liar, Liar Pants on Fire: A New Benchmark Dataset for Fake News Detection, arXiv preprint arXiv:1705.00648, 2017

7. LIAR dataset from fact-checking website PolitiFact 12,836 human-labeled short statements from news releases, TV/radio, interviews, campaign speeches etc labels: pants-fire, false, barely-true, half-true, mostly true, and true

8. Experiment 1, Bi-directional RNN • False dataset (5,113): pants-fire, false, barely-true • True dataset (6,440) : half-true, mostly true, and true • Precision on Training: 0.9734

9. Experiment 2, Bi-directional RNN False dataset (3219): pants-fire, false, barely-true True dataset (4069) : half-true, mostly true, and true Precision on Training: 0.9941

11. Review on Fact Checking Fact checking is a way of knowledge-based news content verification, can be categorized as • Expert-oriented • relies on human domain experts to investigate relevant data and documents to construct the verdicts of claim veracity, e.g., PolitiFact11, Snopes12 • Crowdsourcing-oriented • exploits the “wisdom of crowd” to enable normal people based news content annotations which are then aggregated to produce an overall assessment of the news veracity, e.g., Fiskkit, ‘for real’ account of LINE, Twitter Trails • Computational-oriented • provide an automatic scalable system to classify true and false claims using open web and structured knowledge graph (e.g. Google Knowledge Graph Search API) • identifying check-worthy claims • discriminating the veracity of fact claims

12. Relationship rather than replacing the factchecker, the software's role is to make their work easier

13. Four stage factchecking processes [1] • Factchecking is the same four stage processes whether it’s done by humans or machines • http://fullfact.org/automated

14. Expert-oriented

15. Fact checking methodologies • Selecting process • For what claims to evaluate • Research methods • techniques and sources that fact checkers use when conducting research on claims • the official rules and editorial policies that govern their approaches • Claim evaluation • the systems and processes by which fact checkers establish the veracity of a claim • Three major fact-checking organizations in the United States • PolitiFact • FactCheck.org • The Washington Post Fact Checker

16. PolitiFact • Began in 2007, a project of the Tampa Bay Times, a publishing company • Rates the accuracy of claims by elected officials, candidates, leaders of political parties and political activists based on its Truth-O-Meter,from True to False, down to the lowest rating, Pants on Fire • http://www.politifact.com/truth-o-meter/statements/ • Monitoring Sources of statements to be checked • comb through speeches, news stories, press releases, campaign brochures, TV ads, Facebook postings and transcripts of TV and radio interviews

17. Truth-O-Meter rulings TRUE – The statement is accurate and there’s nothing significant missing. MOSTLY TRUE – The statement is accurate but needs clarification or additional information. HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context. MOSTLY FALSE – The statement contains an element of truth but ignores critical facts that would give a different impression. FALSE – The statement is not accurate. PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.

18. Principles in Truth-O-Meter rulings • Words matter • Pay attention to the specific wording of a claim, e.g. mitigating words or phrases • Context matters • Full context related to a claim, e.g. comments, questions prompt to, points make • Statements can be right and wrong • Rate the overall accuracy of compound statements containing two or more factual assertions • Timing • The date a statement was made and the available information during the period

19. Process for Truth-O-Meter rulings A writer researches the claim and writes the Truth-O-Meter article with a recommended rulings, say, true, mostly true, half true, half false, mostly false, pants on fire After the article is edited, it is reviewed by a panel of at least three editors that determines the Truth-O-Meter ruling

20. Corrections and review policy • For a factual error • an editor's note will be added and labeled "CORRECTION" explaining how the article has been changed • For clarifications or updates • an editor's note will be added and labeled "UPDATE" explaining how the article has been changed • For the significant mistakes • reconvene the three-editor panel • if there is a new ruling, rewrite the item and put the correction at the top indicating how it's been changed

21. FactCheck.org • Aproject of the Annenberg Public Policy Center (APPC) of the University of Pennsylvania addresses public policy issues at the local, state and federal levels • Aims to reduce the level of deception and confusion in U.S. politics • Rates the accuracy of claims by • focusing on presidential candidates in presidential election years • on the top Senate races in midterm elections • In off-election years, our primary focus is on the action in Congress • Sources of statements to be checked • Sunday talk shows; TV ads; C-SPAN; Presidential remarks; CQ Transcripts; Campaign and official websites, press releases and similar materials; Readers

22. Process: focus on false claims Once a reporter or writer find a statement that we suspect may be inaccurate or misleading • engage with the person or organization that is being fact-checked for supporting materials • If supporting material does not support statement • Check with the sources of information: the Library of Congress for congressional testimony; the House Clerk and Senate Secretary’s office for roll call votes; the Bureau of Labor Statistics for employment data; the Securities and Exchange Commission for corporate records; the IRS for tax data; the Bureau of Economic Analysis for economic data; and the Energy Information Administration for energy data • also interview experts on other topics as needed – for instance, in researching issues on foreign countries, we would contact experts on those areas.

23. Process of story publishing • Line editing • Is context missing? Is the writing clear? Is the word choice accurate? • Copy editing • for proper style and grammar • Fact-checking • goes through the story line by line, word by word • Make sure every statement is correct • By the time of publishing, a story has been reviewed by a line editor, copy editor, fact-checker and by the director of the APPC, Prof. Kathleen Hall Jamieson, a former dean of the Annenberg School for Communication at the University of Pennsylvania

24. Corrections and review Policy If any new information comes to light after we publish a story that materially changes that story, we will clarify, correct or update our story and provide a note to readers that explains the change, why it was made and the date it was made

25. Computational-oriented

26. The State of Automated Factchecking- 2016 • fullfact.org/automated • intends to develop products that automate fact-checking tasks wherever possible, using statistical analysis and natural language processing technologiesin real-time • An open standards, with aims: • Standard data formats (schema.org), so that any new automated factchecking tool can work with any known source. • Shared monitoring systems, so we do not duplicate work unnecessarily. • Open and shared evaluation, so we know what works and what it works for. • Published roadmaps, to attract volunteers, researchers, partners and funders to work with us. • Think global, so that where possible new automated factchecking tools are designed with the aim of being able to work for many languages and countries.

27. The state of automated factchecking

28. Full fact’s roadmap • Hawk • monitoring system, also spots claims have been factchecked • technologies: CrowdTangle, Google Trends, Newswhip, Trendolizer, Trendsmap, and Signal • monitor content and conversations on: Twitter, Facebook, YouTube, Reddit and online forums • Stats • automatically checking statistical claims • Trends • with output from Hawk, monitoring how common a claim is, where it is being made, and who is making it • Robocheck • areal time product provides subtitles of live TV, and add verdicts to claims using the results from Stats and Hawk

29. Automated factcheckingFull fact.org

30. 1. Popular Sources need to track (1)

31. 1. Metadata on sources monitoring (2) Who, where, and when • Whether it is true to say “unemployment is rising”

32. 2. Spot claims (1) • Monitoring claims that have been factchecked before in new text • open source search engines:ApacheSolr and Elastic Search • Percolator, Apache Lucence & Luwak • Identifying new factual claims that have not been factchecked before in new text • Use template like “x is rising” • Use machine learning algorithms to detect ‘check-worthy claims’ or factual claims • ClaimBuster: learn from labelled check-worthy sentences, identifies features they have in common, and looks for these features in new sentences • Argumentative zoning : classify sentences into several different types, which can include identifying factual claims

33. 2. Spot claims (2) • Making editorial judgements about the priority of different claims • The content approaches • Through content of claims • The contextual (information is relevant to an understanding of the text) approaches • Identifying important content based on reach and engagement (social influence) • Combining the content and context approaches • Dealing with different phrasing for the same or similar claims • A fully automated factchecking system should identify different phrases making the same claim, and distinguishing very similar phrases making different claims, as humans can • Paraphrasing in factchecking is harder than in most NLP because precise wording can matter so much to a factchecker’s conclusion

34. 3 . Check claims (1) • Make sure the sources human factcheckers rely on are available as structured data computers can use • Peter Norvig, Google’s Chief Scientist, explained with the successfulness of driverless cars: “we didn’t have better algorithms, we just had better data” • Victoria and David Beckham are married • Reference approaches - Look up their names in the register of marriages • Statcheck (psychology), jEugene (legal doc) • Instant answers from search engines do a similar job • Machine learning approaches - Make a mathematical model of things we have known (e.g. from knowledge base [2]) • True claims are likely to be closer together in a knowledge graph • Contextual approaches - Look at how claims that they are married spread from reasoning about the social & other claims • Claims that survive longer with reaction claims in open discussion than contradictory claims, so they probably are

35. 3 . Check claims (2) Automated checking projects vary in what kinds of types of sources they deal with, what kinds of claims they deal with, and what topics they deal with

36. 4. Create and publish (1) • Automated journalism takes data and tries to make stories, while automated factchecking takes stories and tries to reverse-engineer the data or sources • encoding factchecks as structured data so they can be presented in different places, from shareable widgets to search results (schema.org) • Representing automated factcheckingresults • Real time pop ups for audiences • Truth Teller provides automated factchecking annotations for video clips based on previous human factchecks • Claim tracker for factcheckers and journalists • Full Fact provides a graph of how frequently claims have appeared over time, the details of where the claim has appeared

37. References The state of Automated Factchecking, http://fullfact.org/automated Ciampaglia, Giovanni Luca, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and m Knowledge Networks". In: PLoS ONE 10.6 (June 2015), e0128193. http://dx.doi.org/10.1371/journal.pone.0128193