Andrew G. West Wikimania `11 – August 5, 2011

1. Andrew G. West Wikimania `11 – August 5, 2011 Autonomous Detection of Collaborative Link Spam

2. Big Idea Design a framework to detect link spam additions to wikis/Wikipedia, including those employing: (1) Subtlety; aims at link persistence (status quo) (2) Vulnerabilities of recent literature [1] And use this tool’s functionality to: (1) Autonomously undo obvious spam (i.e., a bot) (2) Feed non-obvious, but questionable instances to human patrollers in a streamlined fashion

3. Outline • Motivation • Corpus construction • Features • Performance • Live implementation • Demonstration

4. External Link Spam • Any external link (i.e., not wikilink) which violates the subjective link policy [2] is external link spam: • Issues with the destination URL (the landing site). For example: Commercial intent, shock sites, non-notable sources. • Issues with presentation. For example: Putting a link atop article. [[http://www.google.com Visit Google!]] Article: Visit Google!

5. Research Motivations

6. Motive: Social Spam • Not entirely different from link spam in other UGC/ collaborative applications • Much literature [3] • But wikis/WP are unique: • Not append-only; less formatting constraints • Massive traffic in a single installation. • Large topic space: Spam relevant to landing site. • Mitigation entirely community-driven

7. Motive: Incentive • Much research on wikivandalism (link spam is a subset) • But most vandalism is “offensive” or “nonsense” [4] • Not well incentivized; whereas link spam likely serves poster self-interest (e.g., monetary, lobbying, etc.) • Thus, more aggressive/optimal attack tactics expected • In [1], examined the status quo nature of WP link spam: • “Nuisance”: order of magnitude less frequent than vandalism. See also [[WP:WPSPAM]]. • Less sophistication than seen in other spam domains • Subtlety: Even spam links follow conventions. Perhaps an attempt to deceive patrollers/watch-list: Doesn’t work

8. Motive: Vulnerabilities • Novel attack model [1], exploits human latency and appears economically viable: • High-traffic page targets Peak views/second • Prominent placement • Script-driven operation via autonomously attained privileged accounts • Distributed • Other recent concerns: • Declining labor force [5] • XRumer (blackhat SEO) [6] Calculated Jan.-Aug. 2010 Highest avg. views/day

9. Corpus Construction

10. Corpus: Spam/Ham • SPAM edits are those that: • Added exactly one link to an HTML document • The only changes made in the edit are the link addition and its immediate supporting “context” • Were rolled-back by a privileged editor. Where rollback is a tool used only for “blatantly unproductive edits” • HAM edits are those: • Added by a privileged user • Meeting criteria (1) and (2) above By relying on implicit actions, we save time and allow privileged users to define spam on case-by-case basis

11. Corpus: Context (1) Because the link was the ONLY change made. The privileged user’s decision to roll-back that edit speaks DIRECTLY to that link’s inappropriateness.

12. Corpus: Context (2)

13. Corpus: Collection human 2,865 1,095 238,371 188,210 All links collected Links to HTML doc. Was rolled back only link SPAM “context” LBL NUM# PER% Added by priv. user human HAM only link SPAM 1,095 18.4% “context” 4,867 50,108 4,867 81.6% HAM • ≈2 months of data collection in early 2011 (en.wiki) • Done in real-time viva the STiki framework [7] • Also retrieved the landing site for each link • Be careful of biasing features!

14. Features

15. Features • 55+ features implemented and described in [8] • For brevity, focus on features that: (1) Are strong indicators, and (2) have intuitive presentation • Three major feature groups: • (1) Wikipedia-driven • (2) Landing-site driven • (3) Third-party based

16. Features: Wiki (1) • Examples of Wikipedia features: • URL: Length, sub-domain quantity, TLD (.com, .org, etc.) • Article: Length, popularity, age • Presentation: Where in article, citation, description length • URL History: Addition quantity, diversity of adding users • Metadata: Revision comment length, time-of-day

17. Features: Wiki (2) • Long URLs are good URLs: • www.example.com vs. www.example.com/doc.html • Former more likely to be promotional • Spam is rarely used in citations • Advanced syntax implies advanced editors

18. Features: Wiki (3) TLDs with greater admin. control tend to be good. Also correlates well with registration cost ≈ 85% of spam leaves no “revision comment” vs. < 20% of ham

19. Features: Wiki (4) An article’s non-citation links “stabilize” with time(Non-cites tend to have their own section at article bottom)

20. Features: Site • We fetch and process the HTML source of the landing site • Spam destinations marginally more profane/commercial (SEO?) • Re-implement features from a study of email spam URLs [9] • Opposite results from that work • TAKEAWAY: Subtlety and link diversity impair this analysis

21. Features: 3rd Party (1) Two third-party sources queried: • Alexa WIS [10]: Data from web-crawling and toolbar. Including traffic data, WHOIS, load times… • Google Safe Browsing Project: Lists suspected phishing /malware hosts • Google lists produce NO corpus matches: • So worthless during learning/training • But a match is unambiguously bad… • …. So bring into force via statically authored rules

22. Features: 3rd Party (2) • #1 weighted feature • At median, ham has 850 BLs, spam has 20 BLs (40× difference). • Intuitive: basis for search-engine rank • Continent of WHOIS registration. • Asia especially poor • Other good, non-intuitive Alexa feats.

23. Performance

24. Perform: ADTrees val =0.0 Example ADTree: • Features turned into ML model via ADTree algorithm • Human-readable • Enumerated features • In practice… • Tree-depth of 15 • About 35 features • Evaluation performed using standard 90/10 cross validation. BACKLINKS > 200 Y: +0.8 N: -0.4 IS_CITE == TRUE COMM_LEN> 0 N: -0.1 Y: +0.2 Y: +0.6 N: -0.6 … … … … if (final_value > 0): HAM if (final_value < 0): SPAM

25. Perform: Piecewise • Obviously, much better than random predictions (status quo) • Wikipedia-driven features weighed most helpful • But also must consider robustness issues

26. Perform: All

27. Live Implementation

28. Live: Architecture <HTML>… </HTML> STiki Client Wikipedia Spam-scoring engine STiki Services Scoring/ADTree IRC #enwiki# STiki Client Edit Queue STiki Client Wiki-DB Likely vandalism Alexa Likely vandalism Fetch Edit Likely vandalism ------------------- Likely innocent Safe-Browse Display Queue Mainte--nance Wikipedia Article Landing site Classify 3rd-party If spam, revert and warn • Bringing live is trivial via IRC and implemented ADTree BotHandler • But how to apply the scores: • Autonomously (i.e., a bot); threshold; approval-pending if(score) < thresh: then: elseREVERT • Prioritized human-review via STiki [7] • Priority-queue used in crowd-sourced fashion

29. Live: Demonstration Software Demonstration open-source [7], [[WP:STiki]]

30. Live: Discussion • Practical implementation notes • Multiple links: Score individually, assign worst • Dead links (i.e., HTTP 404): Reported to special page • Non-HTML destinations: Omit corresponding features • Static rules needed to capture novel attack vectors • Features are in place: Page placement, article popularity, link histories, diversity quotients, Safe Browsing lists. • Pattern matches result in arbitrarily high scores • Offline vs. online performance • Bot performance immediate; should be equal • Difficult to quantify decline with prioritized humans (STiki)

31. Live: Gamesmanship • How would an attacker circumvent our system? • Content-optimization (need for robust features) • TOCTTOU attacks (i.e., redirect after inspection) • Rechecking on an interval is very expensive • But a practical concern; LTA case [[WP:UNID]] • Crawler redirection • Determine our system’s IP; serve better content • A more distributed operational base • Denial-of-service (overloading system with links) + STiki • Solutions to these kinds of problems remain future work

32. References [1] A. G. West, J. Chang, K. Venkatasubramanian, O. Sokolsky, and I. Lee. Link spamming Wikipedia for profit. In CEAS ‘11, September 2011. [2] Wikipedia: External links. http://en.wikipedia.org/wiki/Wikipedia:External_links. [3] P. Heymann, G. Koutrika, and H. Garcia-Molina. Fighting spam on social web sites: A survey of approaches and future challenges. IEEE Internet Comp., 11(6), 2007. [4] R. Priedhorsky, J. Chen, S. K. Lam, K. Panciera, L. Terveen, and J. Riedl. Creating, destroying, and restoring value in Wikipedia. In GROUP ’07, 2007. [5] E. Goldman. Wikipedia’s labor squeeze and its consequences. Journal of Telecommunications and High Technology Law, 8, 2009. [6] Y. Shin, M. Gupta, and S. Myers. The nuts and bolts of a forum spam automator. In LEET’11: 4th Wkshp. on Large-Scale Exploits and Emergent Threats, 2011. [7] A.G. West. STiki: A vandalism detection tool for Wikipedia. http://en.wikipedia.org/wiki/Wikipedia:STiki. Software, 2010. [8] A.G. West, A. Agarwal, P. Baker, B. Exline, and I. Lee. Autonomous Link Spam Detection in Purely Collaborative Environments. In WikiSym '11. [9] A. Ntoulas, M. Najork, M. Manasse, and D. Fetterly. Detecting spam web pages through content analysis. In WWW’06: World Wide Web Conference, 2006. [10] Alexa web information service. http://aws.amazon.com/awis/.

33. Backup slides (1)

34. LINKS Prevention Detection Backup slides (2) Blacklists Patrollers Watchlisters Readers Immediate Seconds Mins./Days ∞ Latency: LEFT: Pipeline of typical Wikipedia link spam detection, including both actors and latency RIGHT: log-log plot showing average daily article views versus article popularity rank. Observe the power-law distribution. A spammer could reach large percentages of viewership via few articles.

35. Backup slides (3) ABOVE: Evaluating time-of-day (TOD) and day-of-week (DOW) AROUND: Evaluating feature strength. All features here are “Wikipedia-driven”.