Contributions of SCG to SDG

1. Contributions of SCG to SDG Supported by Novartis SCG = Scientific Community Game = Specker Challenge Game • Karl Lieberherr • Northeastern University • College of Computer and Information Science • Boston, MA • joint work with • Ahmed Abdelmeged and Bryan Chadwick

2. Paper available with more details. Contents • Gamification of Software Development • Scientific Community Game(X) • How it works • Advantages • Disadvantages • Potential benefits for IBM • Collaboration we seek • Conclusions Games for SD

3. Gamification of Software Development etc. at Northeastern • Want reliable software to solve a computational problem? Design a game where the winning team will create the software you want. • (Want to teach a STEM domain? Design a game where the winning students demonstrate superior domain knowledge.) Doesn’t TopCoder already do this? Games for SD

4. SCG = Scientific Community Game • Make software development more scientific. • Software developers build reputation • propose and defend claims about their software • oppose claims made by others • refute claims • strengthen claims • claim defined by refutation protocol Games for SD

5. Claims and Refutation Protocol • Alice claims: I have a program that solves inputs in domain X with quality Q and resources R. • AliceClaim(X,Q,R) • Bob is critical. He prepares an input in X and gives it to Alice who applies her program. Bob refutes AliceClaim(X,Q,R) iff Alice achieves < Q or uses > R. • Refutation protocol Games for SD

6. Why a web application with avatars? Fair Evaluation. Who are Alice and Bob? • They are avatars developed by real Alice and real Bob. • Alice and Bob compete with 10 other avatars in a full-round robin tournament. • Who is the winner: The avatar with the highest reputation, i.e., the avatar who has the strongest, not successfully opposed claims (like in a real scientific community). Games for SD

7. Clear Feedback Sense of Progress What we want Authenticity (Facebook) • Engage software developers • let them produce software that models an organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them. • let them propose claims about the software they produce. Reward them when they • defend their claims successfully or • oppose the claims of others successfully. Possibility of Success Games for SD

8. Software Engineering Properties fostered by SCG • Reliable (otherwise the avatar is removed from the game) • Flexible, modular (otherwise the avatar cannot be easily updated between competitions) • Efficient (otherwise you cannot defend your claims and oppose the claims of others) Games for SD

9. How to define X • Claim definition • Language to define claims • Refutation protocol • information exchanged • language to define inputs, outputs • functions to check inputs, outputs • refute function Games for SD

10. Benefits for IBM of using SCG(X) • Teams perform know-how retrieval and integration and maybe some research. • Participating teams try to find the best knowledge in the area. • Claims language gives control! • The non-opposed claims give hints about new X-specific knowledge. • A well-tested solver for X-problems that integrates the current algorithmic knowledge in field X. Games for SD

11. Benefits for IBM of using SCG(X) • Also great for evaluating potential employees. Games for SD

12. Life of an avatar: (propose+ oppose+ provide* solve*)* Avatars propose and oppose proposed claims egoistic Alice egoistic Bob social welfare CA1 CB1 opposes (1) CA2 CB2 CA3 provides problem (2) CA4 solves problem LOSES WINS! not as well as she expected based on CA2 (3) reputation 10 reputation 1000 transfer 200 Games for SD

13. What is SCG(X)? Team Alice Team Bob Teams Design Problem Solver Develop Software Deliver Avatar I am the best No!! Agent Alice Agent Bob Let’s play constructively Administrator SCG police Games for SD

14. competitive / collaborative Avatar Alice: claim H loses reputation r wins knowledge k Avatar Bob: opposes H, refutes: provides evidence for !H wins reputation r makes public knowledge k Games for SD

15. Disadvantages of SCG • The game is addictive. After Bob having spent 4 hours to fix his avatar and still losing against Alice, Bob really wants to know why! • Overhead to learn to define and participate in competitions. • The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems. • if admin does not, cheap play is possible • watching over the admin Games for SD

16. How to compensatefor those disadvantages • Warn the scholars. • Use a gentleman’s security policy: report administrator problems, don’t exploit them to win. • Occasionally have a non-counting “attack the administrator” competitions to find vulnerabilities in administrator. • both generic as well as X-specific vulnerabilities. Games for SD

17. Opening the development approach • Problem to be solved: Develop the best practical algorithms for solving computational problems in domain X. • Issue: There are probably hundreds of papers on the topic with isolated implementations. What are the best practical algorithms? • Our solution: Use the scientific community game SCG(X) with a suitably designed claims language to compare the software. The winning avatar has the best practical algorithms/software. Games for SD

18. Collaboration with IBM • Offer SCG(X) as a web application for software development and education in STEM areas. • We have some of this generic software in place but a lot more work is needed. • Use SCG(X) as internal tool at IBM. Games for SD

19. Related Work • TopCoder • Renaissance mathematicians • Various benchmark based competitions • What is new? • Software that has an ego • Scientific Community Game Games for SD

20. Example: Independent Set • An independent set in a graph is a set of mutually nonadjacent vertices. The problem of finding a maximum independent set in a graph, is one of most fundamental combinatorial NP-hard problems. Games for SD

21. Example: Independent Set • Alice claims IndSet(m, 0.9, t(m)): • Alice can construct graphs G with at most m edges and she can construct a secret independent set I1 for G so that Bob, given G, size(I1) and t(n) minutes only, cannot find an independent set I2 with • size(I2) >= size(I1)*0.9. Games for SD

22. Example: Refutation Protocol • Alice constructs graph G and deposits her secret independent set I1. • Alice gives G as well as the size of I1 to Bob. • Bob has 10 minutes to construct his independent set I2 which he gives to Alice. • Alice reveals her secret set I1. • Bob refutes iff size(I2) >= size(I1)*0.9 Games for SD

23. Conclusions • To address a problem domain X: • “map it to second life”: define a scientific community game for X on the web: SCG(X) • let the game SCG(X) run a few times and choose the winner • Benefits • Evaluates fairly, frequently, constructively and dynamically. Encourages retrieval of state-of-the-art know-how, integration and discovery. • Challenges humans, drives innovation, both competitive and collaborative. • Agents point humans to what needs attention in problem solution / software. Games for SD

24. Conclusions • SCG(X) provides a structured process for developing software for computational problems. • Benefits • Social Engineering: makes it fun through game. • Fair: Only hard work makes you win. • Engage a large community on one domain X. • Tools Games for SD

25. Conclusions • SCG has many applications of potential value to IBM • Training employees in constructive domains • Software development process • Hiring • Driving innovation in constructive domains Games for SD

26. Thank you Games for SD

27. Axioms for reputation • Confidence factors are real numbers in [0,1] that indicate how confident the proposer is in the claim. • The confidence factor should reflect the amount of effort that has been put into the claim. • For example, one could use confidence 1 for a claim for which the scholar has a proof. • Confidence 1 means that we believe that the claim cannot be opposed. • The scholars are encouraged to give their "true" confidence in a claim. If not, they will be penalized. Games for SD

28. Axioms • Scholars gain reputation either by opposing other scholars' claims or by having their claims recognized by other scholars (when the other scholars fail to oppose). • Scholar's gain from their claim is proportional to both the credibility of their claim and the recognition of the claim after the refuting protocol. • Note: the payoff for strengthening comes after the strengthened claim is recognized (successfully defended or never opposed). Games for SD

29. Axioms • One scholar's reputation gain is another scholar's reputation loss. The sum of all scholars's reputation is preserved. Games for SD

30. Axioms • The refutation protocol recognizes claims by a recognition factor in [-1,1]. • A recognition factor of 1 means that the other scholar has completely failed to refute that claim. • A recognition factor of -1 means that the other scholar has completely succeeded to refute that claim. Games for SD

31. Axioms • Claims have a credibility in [0..]. • The Credibility of a claim made by a \scholar is proportional to either the scholar's ``confidence'' in the claim or the scholar's reputation or both. • The claim's confidence reflects the amount of effort made by the scholar behind the claim to prove the claim (i.e., turning it into a theorem or finding other supporting evidence). • Scholar's reputation is the sum of the scholar's initial reputation plus the reputation gains and losses; thus reflecting the past performance of the scholar. Games for SD

32. Software Development Governance • Software Development Governance (SDG) is defined as: • Establishing chains of responsibility, authority and communication to empower people within a software development organization • Establishing measurement and control mechanisms to enable software developers, project managers and others within a software development organization to carry out their roles and responsibilities Games for SD

33. SCG = Scientific Community Game = Specker Challenge Game Applications • Develop algorithms/software for new computational domain X • Scientific Community Game Software Development: Describe a problem domain X so that SCG(X) provides the best algorithms and their implementations for problems in X. (best within the participating scientific community) Games for SD

34. Games for SD

35. Plan • Why is it relevant, useful? • Larger context: Open Innovation, Wikinomics • Applications: Netflix in the small, teaching • What is it? • What is new? • Map problem domain to “second life”, find best solution there and map it back to real life. • What do we improve: benefits of SCG • How to use SCG • Disadvantages • Experience with current implementation • Related work • Detailed example • Conclusions Games for SD

36. Introduction (2) • Scientific Community Game(X) [SCG(X)] • Goal: Foster innovation and reliable software for solving optimization problems in some domain X • A virtual scientific community consists of virtual scholars that propose and oppose claims maximizing their reputations Games for SD

37. Claim • Subdomain N • subset of problems • Confidence [0,1] • Valuation [0,1] 1 valuation (how well problems in N can be solved) confidence 0 Games for SD

38. Claim 1 over strengthening correct valuation valuation strengthening 0 Games for SD 39

39. Hypothesis • hypothesis by Alice: for all problems F in niche N there exists a solution J: p(F,J) • Bob opposes: F’ to Alice, Alice cannot find J’:p(F’,J’) therefore she loses reputation. Games for SD

40. Full Round Robin Tournaments or Swiss-Style • Agents to play the SCG(X). Repeat a few times with feedback used to update agents. • Within the group of participating agent, the winning agent has the • best solver for X-problems • best supported knowledge about X Games for SD

41. What is the purpose of SCG? • The purpose of playing an SCG(X) competition is to assess the "skills" of the agents in: • "approximating" optimization problems in domain X, • "figuring-out" the wall-clock-time-bounded approximability of niches in domain X, • "figuring-out" hardest problems in a specific niche, and • "being-aware" of the niches in which their own solution algorithm works best. • This multi-faceted evaluation makes SCG(X) more superior to competitions based on benchmarks that only test the player's skills in approximating optimization problems. During SCG, players cross-test each others' skills. Games for SD

42. How to use SCG • Company A provides a problem domain description X and submits it to the SCG server. The game SCG(X) runs on the web (with human algorithm/software developers involved) and company A receives good, tested software and knowledge about problem domain X Games for SD

43. Plan • Why is it relevant, useful? • Larger context: Open Innovation, Wikinomics • Applications: Netflix in the small, teaching • What is it? • What is new? • Map problem domain to “second life”, find best solution there and map it back to real life. • What do we improve: benefits of SCG • How to use SCG • Disadvantages • Experience with current implementation • Related work • Detailed example • Conclusions Games for SD

44. From Benchmark-Driven to SCG-Driven Algorithm Development • Hard to measure and detect what is fraud. • Instead: Design a system that needs a much weaker “gentleman’s agreement” or none at all • The Static Benchmark Problem is ONE problem that SCG solves. Dynamic Benchmarks • Others: crowd sourcing management, new software development process that engages software developers and that fosters ease of evolution (e.g., good separation of concerns, …) Games for SD

45. Problems with Static Benchmarks http://www.cs.kuleuven.be/~dtai/events/ASP-competition/index.shtml Policy against special purpose solutions The purpose of the competition is to be as informative as possible about strengths and weaknesses of … Submission of special purpose programs for solving certain benchmark problems falsifies the information that we get from the rankings and goes against the spirit of the competition. … the use of special purpose programs for certain benchmarks can rightfully be considered as scientific fraud. We appeal to participants … Games for SD

46. SCG-Driven Algorithm Development • Differences to Benchmark-Driven • You don’t rank chess players by giving them a benchmark; you let them play • We turn the algorithms into egoistic virtual scientists that fend for themselves • social welfare: constructive knowledge based on good algorithms Games for SD

47. our focus What is SCG(X) agent Bob Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation agent plays transfer to reliable, efficient software more applications: test constructive knowledge Games for SD

48. Scholars and Agents:Same rules • Are encouraged to • offer results that are not easily improved. • offer results that they can successfully support. • strengthen results, if possible. • stay active and publish new results or oppose current results. • become famous! Games for SD

49. More Applications • Special issue editors for problem domain X. publish top 15 submissions • Professor teaching a software development class: students develop fighting agents for full-round robin tournament • Teaching constructive topics • etc. Games for SD

50. Soundness Theorem • SCG is sound: The agent with the best algorithms / knowledge wins (there is no way to cheat) • best: within the group of participating agents • issues: • Does an agent win because she is good at solving? Or good at proposing, opposing and providing? Answer: proposing, opposing and providing all reduce to solving. Games for SD