1 / 82

NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning

Supported by Novartis and GMO. NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning. Karl Lieberherr joint work with Ahmed Abdelmeged and Bryan Chadwick. Introduction. Scientific Community Game(X) [SCG(X)]

monty
Télécharger la présentation

NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Supported by Novartis and GMO NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning Karl Lieberherr joint work with Ahmed Abdelmeged and Bryan Chadwick SCG Innovation

  2. Introduction • Scientific Community Game(X) [SCG(X)] • Goal: Foster innovation and learning in some domain X • A virtual scientific community consists of virtual scholars that propose and oppose hypotheses maximizing their reputations • Applications: Learning and innovation through focused interaction, “Netflix in the small” Innovation

  3. How to model a scholar? • Solve problems • Provide hard problems • Propose hypotheses about Solve and Provide (Introspection) • Oppose hypotheses • Strengthen hypotheses • Refute hypotheses • Supported opposing failed • Refuted opposing succeeded Innovation

  4. Where SCG comes from • J ACM Lieberherr/Specker 1981 SCG Innovation

  5. Outline • Introduction (done) • Highest safe rung example • SCG Scholar / Agent • SCG Agent in Action • Highest safe rung example (opposition) • Who is the winner? • Competition and collaboration • Disadvantages of SCG • Further Examples • SCG-based Software Development Process • Conclusions SCG Innovation

  6. Example: Jar Stress Testing • You have a ladder with r rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break.  We call this the highest safe rung problem (r,b). • How many experiments do you need? Minimize. • (r,infinity) • (r,1) SCG

  7. Highest Safe Rung ProblemProblems and Solutions • Problems: p=((r,b),secrethsr), secret hsr in [0,r], r,b natural numbers • r = number of rungs • b = number of jars that are allowed to break • (r,b) is called a niche • Solutions: sequence of queries of the form n? to find hsr. Responses: yes/no. • Quality of solution: q = length of sequence of queries SCG Innovation

  8. Highest Safe Rung ProblemHypotheses (from Kleinberg/Tardos) • Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality q: abbreviated H = ((r,b),q) • Problems to be delivered for H = ((r,b),q) are of the form ((r,b), s). Important: A hypothesis defines a family of problems. • Propose: Hypotheses H1 = ((25,2),11), H2 = ((25,2),6) SCG Innovation

  9. Life of a scholar: (propose+ oppose+ provide* solve*)* Scholars propose and oppose proposed hypotheses egoistic Alice egoistic Bob social welfare HB1 opposes (1) HA1 HB2 provides problem (2) HA2 solves problem LOSES WINS! not as well as she expected based on HA2 (3) HA3 Bob increases his reputation HA4 Innovation

  10. What is the purpose of SCG? • The purpose of playing an SCG(X) competition is to assess the "skills" of the agents in: • solving problems in domain X, • making good predictions about niches in domain X, • finding the hardest problems in a specific niche Innovation

  11. What is SCG(X) agent Alice agent Bob degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation agent plays transfer to reliable, efficient software more applications: test constructive knowledge Innovation

  12. What is SCG(X)? Team Alice Team Bob Teams Design Problem Solver Develop Software Deliver Agent I am the best No!! Agent Alice Agent Bob Let’s play constructively Administrator SCG police Innovation

  13. For agents: Full Round Robin Tournaments or Swiss-Style • Agents to play the SCG(X). Repeat a few times with feedback used to update agents. • Within the group of participating agent, the winning agent has the • best solver for X-problems • best supported knowledge about X Innovation

  14. SCG in Action: Competitions • http://www.ccs.neu.edu/home/lieber/courses/cs4500/f09/files/competitions/past_competitions/11_23/tournament_1/final_results_tournament_2009_11_24_12_03_41.html • http://www.ccs.neu.edu/home/lieber/courses/cs4500/f09/files/competitions/past_competitions/10_22/tournament_1/final_results_tournament_2009_10_23_04_35_18.html Innovation

  15. Highest Safe Rung Problemopposing • opposing(refuting, strengthening) • Alice claims: Hypothesis ((25,2),5) • Bob opposes it by refuting it: Bob invents problem ((25,2), secret 22). Alice: 5? no, 10? yes, 6? no, 7? no, 8? no. Already 5 questions asked and answer still unknown. Alice’ claim is refuted. • Alice claims: Hypothesis ((25,2),12) • Bob opposes it by strengthening it to ((25,2),9); and he can successfully support this hypothesis SCG Innovation

  16. Highest Safe Rung Problemsupporting • Alice claims: Hypothesis ((25,2),12) • Bob tries to discount but Alice supports it: Alice: 5? no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes. Only 9 questions asked and problem ((25,2), secret 23) is solved. Alice has supported her hypothesis. SCG Innovation

  17. Motivated by real scientific community Who wins? Alice or Bob? • Reputation of Alice = • the number of hypotheses that Alice proposed that were never successfully opposed by Bob (neither refuted nor strengthened) + • the number of hypotheses that Bob proposed that were successfully opposed by Alice • RA = HAnotOpposedB + HBOpposedA • The scholar with the highest reputation wins • encourages: creating strong knowledge and discounting knowledge created by others SCG Innovation

  18. competitive / collaborative Agent Alice: claims hypothesis H Alice wins knowledge Agent Bob: opposes H, refutes: provides evidence for !H Bob wins reputation makes public knowledge Innovation

  19. Highest Safe Rung Problemcompetition / collaboration • Alice claims: Hypothesis ((25,2),12) • Bob tries to discount but Alice supports it: Alice: 5? no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes. • From this exchange which is prompted by Alice defending her reputation, Bob gets an idea: For problem: p=((r,b),secret hsr), consider f(r,q) =(r/q + q) and find a q so that f(25,q) is minimized. f(25,5)=10; f(25,6)=11;f(25,4)=11. • From this idea Bob knows that he can strengthen the hypothesis to ((25,2),10) • General solution: Given r, find q to minimize (r/q + q). SCG Innovation

  20. Scholars and Agents:Same rules • Are encouraged to • offer results that are not easily improved. • offer results that they can successfully support. • strengthen results, if possible. • expose results that are wrong. • stay active and publish new results. • be well-rounded: solve posed problems and pose difficult problems for others. • become famous! Innovation

  21. Soundness Theorem • SCG is sound: The agent with the best algorithms / knowledge wins (there is no way to cheat) • best: within the group of participating agents Innovation

  22. Highest Safe Rung ProblemAsymptotic Hypotheses • Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality f(r,b) : abbreviated H = ((r,b),f(r,b)) • Problems to be delivered for H = ((r,b),f(r,b)) are of the form ((r,b), secret hsr). • Propose: Hypotheses H1 = ((r,b),(log(r))b), H2 = ((r,b),r1/b) SCG Innovation

  23. Highest Safe Rung Problemdiscounting asymptotic hypothesis • discounting (refuting, strengthening) • Alice claims: Hypothesis ((r,b),(b*log(r))) • Bob discounts it by refuting it: Bob invents problem ((1024,2), secret hsr). log(1024) = 10. 20 questions are not enough! Alice: 30? no, 60? yes, 31? no, 32? no, etc.. Already 20 questions asked and answer still unknown. Alice’ claim is refuted. • Alice claims: Hypothesis ((r,2),r/2) • Bob discounts it by strengthening it to ((r,2),2*r½ ); and he can successfully support this hypothesis. SCG Innovation

  24. Disadvantages of SCG • The game is addictive. After Bob having spent 4 hours to fix his agent and still losing against Alice, Bob really wants to know why! • Overhead to learn to define and participate in competitions. • The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems. • if admin does not, cheap play • watching over the admin Innovation

  25. How to compensatefor those disadvantages • Warn the scholars. • Use a gentleman’s security policy: report administrator problems, don’t exploit them to win. • Occasionally have a non-counting “attack the administrator” competition to find vulnerabilities in administrator. • both generic as well as X-specific vulnerabilities. Innovation

  26. GIGO: Garbage in / Garbage out • If all agents are weak, no useful solver created. Innovation

  27. Physics Maximum Height ProblemProblems and Solutions • Problems: p=(v, a), v, a: positive real numbers • The maximum height obtained by a projectile launched with speed v at angle a to the horizontal is z. • Solutions: real number z. • Quality of solution: Number of correct decimal places. SCG Innovation

  28. Physics Maximum Height Problem Hypotheses • Alice claims the hypothesis: I can solve any maximum height problem p=(v,a) with quality q in 1 minute: abbreviated H = (MHP,q) • Problems to be delivered for H = (MHP,q) are of the form (v,a). • Propose: Hypotheses H1 = (MHP,3), H2 = (MHP,6) http://scienceworld.wolfram.com/physics/Height.html SCG Innovation

  29. Physics Maximum Height Problem discounting • discounting (refuting, strengthening) • Alice claims: Hypothesis (MHP,3) • Bob discounts it by refuting it: Bob invents problem (25,60 degrees). Alice fails to solve the problem in 1 minute with 3 correct digits. Alice’ claim is refuted. Checking is done by experiment or trusted third party. • Alice claims: Hypothesis (MHP,1) • Bob discounts it by strengthening it to (MHP,2); and he can successfully support this hypothesis SCG Innovation

  30. RegExpToAutomata ProblemProblems and Solutions • Problems: p=(r,n); r a regular expression of size n. • r = regular expression; a + b* a + a a a b* • n defines a niche of regular expressions • Solutions: DFA d equivalent to r. • Quality of solution: Number of states of d. SCG Innovation

  31. RegExpToAutomata ProblemProblems and Solutions • Problems: p=(r,n); r a regular expression of size n. • r = regular expression; a + b* a + a a a b* • n defines a niche of regular expressions • Solutions: DFA d equivalent to r. • Quality of solution: Number of states of d. SCG Innovation

  32. RegExpToAutomata Problem Hypotheses • Alice claims the hypothesis: I can solve any problem p=(r,n) with quality q or less: abbreviated H = (n,q) • Problems to be delivered for H = (n,q) are of the form p=(r,n). Important: A hypothesis defines a family of problems. • Propose: Hypotheses H1 = (5,11), H2 = (5,10) SCG Innovation

  33. RegExpToAutomata Problem opposing • opposing(refuting, strengthening) • Alice claims: Hypothesis (5,11) • Bob discounts it by refuting it: Bob invents a regular expression r of size 5, gives it to Alice and she fails to deliver a DFA d with 11 or fewer states. Alice’ claim is refuted. • Alice claims: Hypothesis (5,20) • Bob discounts it by strengthening it to (5,19); and he can successfully support this hypothesis SCG Innovation

  34. RegExpToAutomata Problem supporting • Alice claims: Hypothesis (4,12) • Bob tries to discount but Alice supports it: Bob gives to Alice a regular expression r of size 4. Alice provides and equivalent DFA with 12 or fewer states. Alice has supported her hypothesis. SCG Innovation

  35. Who wins? Alice or Bob? • Reputation of Alice = • the number of hypotheses that Alice proposed that were never successfully opposed by Bob (neither refuted nor strengthened) + • the number of hypotheses that Bob proposed that were successfully opposed by Alice. • RA = HAnotOpposedB + HBOpposedA • The scholar with the highest reputation wins. • encourages: creating minimum automata for regular expressions of a given size. SCG Innovation

  36. Software Development Process • Increase targeted interaction between software developers. Innovation

  37. Traditional Approach Requirements for X Human Developers human1 human3 human2 human4 Static Evaluation. No competition. Testing unit testing integration testing Develop new software for problem solving domain X Benchmark is used to evaluate software Users SCG-SP2010

  38. Why Software Development through a virtual scientific community? Requirements for X Evaluates fairly, frequently, constructively and dynamically. Drives innovation. Challenges humans. Agents point humans to what needs attention in the software. Human Developers human1 human2 Erika Patrick SCG(X) Erika-Patrick-agent winning-agent Develop new software for problem solving domain X Benchmark is used to evaluate software Users SCG-SP2010

  39. Erika-Patrick Agent • Surrogate of combined knowledge of Erika and Patrick successfully transferred to agent. • Transfer knowledge by programming. SCG-SP2010

  40. Conclusions • How to make learning and problem solving fun: design a game and interact. • Scientific Community Game = Specker Challenge Game = SCG • How to create reliable problem solving software? Have it tested through SCG. Innovation

  41. Final Slide • More Questions? Innovation

  42. SCG Innovation

  43. SCG concepts • Scholars working in a domain with niches. Define functions on niches. • Hypotheses: claims about functions on niches: • Discounting protocol for HA: Alice selects niche element ne and Bob applies fBob so that claim about function does not hold • Strengthening protocol • Reputation SCG Innovation

  44. SCG concepts • Scholars working in a domain with niches. Function f: Niche -> S for Alice and Bob. • Hypotheses: claims about niches: belief: f has property b(s, dn, fdn). (Niche,Belief) • Discounting protocol: Alice selects niche element ne and Bob applies fBob creating s, so that !b(s,ne) • Strengthening protocol • Reputation SCG Innovation

  45. Hypothesis Structure • Algorithm Solver: Problems -> Solutions • For all p in Problems with feature f in Features algorithm Solver solves p using resources p(f) with quality(p,Solver(p),f). • Algorithm Provider: Features -> Problems • For feature f, Algorithm Provider provides a problem p, for all solutions of p, !quality(p,Solver(p),f). SCG Innovation

  46. Two person SCG • Alice, Bob • Domain: Source, Target; fA, fB-> Source-> Target; Source defined by niche predicate. • Hypotheses HA (HB): claims about fA (fB) • Discounting protocol for HA: • Bob provides element ne in Source so that fA(ne) contradicts HA. • Alice provides element ne in Source so that fB(ne) contradicts HA. • Strengthening protocol • Bob proposes HB, HA => HB and Alice cannot discount HB. SCG Innovation

  47. Two person SCGSpecialize for problem solving • Alice, Bob • Domain: Problems -> Solutions • Hypotheses • Discounting protocol for hypothesis HA by Alice: • Bob attacks in one of two ways (depends on HA) • Bob provides a problem for which Alice constructs a solution that contradicts HA. • Alice provides a problem for which Bob constructs a solution that contradicts HA. • Strengthening protocol • Bob proposes HB, HA => HB and Alice cannot discount HB. SCG Innovation

  48. Hypotheses • Solution algorithm A: Problems->Solutions • For all elements p in Problems that have feature F and a secret solution ss(p), algorithm A(p) constructs with resource constraint Prediction(F) an element s(p) in set Solutions(p) with property Q(p,s(p),ss(p),F). SCG Innovation

  49. Hypotheses • Problem creation algorithm A-1 SCG Innovation

  50. Discounting protocol SCG Innovation

More Related