Mohammad Masudur Rahman , Shamima Yeasmin, and Chanchal K. Roy Department of Computer Science

1. CSMR-18/WCRE-21 Software Evolution Week(SEW 2014), Antwerp, Belgium Towards a Context-Aware Meta Search Engine For IDE-Based Recommendation About Programming Errors & Exceptions Mohammad Masudur Rahman, Shamima Yeasmin, and Chanchal K. Roy Department of Computer Science University of Saskatchewan

2. Software Maintenance, Bugs & Exceptions A common experience!! Software Research Lab, U of S

3. Exception Handling: IDE Support 2 Software Research Lab, U of S 1

4. Exception Search Query Class can not access a member of class java.util.HashMap$HashIterator with modifiers "public final” Software Research Lab, U of S

5. Exception Handling: Web Search • Traditional web search • No ties between IDE and web browsers • Does not consider problem-context • Environment-switching is distracting & Time consuming • Often not much productive (trial & error approach) Software Research Lab, U of S

6. IDE-Based Web Search • About 80% effort on Software Maintenance,(Ponzanelli et al, ICSE 2013) • Bug fixation– error and exception handling • Developers spend about 19% of time in web search, (Brandt et al, SIGCHI, 2009) • IDE-Based context-aware web searchis the right choice Software Research Lab, U of S

7. Existing Related Works • Rahman et al. (WCRE 2013) • ERA version of this paper • Outlines basic idea, limited experiments • Cordeiro et al. (RSSE 2012) • Based on StackOverflow data dump • Subject to the availability of the dump, not easily updatable • Uses limited context, only stack trace • Very limited experiments • Ponzanelli et al. (ICSE 2013) • Based on StackOverflow data dump • Uses limited context, only context-code • Not specialized for exception handling Software Research Lab, U of S

8. Existing Related Works • Poshyvanyk et al. (IWICSS 2007) • Integrates Google Desktop in the IDE • Not context-aware • Brandt et al. (SIGCHI 2010) • Integrates Google web search into IDE • Not context-aware • Focused on usability analysis Software Research Lab, U of S

9. Motivation Experiments • 75 Exceptions (details later) • Individual engine can provide solutions for 58 exceptions at most. • Maximizing total solutions Software Research Lab, U of S

10. The Key Idea !! Meta Search Engine Software Research Lab, U of S

11. Proposed IDE-Based Meta Search Model Software Research Lab, U of S Web page Start search Results

12. Proposed IDE-Based Meta Search Model • Distinguished Features (5) • IDE-Based solution • Web search, search result and web browsing all from IDE • No context-switching needed • Meta search engine • Captures data from multiple search engines • Also applies custom ranking techniques • Context-Aware search • Uses stack traces information • Uses context-code (surroundings of exception locations) • Software As A Service (SAAS) • Search is provided as a web service, and can be leveraged by an IDE. http://srlabg53-2.usask.ca/wssurfclipse/ Software Research Lab, U of S

13. Proposed IDE-Based Meta Search Model • Two Working Modes • Proactive Mode • Auto-detects the occurrence of an exception • Initiates search for exception by client itself • Aligned withCordeiro et al. (RSSE’ 2012) &Ponzanelli et al. (ICSE 2013) • Interactive Mode • Developer starts search using context menu • Also facilitates keyword-based search • Aligned with traditional web search within the IDE

14. Search Query Generation • Search Query required to collect results from the Search Engine APIs and to develop the corpus. • Query generation • Uses stack trace and Context code • Collects 5 tokens of top-most degree of interestsfrom stack trace. • Collects 5 most frequently invoked methods in the context-code. • Combined both token list to form the recommended keywords for the context. Software Research Lab, U of S

15. Result Ranking Aspects (4) • Content-Relevance • Considers page title, body content against search query • Context-Relevance • Considers stack traces from webpage against target stack trace • Considers code snippets against context-code extracted from IDE • Link Popularity • Considers the Alexa & Compete site rank • Estimates a normalized score from those ranks • Search Engine Confidence • Heuristic measure of confidence for the result • Considers the frequency of occurrence • Considers the weight of each search engine Software Research Lab, U of S

16. Proposed Metrics & Scores • Content Matching Score (Scms) • Cosine similarity based measurement • Stack trace Matching Score (Sstm) • Structural and lexical similarity measurement of stack traces • Code context Matching Score (Sccx) • Code snippet similarity (code clones) • StackOverflow Vote Score (Sso) • Total votes for all posts in the SO result link Software Research Lab, U of S

17. Proposed Metrics & Scores • Site Traffic Rank Score (Sstr)-- Alexa and Compete Rank of each link • Search Engine weight (Ssew)---Relative reliability or importance of each search engine. Experiments with 75 programming queries against the search engines. • Heuristic weights of the metrics are determined through controlled experiments. Software Research Lab, U of S

18. Experiment Overview • 75 Exceptions collected from Eclipse IDE workspaces of grad-students of SR Lab, U of S, and different online sources (StackOverflow, pastebin) • Related to Eclipse plug-in framework and Java Application Development • Solutions chosen from exhaustive web search with cross validations by peers • Recommended results manually validated. • Results compared against existing approaches and search engines. Software Research Lab, U of S

19. Performance Metrics • Mean Precision (MP) • Recall (R) • Mean First False Positive Position (MFFP) • Mean Reciprocal Rank (MRR) Software Research Lab, U of S

20. Results for Score Components Software Research Lab, U of S [ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]

21. Results of Existing Approaches Software Research Lab, U of S [ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]

22. Results of Search Engines Software Research Lab, U of S

23. Threats to Validity • Search not real time yet, generally takes about 20-25 seconds per search. Multithreading used, extensive parallel processing needed. • Search engines constantly evolving, same results may not be produced at later time. • Experimented with common exceptions, which are widely discussed and available in the web. Software Research Lab, U of S

24. Latest Updates • More extensive experimentswith 150 exceptions. Achieved 92% accuracy. • Eclipse plugin release (https://marketplace.eclipse.org/content/surfclipse) • Context-aware Keyword search with automatic query completion feature. • Visual Studio 2012 Plugin under development. • Extensive User Study ongoing. Software Research Lab, U of S

25. SurfClipse Tool Demonstration • Tool Demo video: https://www.youtube.com/watch?v=hGbyF4YveaI Software Research Lab, U of S

26. Thank You !!! Software Research Lab, U of S

27. References [1] M.M. Rahman, S.Y. Mukta, and C.K. Roy. An IDE-Based Context-Aware Meta Search Engine. In Proc. WCRE, pages 467–471, 2013. [2] J. Cordeiro, B. Antunes, and P. Gomes. Context-based Recommendation to Support Problem Solving in Software Development. In Proc. RSSE, pages 85 –89, June 2012. [3] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: StackOverflow in the IDE. In Proc. ICSE, pages 1295–1298, 2013. [4] D. Poshyvanyk, M. Petrenko, and A. Marcus. Integrating COTS Search Engines into Eclipse: Google Desktop Case Study. In Proc. IWICSS, pages 6–, 2007. [5] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proc. SIGCHI, pages 1589–1598, 2009. Software Research Lab, U of S

28. Sample Stack Trace java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at java.net.Socket.<init>(Unknown Source) at java.net.Socket.<init>(Unknown Source) at test.SockTest.main(SockTest.java:13) Software Research Lab, U of S

29. Sample Context Code try { Socket client = new Socket("localhost", 4321); ObjectOutputStream out = newObjectOutputStream( client.getOutputStream()); out.flush(); ObjectInputStream in = newObjectInputStream( client.getInputStream()); System.out.println("Buffer size: " + client.getSendBufferSize()); for (int i = 0; i < 10; i++) { if (i == 3) { Thread.currentThread().interrupt(); System.out.println("Interrupted."); } out.writeObject("From Client: Hellow." + i); out.flush(); System.out.println(in.readObject()); } } catch (Exception e) { e.printStackTrace(); } Software Research Lab, U of S

30. Search Query for Corpus Development java.net.ConnectException Connection refused connect currentThread Software Research Lab, U of S

31. Items used For relevance Checking java.net.ConnectException Connection refused connect currentThread + Sample Stack Trace + Sample Context Code Software Research Lab, U of S

32. Sample Stack Trace (2) java.lang.ClassNotFoundException: org.sqlite.JDBC at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Unknown Source) at core.ANotherTest.main(ANotherTest.java:18) Software Research Lab, U of S

33. Context Code (2) try { //code for making connection with a sqlite database Class.forName("org.sqlite.JDBC"); Connection connection=null; connection=DriverManager.getConnection("jdbc:sqlite:"+"/"+"test.db"); Statement statement=connection.createStatement(); String create_query="create table History ( LinkID INTEGER primary key, Title TEXT not null, LinkURL TEXT not null);"; booleancreated=statement.execute(create_query); System.out.println("Succeeded"); }catch(Exception exc){ exc.printStackTrace(); } Software Research Lab, U of S

34. Search Query for Corpus Development java.lang.ClassNotFoundExceptionorg.sqlite.JDBCdbClassLoader execute Software Research Lab, U of S

35. Items used For relevance Checking java.lang.ClassNotFoundExceptionorg.sqlite.JDBCdbClassLoader execute + Sample Stack Trace + Sample Context Code Software Research Lab, U of S