A Framework for Writing Trigger-Action Todo Comments in Executable Format

1. A Framework for Writing Trigger-Action Todo Comments in Executable Format Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J. Mooney, and Milos Gligoric August 29, 2019 ESEC/FSE 2019 Tallinn, Estonia supported by

2. Motivation (1) From Google Guava protected AbstractStreamingHasher(intchunkSize, intbufferSize) {   // TODO(kevinb): check more preconditions   // (bufferSize >= chunkSize) if this is ever public checkArgument(bufferSize % chunkSize == 0); ...} Trigger The method is public  Action checkArgument(bufferSize >= chunkSize); • The comment is written in natural language, thus a developer needs to … • locate the comment • check if the trigger evaluates to true • execute the action • Tediousand error-prone

3. Motivation (2) 2012/11: The comment was added 2015/5:   The trigger was satisfied 2015/10: The action was executed 2017/10: The comment was removed From Apache Struts void testDynamicAttributesSupport() throws Exception { ... // TODO : remove expectedJDK15 and if() after switching to Java 1.6 String expectedJDK15 = "<input type=\"text\" ..."; String expectedJDK16 = "<input type=\"text\" ..."; if (result.contains("foo=\"bar\" ...")) assertEquals(expectedJDK15, result); else assertEquals(expectedJDK16, result); ...} Trigger switch to Java 1.6 Action remove a variable and if statement

4. Motivation (3) • Trigger-action comments are frequently used • We have found hundreds of trigger-action comments in 10 open-source projects • Trigger-action comments are 10+% of developer comments • Like any other comments written in natural language, they frequently become outdated or forgotten • We contacted 10 developers from large software companies and asked if they write trigger-action comments; 7 (out of 8 replies) answered yes • They confirmed they have a lot of trigger-action comments, and lack of automation is a big issue

5. TrigIt • A framework for writing trigger-action comments as specifications in executable format • Developers write triggers and actions in the TrigIt DSL • Embedded in the host language (Java) • Utilizes native syntax (if-statements) with different semantics • TrigIt checks the triggers after compilation and before execution • If a trigger satisfies, TrigIt executes the associated actions • TrigIt removes all its specifications before execution

6. Example TrigIt Specification (1) From Google Guava protectedAbstractStreamingHasher(intchunkSize, intbufferSize) {   // TODO(kevinb): check more preconditions   // (bufferSize >= chunkSize) if this is ever public if (trigItIsPublic()) checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod booleantrigItIsPublic() { returnTrigIt.getMethod("<init>", int.class, int.class).isPublic(); }

7. Example TrigIt Specification (2) From Apache Struts void testDynamicAttributesSupport() throws Exception { ... // TODO : remove expectedJDK15 and if() after switching to Java 1.6 if (TrigIt.getJavaVersion().ge(TrigIt.JAVA6)) { String expectedJDK16 = "<input type=\"text\" ..."; assertEquals(expectedJDK16, result); } else { String expectedJDK15 = "<input type=\"text\" ..."; String expectedJDK16 = "<input type=\"text\" ..."; if (result.contains("foo=\"bar\" ...")) assertEquals(expectedJDK15, result); else assertEquals(expectedJDK16, result); } ...}

8. Design Choices • Preserves semantics of comments from the execution point • Each TrigIt specification is local to its context similar to inline comments • TrigIt specifications are removed from the compiled code (after TrigIt's processing), thus they do not cause side-effects during execution • Java compiler checks partial correctness of TrigIt specifications • Integrated as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true

9. TrigIt DSL • Query Expression / Trigger • Action Statement / Global Action • Local Action • TrigIt Method

10. TrigIt DSL (1/4) • Query Expression / Trigger • Examples • AST: TrigIt.getClasses().findAny(“C”).isPresent() • Build: TrigIt.getJavaVersion().ge(TrigIt.JAVA8) • Issue: TrigIt.isClosed(“github…issues/1897”) • Time: TirgIt.after(2019, 4, 2) if ( <query expression> ) { // action }

11. TrigIt DSL (2/4) • Action Statement / Global Action • Examples • TrigIt.getMethod(m1()).remove() • TrigIt.getField(f).setPrivate() @TrigItMethod voidglobalAction() { if (/* trigger */) { <action statement> } }

12. TrigIt DSL (3/4) • Local Action if (/* trigger */) { /* code executed if trigger evaluates to true */ } else { /* code executed if trigger evaluates to false */ }

13. TrigIt DSL (4/4) • TrigIt Method • @TrigItMethod annotation • void return type: specifies trigger + global action(s) • boolean return type: specifies a trigger that can be reused @TrigItMethod voidtrigItDeprecateMakeChecked() { if (TrigIt.after(2018, 4, 30)) TrigIt.getMethodByName("immediateCheckedFuture").remove(); } @TrigtItMethod booleantrigItIsPublic() { returnTrigIt.getField(f).isPublic(); } if (trigItIsPublic()) { /* action 1 */ } if (trigItIsPublic()) { /* action 2 */ }

14. TrigIt Workflow Test/Execution Compiler .class Files .java Files

15. TrigIt Workflow TrigIt Compiler Test/Execution .class Files .java Files

16. TrigIt Workflow TrigIt • Rewrite • removes all other code and only keeps TrigIt specifications Compiler Test/Execution rewrite TrigIt specifications .class Files .java Files

17. TrigIt Workflow TrigIt • Rewrite • removes all other code and only keeps TrigIt specifications • Evaluate • checks all triggers Compiler Test/Execution evaluate rewrite TrigIt specifications .class Files .java Files AST Build Configurations Issue Tracker System Time

18. TrigIt Workflow TrigIt • Rewrite • removes all other code and only keeps TrigIt specifications • Evaluate • checks all triggers • Apply • executes actions by changing the bytecode, sends warning messages, or changing source code • removes all TrigIt specifications Compiler Test/Execution evaluate rewrite TrigIt specifications .class Files .java Files apply AST Build Configurations Issue Tracker System Time Warning Messages Patch to .java Files

19. Evaluation • RQ1: Can developers quickly implement correct triggers and actions using TrigIt? • RQ2: What’s the build overhead of TrigIt? • RQ3: What’s the overhead for writing TrigIt specifications over trigger-action comments, measured by number of tokens?

20. User Study Design • Can developers quickly implement correct triggers and actions using TrigIt? • Tasks • Read a brief tutorial of TrigIt syntax (10min) • Migrate 3 trigger-action comments to TrigIt specifications • The first one is warm-up and not counted in results (without telling the participants) • 2 groups of tasks: • Warm-up, Task A from Google Guava, Task B from Apache Struts • Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins • Details available on our webpage: cozy.ece.utexas.edu/trigit

21. User Study Participants • 2 groups of tasks: • Warm-up, Task A from Google Guava, Task B from Apache Struts • Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins • 20 participants • 8 professionals (6 developers in large software companies, 2 researchers) • 12 students (4 undergrads, 8 PhD students)

22. User Study Results

23. User Study Results Participants obtained correct solutions with very little training

24. Build Overhead • RQ2: What’s the build overheadof TrigIt? • Subjects: We migrated 44 trigger-action comments from 10 projects to TrigIt specifications. • Environment: Intel i7-6700U@3.40GHz, 16GB RAM, Ubuntu 18.04 LTS • Results: average 8.85%, median 6.63%, min 1.91%, max 26.75%

25. Complexity of the TrigIt DSL • RQ3: What’s the overhead for writing TrigIt specifications over trigger-action comments, measured bynumber of tokens? • Results: 58.74% • Trigger-action comment: average 12.2, median 12.0, min 6, max 26 • TrigIt specifications: average 18.0, median 16.5, min 11, max 36 • Trigger: average 7.3, median 6.0, min 4, max 19 • Action: average 7.4, median 5.0, min 2, max 27 • Boilerplate code: average 3.3, median 3.0, min 3, max 7 Boilerplatecode Trigger Action

26. Anecdotal Experience • During our inspection of trigger-action comments in open source projects, we observed some comments are forgotten, i.e., the trigger is satisfied but the action is not executed • We reported 6 cases and the developers responded quickly • Some triggers and actions in comments are not specified enough: “Unfortunately Java 8 still causes some issues with some Google internal infrastructure. Java 8 is allowed in tests but not in the main part of the code (yet).” • Knowing the details of the projects, e.g., internal testing reports, will help writing correct TrigIt specifications

27. Conclusion • A framework for writing trigger-action comments in executable format • Introduces DSL for triggers and actions • Preserves semantics and developers’ workflow • TrigIt is easy to learn and introduces small overhead • TrigIt can run as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true • Webpage:  cozy.ece.utexas.edu/trigit Pengyu Nie <pynie@utexas.edu>

28. Backup slides after this point

29. User Study Participants’ Prior Experience • Programming experience: average 8.4 years • Java skills (on a scale of 1-5): 3.7 • None of the participants has contributed to the open-source projects where we selected the tasks from

30. User Study Error Analyze • Trigger error: 1 & action error: 1 • Misunderstanding the trigger-action comments • Syntax errors: 3 • Putting the local action in a wrong place • We added additional checks in TrigIt to prevent this kind of errors protectedAbstractStreamingHasher(intchunkSize, intbufferSize) {   // TODO(kevinb): check more preconditions   // (bufferSize >= chunkSize) if this is ever public checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod booleantrigItIsPublic() { if (TrigIt.getMethod("<init>", int.class, int.class).isPublic()) checkArgument(bufferSize >= chunkSize); }

31. Example TrigIt Specifications (3) From Elastic Elasticsearch publicvoidtestShadowReplicaNaturalRelocation() throws Exception { … // TODO: uncomment the test below when https://github.com/elastic/elasticsearch/issues/17695 // is resolved. if (TrigIt.isClosed("https://github.com/elastic/elasticsearch/issues/17695")) assertIndicesDirsDeleted(nodes); }

32. Example TrigIt Specifications (4) From Google Guava // TODO(b/72241893): Remove by 2018-04 @Deprecated publicstatic <V, X extends Exception> CheckedFuture<V, X> immediateCheckedFuture( @NullableDeclV value) { returnnewImmediateSuccessfulCheckedFuture<>(value); } @TrigItMethod voidtrigItDeprecateMakeChecked() { if (TrigIt.after(2018, 4, 30)) TrigIt.getCurrentClass().getMethodByName("immediateCheckedFuture").remove(); }

33. Study of Trigger-Action Comments • We inspected 471 trigger-action comments and classify the trigger/action’s specificity • high: migration feasible • medium: migration could potentially be done • low: hard to understand, migration is not feasible • 28% have both specific trigger and specific action • 70% have either specific trigger or specific action

34. TrigIt Implementation • Implemented as a standalone Java library, can be easily integrated into existing build process • Command line interface • Maven plugin • Implemented with Java bytecode manipulation, using ASM • Extra checks for the correctness of triggers and actions • TrigIt.getClass("C").getField("f").isPrivate()TrigIt will check if class C and field f exist • Debug options: --assume-true, --no-action

35. Using Bytecode Analysis & Manipulation • The bytecode format was more stable in recent years than the Java language syntax; this may lower the cost of maintenance. • We wanted to avoid dependencies on IDEs and enable integration of TrigIt with any build process, even if the build is running on a continuous integration service. Future versions of TrigIt may have a tighter connection with IDEs based on the adoption and feedback that we received from the community.

36. Example Patch (Before Trigger Satisfies) From Google Guava protectedAbstractStreamingHasher(intchunkSize, intbufferSize) { if (trigItIsPublic()) checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod booleantrigItIsPublic() { returnTrigIt.getMethod("<init>", int.class, int.class).isPublic(); }

37. Example Patch (The Patch) From Google Guava protectedAbstractStreamingHasher(intchunkSize, intbufferSize) { - if (trigItIsPublic()) - checkArgument(bufferSize >= chunkSize); + checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} -@TrigtItMethod -booleantrigItIsPublic() { - returnTrigIt.getMethod("<init>", int.class, int.class).isPublic(); -}

38. Related Work • Pengyu Nie, Junyi Jessy Li, Sarfraz Khurshid, Raymond Mooney, and Milos Gligoric. 2018. Natural language processing and program analysis for supporting todo comments as software evolves. In Workshops of the the AAAI Conference on Artificial Intelligence. 775–778. • Margaret-Anne Storey, Jody Ryall, R. Ian Bull, Del Myers, and Janice Singer. 2008. TODO or to bug. In International Conference on Software Engineering. 251–260. • GiriprasadSridhara. 2016. Automatically detecting the up-to-date status of ToDo comments in Java programs. In India Software Engineering Conference. 16–25. • DorsafHaouari, HouariSahraoui, and Philippe Langlais. 2011. How good is your comment? A study of comments in Java programs. In International Symposium on Empirical Software Engineering and Measurement. 137–146. • Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*iComment: bugs or bad comments?*/. In Symposium on Operating Systems Principles. 145–158. • Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc comments to detect comment-code inconsistencies. In International Conference on Software Testing, Verification, and Validation. 260–269.