Committee Members: Dr. Barrett R. Bryant (Chair) Dr. Jeff Gray, Dr. Marjan Mernik Dr. Alan Sprague, Dr. Murat Tanik

COMPONENT-BASED LANGUAGE IMPLEMENTATION WITH OBJECT-ORIENTED SYNTAX AND ASPECT-ORIENTED SEMANTICSXIAOQING (CARL) WU MARCH 2007 Committee Members: Dr. Barrett R. Bryant (Chair) Dr. Jeff Gray, Dr. Marjan Mernik Dr. Alan Sprague, Dr. Murat Tanik PH.D. THESIS DEFENSE UNIVERSITY OF ALABAMA AT BIRMINGHAM

Ph.D. Defense Xiaoqing Wu March 2007 Page 2 Outline • Background knowledge • Problem statement • Related work • Framework overview • Component-based language development • OOS and AOS • Object-oriented syntax • Aspect-oriented semantics • Framework usage • Demo • Future work • Conclusion

Ph.D. Defense Xiaoqing Wu March 2007 Page 4 Background knowledge • Syntax and semantics • Component-Based Software Engineering (CBSE) • Promote software reuse • Essential properties • Information hiding • Explicit interface • Context Independency • Object-oriented programming • Aspect-Oriented Programming (AOP) and AspectJ • Aspects: special language constructs to modularize crosscutting concerns. • Introduction (inter-type declarations) • Interception (join-points)

Ph.D. Defense Xiaoqing Wu March 2007 Page 6 Compiler construction vs. cooking a wedding cake • Cooking facilities: YACC, JavaCC, CUP, ... • Cooking complexity • Compiler design is known as a “dragon” task • Good modularity enables you to divide-and-conquer the complexity • As long as the pieces can be assembled together

Ph.D. Defense Xiaoqing Wu March 2007 Page 7 No decomposition of language definitions Most parser generators don’t support modular grammar definitions at all Cobol 85 is 2500 lines of specification, more than 1000 variables A Language Comprehensibility Changeability Expressions Reusability Statements Independentdevelopment Classes

Syntax and semantics Syntax analysis -- formal specification Semantic analysis -- programming languages The communication between syntax and semantics makes the specification and code tangled together Among different semantic phases Pure object-oriented design, code scatter all over the syntax tree class hierarchy Ph.D. Defense Xiaoqing Wu March 2007 Page 8 No clear separation of compiler construction phases Hard to maintain and evolve!!

Ph.D. Defense Xiaoqing Wu March 2007 Page 9 Syntax Semantics Semantic phase1 Semantic phase2 Declarative formal specification Imperative programming language code Hand-written code Auto-generated code Syntax analysis Type checking Code generation Ideal separation objectives

Ph.D. Defense Xiaoqing Wu March 2007 Page 11 Related work – decomposition of language syntax Modular grammar • The nature is pure text copying • Modules are still tightly coupled  conflicts • Any update to a particular module requires a re-composition of all the modules and regeneration of a large parse table

Ph.D. Defense Xiaoqing Wu March 2007 Page 12 Related work – separation of compiler phases • Separation between syntax and semantics • Semantics by formal specification • Good separation but not rich enough to fully describe semantics • Separation between semantic phases • The Visitor pattern • Introduce a lot of extra code • Force all concrete visitors share the same interface • New semantics are always introduced by traversal of the whole tree • Cannot access private members of a node class • Aspect-oriented semantics • JastAdd II – only support static introduction

Ph.D. Defense Xiaoqing Wu March 2007 Page 14 Framework overview Structure Component-based LR (CLR) parsing decomposes a large language into a set of smaller languages Function Object-Oriented Syntax (OOS) and Aspect-Oriented Semantics (AOS) facilitate separation of different phases

Ph.D. Defense Xiaoqing Wu March 2007 Page 15 Contribution • CLR decreases the development complexity by reducing the granularity of a language • Syntax composition at the parser level  reduced coupling between grammar modules • More expressive than regular LR parsing • OOS + AOS isolates syntax and semantics as well as semantic phases themselves into different modules • Separation of declarative and imperative behavior • Separation of generated code and handwritten code • OOS - generation of both parser and syntax tree • AOS - transparent to node classes, flexible in tree walking and phase composition. The overall paradigm increases the comprehensibility, reusability, changeability, extendibility and independent development ability of the syntax and semantic analysis with less development workload required from compiler designers.

Ph.D. Defense Xiaoqing Wu March 2007 Page 17 Component-based Context-Free Grammar (CCFG)

Ph.D. Defense Xiaoqing Wu March 2007 Page 18 Java language components (11 components)

A CCFG component G is a quintuple (N, T, C, P, S) N: a set of nonterminal symbols T: a set of terminal symbols C: a set of symbols representing other grammar components P  N  (N  T  C)* is a finite set of production rules, a production of the form A  means A derives . S  N : the start symbol CCFL Let  (N  T  C  L(C))*,  (N  T  C  L(C))*, 1= B  and 2=, then 1 directly derives 2, denoted 1 2, if one of the two conditions is met: 1) B  is a production in P; 2) B  C and  L(B) L(G) = {x| S *x, x  (TL(C))*}) Parser components Grammar component  parser component. The root parser invokes its sub-parsers that will recursively invoke other parsers as needed. Ph.D. Defense Xiaoqing Wu March 2007 Page 19 CCFG & CCFL & parser components

Ph.D. Defense Xiaoqing Wu March 2007 Page 20 Component-based grammar vs. modularized grammar Modularized grammar Component-based grammar Grammar Module Grammar Module Grammar Component Grammar Component Grammar Parser Parser Parser Code-level composition, less coupled definition, smaller parsing table, multiple lexers, etc ...

Pseudo code: flag component_parse(program, stack): repeat state := stack.top() if switch_map (state)  for component  switch_map (state) if component.lr_parse(program) == true record stack configuration return continue_flag endif endfor endif if return_map (state) == true if return action success return termination_flag end if endif if stack  recover stack by one step else return error_flag endif until reach the stack configuration when last switch action happened return error_flag Ph.D. Defense Xiaoqing Wu March 2007 Page 21 CLR parsing algorithm – switch and return

Ph.D. Defense Xiaoqing Wu March 2007 Page 22 CLR parsing example (4 components)

Ph.D. Defense Xiaoqing Wu March 2007 Page 23 Software engineering benefits Comprehensibility Intertwined symbols and productions are reduced Changeability • Changes are isolated inside individual components • Only local recompilation needed Reusability Components can be plugged and played Independent development Dependencies are handled at the code-level instead of the grammar level

Ph.D. Defense Xiaoqing Wu March 2007 Page 24 Language description ability • Expressive power • CLR’s backtracking can resolve the traditional shift-reduce or shift-shift conflicts in LR parsers • Ambiguous tokens • Benefited by multiple lexers • Useful for embedded languages, languages with no reserved words, etc. • SQLJ: count • PL/I example IF IF = THEN THEN IF = THEN;

Ph.D. Defense Xiaoqing Wu March 2007 Page 25 Performance measurement time Parsing speed comparison among 11 versions of CLR implementation of JLS version The increase of external actions as the number of components increases number of external actions version

Ph.D. Defense Xiaoqing Wu March 2007 Page 27 Object-orientation Aspect-orientation Data structures Crosscutting behaviors Semantic analysis Syntax analysis Design principle Syntax can be easily specified by formal specification but semantics cannot due to its arbitrary nature Syntax analysis Semantic analysis

Ph.D. Defense Xiaoqing Wu March 2007 Page 28 Object-oriented syntax × A ::= B C | D;

// Syntax definition Stmt ::= Block | “if” Expr “then” Stmt | Id “:=” Exp Stmt ::= Block | IfStmt | AssignStmt. IfStmt ::= “if” Exp “then” Stmt. AssignStmt ::= Id “:=” Exp. Ph.D. Defense Xiaoqing Wu March 2007 Page 29 // Tree definition abstract Stmt; BlockStmt : Stmt ::= Block; IfStmt : Stmt ::= Exp Stmt; AssignStmt : Stmt ::= Id Exp; JustAdd specification Object-oriented syntax

Ph.D. Defense Xiaoqing Wu March 2007 Page 30 OOCFG specification features and their usage • Object-oriented grammar definition • Enabling an object-oriented relationship between LHS symbol and RHS symbols, therefore removing the need to provide a separated specification for syntax tree construction. • AST and CST • Providing semantic analysis the flexibility in tree selection and ensuring all analysis needs can be easily computed • Typed LHS symbol – no node class generation • Promoting the reuse of existing node classes • Macros – only occur in syntax trees, transparent to parser • Reducing parsing conflicts while providing richer description of the grammar and distinct syntax tree nodes • Templates – generic production definitions • Facilitating OOCFG to support generic production definition in a grammar specification

Ph.D. Defense Xiaoqing Wu March 2007 Page 31 Aspect-oriented semantics implementation • Each semantic concern is modularized as an aspect • An independent semantic pass • A group of action codes • Semantic pass • Implemented as introductions to the syntax tree classes • Crosscutting actions applied to a group of nodes • Weaved into syntax tree classes as interceptions

Ph.D. Defense Xiaoqing Wu March 2007 Page 32 Introduction

Ph.D. Defense Xiaoqing Wu March 2007 Page 33 Aspect-oriented introduction vs. object-oriented Visitor pattern

Ph.D. Defense Xiaoqing Wu March 2007 Page 34 class UnparseVisitor extends Visitor{ protected PrintStream out = System.out; public Object print(Node node){ // ... } // Other utility routines // ... public Object visit(Node node){ return print(node); } public Object visit(CompilationUnit node){ return print(node); } // same visit methods for another 83 nodes. // ... } aspect Unparse{ protected PrintStream out = System.out; publicstaticvoid print(Node node){ // ... } // other utility routines // ... publicvoid Node.unparse(){ Unparse.print(this); } } 500 lines of redundant code have been removed !

Ph.D. Defense Xiaoqing Wu March 2007 Page 35 Occurred 46 times in a parser! Interception pointcut scopeEvaluate(): target(ScopeNode+) && call (* *.evaluate()) ; before() : scopeEvaluate(){ symTabs.push(currentSymTab); SymbolTable tmp = currentSymTab; currentSymTab = new SymbolTable(); currentSymTab.parentScope = tmp; } after() : scopeEvaluate(){ currentSymTab = (SymbolTable)symTabs.pop(); } Executed each time entering a new scope Executed each time leaving a scope

Ph.D. Defense Xiaoqing Wu March 2007 Page 36 before() : scopeEvaluate(){ symTabs.push(currentSymTab); SymbolTable tmp = currentSymTab; currentSymTab = new SymbolTable(); currentSymTab.parentScope = tmp; } after() : scopeEvaluate(){ currentSymTab = (SymbolTable)symTabs.pop(); } … // node is an instance of ScopeNode symTabs.push(currentSymTab); SymbolTable tmp = currentSymTab; currentSymTab = new SymbolTable(); currentSymTab.parentScope = tmp; node.evaluate(); currentSymTab = (SymbolTable)symTabs.pop(); … ASPECT WEAVING … // node is an instance of ScopeNode node.evaluate(); …

Ph.D. Defense Xiaoqing Wu March 2007 Page 37 AOS advantages • Aspect-orientation can isolate crosscutting semantic behavior in an explicit way • Each semantic aspect can be freely attached to (generated) AST nodes without “polluting” the parser or AST node structure. • Different aspects can be selectively plugged in for different purposes at compile time. • Since each aspect is separated with other aspects, developers can always come back to the previous phase while developing a later phase.

Ph.D. Defense Xiaoqing Wu March 2007 Page 38 aspect PrintNodeCreation { pointcut construction(Node n): target(n) && execution((Node+ && !Node).new(..)); after(Node n) returning():construction(n) { System.out.println( thisJoinPointStaticPart.getSignature() .getDeclaringType().getName()+"is created"); } } AOS advantages (cont’d) • Inter-type declaration (Introduction) • Defined within class scope, direct access to syntax tree node class members (include private members) • Extend the object-oriented inheritance relationship and promote semantic reuse • Join-point model (Interception) • Provide flexibility to insert semantic behaviors into AST nodes or parser • Avoid code duplication • Trace facility • Introduction + Interception • Tree traversal • Phase combination

Ph.D. Defense Xiaoqing Wu March 2007 Page 39 Expr component Stmt component Class component Parser Parser Parser Type checking Type checking Type checking Code generation Code generation Code generation Integration with CLR parsing • Syntax specification The restrictions of OOS can be applied to CCFG without generating any side-effects • Syntax tree construction CLR’s parse tree generation process is inlined with OOS tree generation • Semantic analysis Semantic composition follows syntax tree composition

Ph.D. Defense Xiaoqing Wu March 2007 Page 41 Demo – the Robot language • Part I: the integration between syntax and semantics • Part II: the composition between language components

Pam and BasicM RelationJava OOCFG Converter Bootstrap Implementation Google Query Language Java Changeability Reusability Independent development ability Readability Flexible semantics Flexible syntax/tree Ph.D. Defense Xiaoqing Wu March 2007 Page 42 Case studies

Ph.D. Defense Xiaoqing Wu March 2007 Page 44 Future work • CLR backtracking • Time complexity • Error recovery • Module inclusion • Grammar aspects • Support of other parsing paradigms • Rich client platform based on eclipse platform

Ph.D. Defense Xiaoqing Wu March 2007 Page 46 Conclusion • Compiler design is an intricate task because it is hard to be modularized (structure wise and function wise). • The presented framework presents a solution that can attack the modularity problems in two dimensions • CLR decreases the complexity of building a large language by constructing a set of smaller language parsers from grammar components • OOS + AOS provides a clean separation of concerns between syntax and semantics as well as semantic phases themselves • The framework also supersedes conventional language implementation practices by its description power, reduced specification, and support to describe crosscutting semantic behaviors, etc. • Various experiments prove that the methodology increases the comprehensibility, reusability, changeability, extendibility and independent development ability of both syntax and semantics specification with less development workload required from compiler designers.

Ph.D. Defense Xiaoqing Wu March 2007 Page 47 Selected publication list (First author) • Book Chapters • Xiaoqing Wu, Barrett R. Bryant, Jeff G. Gray, and Marjan Mernik, Description and Implementation of Syntax and Semantics of Programming Languages, submitted to Encyclopedia of Information Science and Technology, Second Edition. • Journal Papers • Xiaoqing Wu, Barrett R. Bryant, Jeff G. Gray, and Marjan Mernik, Component-based LR Parsing, in preparation. • Xiaoqing Wu, Barrett R. Bryant, Jeff G. Gray, and Marjan Mernik, Applying Object-Orientation and Aspect-Orientation in Teaching Domain-Specific Language Implementation, Journal of Computing Sciences in Colleges, No. 2, Vol.21, December 2005, pp. 335-340. • Xiaoqing Wu, Barrett R. Bryant and Marjan Mernik, Object-Oriented Pattern-Based Language Implementation, Acta Electrotechnica et Informatica, No.4, Vol.4, 2004, pp.5-12. • Referred Conference and Workshop Papers • Xiaoqing Wu, A Framework for Component-based Compiler Development, Student Research Competition, Programming Language and Design Implementation (PLDI 2007), San Diego, CA, June 2007 • Xiaoqing Wu, Barrett Bryant, Jeff Gray, Suman Roychoudhury, and Marjan Mernik, Separation of Concerns in Compiler Development using Aspect-Orientation, ACM Symposium for Applied Computing (SAC) Programming for Separation of Concerns Track, Dijon, Bourgogne, France, March 2006. • Xiaoqing Wu, Pattern Transformation for Two-Dimensional Separation of Concerns, Student Research Competition, Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2005), San Diego, CA, October 2005. • Xiaoqing Wu, Suman Roychoudhury, Barrett Bryant, Jeff Gray and Marjan Mernik, A Two-Dimensional Separation of Concerns for Software Construction, ACM Symposium for Applied Computing (SAC) Programming for Separation of Concerns Track, Santa Fe, NM, March 2005. • Xiaoqing Wu, Barrett Bryant, Jeff Gray and Marjan Mernik, Pattern Transformation for Two-Dimensional Separation of Concerns, First International Workshop on the Modeling and Analysis of Concerns in Software (MACS), held at ICSE, St. Louis, MO, May 2005 (also appeared in publication as ACM SIGSOFT Software Engineering Notes, volume 30, issue 4, July 2005, pp. 1-5).

Ph.D. Defense Xiaoqing Wu March 2007 Page 48 FAQ – www.cis.uab.edu/wuxi/research • What are the implementation challenges of the CLR algorithm? • How expressive is CLR? • How exactly switch and return work in CLR? • What are the restrictions of CLR grammars ? • Do CLR grammars need any global-analysis? • What is the time complexity of CLR? Why backtracking is not a big deal in CLR? • How to convert a regular CFG into object-oriented CFG? • What are abstract syntax tree and concrete syntax trees? • How macros work? • How templates work? • Can semantics intercept the parser directly?

Ph.D. Defense Xiaoqing Wu March 2007 Page 49 Abstract syntax tree vs. concrete syntax tree OOS specification generates both concrete syntax tree and abstract syntax tree to fully satisfy various semantic analysis requirements

Ph.D. Defense Xiaoqing Wu March 2007 Page 50 /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ parser_code_part ::= PARSER CODE CODE_STRING:user_code opt_semi {: if (emit.parser_code!=null) lexer.emit_error("Redundant parser code (skipping)"); else /* save the user included code string */ emit.parser_code = user_code; :} ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ init_code ::= INIT WITH CODE_STRING:user_code opt_semi {: if (emit.init_code!=null) lexer.emit_error("Redundant init code (skipping)"); else /* save the user code */ emit.init_code = user_code; :} ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ scan_code ::= SCAN WITH CODE_STRING:user_code opt_semi {: if (emit.scan_code!=null) lexer.emit_error("Redundant scan code (skipping)"); else /* save the user code */ emit.scan_code = user_code; :}; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ symbol_list ::= symbol_list symbol | symbol; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ symbol ::= TERMINAL type_id declares_term TERMINAL declares_term | non_terminal type_id declares_non_term | non_terminal declares_non_term | /* error recovery productions -- sync on semicolon */ TERMINAL error {: /* reset the accumulated multipart name */ multipart_name = new String(); :} SEMI {: /* reset the accumulated multipart name */ multipart_name = new String(); :} SEMI ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ declares_term ::= term_name_list {: /* reset the accumulated multipart name */ multipart_name = new String(); :} SEMI ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ declares_non_term ::= non_term_name_list {: /* reset the accumulated multipart name */ multipart_name = new String(); :} SEMI ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ term_name_list ::= term_name_list COMMA new_term_id | new_term_id; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ non_term_name_list ::= non_term_name_list TERMINAL error {: :} SEMI | non_terminal error {: :} SEMI ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ declares_term ::= term_name_list {: :} SEMI ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ declares_non_term ::= non_term_name_list {: :} SEMI ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ term_name_list ::= term_name_list COMMA new_term_id | new_term_id; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ parser_code_part ::= PARSER CODE CODE_STRING:user_code opt_semi {: :} ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ init_code ::= INIT WITH CODE_STRING:user_code opt_semi {: :} ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ scan_code ::= SCAN WITH CODE_STRING:user_code opt_semi {: :} ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ symbol_list ::= symbol_list symbol | symbol; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ symbol ::= TERMINAL type_id declares_term | TERMINAL declares_term | non_terminal type_id declares_non_term | non_terminal declares_non_term | /* error recovery productions -- sync on semicolon */ non_term_name_list ::= non_term_name_list COMMA new_non_term_id | new_non_term_id ; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ precedence_list ::= precedence_l | empty; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ precedence_l ::= precedence_l preced | preced; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ preced ::= PRECEDENCE LEFT {: :} terminal_list SEMI | PRECEDENCE RIGHT {: terminal_list SEMI | PRECEDENCE NONASSOC :} terminal_list SEMI ; if (emit.parser_code!=null) lexer.emit_error("Redundant parser code (skipping)"); else /* save the user included code string */ emit.parser_code = user_code; if (emit.init_code!=nul) lexer.emit_error("Redundant init code (skipping)"); else /* save the user code */ emit.init_code = user_code; if (emit.scan_code!=null) lexer.emit_error("Redundant scan code (skipping)"); else /* save the user code */ emit.scan_code = user_code; /* reset the accumulated multipart name */ multipart_name = new String(); /* reset the accumulated multipart name */ multipart_name = new String(); /* reset the accumulated multipart name */ multipart_name = new String(); /* reset the accumulated multipart name */ multipart_name = new String(); update_precedence(assoc.left); update_precedence(assoc.right); :} {: update_precedence(assoc.nonassoc); COMMA new_non_term_id | new_non_term_id /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ precedence_list ::= precedence_l | empty; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ precedence_l ::= precedence_l preced | preced; /*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . */ preced ::= PRECEDENCE LEFT {: update_precedence(assoc.left); :} terminal_list SEMI | PRECEDENCE RIGHT {: update_precedence(assoc.right); :} terminal_list SEMI | PRECEDENCE NONASSOC {: update_precedence(assoc.nonassoc); :} terminal_list SEMI after(ASTNode n) returning(): target(n) && execution(( NodeA || NodeB || NodeC).new(..)){ System.out.println("you can insert action here!"); } Interception to parser parser.cup Action.aj

Committee Members: Dr. Barrett R. Bryant (Chair) Dr. Jeff Gray, Dr. Marjan Mernik Dr. Alan Sprague, Dr. Murat Tanik