Generic Language Technology and Model Driven SW Engineering

1. Generic Language Technology andModel Driven SW Engineering Prof. Dr. Mark van den Brand Software Engineering and Technology Faculteit Wiskunde en Informatica Technische Universiteit Eindhoven

2. GLT and MDE • Generic Language Technology • Program Generators • ATerms • ApiGen • Model Driven SW Engineering

3. Compiler construction Model Driven Eng. Formal methods Reverse engineering Re-engineering Domain Specific Lang. Aspect Oriented Prog. Software Maintenance Generic Language Technology Tools Fields • Compilers • Code generators • Static checkers • Refactoring • Outlines • Cross-linking • Decompilers • etc, etc, ...

4. Generic Language Technology • Source code juggling tools • all instances of meta programming • Three main levels of representation • Tools are all mappings between representations • GLT uses this similarity • representationshave languages • mappings between languages share functionality

5. Generic Language Technology • Meta programming is complex • Many different (legacy) languages • “Devil in the details” of syntax and (static) semantics • Tools are (company/application/domain) specific • GLT goal: facilitate meta programming • generic tools (reusable across languages) • generated tools (language and task parametric)

6. Syntax highlighters Structure editors Source browsers Fact visualizers Debugging interfaces Un-parsers Generic Language Technology Examples of tools • Parsers • Pretty printers • Tree transformers • Tree fact extractors • Fact analyzers Generated tools Generic tools

7. Generic Language Technology • ASF+SDF Meta-Environment • Since early 90's: generation of IDE components • Generated and generic meta programming tools • Today: focus on generated tools • Parsing • Transformation • Formatting • Reasoning

8. Generic Language Technology • ASF+SDF Meta-Environment • A domain specific language for each task • SDF: parser generation context-free grammars • ASF: tree transformation term rewriting • BOX: formatting box 2D layout • RScript: reasoning relational calculus • Meta-environment: generic (parameterized) IDE

9. Generic Language Technology • Challenges: • Meta programming is not compiler construction • Why? Different quality attributes! • Language and task independent • No heuristics (example: `longest match' in scanners) • No abstraction (example: keep layout and comments) • More flexible • Modular (example: embeddings, dialects) • Reusable (example: several tasks share a parser) • And so the challenges are • Finding the appropriate level of abstraction • Getting enough efficiency without heuristics or abstraction

10. GLT and MDE • Generic Language Technology • Program Generators • ATerms • ApiGen • Model Driven SW Engineering

11. Program Generators • “State-of-the-art” technology: component based software development • Model Driven Architectures • Code Generation • Aspect Oriented Programming • Coordination Architectures • Design Patterns

12. Program Generators • Automatic production of programs by means of other programs • A program generator reads meta-data and produces well-formed source code • Grammars • Database model • UML diagrams • A program generators makes your project “agile”

13. Program Generators Advantages • Increase in productivity: • generating tedious and boring parts of the code • code generators produce thousands of lines of code in seconds • changes are quickly propagated • Agile development

14. Program Generators Advantages • Increase of Quality: • bulky handwritten code tend to have inconsistent quality because increase of knowledge during development • bug fixes and code improvements can be consistently roled out using a generator • Increase of Consistency: • in API design and naming convention • Single point of definition • More design time • Explicit design decisions

15. Program Generators Advantages • Architectural consistency: • Programmers work within the architecture • Well-documented and -maintained code generator provides a consistent structure and approach • Abstraction: language-independent definition • Lifting problem description to a higher level • Easier porting to different languages and platforms • Design can be validate on an abstract level

16. Program Generators Disadvantages • Code generator has to be written first • Not always applicable • There will always be some code that has to be hand written • Generality can be a drawback, e.g.: • Databases must be well-designed and normalized • Grammars must be member of a specific class

17. Program Generators • 10 code-generation rules • Give proper respect to hand-coding • Handwrite the code first • Control the source code, e.g., CVS • Make a considered decision about the implementation language • Integrate the generator into the development process

18. Program Generators • 10 code-generation rules (continued) • Include warnings wrt modifying the generated code • Make it friendly • Include documentation • Keep in mind that generation is a cultural issue • Maintain the generator

19. Program Generators Models of program generators: • Code munging: given information in some input code, one or more output files are generated, e.g., scanner or parser • Inline-code expander: take source code with special mark-up code as input and creates production code in separate output file, e.g., imbedded SQL is replaced by C • Mixed-code generation: take source code with special mark-up code as input and replace this inline

20. Program Generators Examples of program generators: • Programming environment generators • API generators • UML based generators • Domain Specific Languages

21. Program Generators • Template based program generators • Legal code clones • Focus on code templates • Template validation: syntax

22. Program Generators Software templates: • Object code with holes • Holes contain instructions for an evaluator • meta code • Separate treatment of object code and evaluator code • interpretation meta language construct is invisible • Meta-Programming in concrete object code (WYSIWYG)‏ • template is skeleton of resulting code

23. Template validation public class<%class/name%>{ <%foreachclass/attribute do%> private <%type%> <%name%>; public<%type%> <%”get”||name%>(){ return<%name%> } publicvoid<%”set”||name%> (<%type%> <%name%>){ this.<%name%>=<%name%>; } <%od%> } class( name(“Customer”), attribute( name( “address” ), type( “String” )‏ ), attribute( name( “age” ), type( “int” )‏ )‏ ) Template Input data

24. Template validation public classCustomer{ private Stringaddress; publicStringgetaddress(){ returnaddress } publicvoidsetaddress (Stringaddress){ this.address=address; } private intage; public int getage(){ returnage } publicvoidsetage (int age){ this.age=age; } }

25. Template validation public class<%class/name%>{ <%foreachclass/attribute do%> private <%type%> <%name%>; public<%type%> <%”get”||name%>(){ return<%name%> } publicvoid<%”set”||name%> (<%type%> <%name%>){ this.<%name%>=<%name%>; } <%od%> }

26. Template validation • Better than using the compiler errors • detection of syntax errors in branches • more accurate error message • Generation of code not necessary • syntax checking without input data • However, can this technology be applied to other languages?

27. Template validation <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title><%class/name || " form"%></title> </head> <body> <formaction="/commit" method="post"> <%foreachclass/attribute do%> <p> <%name%> <br> < input type="text" name=<%"\""||name||"\""%>size="20"> </p> <%od%> <p> <input type="submit" value="Submit"> <input type="reset" value="Reset"> </p> </form> </body> </html> class( name(“Customer”), attribute( name( “address” ), type( “String” )‏ ), attribute( name( “age” ), type( “int” )‏ )‏ )‏

28. Template validation <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>Customerform</title> </head> <body> <formaction="/commit" method="post"> <p> address <br> < input type="text" name=“address"size="20"> </p><p> age <br> < input type="text" name=“age"size="20"> </p> <p> <input type="submit" value="Submit"> <input type="reset" value="Reset"> </p> </form> </body> </html>

29. Template validation <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>Customerform</title> </head> <body> <formaction="/commit" method="post"> <p> address <br> <input type="text" name=“address"size="20"> </p><p> age <br> <input type="text" name=“age"size="20"> </p> <p> <input type="submit" value="Submit"> <input type="reset" value="Reset"> </p> </form> </body> </html>

30. Template validation <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title><%class/name || " form"%></title> </head> <body> <formaction="/commit" method="post"> <%foreachclass/attribute do%> <p> <%name%> <br> < input type="text" name=<%"\""||name||"\""%>size="20"> </p> <%od%> <p> <input type="submit" value="Submit"> <input type="reset" value="Reset"> </p> </form> </body> </html>

31. Template validation • First parse the template and obtain the parse tree public class<%class/name%>{ <%foreachclass/attribute do%> private <%type%> <%name%>; public<%type%> <%”get”||name%>(){ return<%name%>; } publicvoid<%”set”||name%> (<%type%> <%name%>){ this.<%name%>=<%name%>; } <%od%> }

32. Template validation • Next find placeholders in parse tree • A tree traversal configured to match on the placeholder nodes is used

33. Template validation • Finding the placeholders in the parse tree

34. Template validation • Evaluation of placeholders public class <%class/name%>{ • Evaluate expression class/name • lookup in data specification • return value: “Customer” class( name(“Customer”), attribute( name( “address” ), type( “String” )‏ ), attribute( name( “age” ), type( “int” )‏ )‏ )

35. Template validation • Replace placeholder • Parse result of expression as ID • If parse error return error message • Otherwise, replace placeholder with parse tree of the result of the expression

36. Template validation • Parse template • Combining Object Language with Meta Language • Combination of grammars • Generic • Different Object Languages • Parameterize the Meta Language grammar and evaluator with Object Language • More Grammar, More Checking • More detailed Object Language grammar implies more syntax safety

37. Template validation Combination Meta grammar Java grammar

38. Literature on Program Generators • “Code Generation in Action” by Jach Herrington • First and fourth chapter available at: http://www.codegeneration.net/cgia/download.html • Arnoldus, J., Bijpost, J., Brand, M.G.J. van den (2007). Repleo: a syntax-safe template engine. Proceedings of the 6th International Conference on Generative Programming and Component Engineering (GPCE 2007) 1-3 October 2007, Salzburg, Austria.(pp. 25-32). New York, New York: ACM.

39. GLT and MDE • Generic Language Technology • Program Generators • ATerms • ApiGen • Model Driven SW Engineering

40. ATerms • Generic Term representation • Applicative, prefix terms • Maximal subterm sharing • Cheap equality test • Automatic generational garbage collection • Annotations (text coordinates, dataflow information, ...) • Concise and sharing preserving encoding for shipping: • yextual • binary

41. ATerms • ATerms are used to represent and exchange (tree-like) data structures: • parse trees • abstract syntax trees • parse tables • ATerm-library: • C version • JAVA version

42. ATerms • The ATerm data type is defined as: • INT: (32-bits) integers • REAL: (64-bits) reals • APPL: function application, function symbol with zero or more ATerms as arguments • LIST: a list of zero or more ATerms • PLACEHOLDER: an ATerm representing the type of the placeholder, used for matching and constructing

43. ATerms • The ATerm data type is defined as (continued): • BLOB: Binary Large data Objects, a length indication and a byte array of arbitrary (very large) binary data • a list of ATerm pairs (label, annotation)may be associated with an ATerm

44. ATerms • Examples of ATerms: • Unquoted and quoted function application: a, f(a), “x'”, f(“Hello World\n”) • Integers and Reals: 1, 2, 3.14, 1E-10 • Lists: [1,2.2,f(“greetings”)],[] • Placeholders: <int>, <f(1,2)> • Annotations: or(true,false){[color,white]}

45. ATerms • Simple match-and-make paradigm: • make a new ATerm by providing a pattern, “and(<int>,<appl>)”, and filling in the holes • match an existing ATerm by comparing it with a pattern and decompose it according to the pattern • most important operations: • ATerm ATmake(String p, ATerm a1,..., ATerm an) • ATerm ATmatch(ATerm t, String p, ATerm *a1,..., ATerm *an) • ATbool ATisEqual(ATerm t1, Aterm t2) • Integer ATgetType(ATerm t)

46. ATerms • Previous operations level one operations: • simple to use • high-level • inefficient • Efficient level-two operations: • advanced usage • efficient • ATgetArgument, ATmakeAppl, ATmakeInt, etc.

47. ATerms • Applications: • ASF+SDF Meta-Environment: an IDE for developing programming language definitions • mCRL2: a protocol verification tool uses ATerms to represent the state space. • Stratego compiler uses ATerms to represent abstract syntax trees. • TOM, a pattern matching compiler. • Semantic web ontology

48. ATerms • More information: • www.aterm.org • More reading: • M.G.J. van den Brand, H.A. de Jong, P. Klint, and P.A. Olivier (2000). "Efficient Annotated Terms" Software -- Practice & Experience 30:259--291

49. GLT and MDE • Generic Language Technology • Program Generators • ATerms • ApiGen • Model Driven SW Engineering

50. Application Programmers Interface (Library) ApiGen • Good properties of ATerms: • Simple concept • Simple and complete API • make and match • read and write • Efficient • Bad properties of ATerms • not typed (APPL, LIST, INT, REAL, BLOB)