A domain specific language for Traversal Specification

1. A domain specific language for Traversal Specification Johan Ovlinger Mitchell Wand Northeastern University Center for Software Sciences

2. Center for Software Sciences Northeastern University Talk Outline • Introduction • introduce visitor pattern • point out some of its shortcomings • introduce our motivating, and running, example • show how it is unsatisfactorily solved by visitor pattern, despite the obvious applicability.

3. Introducing the Visitor Pattern • Visitor pattern is useful for • removing structural assumptions from programs, by separating navigational and behavioral concerns • adding behavior over classes independently of their definition.

4. Critiquing Traversals • Visitor is guided around object graph by traversal (iterator, strategy) • Unfortunately, the traversal serializes the hierarchical structure of the object being traversed into a series of visits. • One global traversal is typically hard-coded into classes, restricting its interface to Visitors to LCD.

5. Motivating Example • Taken from real world experience, writing a compiler. • We want to verify that all variables have been declared before use. • Obvious use of visitor pattern (suggested in “Design Patterns”).

6. Useless language • Our example is a very simple language that only has the ability to bind and reference variables. • Lets are non recursive • No Shadowing (for simplicity) • Epitomizes a common task: Finding undeclared variables. • Leave grammar up to imagination.

7. AST Class Diagram Prog Exp Let Ref Fun BindList Empty Bind String

8. A small AST object Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty LET foo = FUN (x) {y} bar = foo IN LET grok = bar IN grok Ref grok

9. Visitor Considerations • Strategy: • Calculate new bindings for one level and also analyze their bodies at same time • Remember which variables are in scope when entering a Let, and reset on exiting. • Since body of binding may also include Lets, must also remember List of new bindings to be added to scope. • Calculate list of bindings to add from let and process their bodies simultaneously.

10. administrator: these are missing the Bind middle visit Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty before Prog visit in scope to add Ref grok

11. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty before Prog before Let visit in scope to add Ref grok

12. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar before Prog before Let before Bind visit in scope to add Empty Ref grok

13. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar before Prog before Let before Bind before Fun x visit in scope to add Empty Ref grok

14. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty before Prog before Let before Bind before Fun x before Ref x visit in scope to add Let Bind grok Ref bar Empty Ref grok

15. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo Empty before Prog before Let before Bind before Fun before Ref x after Ref x visit in scope to add Let Bind grok Ref bar Empty Ref grok

16. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo before Prog before Let before Bind before Fun before Ref x after Ref x after Fun visit in scope to add Empty Let Bind grok Ref bar Empty Ref grok

17. Visitor’s view of Traversal Prog Let Bind foo Fun x Ref y Bind bar Ref foo before Prog before Let before Bind before Fun before Ref x after Ref x after Fun before Bind visit in scope to add Empty Let Bind grok Ref bar Empty Ref grok

18. Visitor’s view of Traversal Prog ... before Bind before Fun before Ref x after Ref x after Fun x before Bind before Ref after Ref before Empty after Empty after Bind bar visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

19. Visitor’s view of Traversal Prog ... before Fun before Ref x after Ref x after Fun x before Bind before Ref after Ref before Empty after Empty after Bind bar after Bind foo,bar visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

20. Visitor’s view of Traversal Prog ... before Ref x after Ref x after Fun x before Bind before Ref after Ref before Empty after Empty after Bind bar after Bind foo,bar middle Let foo,bar visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

21. Visitor’s view of Traversal Prog ... after Bind bar after Bind foo,bar middle Let foo,bar before Let foo,bar before Bind foo,bar before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bar grok middle Let foo,bar,grok visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

22. Visitor’s view of Traversal Prog ... middle Let foo,bar before Let foo,bar before Bind foo,bar before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bar grok middle Let foo,bar,grok before Ref foo,bar,grok before Ref foo,bar,grok visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

23. Visitor’s view of Traversal Prog ... before Let foo,bar before Bind foo,bar before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bar grok middle Let foo,bar,grok before Ref foo,bar,grok before Ref foo,bar,grok after Let foo,bar visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

24. Visitor’s view of Traversal Prog ... before Ref foo,bar after Ref foo,bar before Empty foo,bar after Empty foo,bar after Bind Let foo,bar grok middle Let foo,bar,grok before Ref foo,bar,grok before Ref foo,bar,grok after Let foo,bar after Let after Prog visit in scope to add Let Bind foo Fun x Ref y Bind bar Ref foo Empty Let Bind grok Ref bar Empty Ref grok

25. Implementation in Java • We assume a “typical” traversal distributed through classes which calls appropriate accept method on Visitor. • Visitor is a superclass of all visitors, and has default do-nothing accept methods.

26. Visitor in Java class UDVisitor extends Visitor { Vector inscope,boundbylet; Stack oldscope, oldbound; void before(Ref host) { if (!inscope.contains(host.str)) complain(host); } void after(Bind host) { boundbylet.addElement(host.str); } void before(Fun host) { inscope.addElement(host.str); } void after(Fun host) { inscope.removeElement(host.str); }

27. Visitor Cont void before(Let host) { oldscope.push(inscope.copy()); oldbound.push(boundbylet.copy()); boundbylet = new Vector(); } void middle(Let host) { inscope.add_from(boundbylet); } void after(Let host) { inscope = oldscope.pop(); boundbylet = oldbound.pop(); }

28. Visitor Interface Critique • Since all traversal code is outside our control, visitors have to pass parameters to subsequent visits by side-effecting own variables. Very imperative. • We assume a more intricate interface than normally accepted -- rather than just accept, we have before, middle and after. Otherwise, it would be hard to know when to add new bindings to scope. administrator: We could do with just accept by adding bound at empty and checking on the way down.

29. Structural Critique administrator: the fact that Binding’s body may contain lets forces Let to s/r bound. Also imp is when not to restore vars • The structure cannot be ignored -- the programmer needs to know when to save/restore the parameter passing variables. • Programmer needs to save and restore state manually • Restoring state depends on our non-standard visitor interface. • The order of traversing parts is vitally important. administrator: who ever said that it could? administrator: if we s/r around Bindings, then all is lost administrator: to restore state with only accept would be hard. We know after will close before administrator: Bindings before exp

30. Visitor Critique Summary • Visitors want to return results and accept arguments like recursive functions • The visitor needs to control the order in which parts are traversed. • A more elaborate traversal/visitor interface is desirable. accept is least common denominator. administrator: it is natural to expect that complex behavior will need more complicated behavior than before / middle / after

31. Center for Software Sciences Northeastern University Talk Outline • Introduction • Traversal Specifications • compare to std traversal

32. Traversal Specification • Define traversals with domain specific language. • Traversal is external to classes, and can be tailored to just one visitor, or be generic. • Traversal is explicit about traversing all parts. Assume only local knowledge of the object structure. Non-local knowledge cannot be expressed in Traversal Spec. administrator: surprisingly, we don’t have the faults we pointed out with visitor pattern.

33. administrator: There is a lot of talking about this slide Features of our Language • Separate Behavior, Navigation, and Class Definition • Specify which parts of an object are to be traversed, and which order • Traversals and Visitors return results • Traversal may bind intermediate results • Behavior may influence Navigation • Iterate over collections automatically

34. Go Everywhere Traversal administrator: introduce concept of subtraversal traversal everywhere() = Program => visit before () traverse exp () visit after () Ref => visit before () visit after() Let => visit before () traverse bindings() visit middle () traverse exp() visit after () Binding => visit before () traverse exp () visit middle () traverse bindings() visit after () ... administrator: this makes an interface everywhere_visitor that looks just like the Visitor class. Interface, not class, because we will have many traversals, with different requirements, while the std model has just one traversals, or if several, all w/ subsets of one big interface.

35. The Go Everywhere Traversal • Will generate an interface with before, after, middle methods, just like the traversal will impose its p • Each traversal imposes different requirements on visitors it is used with • By generating Java interfaces instead of abstract classes, a visitor may be used with several traversals.

36. Center for Software Sciences Northeastern University Talk Outline • Introduction • Traversal Specifications • compare to std traversal • overview

37. Syntax of the Language • Variables are declared with C/C++/Java syntax, with optional types • A Traversal is a named set of TraversalEntries, named after the class of objects they traverse. • TraversalEntry describes how to traverse objects of a certain class, using a list of Actions including visit and traverse.

38. More syntax • On entering an object, the most specific TraversalEntry for that object is executed, by performing the Actions in order and returning the result of the last one. • Visit actions name the visitor method to invoke, and which arguments to pass (current object is always passed). • Traversal actions specify the part of the current object to traverse, along with arguments to the traversal.

39. Center for Software Sciences Northeastern University Talk Outline • Introduction • Traversal Specifications • compare to std traversal • overview • running example • detailed interface to visitor • recursion

40. Our Strategy • Write a traversal that calls specific methods on visitor, at appropriate time. • Visits return their results to traversal. • Generate a Java Interface from the traversal to be implemented by visitor. • Traversal uses recursion to save/restore visitor variables automatically.

41. Traversal Spec for AST traversal unbound(Vector scope) = Program => traverse exp (scope) Ref => visit checkref(scope) Let => Vector bound = traverse bindings(scope) Vector newscope = visit concat(scope,bound) traverse exp(newscope) Binding => Vector bound = traverse bindings(scope) visit addtobound(bound) traverse exp(scope) bound Empty => visit emptybound Fun => visit addvar(scope) traverse exp(scope) visit remvar(scope)

42. Interface for AST Traversal interface unbound_visitor { void checkref(Ref host, Vector scope); Vector concat(Let host,Vector a,Vector b); void addtobound(Binding host, Vector bound); Vector emptybound(Empty host, Vector bound); void addvar(Fun host,Vector scope); void remvar(Fun host,Vector scope); }

43. Visitor Definition class UBVisitor implements unbound_visitor { Vector concat(Let host,Vector a,Vector b) {...} void checkref(Ref host, Vector inscope) { if (!inscope.contains(host)) complain(host); } void addtobound(Binding host, Vector bound) { bound.addElement(host.var); } Vector emptybound(Empty host, Vector bound) { return new Vector(); } void addvar(Fun host,Vector scope) { scope.addElement(host.var); } void remvar(Fun host,Vector scope) { scope.removeElement(host.var); } }

44. Compare to Visitor pattern • Surprisingly, we much prefer the separation of our version. • Navigation and Behavior are specified separately, but are tailored

45. Center for Software Sciences Northeastern University Talk Outline • Introduction • Traversal Specifications • compare to std traversal • overview • running example • influencing navigation

46. Behavior influences Navigation • In several cases, it is necessary to decide dynamically the navigational details. • The only way to do this with the visitor pattern is to hard code another traversal both in the classes, and even worse, in the visitor.

47. Example: Recursive Lets • Recursive Let adds bound variables to scope before checking bodies. • Let has boolean variable indicating whether it is recursive. • Strategy: • First calculate variables bound by Let • Add these to scope at appropriate time during processing

48. In Java • Separate traversal, launched from before(Let) collects bound vars, with separate visitor. • The new traversal doesn't enter bodies of Let or Bindings. administrator: so this example needs two traversals, as if we enter the bodies of bindings, we’re hosed.

49. Recursive Lets in Java class UDVisitor extends Visitor { ... void before(Let host) { oldscope.push(inscope.copy()); oldbound.push(boundbylet.copy()); GBVisitor gbv = new GBVisitor(); host.traverse_bindings(gbv); boundbylet = gbv.boundbylet; if (host.rec) inscope.add_from(boundbylet); } void middle(Let host) { if (!host.rec) inscope.add_from(boundbylet); } void after(Let host) { inscope = oldscope.pop(); boundbylet = oldbound.pop(); }

50. GBVisitor class GBVisitor extends Visitor { Vector boundbylet = new Vector(); void before(Bind host) { boundbylet.addElement(host.var); } }