1 / 44

Parser Generators

Parser Generators. Aurochs and ANTLR and Yaccs, Oh My. Tuesday, December 1, 2009 Reading: Appel 3.3. Compiler Structure. Abstract Syntax Tree (AST) . Source Program. Lexer (a.k.a. Scanner, Tokenizer). Tokens. Type Checker. Parser. (character stream).

felicia
Télécharger la présentation

Parser Generators

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Parser Generators Aurochs and ANTLR and Yaccs, Oh My Tuesday, December 1, 2009 Reading: Appel 3.3

  2. Compiler Structure Abstract Syntax Tree (AST) Source Program Lexer (a.k.a. Scanner, Tokenizer) Tokens TypeChecker Parser (character stream) Intermediate Representation Front End (CS235) Semantic Analysis Middle Stages(CS251/ CS301) Intermediate Representation Global Analysis Information (Symbol and Attribute Tables) Optimizer Intermediate Representation Back End (CS301) Code Generator (used by all phases of the compiler) Machine code or byte code Parser Generators

  3. Motivation We now know how to construct LL(k) and LR(k) parsers by hand. But constructing parser tables from grammars and implementing parsers from parser tables is tedious and error prone. Isn’t there a better way? Idea: Hey, aren’t computers good at automating these sorts of things? Parser Generators

  4. Parser Generators A parser generator is a program whose inputs are: a grammar that specifies a tree structure for linear sequences of tokens (typically created by a separate lexer program). a specification of semantic actions to perform when building the parse tree. Its output is an automatically constructed program thatparses a token sequence into a tree, performing the specified semantic actions along the way In the remainder of this lecture, we will explore and compare a two parser generators: Yacc and Aurochs. Parser Generators

  5. ml-yacc: An LALR(1) Parser Generator Slip.yacc Slip.yacc.desc LR parsing tableinfo for debuggingshift-reduce andreduce-reduce conflicts .yacc file (token datatype,parsing rules,precedence &associativity info) ml-yacc Slip.yacc.sig Slip.yacc.sml token datatype Slip.lex parsing program .lex file (token rules &token-handling code) ml-lex Slip.lex.sml token stream ant ant program (as character stream) abstract syntax tree parser program(SML) scanner program (SML) Slip tokens A Slip program Slip AST Parser Generators

  6. Format of a .yacc File Header section with SML code %% Specification of terminals & variables and various declarations (including precedence and associativity) %% Grammar productions with semantic actions Parser Generators

  7. IntexpUnambiguousAST.yacc (Part 1) (* no SML code in this case *) %% (* separates initial section from middle section of .yacc file *) %name Intexp %pos int %term INT of int | ADD | SUB | MUL | DIV | EXPT | LPAREN (* "(" *) | RPAREN (* ")" *) | EOF (* End of file*) %nonterm Start of AST.exp | Exp of AST.exp | Term of AST.exp | Factor of AST.exp | Unit of AST.exp %start Start (* Middle section continued on next slide *) Name used to prefix various names created by ml-yacc Declares the type of positions for terminals. Each terminal has a left and right position. Here, position is just an int (a character position in a string or file) Specifies the terminals of the language. ml-yacc automatically constructs a Tokens module based on this specification. Tokens specified without "of" will have a constructor of two args: (1) its left position and (2) its right position. Tokens specified with "of" will have a constructor of three args: (1) the component datum (whose type follows "of"); (2) its left position; and (3) its the right position. Specifies the non-terminals of the language and the kind of value that the parser will generate for them (an AST in this case). The "start" non-terminal should not appear on the right-hand sides of any productions. Start symbol of the grammar Parser Generators

  8. IntexpUnambiguousAST.yacc (Part 2) (* Middle section continued from previous slide *) %keyword %eop EOF %noshift EOF %nodefault %verbose %value INT(0) (* Grammar productions and semantic actions on next slide *) Lists the keywords of the language. The error handler of the ml-yacc generated parser treats keywords specially. In this example, there aren’t any keywords. Indicates tokens that end the input Tells the parser never to shift the specfied tokens Suppresses generation of default reduction Generates a description of the LALR parser table in filename.yacc.desc. This is helpful for debugging shift/reduce and reduce/reduce conflicts Specifies default values for value-bearing terminals. Terminals with default values may be created by an ml-yacc-generated parser as part of error-correction. Note; in this middle section, you can also specify associativity and precedence of operators. See IntexpPrecedence.yacc (later slide) for an example. Parser Generators

  9. AST module for integer expressions structure AST = struct datatype exp = Int of int | BinApp of binop * exp * exp and binop = Add | Sub | Mul | Div | Expt (* Various AST functions omitted from this slide *) end Parser Generators

  10. IntexpUnambiguousAST.yacc (Part 3) %% (* separates middle section from final section *) (* Grammar productions and associated semantic actions (in parens) go here *) Start: Exp (Exp) Exp : Term (Term) (* The following rules specify left associativity *) | Exp ADD Term (AST.BinApp(AST.Add,Exp,Term)) | Exp SUB Term (AST.BinApp(AST.Sub,Exp,Term)) Term : Factor (Factor) (* The following rules specify left associativity *) | Term MUL Factor (AST.BinApp(AST.Mul,Term,Factor)) | Term DIV Factor (AST.BinApp(AST.Div,Term,Factor)) Factor : Unit (Unit) (* The following rule specifies right associativity *) | Unit EXPT Factor (AST.BinApp(AST.Expt,Unit,Factor)) Unit : INT (AST.Int(INT)) | LPAREN Exp RPAREN (Exp) Within parens, nonterminals stand for the value generated bythe parser for the nonterminal, as specified in the %nonterm declaration (an AST expression in this case). Within parens, terminals stand for the component datumspecified after “of” in the %term declaration. E.g., the datum associated with INT is an integer. Parser Generators

  11. Interfacing with the Lexer (*contents of Intexp.lex *) open Tokens type pos = int type lexresult= (svalue,pos) token fun eof () = Tokens.EOF(0,0) fun integer(str,lexPos) = case Int.fromString(str) of NONE => raise Fail("Shouldn't happen: sequence of digits not recognized as integer -- " ^ str) | SOME(n) => INT(n,lexPos,lexPos) (* For scanner initialization (not needed here) *) fun init() = () %% %header (functor IntexpLexFun(structure Tokens: Intexp_TOKENS)); alpha=[a-zA-Z]; digit=[0-9]; whitespace=[\ \t\n]; symbol=[+*^()\[]; any= [^]; %% {digit}+ => (integer(yytext,yypos)); "+" => (ADD(yypos,yypos)); (* more lexer rules go here *) Parser Generators

  12. Parser interface code, Part 1 structure Intexp = struct (* Module for interfacing with parser and lexer *) (* Most of the following is "boilerplate" code that needs to be slightly edited on a per parser basis. *) structure IntexpLrVals = IntexpLrValsFun(structure Token = LrParser.Token) structure IntexpLex = IntexpLexFun(structure Tokens = IntexpLrVals.Tokens); structure IntexpParser = Join(structure LrParser = LrParser structure ParserData = IntexpLrVals.ParserData structure Lex = IntexpLex) val invoke = fn lexstream => (* The invoke function invokes the parser given a lexer *) let val print_error = fn (str,pos,_) => TextIO.output(TextIO.stdOut, "***Intexp Parser Error at character position " ^ (Int.toString pos) ^ "***\n" ^ str^ "\n") in IntexpParser.parse(0,lexstream,print_error,()) end fun newLexer fcn = (* newLexer creates a lexer from a given input-reading function *) let val lexer = IntexpParser.makeLexer fcn val _ = IntexpLex.UserDeclarations.init() in lexer end (* continued on next slide *) ml-yacc

  13. Parser interface code, Part 2 (* continued from previous slide *) fun stringToLexer str = (* creates a lexer from a string *) let val done = ref false in newLexer (fn n => if (!done) then "“ else (done := true; str)) end fun fileToLexer filename = (* creates a lexer from a file *) let val inStream = TextIO.openIn(filename) in newLexer (fn n => TextIO.inputAll(inStream)) end fun lexerToParser (lexer) = (* creates a parser from a lexer *) let val dummyEOF = IntexpLrVals.Tokens.EOF(0,0) val (result,lexer) = invoke lexer val (nextToken,lexer) = IntexpParser.Stream.get lexer in if IntexpParser.sameToken(nextToken,dummyEOF) then result else (TextIO.output(TextIO.stdOut, "*** INTEXP PARSER WARNING -- unconsumed input ***\n"); result) end val parseString = lexerToParser o stringToLexer (* parses a string *) val parseFile = lexerToParser o fileToLexer (* parses a file *) end (* struct *) Parser Generators

  14. Putting it all together: the load file (* This is the contents of load-intexp-unambiguous-ast.sml *) CM.make("$/basis.cm"); (* loads SML basis library *) CM.make("$/ml-yacc-lib.cm"); (* loads SML YACC library *) use "AST.sml"; (* datatype for integer expression abstract syntax trees *) use "IntexpUnambiguousAST.yacc.sig"; (* defines Intexp_TOKENS and other datatypes *) use "IntexpUnambiguousAST.yacc.sml"; (* defines shift-reduce parser *) use "Intexp.lex.sml"; (* load lexer *after* parser since it uses tokens defined by parser *) use "Intexp.sml"; (* load top-level parser interface *) Control.Print.printLength := 1000; (* set printing parameters so that *) Control.Print.printDepth := 1000; (* we’ll see all details of parse trees *) Control.Print.stringDepth := 1000; (* and strings *) open Intexp; (* open the parsing module so that we can use parseString and parseFile without qualification. *) Parser Generators

  15. Taking it for a spin [fturbak@cardinal intexp-parsers] ml-lex Intexp.lex Number of states = 14 Number of distinct rows = 3 Approx. memory size of trans. tabl e = 387 bytes [fturbak@cardinal intexp-parsers] ml-yacc IntexpUnambiguousAST.yacc [fturbak@cardinal intexp-parsers] Linux shell - use "load-intexp-unambiguous-ast.sml"; (* … lots of printed output omitted here … *) - parseString "1+2*3^4^5*6+7"; val it = BinApp (Add, BinApp (Add,Int 1, BinApp (Mul, BinApp (Mul,Int 2,BinApp (Expt,Int 3,BinApp (Expt,Int 4,Int 5))), Int 6)),Int 7) : IntexpParser.result SML intepreter Parser Generators

  16. IntexpUnambiguousAST.yacc.desc state 0: Start : . Exp INT shift 6 LPAREN shift 5 Start goto 19 Exp goto 4 Term goto 3 Factor goto 2 Unit goto 1 . error state 1: Factor : Unit . (reduce by rule 7) Factor : Unit . EXPT Factor ADD reduce by rule 7 SUB reduce by rule 7 MUL reduce by rule 7 DIV reduce by rule 7 EXPT shift 7 RPAREN reduce by rule 7 EOF reduce by rule 7 . error … lots of states omitted … state 18: Unit : LPAREN Exp RPAREN . (reduce by rule 10) ADD reduce by rule 10 SUB reduce by rule 10 MUL reduce by rule 10 DIV reduce by rule 10 EXPT reduce by rule 10 RPAREN reduce by rule 10 EOF reduce by rule 10 . error state 19: EOF accept . error 72 of 104 action table entries left after compaction 21 goto table entries Parser Generators

  17. IntexpUnambiguousCalc.yacc We can calculate the result of an integer expression ratherthan generating an AST. %% Start: Exp (Exp) Exp : Term (Term) (* The following rules specify left associativity *) | Exp ADD Term (Exp + Term) | Exp SUB Term (Exp - Term) Term : Factor (Factor) (* The following rules specify left associativity *) | Term MUL Factor (Term * Factor) | Term DIV Factor (Term div Factor) (* div is integer division *) Factor : Unit (Unit) (* The following rule specifies right associativity *) | Unit EXPT Factor (expt(Unit,Factor)) Unit : INT (INT) | LPAREN Exp RPAREN (Exp) (* An integer exponentiation function *) fun expt (base,0) = 1 | expt(base,power) = base*(expt(base,power-1)) %% (* Only changed parts shown *) %nonterm Start of int | Exp of int | Term of int | Factor of int | Unit of int ) Parser Generators

  18. Testing IntexpUnambiguousCalc Linux shell [fturbak@cardinal intexp-parsers] ml-yacc IntexpUnambiguousCalc.yacc [fturbak@cardinal intexp-parsers] - use "load-intexp-unambiguous-calc.sml"; (* … lots of printed output omitted here … *) - parseString "1+2*3"; val it = 7 : IntexpParser.result - parseString "1-2-3"; val it = ~4 : IntexpParser.result SML intepreter Parser Generators

  19. IntexpAmbiguousAST.yacc We can use an ambiguous grammar. (In this case, it leads to shift/reduce conflicts.) %% (* Only changed parts shown *) %nonterm Start of AST.exp | Exp of AST.exp ) %% Start: Exp (Exp) Exp : INT (AST.Int(INT)) | Exp ADD Exp (AST.BinApp(AST.Add,Exp1,Exp2)) | Exp SUB Exp (AST.BinApp(AST.Sub,Exp1,Exp2)) | Exp MUL Exp (AST.BinApp(AST.Mul,Exp1,Exp2)) | Exp DIV Exp (AST.BinApp(AST.Div,Exp1,Exp2)) | Exp EXPT Exp (AST.BinApp(AST.Expt,Exp1,Exp2)) Parser Generators

  20. IntexpAmbiguousAST.yacc.desc 25 shift/reduce conflicts error: state 8: shift/reduce conflict (shift EXPT, reduce by rule 6) error: state 8: shift/reduce conflict (shift DIV, reduce by rule 6) error: state 8: shift/reduce conflict (shift MUL, reduce by rule 6) error: state 8: shift/reduce conflict (shift SUB, reduce by rule 6) error: state 8: shift/reduce conflict (shift ADD, reduce by rule 6) error: state 9: shift/reduce conflict (shift EXPT, reduce by rule 5) error: state 9: shift/reduce conflict (shift DIV, reduce by rule 5) error: state 9: shift/reduce conflict (shift MUL, reduce by rule 5) error: state 9: shift/reduce conflict (shift SUB, reduce by rule 5) error: state 9: shift/reduce conflict (shift ADD, reduce by rule 5) error: state 10: shift/reduce conflict (shift EXPT, reduce by rule 4) error: state 10: shift/reduce conflict (shift DIV, reduce by rule 4) error: state 10: shift/reduce conflict (shift MUL, reduce by rule 4) error: state 10: shift/reduce conflict (shift SUB, reduce by rule 4) error: state 10: shift/reduce conflict (shift ADD, reduce by rule 4) error: state 11: shift/reduce conflict (shift EXPT, reduce by rule 3) error: state 11: shift/reduce conflict (shift DIV, reduce by rule 3) error: state 11: shift/reduce conflict (shift MUL, reduce by rule 3) error: state 11: shift/reduce conflict (shift SUB, reduce by rule 3) error: state 11: shift/reduce conflict (shift ADD, reduce by rule 3) error: state 12: shift/reduce conflict (shift EXPT, reduce by rule 2) error: state 12: shift/reduce conflict (shift DIV, reduce by rule 2) error: state 12: shift/reduce conflict (shift MUL, reduce by rule 2) error: state 12: shift/reduce conflict (shift SUB, reduce by rule 2) error: state 12: shift/reduce conflict (shift ADD, reduce by rule 2) … state descriptions omitted … Parser Generators

  21. IntexpPrecedenceAST.yacc We can resolve conflicts with precedence/associativity declarations(can also handle dangling else problem this way). %% (* Only changed parts shown *) %nonterm Start of AST.exp | Exp of AST.exp ) (* Specify associativity and precedence from low to high *) %left ADD SUB %left MUL DIV %right EXPT %% Start: Exp (Exp) Exp : INT (AST.Int(INT)) | Exp ADD Exp (AST.BinApp(AST.Add,Exp1,Exp2)) | Exp SUB Exp (AST.BinApp(AST.Sub,Exp1,Exp2)) | Exp MUL Exp (AST.BinApp(AST.Mul,Exp1,Exp2)) | Exp DIV Exp (AST.BinApp(AST.Div,Exp1,Exp2)) | Exp EXPT Exp (AST.BinApp(AST.Expt,Exp1,Exp2)) Parser Generators

  22. Stepping Back Whew! That’s a lot of work! Is it all worth it? There’s a tremendous amount of initial setup and glue code to get Lex and Yacc working together. But once that’s done, we can focus on specifying and debugging high-level declarations of the token structure (in Lex) and grammar (in Yacc) rather than on SML programming. In Yacc, generally spend most of the time debugging shift/reduce and reduce/reduce conflicts – must be veryfamiliar with the LALR(1) table descriptions in the .yacc.desc file. Parser Generators

  23. Other Parser Generators • There are a plethora of other LALR parser generators for many languages, including: • GNU Bison (Yacc for C, C++) • jb (Bison for Java) • CUP (Yacc for Java) • There are also LL(k) parser generators, particularly ANTLR, which generates parsers and other tree-manipulation programs for Java, C, C++, C#, Python, Ruby, and JavaScript. There is also an sml-antlr. • There are a relatively new (starting in 2002) collection of so-called packrat parsers for Parsing Expression Grammars (PEGs) for a wide range of target languages. We will study one of these: Aurochs. Parser Generators

  24. Parsing Expression Grammars (PEGs) • Key problem in building top-down (recursive-descent) parsers: deciding which production to use on tokens. Three strategies: • Predictive parser: determine which production to use based on k tokens of lookahead. Often mangle grammars to make this work. • Backtracking parser: when have several choices, try them in order. If one fails try the next. Very general and high-level, but can take exponential time to parse input. • PEG Parser: have a prioritized choice operator in which unconditionally use first successful match. • Can directly express common disambiguation rules like longest-match, followed-by, and not-followed-by. • Can express lexing rules and parsing rules in same grammar!This eliminates glue code between lexer and parser. • Can implement PEGs via a packrat parser that memoizes parsing results and guarantees parsing in linear time (but possibly exponential space) Parser Generators

  25. Aurochs: a PEG Parser We will explore PEG parsing using the Aurochs parser generator,developed by BerkeDurak and available from http://aurochs.fr. The input to Aurochs is (1) a file to be parsed and (2) a .peg file specifying PEG declarations for lexing and parsing rules. The output of Aurochs is an XML parse tree representation. (It is also possible to generate parse trees in particular languages,but for simplicity we’ll stick with the XML trees.) Parser Generators

  26. Aurochs Example [lynux@localhostslipmm-parsers]$ aurochs -parse simple.slip SlipmmParens.peg Aurochs 1.0.93 - Parsing file simple.slip using Nog interpreter - Grammar loaded from file SlipmmParens.peg - RESULT <Root> <StmList> <Assign var="x"> <BinAppbinop="+"> <Int value="3"/> <Int value="4"/> </BinApp> </Assign> <Print> <BinAppbinop="*"> <BinAppbinop="-"> <Id name="x"/> <Int value="1"/> </BinApp> <BinAppbinop="+"> <Id name="x"/> <Int value="2"/> </BinApp> </BinApp> </Print> </StmList> </Root> - No targets Contents of the filesimple.slip begin x := (3+4); print ((x-1)*(x+2)); end Parser Generators

  27. Aurochs Grammar Expressions Parser Generators

  28. Some Simple Expressions Parser Generators

  29. Numerical Expressions in Aurochs (* Tokens *) float ::= '-'? decimal+ ('.' decimal+)? ('e' ('-'|'+')? decimal+)?; integer ::= '-'? decimal+; decimal ::= [0-9]; sp ::= (' '|'\n'|'\r'|'\t')*; (* any number of whitespaces *) ADD ::= sp '+' sp; SUB ::= sp '-' sp; MUL ::= sp '*' sp; DIV ::= sp '/' sp; LPAREN ::= sp '(' sp; RPAREN ::= sp ')' sp; (* Expressions *) start ::= exp sp EOF; exp ::= term ADD exp (* right associative *) | term SUB exp (* right associative *) | term; term ::= factor MUL term (* right associative *) | factor DIV term (* right associative *) | SUB factor (* unary negation *) | factor; factor ::= number | LPAREN exp RPAREN; number ::= float|integer; Parser Generators

  30. XML Parse Trees for Numexps in Aurochs (* Tokens the same as before, so omitted *) (* Expressions *) start ::= exp sp EOF; exp ::= <add> term ADD exp </add> (* right associative *) | <sub> term SUB exp </sub> (* right associative *) | term; term ::= <mul> factor MUL term </mul> (* right associative *) | <div> factor DIV term </div> (* right associative *) | <neg> SUB factor </neg> | factor; factor ::= number | LPAREN exp RPAREN; number ::= <number> value:(float|integer) </number>; Parser Generators

  31. Testing Numexp Parser [lynux@localhost calculator]$ aurochs -quiet -parse exp.txt numexp.peg <Root> <mul> <sub> <number value="1"/> <sub> <number value="2"/> <number value="3"/> </sub> </sub> <neg> <add> <neg> <number value="4"/> </neg> <add> <number value="5.6"/> <number value="7.8e-9"/> </add> </add> </neg> </mul> </Root> Contents of the file exp.txt (1-2-3)*-(-4+5.6+7.8e-9) Parser Generators

  32. Transforming Left Recursion (* Expressions *) start ::= exp sp EOF; exp ::= term <addsub> (op:(ADD|SUB) exp)* </addsub>; term ::= <neg> SUB factor </neg> | factor <muldiv> (op:(MUL|DIV) term)* </muldiv>; factor ::= number | LPAREN exp RPAREN; number ::= <number> value:(float|integer) </number>; Parser Generators

  33. Left Recursion Transformation Exampl [lynux@localhost calculator]$ aurochs -quiet -parse sub.txt numexp-leftrec.peg <Root> <number value="1"/> <muldiv/> <addsub op="-"> <number value="2"/> <muldiv/> <addsub op="-"> <number value="3"/> <muldiv/> <addsub op="-"> <number value="4"/> <muldiv/> <addsub/> </addsub> </addsub> </addsub> <muldiv/> <addsub/> </Root> Contents of the file sub.txt (1-2-3-4) Parser Generators

  34. Moving Spaces from Tokens to Expressions (* Tokens *) float ::= '-'? decimal+ ('.' decimal+)? ('e' ('-'|'+')? decimal+)?; integer ::= '-'? decimal+; decimal ::= [0-9]; sp ::= (' '|'\n'|'\r'|'\t')*; (* any number of whitespaces *) ADD ::= '+'; SUB ::= '-'; MUL ::= '*'; DIV ::= '/'; LPAREN ::= '('; RPAREN ::= ')'; (* Expressions *) start ::= sp exp sp EOF; exp ::= <add> sp term sp ADD sp exp sp </add> (* right associative *) | <sub> sp term sp SUB sp exp sp </sub> (* right associative *) | term; term ::= <mul> sp factor sp MUL sp term sp </mul> (* right associative *) | <div> sp factor sp DIV sp term sp </div> (* right associative *) | <neg> sp SUB sp factor sp </neg> | sp factor sp; factor ::= sp number sp | sp LPAREN sp exp sp RPAREN sp; number ::= <number> sp value:(float|integer) sp </number>; Parser Generators

  35. Modifiers: Surrounding, Outfixing, and Friends • The following modifiers take an expression x as parameter and transform each production a ::= b as follows: • appending x do ... done transforms them into a ::= b x • prepending x do ... done transforms them into a ::= x b • surrounding x do ... done transforms them into a ::= x b x • The following modifiers transform each concatenation a b c: • outfixing x do ... done transforms them into x a x b x c x • suffixing x do ... done transforms them into a x b x c x • prefixing x do ... done transforms them into x a x b x c • Note that these modifiers also act within implicit concatenations produced by the operators '*' and '+'. These rules found at http://aurochs.fr Parser Generators

  36. A Shorthand for all those Expression Spaces start ::= sp exp sp EOF; outfixing sp do surrounding sp do exp ::= <add> term ADD exp </add> (* right associative *) | <sub> term SUB exp </sub> (* right associative *) | term; term ::= <mul> factor MUL term </mul> (* right associative *) | <div> factor DIV term </div> (* right associative *) | <neg> SUB factor </neg> | factor; factor ::= number | LPAREN exp RPAREN; number ::= <number> value:(float|integer) </number>; done; done; Parser Generators

  37. Aurochs Productions for Slip-- Tokens whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) (* sp is at least one ignorable unit (space or comment) *) sp ::= whitespace*; PRINT ::= "print"; BEGIN ::= "begin"; END ::= "end"; keyword ::= <keyword> value: (PRINT | BEGIN | END) </keyword>; ADD ::= '+'; SUB ::= '-'; MUL ::= '*'; DIV ::= '/'; op ::= <op> value: (ADD | SUB | MUL | DIV) </op>; LPAREN ::= "("; RPAREN ::= ")"; SEMI ::= ";"; GETS ::= ":="; punct ::= <punct> value: (LPAREN | RPAREN | SEMI | GETS) </punct> ; alpha ::= [a-zA-Z]; alphaNumUnd ::= [a-zA-Z0-9_]; digit ::= [0-9]; reserved ::= "print" | "begin" | "end"; ident ::= <ident> value: (~(reserved sp) alpha alphaNumUnd*) </ident>; integer ::= <integer> value: digit+ </integer>; token ::= sp (keyword|punct|op|ident|integer) sp; tokenList ::= <tokenList> token* </tokenList>; start ::= tokenList EOF; Parser Generators

  38. Slip-- Tokenizing example [lynux@localhost slipmm-parsers]$ aurochs -quiet -parse simple.slip SlipmmTokens.peg <Root> <tokenList> <keyword value="begin"/> <ident value="x"/> <punct value=":="/> <punct value="("/> <integer value="3"/> <op value="+"/> <integer value="4"/> <punct value=")"/> <punct value=";"/> <keyword value="print"/> <punct value="("/> <punct value="("/> <ident value="x"/> <op value="-"/> <integer value="1"/> <punct value=")"/> <op value="*"/> <punct value="("/> <ident value="x"/> <op value="+"/> <integer value="2"/> <punct value=")"/> <punct value=")"/> <punct value=";"/> <keyword value="end"/> </tokenList> </Root> Contents of the filesimple.slip begin x := (3+4); print ((x-1)*(x+2)); end Parser Generators

  39. Slip-- Tokens with Positions [lynux@localhost slipmm-parsers]$ aurochs -quiet -parse simple.slip SlipmmTokensPositions.peg <Root> <tokenList> <token start="0" stop="5"> <keyword value="begin"/> </token> <token start="9" stop="10"> <ident value="x"/> </token> <token start="11" stop="13"> <punct value=":="/> </token> <token start="14" stop="15"> <punct value="("/> </token> <token start="15" stop="16"> <integer value="3"/> </token> <token start="16" stop="17"> <op value="+"/> </token> … many more tokens …. (* Only Change to .peg file. *) token ::= <token> sp start:position (keyword|punct|op|ident|integer) stop:position sp </token>; Contents of the filesimple.slip begin x := (3+4); print ((x-1)*(x+2)); end Parser Generators

  40. Line Comments How to add line-terminated comments introduced by #? begin sum := 5+3; # Set sum to 8 print(sum); # Display sum end (* Only changes in .peg file *) whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to end of line *) (* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace | linecomment )*; Parser Generators

  41. Nonnestable Block Comments begin sum := 5+3; # Set sum to 8 { Comment out several lines: x := sum * 2; z := x * x; } print(sum); # Display sum end (* Only changes in .peg file *) whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to end of line *) blockcomment ::= '{' ((~ '}') sigma)* '}'; (* Nonnestable block comment *) (* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace | linecomment | blockcomment)*; Parser Generators

  42. Nestable Block Comments begin sum := 5+3; # Set sum to 8 { Comment out several lines: x := sum * 2; { Illustrate nested block comments: y = sum - 3:} z := x * x; } print(sum); # Display sum end (* Only changes in .peg file *) whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to end of line *) blockcomment ::= '{' (blockcomment| ((~ '}') sigma))* '}'; (* Properly nesting block comment *) (* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace | linecomment | blockcomment)*; Parser Generators

  43. Slip -- Programs (* Assume tokens unchanged from before *) start ::= program; outfixing sp do appending sp do program ::= stm EOF; stm ::= <Assign> var:ident GETS exp </Assign> | <Print> PRINT exp </Print> | <StmList> BEGIN stmList END </StmList>; stmList ::= (stm SEMI)*; exp ::= <Id> name:ident </Id> | <Int> value:integer </Int> | <BinApp> LPAREN exp binop:op exp RPAREN </BinApp>; done; done; Parser Generators

  44. Aurouchs Errors • Fatal error: exception Aurochs.Parse_error(1033) • This means that the .peg file has an error at the specified character number • (1033 in this case). • Note: start position is 0-indexed; compared to 1-indexed emacs position in M-x goto-char. • PARSE ERROR IN FILE nesting-block-comment-test.slip AT CHARACTER 12 • This means that the parser encountered an error in parsing the input file at the specified character number. Parser Generators

More Related