1 / 63

Introduction

GOLD: A Grammar Oriented Parsing System Devin Cook and Du Zhang Department of Computer Science California State University Sacramento, CA 95819-6021. Introduction. What is a Parser? Software which breaks a source program into its various grammatical units w.r.t. a formal grammar

yama
Télécharger la présentation

Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. GOLD: A Grammar Oriented Parsing SystemDevin Cook and Du ZhangDepartment of Computer ScienceCalifornia State UniversitySacramento, CA 95819-6021 SEKE 2004

  2. Introduction • What is a Parser? • Software which breaks a source program into its various grammatical units w.r.t. a formal grammar • Used to convert a source program into an internal representation • Parsing Algorithms • LL Parsers: top-down, predictive • LR / LALR Parsers: bottom-up, shift-reduce

  3. Motivation • The common approach to create parsers is through compiler-compiler, or parser generator • Each parser generator is designed for a specific programming language. There is no consistent parser generator • Different grammatical notations • Features and interfaces of tools vary in both the look and the behavior

  4. Goals • Design and implement a generalized parsing system that supports development of multiple programming languages • Offer a consistent development environment for the language developers

  5. GOLD • Grammar Oriented Language Developer. • Separating the component that generates parse tables for a target grammar from the component that does the actual parsing. • Support the full Unicode character set. • Include a set of tools that can aid language development process.

  6. System Structure Builder • Analyzes a target grammar and creates DFA and LALR parse tables • These tables are saved to a Compiled Grammar Table file Compiled Grammar Table file • Intermediary between the Builder and the Engine • The file format is platform independent • Format is designed to be very easy to read and extend in future versions Engine • Reads the tables & parses the source text • Can be implemented in different programming languages – as needed

  7. Development Flow • Grammar is defined and loaded • Any text editor can be used • Builder • Grammar is analyzed and errors reported • The parse tables are created and saved to .cgt file • Engine • Reads the tables, parses the source string, and produces parsing results • Can be implemented in different programming languages – as needed

  8. The Builder • GOLD meta-language • Compiled grammar table (.cgt) file • Skeleton program creation for the Engine from program templates • Interactive source string testing • Display of various parse table information • Export parse tables to a web page, XML file, or formatted text

  9. GOLD Meta-Language • The GOLD Meta-Language is used to define a target grammar • It must not contain features that are programming language dependent • Its notation is very close to the standards • It supports all language attributes (including those which cannot be specified using BNF or regular expressions)

  10. GOLD Meta-Language (contd.) • Format • Parameters are used to specify attributes about the grammar • Character Sets are used to define the character domain for terminals • Terminals are defined using regular expressions • Rules are defined using Backus-Naur Form

  11. Defining Parameters • Used for Name, Author, Case Sensitive, Start Rule, .... • Parameter names are delimited by double quotes • Parameters • "Name", "Author", "Version", "About" are informative • "Start Symbol" specifies the initial / start rule in the grammar

  12. Parameters

  13. Example Parameters "Name"    = 'My Programming Language' "Version" = '1.0 beta' "Author"  = 'John Q. Public' "About"   = 'This is a test declaration.' | 'Multiple lines are available' | 'by using the "pipe" symbol' "Case Sensitive" = 'False' "Start Symbol" = <Statement>

  14. Defining Sets • Character sets are used to aid the construction of regular expressions used to define terminals • Literal sets of characters are delimited using ‘[’ and ‘]’ • Names of user-defined sets are delimited by ‘{’ and ‘}’ • Sets can be defined by adding and subtracting previously declared sets

  15. Example Sets

  16. Pre-defined Character Sets • There are many sets of characters which are not accessible via keyboard, or so commonly used that it would be repetitive and time-consuming to redefine in each grammar • GOLD meta-language contains a collection of useful pre-defined sets • These include sets often used for defining terminals as well as characters not accessible via keyboard

  17. Individual Characters • Some control characters that cannot be specified on a standard keyboard

  18. Commonly used Character Sets {Digit} {Letter} {Alphanumeric} {Printable} {Whitespace} {Letter Extended} {Printable Extended} {ANSI Mapped} {ANSI Printable}

  19. Unicode Character Sets • GOLD meta-language contains 43 pre-defined Unicode character sets • The names of those sets are based on standard names of the Unicode Consortium

  20. Comments • GOLD meta-language allows both line comments and block comments

  21. Defining Terminals • Terminals are used to define reserved words, symbols, and recognized patterns (identifiers) in a grammar • Each terminal is defined using a regular expression which is used to construct the Deterministic Finite Automata used by the tokenizer • Implicit declaration of frequently used reserved words and symbols

  22. Example Terminals

  23. Defining Rules • Use Backus-Naur Form • Nonterminals are delimited by angle brackets < and > • Terminals are delimited by single quotes or not delimited at all

  24. Example: Lists • Lists are specified using recursive rules Recursion

  25. Example: Optional Rules • Optional rules are specified with a production containing no terminals • This allows the developer to both specify a list containing 0 or more members zero or more Optional Rule

  26. Example: LISP Grammar

  27. Example: LISP Grammar Parameters Initial Rule

  28. Example: LISP Grammar Set Definition Set Literal

  29. Example: LISP Grammar Terminal Definition

  30. Example: LISP Grammar Rules Recursive Rule Optional Rule

  31. Compiled Grammar Table File • A file format designed to store parse tables and other information generated by the Builder • Design considerations • Easy to implement on different platforms • Flexibility for data structures to be added or expanded • Room for future growth (additional new types of data)

  32. .cgt File Structure • The file consists of a number of records • Each record contains a number of entries

  33. .cgt Record • The header contains name and version info • A record has the following format

  34. Parameter Record • Parameter record which only occurs once in the .cgt file. It contains information about the grammar as well as attributes that affect how the grammar functions. The record is preceded by a byte field contains the value 80, the ASCII code for the letter 'P'.

  35. Table Size Record • Table size record : that appears before any records containing information about symbols, sets, rules or state table information. The first field of the record contains a byte with the value 84 - the ASCII  code for the letter 'T’ Each value contains the total number of objects for each of the listed tables

  36. Other Types of Records • Character set table member • Symbol table member • Initial states (for both DFA and LALR) • Rule table member • DFA state table member • LALR state table member

  37. An Example cgt File • An example grammar "Name" = 'Example' "Version" = '1.0‘ "Author" = 'Devin Cook' "About" = 'N/A' "Start Symbol" = <Stms> <Stms> ::= <Stm> <Stms> | <Stm> <Stm> ::= if <Exp> then <Stms> end | Read Id | Write <Exp> <Exp> ::= Id '+' <Exp> | Id '-' <Exp> | Id

  38. Table Content • Symbol Table ======================================== Symbol Table ======================================== Index Name ----- ------------ 0 (EOF) 1 (Error) 2 (Whitespace) 3 '-' 4 '+' 5 end 6 Id 7 if 8 Read 9 then 10 Write 11 <Exp> 12 <Stm> 13 <Stms>

  39. Table Content (2) • Rules ======================================== Rules ======================================== Index Name ::= Definition ----- ------ --- ------------------------ 0 <Stms> ::= <Stm> <Stms> 1 <Stms> ::= <Stm> 2 <Stm> ::= if <Exp> then <Stms> end 3 <Stm> ::= Read Id 4 <Stm> ::= Write <Exp> 5 <Exp> ::= Id '+' <Exp> 6 <Exp> ::= Id '-' <Exp> 7 <Exp> ::= Id

  40. Table Content (3) • Character Set Table ======================================== Character Set Table ======================================== Index Characters ----- --------------------------------- 0 {HT}{LF}{VT}{FF}{CR}{Space}{NBSP} 1 + 2 - 3 Ee 4 Ii 5 Rr 6 Tt 7 Ww 8 Nn 9 Dd 10 Ff 11 Aa 12 Hh

  41. Table Content (3) • DFA states ======================================== DFA States ======================================== Index Description Character Set -------- ------------------- ------------- 0 Goto 1 0 Goto 2 1 Goto 3 2 Goto 4 3 Goto 7 4 Goto 10 5 Goto 14 6 Goto 18 7 1 Goto 1 0 Accept (Whitespace) …………

  42. Table Content (4) • LALR states ======================================== LALR States ======================================== Index Configuration/Action -------- ------------------------------------ 0 if Shift 1 Read Shift 9 Write Shift 11 <Stm> Goto 13 <Stms> Goto 17 1 <Stm> ::= if · <Exp> then <Stms> end Id Shift 2 <Exp> Goto 7 …………

  43. cgt File for the grammar • To illustrate, only one of each record type is included

  44. The Remaining Builder Features • Besides meta-language and .cgt file, • Skeleton program creation for the Engine from program templates • Interactive source string testing • Display of various parse table information • Export parse tables to a web page, XML file, or formatted text

  45. Application Layout Online Help Toolbar Grammar Editor Next Button Status Message

  46. Program Templates • When developing the Engine which is interacting with tables of rules and symbols in the .cgt file, manually typing constant definitions can be tedious and problematic • Program templates are designed to help automate the Engine development • The Builder can use a program template to create a “skeleton program” for an implementation of the Engine

  47. Program Templates (contd.) • Skeleton program contains • Necessary declarations of constants and variables • Function calls • Case statements, pre-processor statements • Ready-to-use programs • Notation designed to not conflict with known languages • Program templates are saved in a subfolder

  48. Display of Symbol Table • Symbol table display

  49. Display of Rule Table • Rule table display

  50. Display of Log Information • Log info: general information about the number of symbols, which ones were defined implicitly, table counts, and any errors that occur

More Related