Lexical Analysis, Regular Expressions & Finite State Machines

Lexical Analysis, Regular Expressions & Finite State Machines

Processing English • Consider the following two sentences • Hi, I am 22 years old. I come from Alabama. • 22 come Alabama I, old from am. Hi years I. • Are they both correct? • How do you know? • Same words, numbers and punctuation • What did you do first? • Find words, numbersand punctuation • Then, check order (grammar rules)

Finding Words and Numbers • How did you find words, numbers and punctuation? • You have a definition of what each is, or looks like • For example, what is a number? a word? • Although your are a bit more agile, the process was: • Start with first character • If letter, assume word; if digit, assume number • Scan left to right 1 character at a time, until punctuation mark (space, comma, etc.) • Recognize word or number • If no more characters, done; otherwise return to 1

Processing Code How do you process the following? What are the main parts in which to break the input? Schemes: childOf(X,Y) marriedTo(X,Y) Facts: marriedTo('Zed','Bea'). marriedTo('Jack','Jill'). childOf('Jill','Zed'). childOf('Sue','Jack'). Rules: childOf(X,Y) :- childOf(X,Z), marriedTo(Y,Z). marriedTo(X,Y) :- marriedTo(Y,X). Queries: marriedTo('Bea','Zed')? childOf('Jill','Bea')? • void quote() { • print( • "To iterate is human, to recurse divine." + • " - L. Peter Deutsch" • ); • } defaddABC(x): s = “ABC” return x + s addABC(input(“String: ”))

Example • defaddABC ( x ) : • s = “ABC” • return x + s • addABC ( input ( “String: ” ) )

What are the Parts? • They are called TOKENS • Process similar to English processing • Lexical Analysis • Input: • A program in some language • Output: • A list of tokens • (type, value, location)

Example Revisited

Program Code Program Tokens Internal Data Code Lexical Analyzer Parser Code Generator Syntax Analysis Error messages Program Compilation • Lexical Analysis is first step of process Compiler Or Interpreter (Executed directly) Keywords String literals Variables …

Token Specification • Regular Expressions • Pattern description for strings • Concatenation: abc -> “abc” • Boolean OR: ab|ac -> “ab”, “ac” • Kleene closure: ab* -> “a”, “ab”, “abbb”, etc. • Optional: ab?c -> “ac”, “abc” • One or more: ab+ -> “ab”, “abbb” • Group using () • (a|b)c -> “ac”, “bc” • (a|b)*c -> “c”, “ac”, “bc”, “bac”, “abaaabbbabbaaaaac”, etc.

RegEx Extensions • Exactly n: a3b+ -> “aaab”, “aaabb”, … • [A-Z] = A|B|…|Z • [ABC] = A|B|C • [~aA] = any character but “a” or “A” • \ = escape character (e.g., \* -> “*”) • Whitespace characters • \s, \t, \n, \v

Token Recognition • Finite State Machine • A DFSM is a 5-tuple (Σ,S,s0,δ,F) • Σ: finite, non-empty set of symbols (input alphabet) • S: finite, non-empty set of states • s0: member of S designated as start state • δ: state-transition function δ: S x Σ-> S • F: subset of S (final states, may be empty)

FSM & RegEx • abc • a(b|c) • ab* • (a(b?c))+ a b c b a c b a Note the special double-circle designation of a final/accepting state. b c a c a

Finite State Transducer • Extended FSM: • Γ: finite, non-empty set of symbols (output alphabet) • δ: state-transition function δ: S x Σ-> S x Γ • FST consumes input symbols and emits output symbols • Lexical analyzer • consume raw characters • emit tokens

CS 236 Coolness Factor! • Design our own language • Subset of Datalog (LP-like) • Build an interpreter for our language • Lexical Analyzer (Project 1) • Parser (Project 2) • Interpreter (Projects 3 and 4) • Optimization (Project 5)

Designing a Language • Define the tokens • Elements of the language, punctuation, etc. • For example, what are they in C++? • Recognize the tokens (lexical analysis) • Define the grammar • Forms of correct sentences • For example, what are they in C++? • Recognize the grammar (parsing) • Interpret and execute the program • C++ is a bit too complicated for us…

Varied World Views • fctpersonlist siblings(person x) { • return x’s siblings • } • fctint square(int x) { • return x * x • } • fctboolean succeeds(person x) { • if studies(x) return T else return F • } • fctboolean sibling(person x, person y) { • if y is x’s sibling return T else return F • } • fctboolean square(int x, int y) { • if y == x * x return T else return F • } • fctboolean succeeds(person x) { • if studies(x) return T else return F • } Look up table or oracle No concerns with efficiency

Logic Programming • Assume: all functions are Boolean • Compute using facts and rules • Facts are the known true values of the functions • Rules express relations among functions • Example: studies(x), succeeds(x) • Facts: studies(Matt), studies(Jenny) • Rule: succeeds(x) :- studies(x) • Closed-world Assumption

Logic Programming • Computing is like issuing queries • First check if it can be answered with facts • Second check if rules can be applied • Examples • studies(Alex)? • NO (neither facts nor rules to establish it) • studies(Matt)? • YES (there is fact about that) • succeeds(Jenny)? • YES (no fact, but a rule that if Jenny studies then she succeeds and a fact that Jenny studies)

Functions of Several Arguments • Examples • loves(x,y), parent(x,y), inclass(x,y) • loves(x,y) :- married(x,y) • Computing • parent(Christophe, Samuel)? • Yes, if there is a fact that matches • parent(Christophe, X)? • Yes, if there is a value of X that would cause it to match a fact – return value of X • loves(X, Y)? • Yes, if there are values of X and Y that would make this true, either by matching a fact or via rules (e.g., married(Christophe, Isabelle)) – return values of X and Y

When We Are Done Demo…

Project 1: Lexical Analyzer Define and find the tokens

Basic FST for Project 1 start <character (except <cr> and <eof>)> ‘ ‘ string <cr> or <eof> error : : - or :- :- … <space> | <tab> | <cr> white space <space> | <tab> | <cr> ident. <letter> or keywd. Special check for Keywords (Schemes, Facts, Rules, Queries) <letter> | <digit> <eof> eof <any other char> error

Implementing a FST State in Variable State in Position in Code input = readChar(); // begin in START state if (input == QUOTE) { input = readChar(); // now in STRING state while (input != QUOTE) { input = readChar(); // stay in STRING state } input = readChar(); // now in ACCEPT state } else if (input == ...) { ... other kinds of tokens ... } • state = START; • input = readChar(); • while (state != ACCEPT) { • if (state == START) { • if (input == QUOTE) { • input = readChar(); • state = STRING; • } else if (input == ...) { • ... other kinds of tokens ... • } • } else if (state == STRING) { • if (input == QUOTE) { • input = readChar(); • state = ACCEPT; • } else { • input = readChar(); • state = STRING; • } • } • }

Lexical Analysis, Regular Expressions & Finite State Machines