Constraint Based Systems

1. Constraint Based Systems Reading: Chapter 6 Chess article (see links)

2. Agenda • Finishing up search for machine translation • Constraint satisfaction

3. Hill climbing function HillClimbing(problem, initial-state, queuing-fn) node← MakeNode(initial-state(problem)); whileTdo next ← Best(SearchOperator-fn(node,cost-fn)); if(IsBetter-fn(next, node)) then continue; else if(GoalTest(node)) then return node; else exit; end while return Failure; MT (Germann et al., ACL-2001) node ← targetGloss(sourceSentence); whileTdo next ← Best( LocallyModifiedTranslationOf(node)); if(IsBetter(next, node)) then continue; else print node; exit; end while

4. Types of changes • Translate one or two words (j1e1j2e2) • Translate and insert (j e1 e2) • Remove word of fertility 0 (i) • Swap segments (i1 i2 j1 j2) • Join words (i1 i2)

5. Example • Total of 77,421 possible translations attempted

8. How to search better? • MakeNode(initial-state(problem)) • RemoveFront(Q) • SearchOperator-fn(node, cost-fn); • queuing-fn(problem, Q, (Next,Cost));

9. Example 1: Greedy Search MakeNode(initial-state(problem)) Machine Translation (Marcu and Wong, EMNLP-2002) node ← targetGloss(sourceSentence); whileTdo next ← Best( LocallyModifiedTranslationOf(node)); if(IsBetter(next, node)) then continue; else print node; exit; end while

10. Model validation Model stress-testing Climbing the wrong peak What sentence is more grammatical? 1. better bart than madonna , i say 2. i say better than bart madonna , Can you make a sentence with these words? a and apparently as be could dissimilar firing identical neural really so things thought two

11. Language-model stress-testing • Input: bag of words • Output: best sequence according to a linear combination of an • ngram LM • syntax-based LM (Collins, 1997)

12. Size: 10-25 words long • Best searched • 51.6: and so could really be a neural • apparently thought things as • dissimilar firing two identical • Original word order • 64.3: could two things so apparently • dissimilar as a thought and neural • firing really be identical Size: 3-7 words long • Best searched • 32.3:i say better than bart madonna , • Original word order • 41.6: better bart than madonna, i say SBLM*: trained on an additional 160k WSJ sentences.

13. Cryptograms QFL HCVPS PX V ANSWLCEZK NCJVS; PQ XQVCQX QFL BPSZQL RNZ JLQ ZT PS QFL BNCSPSJ VSW WNLX SNQ XQNT ZSQPK RNZ JLQ QN QFL NEEPGL CNHLCQ ECNXQ

14. Cryptograms QFL HCVPS THE BRAIN PX V ANSWLCEZK NCJVS; PQ XQVCQX IS A WONDERFUL ORGAN; IT STARTS QFL BPSZQL RNZ JLQ ZT PS QFL BNCSPSJ THE MINUTE YOU GET UP IN THE MORNING VSW WNLX SNQ XQNT ZSQPK RNZ JLQ QN AND DOES NOT STOP UNTIL YOU GET TO QFL NEEPGL THE OFFICECNHLCQ ECNXQ ROBERT FROST

15. Constraint Satisfaction Problems Standard representation allows a general-purpose approach to search • Set of variables, X1, X2 … Xn • Each variable has domain Di of possible values • Set of constraints, C1,C2, …Cn • State = assignment of values to some or all variables • Xi=v,i , Xj=vj • Solution = complete assignment that satisfies all constraints • Some solutions maximize an objective function

16. Cryptograms QFL HCVPS PX V ANSWLCEZK NCJVS; PQ XQVCQX QFL BPSZQL RNZ JLQ ZT PS QFL BNCSPSJ VSW WNLX SNQ XQNT ZSQPK RNZ JLQ QN QFL NEEPGL CNHLCQ ECNXQ

17. Cryptograms QFL HCVPS THE BRAIN PX V ANSWLCEZK NCJVS; PQ XQVCQX IS A WONDERFUL ORGAN; IT STARTS QFL BPSZQL RNZ JLQ ZT PS QFL BNCSPSJ THE MINUTE YOU GET UP IN THE MORNING VSW WNLX SNQ XQNT ZSQPK RNZ JLQ QN AND DOES NOT STOP UNTIL YOU GET TO QFL NEEPGL THE OFFICECNHLCQ ECNXQ ROBERT FROST

18. CSP algorithm Depth-first search often used • Initial state: the empty assignment {}; all variables are unassigned • Successor fn: assign a value to any variable, provided no conflicts w/constraints • All CSP search algorithms generate successors by considering possible assignments for only a single variable at each node in the search tree • Goal test: the current assignment is complete • Path cost: a constant cost for every step

19. Local search • Complete-state formulation • Every state is a compete assignment that might or might not satisfy the constraints • Hill-climbing methods are appropriate

20. Discrete/finite domains Map coloring 8 queens Nonlinear constraints No general solution Unary constraints Absolute constraints Infinite domains Scheduling calendars (discrete) Schedule experiments for Hubble Linear constraints Binary/N-ary constraints Preferences Prof. McKeown prefers after 1pm Dimensions of CSPs

21. M GWD’X URNMRPR MD QD QCXRLNMCR, QNXFWZOF M QH ULMDOMDO Q KFQDOR WC ZDGRLARQL. AWWGS QNNRD

22. M GWD’X URNMRPR MD QD A BCD’E A AD DQCXRLNMCR, QNXFWZOF M QHE A E ULMDOMDO Q KFQDOR WC AD AD D CZDGRLARQL. DB AWWGS QNNRD B D

23. M GWD’X URNMRPR MD QD A BCD’I A AD DQCXRLNMCR, QNXFWZOF M QHI A I ULMDOMDO Q KFQDOR WC AD AD D CZDGRLARQL. DB AWWGS QNNRD B D

24. M GWD’X URNMRPR MD QD A BCD’O A AD DQCXRLNMCR, QNXFWZOF M QHO A O ULMDOMDO Q KFQDOR WC AD AD D CZDGRLARQL. DB AWWGS QNNRD B D

25. M GWD’X URNMRPR MD QD A BCD’U A AD DQCXRLNMCR, QNXFWZOF M QHU A U ULMDOMDO Q KFQDOR WC AD AD D CZDGRLARQL. DB AWWGS QNNRD B D

26. M GWD’X URNMRPR MD QD A BCE’ A AE EQCXRLNMCR, QNXFWZOF M QHA ULMDOMDO Q KFQDOR WC AE AE E CZDGRLARQL. EB AWWGS QNNRD B E

27. What additional knowledge about language would be helpful?

28. General purpose methods for efficient implementation • Which variable should be assigned next? • in what order should its values be tried? • Can we detect inevitable failure early? • Can we take advantage of problem structure?

29. Order • Choose the most constrained variable first • The variable with the fewest remaining values • Minimum Remaining Values (MRV) heuristic • What if there are >1? • Tie breaker: Most constraining variable • Choose the variable with the most constraints on remaining variables

30. Order on value choice • Given a variable, chose the least constraining value • The value that rules out the fewest values in the remaining variables

31. M GWD’X URNMRPR MD QD I I I AQCXRLNMCR, QNXFWZOF M QHA I A I A ULMDOMDO Q KFQDOR WCI I A A ZDGRLARQL.A AWWGS QNNRD A

32. M GWD’X URNMRPR MD QD I N I IN ANQCXRLNMCR, QNXFWZOF M QHA I A I A ULMDOMDO Q KFQDOR WCIN IN A AN ZDGRLARQL.NA AWWGS QNNRD A N

33. Forward Checking • Keep track of remaining legal values for unassigned variables • Terminate search when any variable has no legal values

34. M GWD’X URNMRPR MD QD I I I AQCXRLNMCR, QNXFWZOF M QHA I A I A ULMDOMDO Q KFQDOR WCI I A A ZDGRLARQL.AWhen M I selected D (F N S T) When QA selected D (N S T)

35. M GWD’X URNMRPR MD QD I DON’T EL I E E IN ANQCXRLNMCR, QNXFWZOF M QHA FTERL I FE ALTHO GH I AM ULMDOMDO Q KFQDOR WCRIN G I NG A HANGE OFZDGRLARQL.NDERWEAR AWWGS QNNRD WOODYALLEN

36. M GWD’X URNMRPR MD QD I DON’T CEL I E E IN ANQCXRLNMCR, QNXFWZOF M QHA FTERL I FE ALTHO GH I AM ULMDOMDO Q KFQDOR WCCRIN G I NG A HANGE OFZDGRLARQL.NDERWEAR Select UC -> the domain of P is empty. Requires backtracking even before taking the next branch. Avoids selection Z, K next before discovering the error.

37. M GWD’X URNMRPR MD QD I DON’T BEL I E VE IN ANQCXRLNMCR, QNXFWZOF M QHA FTERL I FE ALTHO UGH I AM ULMDOMDO Q KFQDOR WCBRIN G I NG A CHANGE OFZDGRLARQL.UNDERWEAR AWWGS QNNRD WOODYALLEN

38. Speed-ups

39. Other types of improvements • Constraint propagation • Propagate implications of a constraint on one variable onto other variables. Not only values, but constraints between other variables • Intelligent backtracking • Conflict-directed backtracking: Save possible conflicts for a variable value on assignment. • Back jump to assignment of values

40. Crossword Puzzles