1. Graph-Cuts Michael Bleyer LVA Stereo Vision

2. What happened last time? (1) • We have defined an energy function to measure the quality of a disparity map D: where • m(p,dp) computes color dissimilarity for matching pixel p at disparity dp • N denotes all spatial neighbors in 4-connectivity • s() is the smoothness function. We use the Potts model: • This energy function is important for many computer vision problems. 0 if dp = dq P otherwise.

3. What happened last time? (2) • Smoothness interactions define a graph known as 4-connected grid. • Computing the energy optimum on the 4-connected grid is an np-complete problem. • We have learned about dynamic programming: • Computes exact energy optimum • Requires the graph to be a tree=> We had to remove smoothness interactions 4-Connected Grid

4. What is Going to Happen Today? • Just one point on the agenda: • Graph-Cuts

5. What is Graph-Cuts? • Powerful optimization method. • Finds strong local minima of our np-complete energy function. • Graph-cuts have been around in computer vision for quite some time (e.g. [Roy,ICCV98]). • I will speak about modern graph-cuts, i.e. move making algorithms

6. Move Making Algorithms β γ • We are given a labeled image as input. • (In our case, the image is labeled with disparity values, i.e. label α can for example mean a disparity of 10 pixels.) • We want to modify the assignment of pixels to labels to obtain a better solution, i.e. one of lower energy. • An operation that changes labels is called a move. • We will learn about 3 types of moves: • αβ-swap • α-expansion • fusion move α Current labeling Move New labeling (preferably of lower energy than current labeling)

7. αβ-Swap [Boykov,PAMI01] β γ α • Select two labels: α and β. • A pixel that is assigned to α in the current labeling can either: • switch its label to β or • keep its old label α in the new labeling. • Analogously, a pixel that is currently assigned to β can either: • switch its label to α or • keep its old label β in the new labeling. • Simply spoken: • Some pixels that had the label α are now assigned to β. • Some pixels that had the label β are now assigned to α. Current labeling β γ α • One possible labeling after αβ-swap

8. α-Expansion [Boykov,PAMI01] β γ α • Select one label: α. • Any pixel can either • switch its label to α or • keep its old label. • More global than αβ-swap: • All pixels can change their labels simultaneously. • In experiments, α-expansion moves typically outperform αβ-swaps. • We will therefore concentrate on α-expansions. Current labeling β γ α • One possible labeling after α-expansion

9. The Key Problem β α γ α-exp 2 α-exp 1 α β γ • There is an extremely large number of possible α-expansions. • The key challenge is to find the “best” α-expansion, i.e. the one that leads to the largest decrease of our energy. • Good news: • For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph. α β β α-exp 3 α-exp 4 γ γ Current labeling α α

10. E = 2000 E = 1000 The Key Problem β α γ E = 1000 α-exp 2 α-exp 1 α β γ • There is an extremely large number of possible α-expansions. • The key challenge is to find the “best” α-expansion, i.e. the one that leads to the largest decrease of our energy. • Good news: • For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph. α β β α-exp 3 α-exp 4 γ γ Current labeling α α We should take this one E = 500 E = 750

11. The Key Problem β α γ α-exp 2 α-exp 1 α β γ • There is an extremely large number of possible α-expansions. • The key challenge is to find the “best” α-expansion, i.e. the one that leads to the largest decrease of our energy. • Good news: • For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph. α β β α-exp 3 α-exp 4 γ γ Current labeling α α

12. Iterative Algorithm – α-Expansion • Let us for now assume that we know how to compute the optimal α-expansion. • We can incorporate the α-expansion as follows. Iterative Algorithm: • Start with an arbitrary labeling f. • Loop (e.g. 3 times) • For each allowed label α: • Find f* = argmin E(f’) among f’ within one α-expansion of f • f := f* • Comment: • Note that we compute the optimal α-expansion. • Therefore, the energy will either decrease after α-expansion or stay the same (not changing the labeling at all is a feasible α-expansion). • The algorithm will in any case converge to a (strong) local energy optimum.

13. Iterative Algorithm – Example Video (α-expansions for stereo matching)

14. Computing the Optimal α-Expansion • There are 3 things you have to do to find the optimal α-expansion via graph-cuts: • Write your energy as a pseudo-boolean function • Construct a graph that represents your boolean function • Compute the Minimum Cut in this graph • These steps are discussed in the following.

15. Writing the Energy as a Pseudo-Boolean Function (1) • We associate a boolean variable xp with each pixel p where: • xp = 0 means that pixel p keeps its old label after α-expansion • xp = 1 means that pixel p takes label α after α-expansion • For example, if this is the current labeling:then x = leads to the label configuration:after α-expansion. • We can represent all possible α-expansions by the boolean variables x. β β β γ γ 1 1 0 0 1 α α β γ α

16. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling:

17. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: Potts model (Impose penalty P if p and q have different labels) Dissimilarity function

18. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:

19. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:

20. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:

21. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq: We call these terms unary terms, since they depend on one variable. We call this term a pairwise term, since it depends on two variables.

22. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where:

23. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where:

24. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where: β β β α α β α α

25. Writing the Energy as a Pseudo-Boolean Function (2) We have to find the settings of binary variables xp and xq that minimize the energy. This comes next. β β • Let us assume we have two pixels p and q. • Both pixel are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where: β β β α α β α α

26. The Min-Cut Problem source 2 9 • We have two dedicated nodes, the source and the sink. 2 p q 1 5 4 sink ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

27. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where 2 p q 1 5 4 T sink ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

28. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where • The cut consists of all edges that lead from S to T. 2 p q 1 5 4 T sink ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

29. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where • The cut consists of all edges that lead from S to T. • The costs of a cut are the sum of weights of these edges. 2 p q 1 5 4 T sink Costs: 5 + 2 + 9 = 16 ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

30. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where • The cut consists of all edges that lead from S to T. • The costs of a cut are the sum of weights of these edges. • The minimum cut is the cut of minimum costs among all possible cuts. 2 p q 1 5 4 T sink Costs: 2 + 1 + 4 = 7 ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

31. The Min-Cut Problem • The min-cut problem has been extensively studied in graph theory. • There exists a variety of algorithms that • Can find the exact solution • Are computationally very fast. • Side notes: • The min-cut problem and the max-flow problem are dual problems: => Solving min-cut also gives the solution for max-flow and vice versa. • Max-flow and min-cut are therefore often used synonymously. • If you are interested in algorithms for computing min-cut/max-flow: • Read [Boykov,PAMI04]

32. The Min-Cut Problem Nice, but does this help us to optimize our pseudo-boolean function? • The min-cut problem has been extensively studied in graph theory. • There exists a variety of algorithms that • Can find the exact solution • Are computationally very fast. • Side notes: • The min-cut problem and the max-flow problem are dual problems: => Solving min-cut also gives the solution for max-flow and vice versa. • Max-flow and min-cut are therefore often used synonymously. • If you are interested in algorithms for computing min-cut/max-flow: • Read [Boykov,PAMI04]

33. Optimization of our Pseudo-Boolean Function source • We insert a node for each pixel. p q sink

34. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. => xp = 0 p q sink

35. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 => xp = 0 p q => xq = 1 T sink

36. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 • We adjust the edges so that the costs of the cut are equal to the energy of our binary variables x. => xp = 0 p q => xq = 1 T sink The costs of this cut have to be equal to the energy of xp = 0 and xq= 1.

37. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 • We adjust the edges so that the costs of the cut are equal to the energy of our binary variables x. • The minimum cut therefore also represents the minimum of our energy. => xp = 0 p q => xq = 1 T sink The costs of this cut have to be equal to the energy of xp = 0 and xq= 1.

38. Optimization of our Pseudo-Boolean Function S How can we do this for our example? source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 • We adjust the edges so that the costs of the cut are equal to the energy of our binary variables x. • The minimum cut therefore also represents the minimum of our energy. => xp = 0 p q => xq = 1 T sink The costs of this cut have to be equal to the energy of xp = 0 and xq= 1.

39. Optimization of our Pseudo-Boolean Function source • Our unary terms: • Our pairwise term: p q sink

40. Optimization of our Pseudo-Boolean Function source m(p,α) • Our unary terms: • Our pairwise term: p q m(p,β) sink

41. Optimization of our Pseudo-Boolean Function source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: p q m(q,β) m(p,β) sink

42. Optimization of our Pseudo-Boolean Function source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) sink

43. Optimization of our Pseudo-Boolean Function Let us check whether this graph really represents our energy. source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) sink

44. Optimization of our Pseudo-Boolean Function => xp = 0, xq = 0 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(0)+Eq(0)+Ep,q(0,0) = m(p,β)+m(q,β)+0 Cut Costs:C = m(p,β)+m(q,β)

45. Optimization of our Pseudo-Boolean Function => xp = 0, xq = 1 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(0)+Eq(1)+Ep,q(0,1) = m(p,β)+m(q,α)+P Cut Costs:C = m(p,β)+m(q,α)+P

46. Optimization of our Pseudo-Boolean Function => xp = 1, xq = 0 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(1)+Eq(0)+Ep,q(1,0) = m(p,α)+m(q,β)+P Cut Costs:C = m(p,α)+m(q,β)+P

47. Optimization of our Pseudo-Boolean Function => xp = 1, xq = 1 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(1)+Eq(1)+Ep,q(1,1) = m(p,α)+m(q,α)+0 Cut Costs:C = m(p,α)+m(q,α)

48. Optimization of our Pseudo-Boolean Function => xp = 1, xq = 1 S We have shown that the graph represents our energy. source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(1)+Eq(1)+Ep,q(1,1) = m(p,α)+m(q,α)+0 Cut Costs:C = m(p,α)+m(q,α)

49. What Energy Function Can be Optimized via Graph-Cuts? • Not every boolean energy function can be represented by a graph! • The pairwise terms have to fulfill the following constraint [Kolmogorov,PAMI04]: • In our example, this has been the case: • If there is at least one pairwise term in the boolean energy function that violates this constraint, the energy is said to be non-submodular. • Otherwise, it is called submodular. • Optimizing non-submodular energies is an np-complete problem.=> Computing the optimal α-expansion becomes very difficult.  (but not impossible)

50. Max-Flow/Min-Cut Library • “I would like to use graph-cuts, but I do not want to mess around with graphs.” • Good news: • You don’t have to. • It is sufficient to define your energy as a pseudo-boolean function. • You can then download the Max-Flow/Min-Cut library from http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software.html • The library will: • Construct the graph that represents your boolean function • Compute the min-cut • Provide you the optimal labeling • See example on next slide.