A Joint Conditional Model for Grouping And Labeling Ink Fragments

1. 1 A Joint Conditional Model for Grouping And Labeling Ink Fragments MSRC Internship Talk Philip Cowans Mentor: Martin Szummer June 10th 2004

2. 2 Overview Task overview Description of the model Labeling only model Labeling and grouping model Training and inference Examples and results Further work

3. 3 The Task – Interpreting Ink

4. 4 Previous Work Kumar, Hebert: Discriminative random fields. Qi, Szummer: Bayes Conditional Networks Peña Centeno, Svensén, Bishop: Variational CRFs. Saund: Perceptual organization.

5. 5 Task Stages Divide the input data into ink fragments Then… Group the fragments into perceptually salient objects And… Label each object as either a container or connector

6. 6 Example - Input

7. 7 Example - Fragmentation

8. 8 Example – Grouping / Labeling

9. 9 Fragmentation Input data consists of ink strokes, described by sampled pen locations. Strokes may span multiple objects. Divide strokes into straight (within a tolerance) fragments. Fragments are assumed to lie in a single object.

10. 10 The Model Use a Conditional Random Field: Undirected graphical model conditioned on observed data. No need to model ink data (which we already know). For now, just consider labeling.

11. 11 Potentials

12. 12 Nonlinearities Exponential: Probit: (Gaussian CDF) Epsilon is a ‘noise’ parameter.

13. 13

14. 14 Graph Construction Construct a graph with one vertex per ink fragment. Add edges between nearby fragments. Triangulate graph. Calculate interaction potentials for all edges.

15. 15 Graph Construction II Speed is dependent on the tree-width of the graph. By constraining the tree-width it should be possible to limit the complexity of the algorithm.

16. 16 Features Spatial features Angles, lengths, distances… Temporal features Stroke ordering, same stroke… Template features T-Junction identification… 61 observation, 37 interaction features in total

17. 17 Priors Three feature categories: Binary Counting Continuous Continuous features are replaced by histograms Use a correlated Gaussian prior on continuous features

18. 18 Example Weights

19. 19 Labeling And Grouping Use a slight modification of the model Now three interaction categories: Same object (implies same label) Different object, same label Different object, different label

20. 20 Advantages Of This Approach Joint labeling Makes use of contextual information Simultaneous labeling and grouping Processes can reinforce each other Handles groups directly Doesn’t define groups via labeling Results in a ‘cleaner’ model

21. 21 Training Train by finding MAP weights on example data. Training data set was 40 diagrams, from 17 subjects with a total of 2157 fragments. Maximise using quasi-Newton gradient ascent (BFGS in MATLAB). Evaluate function and gradient using message passing.

22. 22 Message Passing Just like standard sum-product, but we are summing over groupings and labelings together rather than just labelings. Fortunately efficient computation via message passing is still possible. Messages are now a list of possible groupings and labelings of the separator variables.

23. 23 Message Passing II Message update rule: Marginals:

24. 24 Message Passing III

25. 25 Inference Unseen data is processed by finding the most probable grouping and labeling. Use the max-product variant of the message passing algorithm. Messages are now the ‘best you can do’ given the separator configuration.

26. 26 Example - Raw Ink

27. 27 Example – Labeling Only

28. 28 Example – Labeling And Grouping

29. 29 Example 2 - Raw Ink

30. 30 Example 2 – Labeling Only

31. 31 Example 2 – Labeling And Grouping

32. 32 MATLAB Demo

33. 33 Grouping Error Measured using the metric: Not really sure how this relates to ‘perceptual’ error. There’s potentially a better way of doing this via mutual information.

34. 34 Results: Accuracy ---- Grouping error should have a more intuitive measure---- Grouping error should have a more intuitive measure

35. 35 Results: Speed Median evaluation time for labeling only: 0.35 seconds. Median evaluation time for labeling and grouping: 13 seconds. (And that’s not very optimised, using MATLAB!)

36. 36 Conclusions Exact inference is possible in reasonable time. This approach is capable of providing high-quality results. Joint labeling improves performance. Labeling and grouping simultaneously helps too.

37. 37 Further Work Multiclass model: Dashed lines, text, arrows, … Relationships: Which containers are connected together. More ‘global’ features: Closed paths, convex containers, … Possibly via re-ordering of N-best lists. More data, better features, …

38. 38 Finally… Thanks to everyone who has made my stay at Microsoft so enjoyable.

39. 39 Observation Weights

40. 40 Interaction Weights