Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic

1. Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic Del Bimbo, E. Vicario and D. Zingoni IEEE Transactions on Knowledge and Data Engineering, 7(4), August 1995.

2. Outline • Overview • Spatial logic • Region-based formulation • Object-based formulation • Temporal logic • Sequence Description using STL • A System for Image Sequence Retrieval

3. Overview:Spatio-temporal logic (STL) • A symbolic representation of spatio-temporal relationships between objects within image sequences. • Completely based on logic. • Two main components: • Spatial logic for individual scenes • Temporal logic for scene sequences

4. Spatial logic • Expresses geometric ordering and relationships between objects • In fact, between the projections of the objects in N-dimensional space. • Based on knowing the points (or regions) that are covered by the object.

5. Spatial logic (cont.) • A scene is a represented as a triple (V, Obj, F): • V: N-dimensional space • Mostly N=2 or 3 • Obj: a set of objects in V • F: a mapping from Obj to the powerset of V, i.e. the set of all subsets of V. • Associates each object with a set of points over which it stands, i.e. F(p) := { r  V | p stands over r)

6. Spatial logic (cont.) • V can be partitioned into a grid of rectangular regions. • A scene will be then represented as a discrete scene. • And F will be then a mapping between objects and the regions they cover.

7. Example e2 I24 I23 p2 I22 I21 p1 e1 I11 I12 I13 I14

8. Spatial logic (cont.) • Two possible formulations: • Region-based formulation • Object-based formulation

9. Region-based formulation • Express the positioning of a set of objects with respect to a single region in the scene. • A spatial assertion has the form: where • In addition, there are some shorthand operations: • spatial-eventually (can be +ve or –ve) • spatial-always (can be +ve or –ve)

10. Semantics • (SE, J , en ) |= p- iff the orthogonal projections on axis en of region g(J) and object p have a nonempty intersection • (SE, J , en ) |= - iff the spatial assertion (SE, J , en ) |=  does not hold • (SE, J , en ) |= 12- iff both (SE, J , en ) |= 1 and (SE, J , en ) |= 2

11. Semantics (Cont.) • (SE, J , en ) |= 1untS+2- iff there esists a region g(J’) which is reached from region g(J) moving along the positive direction of axis en such that assertion 2 holds in region g(J’) and 1 holds in all the regions from g(J) to g(J’) • (SE, J , en ) |= 1untS-2- iff there esists a region g(J’) which is reached from region g(J) moving along the negetive direction of axis en such that assertion 2 holds in region g(J’) and 1 holds in all the regions from g(J) to g(J’)

12. Shorthands • (SE, J , en ) |= ◊S+/--  will eventually hold in some of the regions encountered moving from region g(J) along axis en • (SE, J , en ) |= S+/--  holds in all the regions encountered moving from region g(J) along axis en

13. Example em g(J) g(K) p en

14. Object-based formulation • Express the relationships between objects. • A spatial assertion has the form: ( read:  holds in q) where q is an object, expresses that (SE, J , en ) |= holds in any region g(J) containing q • Five different situations occur as follows:

15. q p p p p (SE, q , x ) |= p - expresses that the projection of q on the x axis is entirely contained in the projection of p

16. (SE, q , x ) |= ◊S+ p - expresses that every point of the projection of q on the x axis has at least one point of the projection of p to its right side q p p p p p

17. (SE, q , x ) |= q untS+ p - expresses that the projection of q extends until one point is found that belongs to the projection of p q q p p p p

18. (SE, q , x ) |=  p - expresses that the projection of q on the x axis does not intersect with the projection of p q p p

19. - expresses that some of the regions of q are aligned with some of the regions of p ((SE, q , x ) |=  p ) q q p p p p p p p p p p p

20. Temporal logic • In general, used to express the ordering of states or actions in time. • A temporal (or state assertion) has the form: where

21. Semantics • (, k) |= iff the spatial assertion  holds in the kth scene of sequence  • (, k) |= iff the temporal assertion (, k) |=  does not hold • (, k) |= 12iff both (, k) |= 1 and (, k) |= 2 hold • (, k) |= 1untt2iff 2 holds in a scene with index k’ > k and 1 holds in all the scenes form k to k’

22. Shorthands Two shorthands can be derived • (, k) |= ◊tmeans that  will hold in some scene subsequent to the kth one◊t := true untt • (, k) |= tmeans that  holds in all the scenesubsequent to the kth onet := ( true untt(  ) )

23. Describing Scene Sequences S1 S2 S3 S4 S5 1 1 1 2 • each node is labeled with the spatial assertion, satisfied in its scene • S1, S2 , S3, S4 and S5 are different states on a time line • (, k) |= ( 1 untt2 ) holds for …………… • (, k) |= ◊t2 holds for ……………… • (, k) |= t ( 1 2 ) holds for ……………..

24. Sequence Description Using STL • 3D scene-based descriptions are used to avoid ambiguity • Two different descriptions are possible 1) Observer-centered description 2) Object-centered description

25. Two Different Descriptions • Observer-centered description -- images of the same scene taken from different viewpoints -- valid only when the camera is fixed • Object-centered description -- how one object sees the rest of the scene-- does not depend on the camera position

26. Example

27. Example • the spatial positions of the house, h as perceived by the car, c along the course of the scene • considering object-centered descriptions and referring to the reference system Ec associated with the car c • 1 := (SEc , c, x) |= h • 2 :=  ((SEc , c, x) |= h ) • 3 := (SEc , c, y) |= h • 4 := (SEc , c, z) |= h • 5 :=  ((SEc , c, z) |= h ) The description of the scene using the above assertions can be as follows:

28. A System for Image Sequence Retrieval A. Iconic Querying 1) Visual Querying 2) Automatic Parsing B. Retrieval from Database1) Sequence Representation (created manually) 2) Sequence Retrieval

29. Retrieval from Database • To allow for different level of details in queries, three levels of operations are used. • Level 1: Least details • 3 possibilities (before, overlapping, after) • Level 2: Modest details • 6 possibilities • Level 3: Very fine details • 13 possibilities (STL full representational power)

30. Level-1 Operators

31. Level-2 Operators

32. User Interface of the Retrieval System

33. Example 1 (Query Specification)

34. Result of Retrieval Only two frames of the Sequence are shown

35. Example 2 (Query Specification) Definition of motion of one car during the play back of the other

36. Result of Retrieval Two frames of the sequence