1 / 46

Agenda Today

Agenda Today. We will discuss a few interesting spatial data mining patterns Then come back to summarize what we have learned in this course so far. Spatial Data Management: Summary. Course Summary. 1. Introduction to Spatial Databases 2. Spatial Concepts and Data Models

griffin-mayer
Télécharger la présentation

Agenda Today

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Agenda Today • We will discuss a few interesting spatial data mining patterns • Then come back to summarize what we have learned in this course so far

  2. Spatial Data Management: Summary

  3. Course Summary • 1. Introduction to Spatial Databases • 2. Spatial Concepts and Data Models • 3. Spatial Query Languages: SQL3 • 4. Spatial Storage and Indexing: R-tree, Grid File • 5. Query Processing and Query Optimization • Strategies for range query, nearest neighbor query • Spatial joins (e.g. tree matching), cost models • 6. Spatial Network Model • 7. Spatial Data Mining • Spatial auto-correlation, co-location patterns, spatial outliers, classification methods • 8. Trends in Spatial Database (Moving Object)

  4. 1. Introduction • Traditional (non-spatial) database management systems provide: • Persistence across failures • Allows concurrent access to data • Scalability to search queries on very large datasets which do not fit inside main memories of computers • Efficient for non-spatial queries, but not for spatial queries • Non-spatial queries: • List the names of all bookstore with more than ten thousand titles. • List the names of ten customers, in terms of sales, in the year 2001 • Use an index to narrow down the search • Spatial Queries: • List the names of all bookstores with ten miles of Minneapolis • List all customers who live in Tennessee and its adjoining states • List all the customers who reside within fifty miles of the company headquarter

  5. 1. Spatial Data Examples • Examples of non-spatial data • Names, phone numbers, … • Examples of Spatial data • Census Data • NASA satellites imagery - terabytes of data per day • Weather and Climate Data • Rivers, Farms, ecological impact • Medical Imaging

  6. 2. Spatial Object Model • Object model concepts • Objects: distinct identifiable things relevant to an application • Objects have attributes and operations • Attribute: a simple (e.g. numeric, string) property of an object • Operations: function maps object attributes to other objects • Example from a roadmap • Objects: roads, landmarks, ... • Attributes of road objects: • spatial: location, e.g. polygon boundary of land-parcel • non-spatial: name (e.g. Route 66), type (e.g. interstate, residential street), number of lanes, speed limit, … • Operations on road objects: determine center line, determine length, determine intersection with other roads, ...

  7. 2. Classifying Spatial objects • Spatial objets are spatial attributes of general objects • Spatial objects are of many types • Simple • 0- dimensional (points), 1 dimensional (curves), 2 dimensional (surfaces) • Example given at the bottom of this slide • Collections • Polygon collection (e.g. boundary of Japan or Hawaii), … • See more complete list in Figure 2.2

  8. 2. Spatial Object Types in OGIS Data Model Fig 2.2: Each rectangle shows a distinct spatial object type

  9. 2. Classifying Operations on spatial objects in Object Model • Classifying operations • Set based: 2-dimensional spatial objects (e.g. polygons) are sets of points • A set operation (e.g. intersection) of 2 polygons produce another polygon • Topological operations: Boundary of USA touches boundary of Canada • Directional: New York city is to east of Chicago • Metric: Chicago is about 700 miles from New York city.

  10. 2. Specifying topological operation Fig 2.3: 9 intersection matrices for a few topological operations

  11. 2. Conceptual DM: The ER Model • 3 basic concepts • Entities have an independent conceptual or physical existence. • Examples: Forest, Road, Manager, ... • Entities are characterized by Attributes • Example: Forest has attributes of name, elevation, etc. • An Entity interacts with another Entity through relationships. • Road allow access to Forest interiors. • This relationship may be name “Accesses”

  12. 2. ER Diagram for “State-Park” Fig 2.4

  13. Pictorial Enhanced ER Diagram for “State-Park

  14. 2. Mapping ER to Relational • Highlights of translation rules • Entity becomes Relation • Attributes become columns in the relation • Multi-valued attributes become a new relation • includes foreign key to link to relation for the entity • Relationships (1:1, 1:N) become foreign keys • M:N Relationships become a relation • containing foreign keys or relations from participating entities

  15. 3. Three Components of SQL? • Data Definition Language (DDL) • Creation and modification of relational schema • Schema objects include relations, indexes, etc. • Data Manipulation Language (DML) • Insert, delete, update rows in tables • Query data in tables • Data Control Language (DCL) • Concurrency control, transactions • Administrative tasks, e.g. set up database users, security permissions

  16. 3. Creating Tables in SQL • Table definition • “CREATE TABLE” statement • Specifies table name, attribute names and data types • Create a table with no rows. • See an example at the bottom • Related statements • ALTER TABLE statement modifies table schema if needed • DROP TABLE statement removes an empty table

  17. 3. Populating Tables in SQL • Adding a row to an existing table • “INSERT INTO” statement • Specifies table name, attribute names and values • Example: • INSERT INTO River(Name, Origin, Length) VALUES(‘Mississippi’, ‘USA’, 6000) • Related statements • SELECT statement with INTO clause can insert multiple rows in a table • Bulk load, import commands also add multiple rows • DELETE statement removes rows • UPDATE statement can change values within selected rows

  18. 3. SELECT Statement- General Information • Clauses • SELECT specifies desired columns • FROM specifies relevant tables • WHERE specifies qualifying conditions for rows • ORDER BY specifies sorting columns for results • GROUP BY, HAVING specifies aggregation and statistics • Operators and functions • arithmetic operators, e.g. +, -, … • comparison operators, e.g. =, <, >, BETWEEN, LIKE… • logical operators, e.g. AND, OR, NOT, EXISTS, • set operators, e.g. UNION, IN, ALL, ANY, … • statistical functions, e.g. SUM, COUNT, ... • many other operators on strings, date, currency, ...

  19. 4. Query Operation & Spatial Index • Filter Step: • Select the objects whose mbb satisfies the spatial predicate • Traverse the index apply the spatial test on the mbb • Output: set of oids • Refinement Step: • Spatial test is done on the actual geometries of objects whose mbb satisfied the filter step • Costly operation • Executed only on a limited number of objects • Concentrate on the design of efficient SAMs for the filter step

  20. 4. Why spatial index method? • B-tree & hash tables • Guarantee the number of I/O operations is respectively logarithmic and constant in the collection sized • Index a collection on a key • Rely on a total order on the key domain, the order of natural numbers, or the lexicographic order on strings • There is no such total order for geometric objects • SAMs were designed to try as much as possible to preserve spatial object proximity

  21. 4. Space-Driven v.s. Data-Driven SAMs • Space-Driven structures: • Partition the embedding 2D Space into rectangular cells • Independently of the distribution of the objects • Objects are mapped to the cells based on some geometric criterion • Grid file, linear structure • Data-Driven structures: • Organized by partitioning the set of objects, as opposed to the embedding space • Adapts to the objects’ distribution in the embedding space • R-tree, R* tree, R+ tree

  22. 4. Grid File – point indexing • One page is associated with each cell • When a cell overflow, it is split into two cells and the points are assigned to the new cell • Two adjacent cells can reference the same page • The cells are of different size and the partition adapts to the point distribution

  23. 4. The Quad tree • The index is represented as a quaternary tree • Each internal node has four children, one per quadrant • NW, NE, SW, SE • Each leaf is associated a disk page, which stores the index entries

  24. 4. The original R-Tree • A leaf entry is a pair (mbb, oid) • A non-leaf node contains an array of node entries • The number of entries is between m and M • For each entry (dr, node_id) in a non-leaf node N, dr is the directory rectangle of a child node of N, whose page address is node_id • All leaves are at the same level • An object appears in one, and only one of the tree leaves

  25. 4. The R+ Tree • The directory rectangles at a given level do not overlap • For a point query, a single path is followed from the root to a leaf • The I/O complexity is bounded by the depth of the tree

  26. 5. What is Query Processing and Optimization (QPO)? • Basic idea of QPO • In SQL, queries are expressed in high level declarative form • QPO translates a SQL query to an execution plan • over physical data model • using operations on file structures, indices, etc. • Ideal execution plan answers Q in as little time as possible • Constraints: QPO overheads are small • Computation time for QPO steps << that for execution plan

  27. 5. QPO Challenges in SDBMS • Building Blocks for spatial queries • Rich set of spatial data types, operations • A consensus on “building blocks” is lacking • Current choices include spatial select, spatial join, nearest neighbor • Choice of strategies • Limited choice for some building blocks, e.g. nearest neighbor • Choosing best strategies • Cost models are more complex since • Spatial Queries are both CPU and I/O intensive • While traditional queries are I/O intensive • Cost models of spatial strategies are not mature.

  28. 5. Choice of building blocks • Choice of building blocks • Varies across software vendors and products • List of representative building blocks • Point Query- Name a highlighted city on a digital map. • Return one spatial object out of a table • Range Query- List all countries crossed by of the river Amazon. • Returns several objects within a spatial region from a table • Spatial Join: List all pairs of overlapping rivers and countries. • Return pairs from 2 tables satisfying a spatial predicate • Nearest Neighbor: Find the city closest to Mount Everest. • Return one spatial object from a collection

  29. 5. Strategies for Spatial Joins • Recall Spatial Join Example: • List all pairs of overlapping rivers and countries. • Return pairs from 2 tables satisfying a spatial predicate • List of strategies • Nested loop: • Test all possible pairs for spatial predicate • All rivers are paired with all countries • Space Partitioning: • Test pairs of objects from common spatial regions only • Rivers in Africa are tested with countries in Africa only! • Tree Matching • Hierarchical pairing of object groups from each table, section 5.1.6 pp.121 • Other, e.g. spatial-join-index based, external plane-sweep, …

  30. 5. Query Processing and Optimizer process • A site-seeing trip • Start: A SQL Query • End: An execution plan • Intermediate Stopovers • query trees • logical tree transforms • strategy selection • What happens after the journey? • Execution plan is executed • Query answer returned Fig 5.2

  31. 5. Query Trees • Nodes = building blocks of (spatial) queries • See section 3.2 (pp.55) for symbols sigma, pi and join • Children = inputs to a building block • Leafs = Tables • Example SQL query and its query tree follows: Fig 5.3

  32. 5. Logical Transformation of Query Trees • Motivation • Transformation do not change the answer of the query • But can reduce computational cost by • reducing data produced by sub-queries • reducing computation needs of parent node • Example Transformation • Push down select operation below join • Example: Fig. 5.4 (compare w/ Fig 5.3, last slide) • Reduces size of table for join operation • Other common transformations • Push project down • Reorder join operations • ... Fig 5.4

  33. 5. Execution Plans • An execution plan has 3 components • A query tree • An ordering of evaluation of non-leaf nodes • A strategy selected for each non-leaf node • Example • Strategies for Query tree in Fig. 5.5 • Use scan for Area(L.Geometry) > 20 • Use index for Fa.Name = ‘Campground’ • Use space-partitioning join for • Distance(Fa, L) < 50 • Use on-the-fly for projection • Ordering • As listed above Fig 5.5

  34. 7. What is Spatial Data Mining? • Non-trivial search for interesting and unexpected spatial pattern • Non-trivial Search • Large (e.g. exponential) search space of plausible hypothesis • Ex. Asiatic cholera : causes: water, food, air, insects, …; water delivery mechanisms - numerous pumps, rivers, ponds, wells, pipes, ... • Interesting • Useful in certain application domain • Ex. Shutting off identified Water pump => saved human life • Unexpected • Pattern is not common knowledge • May provide a new understanding of world • Ex. Water pump - Cholera connection lead to the “germ” theory

  35. 7. Choice of Methods • Two Approaches to mining Spatial Data • Pick spatial features; use classical DM methods • Use novel spatial data mining techniques • Possible Approach: • Define the problem: capture special needs • Explore data using maps, other visualization • Try reusing classical DM methods • If classical DM perform poorly, try new methods • Evaluate chosen methods rigorously • Performance tuning as needed

  36. 7. Location Prediction as a classification problem Given: 1. Spatial Framework 2. Explanatory functions: 3. A dependent class: 4. A family of function mappings: Find: Classification model: Objective:maximize classification_accuracy Constraints: Spatial Autocorrelation exists Nest locations Distance to open water Vegetation durability Water depth Color version of Fig. 7.3, pp. 188

  37. 7. Techniques for Location Prediction • Classical method: • logistic regression, decision trees, bayesian classifier • assumes learning samples are independent of each other • Spatial auto-correlation violates this assumption! • Q? What will a map look like where the properties of a pixel was independent of the properties of other pixels? (see below - Fig. 7.4, pp. 189) • New spatial methods • Spatial auto-regression (SAR), • Markov random field • bayesian classifier

  38. 7. Spatial AutoRegression (SAR) • Spatial Autoregression Model (SAR) • y = Wy + X +  • W models neighborhood relationships •  models strength of spatial dependencies •  error vector • Solutions •  and  - can be estimated using ML or Bayesian stat. • e.g., spatial econometrics package uses Bayesian approach using sampling-based Markov Chain Monte Carlo (MCMC) method. • Likelihood-based estimation requires O(n3) ops. • Other alternatives – divide and conquer, sparse matrix, LU decomposition, etc.

  39. 7. Associations, Spatial associations, Co-location Answers: and

  40. 7. Association Rules: Formal Definitions • Consider a set of items, • Consider a set of transactions • where each is a subset of I. • Support of C • Then iff • Support: occurs in at least s percent of the transactions: • Confidence: At least c% • Example: Table 7.4 (pp. 202) using data in Section 7.4

  41. Association rules Co-location rules Underlying space discrete sets continuous space item-types item-types events /Boolean spatial features collection Transaction (T) Neighborhood (N) prevalence measure support participation index conditional probability metric Pr.[ A in T | B in T ] Pr.[ A in N(L) | B at location L ] 7. Co-location rules vs. association rules Participation index = min{pr(fi, c)} Where pr(fi, c) of feature fi in co-location c = {f1, f2, …, fk}: = fraction of instances of fi with feature {f1, …, fi-1, fi+1, …, fk} nearby N(L) = neighborhood of location L

  42. 7. Spatial Outlier Detection • Compute where • Select points (e.g. S with Z(S(x)) above 3

  43. 7. Spatial Outlier Detection: Example Color version of Fig. 7.19 pp. 219 Given A spatial graph G={V,E} A neighbor relationship (K neighbors) An attribute function : V -> R Find O = {vi | vi V, vi is a spatial outlier} Spatial Outlier Detection Test 1. Choice of Spatial Statistic S(x) = [f(x)–E y N(x)(f(y))] 2. Test for Outlier Detection | (S(x) - s) / s | >  Rationale: Theorem: S(x) is normally distributed if f(x) is normally distributed

  44. 8. Spatiotemporal Data • Two types of problems: • Indexing the current positions and movements of objects and querying their anticipated future positions. • Indexing and querying the past movements of mobile objects. • On Indexing Mobile Objects • Indexing the Positions of Continuously Moving Objects

  45. Spatiotemporal Data (cont’d) • Indexing current/future locations mobile objects • The TPR-tree • Like the R-tree, but the MBRs are time-parameterized to conservative bounding intervals (CBI). • How are the CBI computed? What is the best way to group objects into a CBI? • By minimizing an objective function (e.g., overlap) over the time the TPR-tree is valid. • How do we answer queries using the TPR-tree?

  46. Conclusion • Good progress… still more work is needed: • Devising clean and complete semantics for data models and operators for spatial data, spatial-temporal data • Efficient implementation • Indexing, query processing, query optimization, cost model • Develop efficient algorithms to mine spatial data • Alternatives architectures • spatial-temporal data, moving objects • mobile, wireless applications • web GIS

More Related