Overview of Data Mining & The Knowledge Discovery Process

1. Overview of Data Mining&The Knowledge Discovery Process Bamshad Mobasher DePaul University

2. Why Data Mining? • The Explosive Growth of Data: from terabytes to petabytes • Data collection and data availability • Automated data collection tools, database systems, Web, computerized society • Major sources of abundant data • Business: Web, e-commerce, transactions, stocks, … • Science: Remote sensing, bioinformatics, scientific simulation, … • Society and everyone: news, images, video, documents, ….

3. Source: Intel, 2012

4. From Data to Wisdom • Data • The raw material of information • Information • Data organized and presented by someone • Knowledge • Information read, heard or seen and understood and integrated • Wisdom • Distilled knowledge and understanding which can lead to decisions Wisdom Knowledge Information Data The Information Hierarchy

5. What is Data Mining • What do we need? • Extract interesting and useful knowledge from the data • Find rules, regularities, irregularities, patterns, constraints • hopefully, this will help us better compete in business, do research, learn concepts, make money, etc. • Data Mining: A Definition The non-trivial extraction of implicit, previously unknown and potentially useful knowledge from data in large data repositories • Non-trivial: obvious knowledge is not useful • implicit: hidden difficult to observe knowledge • previously unknown • potentially useful: actionable; easy to understand

6. The Knowledge Discovery Process • Data Mining v. Knowledge Discovery in Data (KDD) • DM and KDD are often used interchangeably • actually, DM is only part of the KDD process - The KDD Process

7. Types of Knowledge Discovery • Two kinds of knowledge discovery: directed and undirected • Directed Knowledge Discovery • Purpose: Explain value of some field in terms of all the others (goal-oriented) • Method: select the target field based on some hypothesis about the data; ask the algorithm to tell us how to predict or classify new instances • Examples: • what products show increased sale when cream cheese is discounted • which banner ad to use on a web page for a given user coming to the site • Undirected Knowledge Discovery • Purpose: Find patterns in the data that may be interesting (no target field) • Method: clustering, affinity grouping • Examples: • which products in the catalog often sell together • market segmentation (find groups of customers/users with similar characteristics or behavioral patterns)

8. From Data Mining to Data Science

9. Data Mining: On What Kinds of Data? • Database-oriented data sets and applications • Relational database, data warehouse, transactional database • Object-relational databases, Heterogeneous databases and legacy databases • Advanced data sets and advanced applications • Data streams and sensor data • Time-series data, temporal data, sequence data (incl. bio-sequences) • Structure data, graphs, social networks and information networks • Spatial data and spatiotemporal data • Multimedia database • Text data and other semi-structured data • The World-Wide Web

10. Data Mining: What Kind of Data? • Structured Databases • relational, object-relational, etc. • can use SQL to perform parts of the process e.g., SELECT count(*) FROM Items WHERE type=video GROUP BY category

11. Data Mining: What Kind of Data? • Flat Files • most common data source • can be text (or HTML) or binary • may contain transactions, statistical data, measurements, etc. • Transactional databases • set of records each with a transaction id, time stamp, and a set of items • may have an associated “description” file for the items • typical source of data used in market basket analysis

12. Data Mining: What Kind of Data? • Other Types of Databases • legacy databases • multimedia databases (usually very high-dimensional) • spatial databases (containing geographical information, such as maps, or satellite imaging data, etc.) • Time Series Temporal Data (time dependent information such as stock market data; usually very dynamic) • World Wide Web • basically a large, heterogeneous, distributed database • need for new or additional tools and techniques • information retrieval, filtering and extraction • agents to assist in browsing and filtering • Web content, usage, and structure (linkage) mining tools • The “social Web” • User generated meta-data, social networks, shared resources, etc.

13. What Can Data Mining Do • Many Data Mining Tasks • often inter-related • often need to try different techniques/algorithms for each task • each tasks may require different types of knowledge discovery • What are some of data mining tasks • Classification • Prediction • Clustering • Affinity Grouping / Association discovery • Sequence Analysis • Characterization • Discrimination

14. Some Applications of Data mining • Business data analysis and decision support • Marketing focalization • Recognizing specific market segments that respond to particular characteristics • Return on mailing campaign (target marketing) • Customer Profiling • Segmentation of customer for marketing strategies and/or product offerings • Customer behavior understanding • Customer retention and loyalty • Mass customization / personalization

15. Some Applications of Data mining • Business data analysis and decision support (cont.) • Market analysis and management • Provide summary information for decision-making • Market basket analysis, cross selling, market segmentation. • Resource planning • Risk analysis and management • "What if" analysis • Forecasting • Pricing analysis, competitive analysis • Time-series analysis (Ex. stock market)

16. Some Applications of Data mining • Fraud detection • Detecting telephone fraud: • Telephone call model: destination of the call, duration, time of day or week • Analyze patterns that deviate from an expected norm • British Telecom identified discrete groups of callers with frequent intra-group calls, especially mobile phones, and broke a multimillion dollar fraud scheme • Detection of credit-card fraud • Detecting suspicious money transactions (money laundering) • Text mining: • Message filtering (e-mail, newsgroups, etc.) • Newspaper articles analysis • Text and document categorization • Web Mining • Mining patterns from the content, usage, and structure of Web resources