Data Mining

1. Data Mining References: U.S. News and World Report's Business & Technology section, 12/21/98, by William J. Holstein Prof. Juran’s lecture note 1 (at Columbia University) J.H. Friedman (1999) Data Mining and Statistics. technical report, Dept. of Stat., Stanford University

2. Main Goal • Study statistical tools useful in managerial decision making. • Most management problems involve some degree of uncertainty. • People have poor intuitive judgment of uncertainty. • IT revolution... abundance of available quantitative information • data mining: large databases of info, ... • market segmentation & targeting • stock market data • almost anything else you may want to know... • What conclusions can you draw from your data? • How much data do you need to support your conclusions?

3. Applications in Management • Operations management • e.g., model uncertainty in demand, production function... • Decision models • portfolio optimization, simulation, simulation based optimization... • Capital markets • understand risk, hedging, portfolios, beta's... • Derivatives, options, ... • it is all about modeling uncertainty • Operations and information technology • dynamic pricing, revenue management, auction design, ... • Data mining... many applications

4. Portfolio Selection • You want to select a stock portfolio of companies A, B, C, … • Information: Stock Annual returns by year A 10%, 14%, 13%, 27%, … B 16%, 27%, 42%, 23%, … • Questions: • How do we measure the volatility of each stock? • How do we quantify the risk associated with a given portfolio? • What is the tradeoff between risk and returns?

6. Currency Value (Relative to Jan 2 1998)

7. Introduction • Premise: All business becomes information driven. • The concept of Data Mining is becoming increasingly popular as a business information management tool where it is expected to reveal knowledge structures that can guide decisions in conditions of limited certainty. • Competitiveness: How you collect and exploit information to your advantage? • The challenges • Most corporate data systems are not ready. • Can they share information? • What is the quality of the information going in • Most data techniques come from the empirical sciences; the world is not a laboratory. • Cutting through vendor hype, info-topia. • Defining good metrics; abandoning gut rules of thumb may be too "risky" for the manager. • Communicating success, setting the right expectations.

8. Wal-Mart • U.S. News and World Report's Business & Technology section, 12/21/98, by William J. Holstein Data-Crunching Santa Wal-Mart knows what you bought last Christmas • Wal-Mart is expected to finish the year with $135 billion in sales, up from $118 billion last year. • It hurts department stores such as Sears, J. C. Penney, and Federated's Macy's and Bloomingdale's units, which have been slower to link all their operations from stores directly to manufacturers. . • For example, Sears stocked too many winter coats this season and was surprised by warmer than average weather. • The field of business analytics has improved significantly over the past few years, giving business users better insights, particularly from operational data stored in transactional systems. business analytics in its everyday activities. • Analytics are now routinely used in sales, marketing, supply chain optimization, and fraud detection.

9. A visualization of a Naive Bayes model for predicting who in the U.S. earns more than $50,000 in yearly salary. The higher the bar, the greater the amount of evidence a person with this attribute value earns a high salary.

10. Telecommunications • Data mining flourishes in telecommunications due to the availability of vast quantities of high-quality data. • A significant stream of it consists of call records collected at network switches used primarily for billing; it enables data mining applications in toll fraud detection and consumer marketing. • The best-known marketing application of data mining, albeit via unconfirmed anecdote, concerns MCI’s “Friends & Family” promotion launched in the domestic U.S. market in 1991. • As the anecdote goes, market researchers observed relatively small subgraphs in this long-distance phone company’s large call-graph of network activity. • It reveals the promising strategy of adding entire calling circles to the company’s subscriber base, rather than the traditional and costly approach of seeking individual customers one at a time. Indeed, MCI increased its domestic U.S. market share in the succeeding years by exploiting the “viral” capabilities of calling circles; one infected member causes others to become infected. • Interestingly, the plan was abandoned some years later (not available since 1997), possibly because the virus had run its course but more likely due to other competitive forces.

11. Telecommunications • In toll-fraud detection, data mining has been instrumental in completely changing the landscape for how anomalous behaviors are detected. • Nearly all fraud detection systems in the telecommunications industry 10 years ago were based on global threshold models. • They can be expressed as rule sets of the form “If a customer makes more than X calls per hour to country Y, then apply treatment Z.” • The placeholders X, Y, and Z are parameters of these rule sets applied to all customers. • Given the range of telecommunication customers, blanket application of these rules produces many false positives. • Data mining methods for customized monitoring of land and mobile phone lines were subsequently developed by leading service providers, including AT&T, MCI, and Verizon, whereby each customer’s historic calling patterns are used as a baseline against which all new calls are compared. • For customers routinely calling country Y more than X times a day, such alerts would be suppressed, but if they ventured to call a different country Y’, an alert might be generated.

12. Risk management and targeted marketing • Insurance and direct mail are two industries that rely on data analysis to make profitable business decisions. • Insurers must be able to accurately assess the risks posed by their policyholders to set insurance premiums at competitive levels. • For example, overcharging low-risk policyholders would motivate them to seek lower premiums elsewhere; undercharging high-risk policyholders would attract more of them due to the lower premiums. • In either case, costs would increase and profits inevitably decrease. • Effective data analysis leading to the creation of accurate predictive models is essential for addressing these issues. • In direct-mail targeted marketing, retailers must be able to identify subsets of the population likely to respond to promotions in order to offset mailing and printing costs. • Profits are maximized by mailing only to those potential customers most likely to generate net income to a retailer in excess of the retailer’s mailing and printing costs.

13. Medical applications (diabetic screening) • Preprocessing and postprocessing steps are often the most critical elements determining the effectiveness of real-life data-mining applications, as illustrated by the following recent medical application in diabetic patient screening. • In the 1990s in Singapore, about 10% of the population was diabetic, a disease with many side effects, including increased risk of eye disease kidney failure, and other complications. • However, early detection and proper care management can make a difference in the health and longevity of individual sufferers. • To combat the disease, the government of Singapore introduced a regular screening program for diabetic patients in its public hospitals in 1992. • Patient information, clinical symptoms, eye-disease diagnosis, treatments, and other details, were captured in a database maintained by government medical authorities. • After almost 10 years of collecting data, a wealth of medical information is available. This vast store of historical data leads naturally to the application of data mining techniques to discover interesting patterns. • The objective is to find rules physicians can use to understand more about diabetes and how it might be associated with different segments of the population.

14. Christmas Season: Georgia Stores • Store at Decatur (just east of Atlanta) • A black middle-income community • Decoration display: African-American angels and ethnic Santas aplenty • Music section: Promoting seasonal disks like "Christmas on Death Row," which features rapper Snoop Doggy Dogg. • Toy department: a large selection of brown-skinned dolls • Store at Dunwoody (20 miles away fom Decatur) • An affluent, mostly white suburb (north of Atlanta) • Music section: Showcasing Christmas tunes by country superstar Garth Brooks. • Toy department: a few expensive toys that aren't available in the Decatur store; Out of the hundreds of dolls in stock, only two have brown skin. • How to determine the kinds of products that are carried by various Wal-Marts across the land?

15. Wal-Mart system • Every item in the store has a laser bar code, so when customers pay for their purchases a scanner captures information about • what is selling on what day of the week and at what price. • The scanner also records what other products were in each shopper's basket. • Wal-Mart analyzes what is in the shopping cart itself. • The combination of [what's in a purchaser's cart] gives you a good indication of the age of that consumer and the preferences in terms of ethnic background. • Wal-Mart combines the in-store data with information about the demographics of communities around each store. • The end result is surprisingly different personalities for Wal-Marts. • It also help Wal-Mart figure out how to place goods on the floor to get what retailers call "affinity sales," or sales of related products.

16. Wal-Mart system (Cont.) • One big strength of the system is that about 5,000 manufacturers are tied into it through the company's Retail Link program, which they access via the Internet. • Pepsi, Disney, or Mattel, for example, can tap into Wal-Mart's data warehouse to see how well each product is selling at each Wal-Mart. • They can look at how things are selling in individual areas and make decisions about categories where there may be an opportunity to expand. • That tight information link helps Wal-Mart work with its suppliers to replenish stock of products that are selling well and to quickly pull those that aren't.

17. Data Mining and Statistics • Data Mining is used to discover patterns and relationships in data with an emphasis on large observational data bases. • It sits at the common frontiers of several fields including Data Base Management, Artificial Intelligence, Machine Learning, Pattern Recognition and Data Visualization. • From a statistical perspective it can be viewed as computer automated exploratory data analysis of large complex data sets. • Many organizations have large transaction oriented data bases used for inventory billing accounting, etc. These data bases were very expensive to create and are costly to maintain. For a relatively small additional investment DM tools offer to discover highly profitable nuggets of information hidden in these data. • Data, especially large amounts of it reside in data base management systems DBMS. • Conventional DBMS are focused on online transaction processing (OLTP); that is the storage and fast retrieval of individual records for purposes of data organization. They are used to keep track of inventory payroll records, billing records, invoices, etc.

18. Data Mining Techniques • Data Mining as an analytic process designed to • explore data (usually large amounts of - typically business or market related - data) in search for consistent patterns and/or systematic relationships between variables, and then • to validate the findings by applying the detected patterns to new subsets of data. • The ultimate goal of data mining is prediction - and predictive data mining is the most common type of data mining and one that has most direct business applications. • The process of data mining consists of three stages: • the initial exploration, • model building or pattern identification with validation and verification, and it is concluded with • deployment (i.e., the application of the model to new data in order to generate predictions).

19. Stage 1: Exploration • It usually starts with data preparation which may involve cleaning data, data transformations, selecting subsets of records and - in case of data sets with large numbers of variables ("fields") - performing some preliminary feature selection operations to bring the number of variables to a manageable range (depending on the statistical methods which are being considered). • Depending on the nature of the analytic problem, this first stage of the process of data mining may involve anywhere between a simple choice of straightforward predictors for a regression model, to elaborate exploratory analyses using a wide variety of graphical and statistical methods in order to identify the most relevant variables and determine the complexity and/or the general nature of models that can be taken into account in the next stage.

20. Stage 2: Model building and validation • This stage involves considering various models and choosing the best one based on their predictive performance • Explain the variability in question and • Producing stable results across samples. • How do we achieve these goals? • This may sound like a simple operation, but in fact, it sometimes involves a very elaborate process. • "competitive evaluation of models," that is, applying different models to the same data set and then comparing their performance to choose the best. • These techniques - which are often considered the core of predictive data mining - include: Bagging (Voting, Averaging), Boosting, Stacking (Stacked Generalizations), and Meta-Learning.

21. Models for Data Mining • In the business environment, complex data mining projects may require the coordinate efforts of various experts, stakeholders, or departments throughout an entire organization. • In the data mining literature, various "general frameworks" have been proposed to serve as blueprints for how to organize the process of gathering data, analyzing data, disseminating results, implementing results, and monitoring improvements. • CRISP (Cross-Industry Standard Process for data mining) was proposed in the mid-1990s by a European consortium of companies to serve as a non-proprietary standard process model for data mining. • The Six Sigma methodology - is a well-structured, data-driven methodology for eliminating defects, waste, or quality control problems of all kinds in manufacturing, service delivery, management, and other business activities.

22. CRISP • This general approach postulates the following (perhaps not particularly controversial) general sequence of steps for data mining projects:

23. Six Sigma • This model has recently become very popular (due to its successful implementations) in various American industries, and it appears to gain favor worldwide. It postulated a sequence of, so-called, DMAIC steps • The categories of activities: Define (D), Measure (M), Analyze (A), Improve (I), Control (C ). • Postulates the following general sequence of steps for data mining projects: Define (D) → Measure (M) → Analyze (A) → Improve (I) → Control (C ) - It grew up from the manufacturing, quality improvement, and process control traditions and is particularly well suited to production environments (including "production of services," i.e., service industries). • Define. It is concerned with the definition of project goals and boundaries, and the identification of issues that need to be addressed to achieve the higher sigma level. • Measure. The goal of this phase is to gather information about the current situation, to obtain baseline data on current process performance, and to identify problem areas. • Analyze. The goal of this phase is to identify the root cause(s) of quality problems, and to confirm those causes using the appropriate data analysis tools. • Improve. The goal of this phase is to implement solutions that address the problems (root causes) identified during the previous (Analyze) phase. • Control. The goal of the Control phase is to evaluate and monitor the results of the previous phase (Improve).

24. Six Sigma Process • A six sigma process is one that can be expected to produce only 3.4 defects per one million opportunities. • The concept of the six sigma process is important in Six Sigma quality improvement programs. • The term Six Sigma derives from the goal to achieve a process variation, so that 6×sigma (the estimate of the population standard deviation) will "fit" inside the lower and upper specification limits for the process. • In that case, even if the process mean shifts by 1.5×sigma in one direction (e.g., to +1.5 sigma in the direction of the upper specification limit), then the process will still produce very few defects. • For example, suppose we expressed the area above the upper specification limit in terms of one million opportunities to produce defects. The 6×sigma process shifted upwards by 1.5 ×sigma will only produce 3.4 defects (i.e., "parts" or "cases" greater than the upper specification limit) per one million opportunities

25. Statisticians’s remark on DM paradigms • The DM community may have to moderate its romance with big. • A prevailing attitude seems to be that unless an analysis involves gigabytes or terabytes of data, it can not possibly be worthwhile. • It seems to be a requirement that all of the data that has been collected must be used in every aspect of the analysis. • Sophisticated procedures that cannot simultaneously handle data sets of such size are not considered relevant to DM. • Most DM applications routinely require data sets that are considerably larger than those that have been addressed by traditional statistical procedures (kilobytes). • It is often the case that the questions being asked of the data can be answered to sufficient accuracy with less than the entire giga or terabyte data base. • Sampling methodology which has a long tradition in Statistics can profitably be used to improve accuracy while mitigating computational requirements. • Also a powerful computationally intense procedure operating on a subsample of the data may in fact provide superior accuracy than a less sophisticated one using the entire data base.

26. Sampling • Objective: Determine the average amount of money spent in the Central Mall. • Sampling: A Central City official randomly samples 12 people as they exit the mall. • He asks them the amount of money spent and records the data. • Data for the 12 people: Person $ spent Person $ spent Person $ spent 1 $132 5 $123 9 $449 2 $334 6 $ 5 10 $133 3 $ 33 7 $ 6 11 $ 44 4 $ 10 8 $ 14 12 $ 1 • The official is trying to estimate mean and variance of the population based on a sample of 12 data points.

27. Population versus Sample • A population is usually a group we want to know something about: • all potential customers, all eligible voters, all the products coming off an assembly line, all items in inventory, etc.... • Finite population: {u1, u2, ... , uN} versus Infinite population • A population parameter is a number (q) relevant to the population that is of interest to us: • the proportion (in the population) that would buy a product, the proportion of eligible voters who will vote for a candidate, the average number of M&M's in a pack.... • A sample is a subset of the population that we actually do know about (by taking measurements of some kind): • a group who fill out a survey, a group of voters that are polled, a number of randomly chosen items off the line.... • {x1, x2, ... , xn} • A sample statisticg(x1, x2, ... , xn) is often the only practical estimate of a population parameter. • We will use g(x1, x2, ... , xn) as proxies for q, but remember their difference.

28. Average Amount of Money spent in the Central Mall • A sample (x1, x2, ... , xn) • Its mean is the sum of their values divided by the number of observations. • The sample mean, the sample variance, and the sample standard deviation are $107, $220,854, and $144.40, respectively. • It claims that on average $107 are spent per shopper with a standard deviation of $144.40. • Why can we claim so?

29. The variance s2of a set of observations is the average of the squares of the deviations of the observations from their mean. • The standard deviation s is the square root of the variance s2 . • How far the observations are from the mean? s2and swill be • large if the observations are widely spread about their mean, • small if they are all close to the mean.

30. Stock Market Indexes • It is a statistical measure that shows how the prices of a group of stocks changes over time. • Price-Weighted Index: DJIA • Market-Value-Weighted Index: Standard and Poor’s 500 composite Index • Equally Weighted Index: Wilshire 5000 Equity Index • Price-Weighted Index: It shows the change in the average price of the stock that are included in the index. • Price per share in current period P0 and price per share in next period P1. • Number of shares outstanding in current period Q0 and number of shares outstanding in next period Q1.

31. DJIA • Dow Jones industrial average (DJIA): • Charles Dow first concocted his 12-stock industrial average in 1896 (expanding to 30 in 1928) • Original: It is an arithmetic average of the thirty stock prices that make up the index. DJIA = [(P01 + P02 +… + P0,30)/30]/[(P11 + P12 +… + P1,30)/30] • Current: It is adjusted for stock splits and the insurance of stock dividends. DJIA = [(P01+ P02 +… + P0,30)/AD1]/(P11 + P12 +… + P1,30) where AD1 is the appropriate divisor. • How do we adjust AD1 to account for stock splits, adding new stocks,...? • The adjustment process is designed to keep the index value the same as it would have been if the split had not occurred. • Suppose X30 splits 2:1 from $100 to $50. Then change c to c0 such that (X1 + X2 +… + 100)/c = (X1 + X2 +… + 50)/c0 • change to c0 < c to keep index constant before & after split. • How about when new stocks are added and others are removed?

32. DJIA • How each stock in the Dow performed during the period when the Dow rose 100 percent (from its close above 5,000 on Nov. 21, 1995 until it closed above 10,000 on March 29, 1999). *Companies not in the Dow when it crossed 5,000. **Adjusted for spinoffs. Does not reflect performance of stocks spun off to shareholders. Company Weight in the Dow (%) Change in Price (%) Alcoa 1.9 + 52 AlliedSignal 2.3 +129 Amer. Express 5.5 +185 AT&T** 3.6 + 87 Boeing 1.5 -5 Caterpillar 2.1 +59 Chevron 4.0 +77 Citigroup* 2.8 +262 Coca-Cola 3.0 +69 Du Pont 2.5 +76 Eastman Kodak 2.9 -6

33. DJIA Company Weight in the Dow (%) Change in Price (%) Exxon 3.2 + 83 General Electric 5.3 +232 General Motors** 3.9 +89 Goodyear 2.2 + 23 Hewlett-Packard* 3.1 +66 I.B.M. 1.9 +276 International Paper 2.0 +24 J. P. Morgan 5.0 +63 Johnson & Johnson* 4.2 +120 McDonald's 2.0 +102 Merck 3.6 +175 Minnesota Mining**3.2+ 15 Philip Morris 1.8 + 37 Procter & Gamble 4.5 +134 Sears, Roebuck 2.1 + 18 Union Carbide 2.1 + 19 United Technologies 6.0 +196 Wal-Mart* 4.2 +288 Walt Disney 1.5 + 62

34. S&P 500 • The S&P 500, which started in 1957, weights stocks on the basis of their total market value. • Suppose X30 splits 2:1 from $100 to $50. Then change c to c0 such that (X1 + X2 +… + 100)/c = (X1 + X2 +… + 50)/c0 • change to c0 < c to keep index constant before & after split. • How about when new stocks are added and others are removed? • S&P 500 is computed by S&P 500 = (w1X1 + w2X2 +… + w500X500)/c where Xi=price of ith stock and wi=# of shares of ith stock. • What happens when a stock splits? • It is a weighted average.

35. Sample vs Population • For both problems, we try to infer properties of a large group (population) by analyzing a small subgroup (the sample). • The population is the group we are trying to analyze; e.g., all eligible voters, etc. • A sample is a subset of the total population that we have observed or collected data from; e.g., voters that are actually polled, etc. • How to draw a sample which can be used to make statements about the population? • Sample must be representative of the population • Sampling is the way to obtain reliable information in a cost effective way (why not census?)

36. Issues in sampling • Representativeness • Interviewer discretion • Respondent discretion - non-response • Key question: is the reason for non-response related to the attribute you are trying to measure? Illegal aliens/Census. Start-up companies/not in phone book. Library exit survey. • Good samples; • Good samples; probability samples; each unit in the population has a known probability of being in the sample • Simplest case; equal probability sample, each unit has the same chance of being in the sample.

37. Utopian Sample for Analysis • You have a complete and accurate list of ALL the units in the target population (sampling frame) • From this you draw an equal probability sample (generate a list of random numbers) • Reality check; incomplete frame, impossible frame, practical constraints on the simple random sample (cost and time of sampling) • Precision considerations • How large a sample do I need? • Focus on confidence interval - choose coverage rate (90%, 95%, 99%) margin of error (half the width). Typically trade off width against coverage rate. • Simple rule of thumb for a population proportion - if it's a 95% CI, then use n = 1/(margin of error)**2.

38. Data Analysis • Statistical Thinkingis understanding variation and how to deal with it. • Move as far as possible to the right on this continuum: Ignorance-->Uncertainty-->Risk-->Certainty • Information science：learning from data • Probabilistic inference based on mathematics • What is Statistics? • What is the connection if any • elds including Data Base Management Articial In