COMP5318 Knowledge Discovery and Data Mining

COMP5318 Knowledge Discovery and Data Mining Week 8: Mining Association Rules Reference: TSK pp.328-353, 363-370 Dunham 125-142

Outline • What is Association Rule Mining? • Basic concepts • Item, Itemset, Transaction, Support, Confidence, … • Association rule problem definition • Apriori principle • What, why, how • Apriori algorithm • FP-growth algorithm • Discussion

What is Association Rule Mining? • Association rule mining finds • combinations of items that typically occur together in a database (market-basket analysis) • Sequences of items that occur frequently (sequential analysis) in a database • Originally introduced for Market-basket analysis -- useful for analysing purchasing behaviour of customers.

Market-Basket Analysis – Examples • Where should strawberries be placed to maximize their sale? • Services purchased together by telecommunication customers (e.g. broad band Internet, call forwarding, etc.) help determine how to bundle these services together to maximize revenue • Unusual combinations of insurance claims can be a sign of a fraud • Medical histories can give indications of complications based on combinations of treatments • Sport: analyzing game statistics (shots blocked, assists, and fouls) to gain competitive advantage • “When player X is on the floor, player Y’s shot accuracy decreases from 75% to 30%” • Bhandari et.al. (1997). Advanced Scout: data mining and knowledge discovery in NBA data, Data Mining and Knowledge Discovery, 1(1), pp.121-125

Basic Concepts • Set of items: I={i1, i2,…,im}; • Set of transactions: T={t1, t2, …,tn}; • Each transaction tn is a subset of I • Example: • 5 transactions: T={t1, t2, …,t5}; • 5 items: I={Bread, Jelly, PeanutButter, Milk, Beer} • Itemset– a collection (set) of 1 or more items • If an itemset contains k items, it is called k-itemset e.g. {Jelly, Milk, Bread} is an example of 3-itemset Example Dataset

Basic Concepts • Searching for rules of the form XY, where X and Y are itemsets, e.g. • {Bread}  {Jelly} • {Bread, Jelly}  {PeanutButter} • Formally: Given I={i1, i2,…,im} and T={t1, t2, …,tn}, an association rule is an implication of the form XY, where X,Y  I and XY= (i.e. X and Y are disjoint itemsets) • Association rules have 2 important ‘properties’ • Support • Confidence These measure how “interesting” the rule is.

Support of an Itemset • The support of an itemset X is the number (or percentage) of transactions containing that itemset. • Example: Question: What is support({Bread, PeanutButter})? Answer: 3 (or 3/5 = 60%)

Support and Confidence of an Association Rule • Support of an association rule XY is the number (or percentage) of transactions that contain X Y • support(XY)=support(X Y) • Measures how often the rule occurs in the dataset • Low support: “uninteresting rule; occurs by chance” • Confidence of an association rule XY is the number of transactions that contain X Y divided by the number of transactions that contain X • confidence(XY)=support(XY)/support(X) • Measures the reliability (strength) of the rule • confidence can be seen as approximating (estimating) P(Y|X) • Intuitive: Question: given X, what is the most likely Y? Answer: The Y so that P(Y|X) is the highest.

Support and Confidence - Example • What is the support and confidence of the following rules? • {Beer}{Bread} • {Bread, PeanutButter}{Jelly} ? Support(XY)=support(X Y) confidence(XY)=support(XY)/support(X)

Association Rule Mining Problem Definition • Given a set of transactions T={t1, t2, …,tn} and 2 thresholds; minsup and minconf, • Find all association rules XY with support  minsup and confidence  minconf • I.E: we want rules with high confidence and support • We call these rules interesting • We would like to • Design an efficient algorithm for mining association rules in large data sets • Develop an effective approach for distinguishing interesting rules from spurious ones

Generating Association Rules – Approach 1 (Naïve) • Enumerate all possible rules and select those of them that satisfy the minimum support and confidence thresholds • Not practical for large databases • For a given dataset with m items, the total number of possible rules is 3m-2m+1+1 (Why?*) • And most of these will be discarded! • We need a strategy for rule generation -- generate only the promising rules • rules that are likely to be interesting, or, more accurately, don’t generate rules that can’t be interesting. *hint: use inclusion-exclusion principle

Generating Association Rules –Approach 2 • What do these rules have in common? A,BC A,CB B,CA • The support of a rule XY depends only on the support of its itemset X Y Answer: they have the same support: support({A,B,C}) • Hence, a better approach: find Frequent itemsets first, then generate the rules • Frequent itemset is an itemset that occurs more than minsup times • If an itemset is infrequent, all the rules that contain it will have support<minsup and there is no need to generate them

Generating Association Rules –Approach 2 • 2 step-approach: Step 1: Generate frequent itemsets -- Frequent Itemset Mining (i.e. support  minsup) • e.g. {A,B,C} is frequent (so A,BC, A,CB and B,CA satisfy the minSup threshold). Step 2: From them, extract rules that satisfy the confidence threshold (i.e. confidence  minconf) • e.g. maybe only A,B C and C,BA are confident • Step 1 is the computationally difficult part (the next slides explain why, and a way to reduce the complexity….)

Frequent Itemset Generation (Step 1) – Brute-Force Approach • Enumerate all possible itemsets and scan the dataset to calculate the support for each of them • Example: I={a,b,c,d,e} Search space showing superset / subset relationships Given d items, there are 2d-1 possible (non-empty) candidate itemsets => not practical for large d

Frequent Itemset Generation (Step 1) -- Apriori Principle (1) A subset of any frequent itemset is also frequent • Example: If {c,d,e} is frequent then {c,d}, {c,e}, {d,e}, {c}, {d} are also frequent

Frequent Itemset Generation (Step 1) -- Apriori Principle (2) If an itemset is not frequent, a superset of it is also not frequent • Example: If we know that {a,b} is infrequent, the entire sub-graph can be pruned. • Ie: {a,b,c}, {a,b,d}, {a,b,e}, {a,b,c,d}, {a,b,c,e}, {a,b,d,e} and {a,b,c,d}are infrequent

Frequent Itemset Generation (Step 1) -- Apriori Principle (3) That is: If an itemset is frequent then all its subsets are frequent Equivalently (more useful): If an itemset is not frequent, a superset of it is also not frequent “Support is anti-monotonic” – support monotonically decreases as we add items to the itemset. Use this to prune the search space – “support-based pruning”.

Recall the 2 Step process for Association Rule Mining Step 1: Find all frequent Itemsets So far: main ideas and concepts (Apriori principle). Later: algorithms Step 2: Generate the association rules from the frequent itemsets.

ARGen Algorithm (Step 2) • Generates interesting rules from the frequent itemsets • Already know the rules are frequent (Why?), just need to check confidence. ARGen algorithm for each frequent itemset F generate all non-empty subsets S. for each s in S do if confidence(s F-s) ≥ minConf then output rule s F-s end Example:F={a,b,c} S={{a,b}, {a,c}, {b,c}, {a}, {b}, {c}} rules output: {a,b} {c}, etc.

ARGen - Example • minsup=30%, minconf=50% • The set of frequent itemsets L={{Beer},{Bread}, {Milk}, {PeanutButter}, {Bread, PeanutButter}} • Only the last itemset from L consists of 2 nonempty subsets of frequent itemsets – Bread and PeanutButter. • => 2 rules will be generated

Summary so far • Concepts (item, itemset, transaction, support, confidence, Association Rules) • 2 step process for Association rule Mining • Step 1: Frequent Itemset Mining • The most computationally difficult step in Association Rule Mining. • Apriori Principle – support is anti-monotonic. • Step 2: Extract rules from frequent itemsets (ARGen).

What’s next? • Algorithms for finding frequent itemsets (ie: Step 1) • Apriori Algorithm • FP-Growth Algorithm

Apriori Algorithm

Frequent Itemset Generation – Apriori Algorithm • Generate candidate itemsets with size = 1 (all items) • Scan the database to see which of them are frequent (database scan step) • Use only the frequent itemsets to generate the set of candidates with size = size + 1 (candidate generation step -- AprioriGen) • If candidates were generated, goto 2. • Stop. All frequent itemsets found. Recall that we would then generate the association rules from these… ARGen

Apriori Algorithm I={Beer, Bread, Jelly, PeanutButter, Milk} T={t1, t2, …,t5} • Let minSup=30% • Level Candidate itemsets Frequent itemsets 1. {Beer} (40%), {Bread} (60%), {Beer}, {Bread}, {Milk}, {Jelly} (20%), {Milk} (40%), {PeanutButter} {PeanutButter} (60%) 2. {Beer, Bread}(20%), {Beer, Milk}(20%), {Bread, PeanutButter} {Beer, PeanutButter}(0%), {Bread, Milk} (20%), {Bread, PeanutButter} (40%), {Milk, PeanutButter} (20%) 3.  * itemset sizes * Why not {Jelly,…}? * AprioriGen – later…

Note the benefits of using Apriori Principle for Candidate Generation • For our simple example (5 items) • Brute-force approach – generate all candidate itemsets of a given size • Apriori – generate candidate itemsets from frequent itemsets • => Apriori is much more efficient • Lets look into this in more detail… 4 = 5+6 = 11

Apriori-Gen (Candidate Generation) • Apriori-Gen = The algorithm for generating candidate itemsets with size k from frequent itemsets with size k-1 • Initially (k = 1), all itemsets of size 1 are considered as candidate itemsets • From k = 2 onwards – different strategies • Brute force (for comparison only – not part of Apriori) • Fk-1 x F1 • Fk-1 x Fk-1

Brute-force Approach for Candidate Generation • Not part of Apriori • Generate all possible combinations of size k (from the original items) and then prune the infrequent => will generate candidate itemsets at level k, d is total number of items • Pruning of so many items is expensive • Total cost of generation and pruning: O(d*2d-1) plus we would already know that most of these are not frequent

Apriori-Gen – Fk-1 x F1 • Extend frequent (k-1)-itemset with frequent 1-itemset • Will generate all frequent itemsets of size k as each frequent k-itemset consists of a frequent (k-1)-itemset and a frequent 1-itemset => the procedure is complete • Less computationally expensive • However, may generate the same candidate itemsets more than once But we already know some of these are not frequent (Apriori Principle) • Solution: lexicographic order of the frequent itemsets and extension of (k-1)-itemset allowed only with lexicographically larger items, e.g. • {Bread, Diapers} can be extended with {Milk} but {Diapers, Milk} cannot be extended with {Bread}

Apriori-Gen – Fk-1 x F1 (cont.) • Although an improvement, it may still produce unnecessary candidate itemsets • E.g. merging {Beer, Diapers} with {Milk} is not necessary as one of the subsets ( {Beer, Milk}) is infrequent • can be checked at the candidate generation time and the itemset discarded or • the itemset can be generated and then pruned

Apriori-Gen – Fk-1 x Fk-1 • Assumes lexicographic ordering • Merges a pair of frequent (k-1)-itemsets only if their first k-2 items are identical • E.g. k=3, merging itemsets of size 2 {Bread, Diapers} and {Bread, Milk} will be merged: {Bread, Diapers, Milk} {Beer, Diapers} and {Diapers, Milk} will not be merged • Complete procedure • Will not generate duplicates • Does not guarantee that all generated candidate itemsets as frequent => pruning is needed

Frequent Itemset Generation in Apriori – Clothing Example • Given: 20 clothing transactions; minSup=20%, minConf=50% • Generate frequent itemset using the Apriori algorithm and the Fk-1 x Fk-1 strategy for candidate itemset generation 1. Level 1 – generate all 1-itemsets and find the frequent ones Level Candidate Frequent 1 {Blouse}(3), {Jeans}(14), {Shoes}(10) {Jeans}(14), {Shoes}(10) {Shorts}(5), {Skirt}(6), {TShirt}(13) {Shorts}(5), {Skirt}(6), {TShirt}(13)

Frequent Itemset Generation in Apriori – Clothing Example (cont.) Level Candidate Frequent 1 {Blouse}(3), {Jeans}(14), {Shoes}(10) {Jeans}(14), {Shoes}(10) {Shorts}(5), {Skirt}(6), {TShirt}(13) {Shorts}(5), {Skirt}(6), {TShirt}(13) 2. Use AR-Gen to generate candidate 2-itemsets from frequent 1-itemsets and F1xF1 • {Jeans, Shoes} (7), {Jeans, Shorts} (5), {Jeans, Shoes} (7), {Jeans, Shorts} (5) (Jeans, Skirt} (2), {Jeans, TShirt} (8), {Jeans, TShirt} (8), {Shoes, Shorts} (4), {Shoes, Skirt} (3), {Shoes, Shorts} (4), {Shoes, TShirt} (9), {Shorts, Skirt} (0), {Shoes, TShirt} (9), {Shorts, TShirt} (4), {Skirt, TShirt} (3) {Shorts, TShirt} (4) 3. Use AR-Gen to generate candidate 3-itemsets from frequent 2-itemsets and F2xF2 (1st item should be identical) 3{Jeans, Shoes, Shorts} (4), {Jeans, Shoes, Shorts} (4), {Jeans, Shoes, TShirt} (7), {Jeans, Shoes, TShirt} (7), {Jeans, Shorts, TShirt} (4) {Jeans, Shorts, TShirt} (4) {Shoes, Shorts, TShirt} (4) {Shoes, Shorts, TShirt} (4)

Frequent Itemset Generation in Apriori – Clothing Example (cont.) . . . 3{Jeans, Shoes, Shorts} (4), {Jeans, Shoes, Shorts} (4), {Jeans, Shoes, TShirt} (7), {Jeans, Shoes, TShirt} (7), {Jeans, Shorts, TShirt} (4) {Jeans, Shorts, TShirt} (4) {Shoes, Shorts, TShirt} (4) {Shoes, Shorts, TShirt} (4) 3. Use AR-Gen to generate candidate 4-itemsets from frequent 3-itemsets and F2xF2 (1st and 2d items should be identical) {Jeans, Shoes, Shorts, TShirt}(4) {Jeans, Shoes, Shorts, TShirt}(4) 4. Use AR-Gen to generate candidate 5-itemsets from frequent 4-itemsets and F2xF2 (1st ,2d and 3d items should be identical)  stop (there are no 4-itemset candidates that can be generated)

Clothing Example – Generation of AR Rules • The next step is to use the frequent itemsets and generate association rules using the ARGen algorithm (slide 14) • =50% • The set of frequent itemsets is L={{Jeans},{Shoes}, {Shorts}, {Skirt}, {TShirt}, {Jeans, Shoes}, {Jeans, Shorts}, {Jeans, TShirt}, {Shoes, Shorts}, {Shoes, TShirt}, {Shorts, TShirt}, {Skirt, TShirt}, {Jeans, Shoes, Shorts}, {Jeans, Shoes, TShirt}, {Jeans, Shorts, TShirt},{Shoes, Shorts, TShirt}, {Jeans, Shoes, Shorts,TShirt} } • We ignore the first 5 as they do not consists of 2 nonempty subsets of frequent itemsets. We test all the others, e.g.: etc.

Frequent Itemset Generation in Apriori Pseudo Code e.g. Fk-1 x F1 or Fk-1 x Fk-1, etc.

FP-Growth Algorithm

Frequent Pattern Growth (FP-Growth) Algorithm • Apriori: generate-and-test approach – generates candidate itemsets and tests if they are frequent • Problem: the generation of candidate itemsets is expensive • FP-growth – the first algorithm that allows frequent itemset discovery without candidate itemsets generation • Uses a compact data structure called FP-tree and extracts frequent itemsets directly from the FP-tree

FP-Tree • Nodes correspond to items + have a counter • 1 path = 1 transaction • Reads 1 transaction at a time and maps it to a path • Pointers between nodes containing same item • Paths may overlap as transactions share items => increment the counter and add pointers • More paths overlap -> higher compression • => FP-tree may fit in the memory => direct extraction of frequent itemsets from FP-tree instead of many passes over data stored on disk

FP-Tree Construction • Pass 1: Scan data and find support for each item. Discard infrequent items. Sort frequent items in decreasing order based on their support. • For our example: a, b, c, d, e • Pass 2: construct the FP-Tree • Read trans. 1 {a, b}. Create 2 nodes a and b and the path null->a->b. Set counts of a and b to 1. • Read trans. 2 {b, c, d}. Create 3 nodes for b, c and d and the path null->b->c->d. Set counts to 1. Note that although trans. 1 and 2 share b, the paths are disjoint as they don’t share a common prefix. • Read trans. 3 {a, c, d, e}. It shares common prefix item {a} with trans. 1 => the path for trans. 1 and 3 will overlap and the frequency count for node a will be incremented with 1. • Continue until all transactions are mapped to a path in the FP-tree.

FP-Tree Size • FP-tree has a smaller size than the uncompressed data as many transactions share items • Best case scenario – all transactions contain the same set of items. • 1 path in the FP-Tree • Worst case scenario – every transaction has a unique set of items (no items in common) • the size of the FP-tree = size of the original data • However, the storage requirements for the FP-tree are higher – need to store the pointers between the nodes and the counters • The size of the FP-tree depends on how the items are ordered • Ordering by decreasing support is typically used but it does not always lead to the smallest tree

Complete FP-tree FP-Growth Algorithm • Extracts frequent itemsets from FP-tree • Bottom-up algorithm – from the leaves to the root • For our example – first look for frequent itemsets ending in e, than in d, c, b and a (Note: reverse lexiographic order!) • Extract the paths ending in e, d, c, b and a (called also prefix paths) Prefix paths ending in e, d, c, b and a

FP-Growth Algorithm (cont. 1) • Each prefix path sub-tree is processed recursively to extract the frequent itemsets and the solutions are then merged • e.g. the prefix path sub-tree for e will be used to extract frequent itemsets ending in e, than in de, ce, be and ae. Each of them can be decomposed into problems, e.g. de into cde,bde, cde, etc. e -> de -> cde bde ade ce -> bce ace be-> abc ae d . . . c . . . b . . . a . . . • End of recursion: no more frequent itemsets can be extracted, i.e. empty tree or tree with 1 item where • tree=prefix path sub-tree or conditional FP-tree

FP-Growth Algorithm - Example • Extract frequent itemsets for e. Let minsup = 2. • 1) Obtain the prefix path sub-tree for e • 2) Check if {e} is a frequent item by adding the counts. If so, extract it. • Yes, count =3 => {e} is extracted as a frequent itemset. • 3) As {e} is frequent, find frequent itemsets ending in de, ce, be and ae. To do this, we need first to obtain the conditional FP-tree for e. Prefix paths ending in e, d, c, b and a

2 2 2 1 1 1 1 1 1 cut cut FP-Growth Example (cont. 2) • To obtain the conditional FP-tree for e: • Update the support counts along the prefix paths to reflect the number of transactions containing e => b and c should be set to 1, a to 2 • Remove the nodes containing e – information about node e is no longer needed because of the previous step • Remove infrequent items (nodes) from the prefix paths, e.g. b has a support of 1 and appears once => there is only 1 trans. containing b and e => be is infrequent => remove b. Final Conditional FP-tree for e

FP-Growth Example (cont. 3) • 4) Use the the conditional FP-tree for e to find frequent itemsets endingin de, ce and ae • Note that be is not considered as b is not in the conditional FP-tree • For each of them (e.g. de) find the prefix paths from the conditional tree for e and extract frequent itemsets; generate conditional FP-tree etc. Extract {e} Frequent itemsets ending with de? Contains 1 item; no need to generate prefix paths ending in ade (will be the same as the cond. FP tree for de); extract frequent itemsets (if any) and stop this branch of the recursion. Continue with itemsets ending with ce. Extract {d,e} Frequent itemsets ending with ade? Extract {a,d,e}

FP-Growth Example - Solution • FP-Growth algorithm will find the following frequent itemsets:

Discussion • Association rules are typically sought for very large databases => efficient algorithms are needed • The Apriori algorithm makes 1 pass through the dataset for each different itemset size • The maximum number of database scans is k+1, where k is the cardinality of the largest frequent itemset (4 in the clothing ex.) • potentially large number of scans – weakness of Apriori • Sometimes the database is too big to be kept in memory and must be kept on disk • The amount of computation also depends on the minimum support; the confidence has less impact as it does not affect the number of passes • Variations • Using sampling of the database • Using partitioning of the database • Generation of incremental rules

Discussion (2) • FP-growth is typically an order of magnitude faster than Apriori • No candidate generation • Uses compact data structure • Only 2 scans of the database: 1 to count the support of each item and 2 to build the FP-tree • Basic operation is FP-tree building and counting

COMP5318 Knowledge Discovery and Data Mining