Automated Failure Detection in Internet Services via Changes in User Behavior Lukas Biewald Gregory Friedman
Hypothesis: We can detect web app bugs through user behavior Session 1 Session 2 Session n index.html index.html index.html … filter.html filter.html filter.html news.html news.html news.html OK OK filter.html BUG?
Classification/Anomaly Detection One Class Support Vector Machine (Burges 1998) Features: Strings of session data, request data. (Try a string kernel…) Index,News,Filter,Index… Advantages: SVM can handle the large feature space, easy to implement and known to work well for a large range of problems. Index,Filter,Filter,News… … Hidden Markov Model (HMM) Advantages: HMM can easily handle partially observed training data. Features aren’t as rich. (Could be good or bad) Bug Bug Bug Bug Data Data Data Data Here Data could be webpage bigrams and trigram frequencies over sessions.
Difficulties Can we find web logs with sessions and matching bug reports? Talking to Amazon, Ofoto, Ebates… How reliable is the bug report data? Can use unsupervised/partially supervised training… Do the bugs really change user behavior in interesting ways? Unknown, but it seems like some of them should…