1 / 36

Statistical Models for Probabilistic Forecasting

Statistical Models for Probabilistic Forecasting. Ian Jolliffe i.t.jolliffe@reading.ac.uk. Introduction –Statistical Models Linear Regression Logistic Regression Discriminant Analysis Other methods/models. Statistical Models. A model can be written in general form as y = g(x) + e.

Anita
Télécharger la présentation

Statistical Models for Probabilistic Forecasting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical Models for Probabilistic Forecasting Ian Jolliffe i.t.jolliffe@reading.ac.uk • Introduction –Statistical Models • Linear Regression • Logistic Regression • Discriminant Analysis • Other methods/models AMS Short Course on Probabilistic Forecasting, 9th January 2005

  2. Statistical Models • A model can be written in general form as y = g(x) + e. • y is something to be predicted (a predictand). It might be a spatial field, but to keep things simple, suppose it is a single quantity at a single location, perhaps temperature, windspeed or probability of precipitation. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  3. Statistical Models II y = g(x) + e • x denotes one or (often) more predictors – it could be something like yesterday’s temperature, or contain many variables. • g(.) is some function. • e is an error term, included because we don’t have perfect predictions (life would be boring if we did). • A crucial point regarding e is that it is random – it has a probability distribution. It quantifies one aspect of uncertainty – there are others associated with the function g(x). AMS Short Course on Probabilistic Forecasting, 9th January 2005

  4. Statistical Models III • Many operational schemes, for example Model Output Statistics (MOS), use approaches where y is tomorrow’s value of a single weather element and x consists of forecasts made today, using dynamical numerical weather prediction (NWP), of tomorrow’s values of relevant weather elements. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  5. Linear Regression • The simplest and most common type of model is when g(.) is linear, so The variables x1, x2, …, xm, are the predictors. If m=1, we have simple linear regression; otherwise multiple linear regression. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  6. Linear regression II • In this course, we are concerned with probability forecasts, so our predictand is a probability, for example • Probability of precipitation • Probability of temperature below a threshold such as 0°C • Probability of windspeed greater than some threshold AMS Short Course on Probabilistic Forecasting, 9th January 2005

  7. Linear regression III • For continuous variables it is arguably more informative to forecast the actual temperature or windspeed, rather than a threshold probability. • This is not true if, as often happens, only a single predicted value is given. • It would be true if we use distributional assumptions about the error term, e, to construct a probability distribution for our prediction (forecast). • Creating such a probability distribution, and perhaps creating prediction intervals from it, is one type of probability forecast. In this session, we concentrate on the (simpler) case where we forecast the probability of a binary event. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  8. Simple (toy) example • Data from Collieston, Scotland, December 2001-March 2002, (105 days, 16 missing values) on • Air frost (presence/absence) • Maximum temperature, previous day • Wind speed (Beaufort), previous evening • Attempt to forecast probability of air frost, given the values of the other two variables. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  9. Example – linear regression • A naïve approach uses • where y, x1, x2 denote frost (0,1), maximum temperature and wind speed respectively. • Values of , 1, 2 are chosen to minimise the sum of squared differences between y predicted from the fitted equation and its actual value (a least squares fit). • The predicted value of y can be used as an estimate of the probability of frost. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  10. Example – linear regression II • For any pair of values of max. temp. and wind we get an estimated probability of frost. • Because of linearity, there is no bound on predicted values, but actual values must be between 0 and 1. Max. predicted value is 1.03, min. is –0.20. • Also there is no straightforward way to assess the uncertainty associated with the estimates – the usual method makes unrealistic (Gaussian) assumptions. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  11. Logistic regression • Data are transformed so that predictions are restricted to be between 0 and 1. • Let p = probability of air frost, and consider the model AMS Short Course on Probabilistic Forecasting, 9th January 2005

  12. Some comments on logistic regression • We now appear to model p, rather than the 0-1 variable y. However, p is the expected value of y. • If y is modelled there will be an error term. We have omitted it in both forms of the equation for simplicity. • We have more than one predictor so this is multiple logistic regression. (Simple) logistic regression has a single predictor – see example later. • The equation is sometimes written in reciprocal form: AMS Short Course on Probabilistic Forecasting, 9th January 2005

  13. (Multiple) Logistic regression - example Same data as before – predict probability of frost from wind and max. temp. Fitted equation to predict p is Note the curvature in the plot. The estimated probability cannot go outside the range (0,1). The different points for the same temperature correspond to different wind speeds at that temperature. For example for t=3.5, w=3, the estimated probability is 0.630; for t=3.5, w=1 it is 0.823; for t=3.5, w=0 it is 0.885 – see diagram. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  14. (Simple) Logistic Regression - Example Same data as before but predict probability of frost only from temperature. Equation is: We now have a single estimated probability of frost for each (yesterday’s max.) temperature. For example, it is 0.658 for t=3.5; 0.361 for t=6. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  15. Fitting a logistic regression equation • In theory we could use least squares (LS), but it is more usual to use maximum likelihood (ML) estimation as this allows us to make inferences about the model (see later). • To implement ML we write down the joint probability distribution of the data – the predictand on the ith day, yi, has a Bernouilli distribution with ‘probability of success’ with obvious notation. Take the product of pi for ‘successes’ and (1-pi) for ‘failures’; then maximum this joint probability (likelihood) with respect to , 1, and 2 . AMS Short Course on Probabilistic Forecasting, 9th January 2005

  16. More on fitting and model choice • The data can be presented in two different ways: • As a sequence of 0s and 1s • If there are only a small number of different values for the predictors, then we may have the proportion of successes for each combination of predictor values. This seems more natural – if we are modelling the probability of an event it can be estimated by the proportion of times it occurs. • In fact, the ML approach gives the same fitted curve in both cases. • If there are many potential predictors, step-wise selection methods are available, as in linear regression. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  17. Inference for regression • Back to linear regression: • If we assume that the error term is Gaussian we can make various types of inference • Inference about the parameters, usually hypothesis tests or confidence intervals for the slope parameter(s) • Confidence intervals for g(x), the ‘true’ line • Prediction intervals for y – these take into account uncertainty about g(x) and uncertainty associated with the error term. • Similar things can be done for logistic regression, though the distributional assumption is different. Aside: for linear regression LS fitting is equivalent to ML when a Gaussian assumption (inappropriate here) is made. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  18. Inference for logistic regression The output below (from Minitab) relates to the fitted model above, with test statistics and p-values for the null hypotheses that ,  are zero; an estimate of odds ratio = exp() – see later; a confidence interval for exp(); various goodness-of-fit statistics. Logistic Regression Table Odds 95% CI Predictor Coef SE Coef Z P Ratio Lower Upper Constant 2.36415 0.884408 2.67 0.008 Temperature -0.488954 0.129382 -3.78 0.000 0.61 0.48 0.79 Goodness-of-Fit Tests Method Chi-Square DF P Pearson 15.8375 18 0.604 Deviance 18.8651 18 0.400 Hosmer-Lemeshow 10.2924 7 0.173 AMS Short Course on Probabilistic Forecasting, 9th January 2005

  19. Inference for logistic regression inference II • ‘Odds ratio’, exp(), measures by how much the odds p/(1-p) change when max. temp. increases by 1°C. • Confidence intervals can be found for ,  [as well as exp()] in a number of ways. The simplest is to add/subtract to/from the estimate a multiple (e.g 1.96 for a 95% interval) of the standard error of the estimate. Such intervals are approximate – OK for large samples. • For small samples, ‘exact’ inference is possible but more complicated. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  20. Inference for logistic regression - inference III • If logistic regression is used for probabilistic forecasting: • The ‘point estimate’ of the probability is of greatest interest. • Inference mentioned so far is not of direct interest, except in understanding the predictions. • Intermediate in interest is a confidence interval/band for g(x), the ‘true’ value of p for given values of the predictor(s). This can be done approximately, using estimated variances of the estimated ,  and their covariance, first for  + x, then transform to p - not available in Minitab or SPSS. • In Iinear regression, prediction intervals for future random variables are often of interest, but not relevant here because our random variable takes only the values 0 and 1. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  21. Generalised linear models (GLMs) • Both linearregression and logistic regression are special cases of y = g(x) + e. They are also special cases of GLMs. • These models specify a probability distribution for y, and a link function which expresses a function of the mean of y as a linear function of the predictors: • For linear regression the link function is the identity function, and the distribution often Gaussian • For logistic regression the link function is ln(p/(1-p)) [the mean of y is p] and the distribution is Bernouilli. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  22. GLMs II • When predicting probabilities, with binary data, the Bernouilli distribution is the only candidate distribution but the link can be different from our logit link, for example: • Probit link – the inverse of the distribution function of a standard Gaussian -1(p) replaces the logit link ln[p/(1-p)] and has a very similar shape to it. • Complementary log-log link has link function ln[-ln(1-p)]. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  23. Discriminant Analysis • Given data x from two or more groups we wish to find rules that use the value of x to ‘discriminate between the groups’. • There are a number of different possible objectives covered by this phrase: • Use the rule to assign each observation to one of the groups • Use the rule to estimate probabilities of an observation belonging to each group • For high-dimensional x find a lower-dimensional space that best displays differences between groups (sometimes called canonical variate analysis - it is rather like EOF analysis, but with a different criterion to optimise) • Objective 2 is clearly relevant to predicting probabilities. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  24. Discriminant Analysis II How to construct a rule? • Use a linear function of x, with a threshold if definite assignment is to be made (linear discriminant analysis). • Replace ‘linear’ by ‘quadratic’ in 1. • Some sort of tree approach – binary splits on individual variables until subgroups are sufficiently homogeneous with respect to groups e. g. CART (Classification and Regression Trees). • Use group membership of nearest neighbours. • Various others – the general problem is called ‘supervised learning’ in the neural network literature. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  25. Linear discriminant analysis for our earlier example Summary of classification True Group Put into Group 0 1 0 57 5 1 19 24 Total N 76 29 N = 105 N Correct = 81 Proportion Correct = 0.771 Linear Discriminant Function for Groups Rule for determining which group to 0 1 assign observations to. Calculate Constant -9.1299 -4.5860 these linearfunctions and see which Temperature 1.8172 1.3303 is maximum. Wind 1.2546 0.7661 Probabilities of selected Summary of Misclassified Observations observations fallingin each group, with/without cross-validation True Pred X-val Squared Distance Probability Observation Group Group Group Group Pred X-val Pred X-val 15** 0 1 1 0 1.7532 1.8141 0.34 0.34 1 0.4590 0.4554 0.66 0.66 16** 0 1 1 0 3.232 3.395 0.34 0.33 1 1.935 1.947 0.66 0.67 17** 0 1 1 0 2.4856 2.5908 0.24 0.23 1 0.2176 0.2183 0.76 0.77 AMS Short Course on Probabilistic Forecasting, 9th January 2005

  26. Linear discriminant analysis example – some explanations • Linear discriminant function(s) [l.d.f.s]. One per group given here. Given values of predictors, assign a observation to the group with the largest value of the l.d.f. Sometimes (equivalently) a single l.d.f. is given, with a threshold determining assignment of observations. • With many predictors, stepwise selection of predictors, as in regression, can be used. • Canonical variate analysis gives the same single l.d.f., but with many predictors one or more extra linear functions can be derived and observations plotted in 2 (or higher) dimensions. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  27. More explanations and modifications • Also given in the output are posterior probabilities for the two groups, for selected observations. • These are calculated by multiplying prior probabilities by likelihoods (a Bayesian approach) • Prior probabilities must be supplied – here we assume equal probabilities for the two groups, but see the next slide • Likelihood is just the probability density function of an observation, assuming membership of a particular group. To calculate this, a Gaussian assumption is usually made for the predictors. • The software also mentions cross-validation – for a given observation, use rules/calculations computed leaving out that observation. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  28. Changing the prior probabilities In the first case below, prior probabilities are equal and 15, 16 are misclassified. In the second, prior probabilities of no frost/frost are taken to be 0.75, 0.25, so posterior probabilities of no frost also increase and we are more likely than before to classify observations as ‘no frost’ Squared Posterior True Pred X-val Distance Probability Observation Group Group Group Group Pred X-val Pred X-val 15** 0 1 1 0 3.139 3.200 0.34 0.34 1 1.845 1.842 0.66 0.66 16** 0 1 1 0 4.618 4.781 0.34 0.33 1 3.321 3.334 0.66 0.67 15 0 0 0 0 2.329 2.389 0.61 0.60 1 3.232 3.228 0.39 0.40 16 0 0 0 0 3.807 3.970 0.61 0.59 1 4.707 4.720 0.39 0.41 Overall, with equal priors 19 no frost days and 5 frost days are misclassified; for a 0.75:0.25 prior, the figures are 6 and 17. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  29. Regression and discrimination – some links • If we formulate the discrimination problem as a regression analysis with a binary predictand and appropriate coding for the two values, the l.d.f. pops out as the linear prediction function. • Logistic discrimination doesn’t assume specific distributions for its likelihoods, but does assume that the log of the ratio of the likelihoods in the two groups is a linear function of the predictors. This implies a linear form in the predictors for the log of the ratio of the posterior probabilities (the priors are incorporated in the constant term) • This is equivalent to logistic regression • Several distributions, including Gaussian, satisfy the assumption, so there is an equivalence to linear discriminant analysis as well. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  30. Extensions (but first something simple) • For a binary forecast we can look at historical data and see what proportion of the time the event occurred. This proportion can then be used as the probability forecast (a climatological forecast). • We can be slightly more sophisticated if there are only a few combinations of predictor values and/or lots of data. For a given combination, find the proportion of the time the event occurred. This proportion can then be used as the probability forecast for that combination ( a contingency table approach). AMS Short Course on Probabilistic Forecasting, 9th January 2005

  31. Extensions II • We have concentrated on binary forecasts, though continuous data have been mentioned briefly. • Another major data type is categorical with 3 or more categories, where we want probability forecasts for each category • We can use linear regression separately for each category (REEP – regression estimation of event probabilities) • We can use other techniques for binary forecasts (logistic regression, discriminant analysis etc.) for each category • There are extensions of logistic regression for ordered categories (the most usual case) or unordered (nominal) categories • Discriminant analysis can also be extended to more than 2 groups (any ordering is not taken into account). AMS Short Course on Probabilistic Forecasting, 9th January 2005

  32. Readings I – Gahrs et al. (2003) • Data – 24-hour precipitation amounts above various thresholds. 234 potential predictors – reduced to 20, then fewer. • Methods – linear regression, logistic regression, ‘binning’ (a contingency table approach). Methods for variable selection and parameter estimation discussed. • Results – logistic regression outperforms linear regression. ‘Binning’ is competitive at some, but not all, thresholds. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  33. Readings II – Hamill et al. (2004) • Data – 6-10 day and week 2 forecasts of surface temperature and precipitation over thresholds, corresponding to terciles, for 484 stations. 15-member ensembles of forecasts are available. • Method – logistic regression using as a single predictor, ensemble mean forecast (precipitation) or forecast anomaly (temperature). A MOS approach. • Results – method outperforms operational NCEP forecasts for data set examined. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  34. Readings III – Hennon & Hobgood (2003) • Data – properties of tropical cloud clusters during Atlantic hurricane seasons, and whether they develop into tropical depressions(DV) or not – 8 predictors. • Method – linear discriminant analysis giving a probability, p, of DV. • DV is unlikely if p < 0.7, likely if p > 0.9. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  35. Readings IV – Mason & Mimmack (2002) • Data – monthly ENSO anomalies (Niño-3.4) grouped into 5 equiprobable categories. Predictors are 5 leading principal components of tropical Pacific sea surface temperatures in an earlier month. • Methods – various methods for two categories implemented separately for each category: linear regression (two variants including a ‘contingency table’ approach), two forms of discriminant analysis, 3 GLMs including logistic regression. Also extensions of GLMs to more than two categories. • Results – the main comparison is models vs. (damped) persistence, not between the models themselves – see next slide. AMS Short Course on Probabilistic Forecasting, 9th January 2005

  36. FIG. 2. Ranked probability skill scores for retroactive forecasts at increasing lead times of monthly Niño-3.4 sea surface temperature anomaly categories for the 20-yr period Jan 1981–Dec 2000. The skill scores are calculated with reference to a strategy of random guessing. The black bars represent the scores for the models, and the dark (light) gray bars are for forecasts of persisted anomaly categories (damped toward climatology). [previous][next] Example with several groups/categories From Mason and Mimmack (2002) - compares various methods when there are 5 categories to forecast. Their ‘canonical variate’ method produces probabilities equivalent to our ‘discriminant analysis; their discriminant analysis (predictive) is slightly different. Figure 2. Ranked probability skill scores for retroactive forecasts at increasing lead times of monthly Niño-3.4 sea surface temperature anomaly categories for the 20-yr period Jan 1981–Dec 2000. The skill scores are calculated with reference to a strategy of random guessing. The black bars represent the scores for the models, and the dark (light) gray bars are for forecasts of persisted anomaly categories (damped toward climatology). AMS Short Course on Probabilistic Forecasting, 9th January 2005

More Related