G. Punzi - PHYSTAT 05 - Oxford, UK

1. Ordering algorithms and Confidence Intervals in the presence of nuisance parameters Giovanni Punzi SNS & INFN-Pisa giovanni.punzi@pi.infn.it G. Punzi - PHYSTAT 05 - Oxford, UK

2. My benchmark problem • Poisson(+background), with a systematic uncertainty on efficiency: x : Pois(  b) e : G(, ) • • • e is a measurement of the unknown efficiency , with resolution    is the efficiency (a “normalization factor”, can be larger than 1). A common example, the CDF statistics committee is using it as a benchmark in comparing various techniques in setting limits. We usually take G to be a Poisson, in order to allow Bayesian solutions. Today I will take G Gaussian for simplicity , as in the frequentist context e=0 or e<0 is not a problem. (Negative values can actually occur in practice, e.g., due to background-subtracted measurements) • • Method: full Neyman construction. This led me to revisiting some old stuff and adding something new G. Punzi - PHYSTAT 05 - Oxford, UK

3. Neyman construction with nuisance parameters 1) Build a confidence band, by treating the nuisance parameter as any other parameter: p( (x, e) | (µ, ) ) = p(x|µ, )*p(e | ) 2) Evaluate CR in (µ, ) from the measurement (x0, e0) 3) Project onto µ space to get rid of information on  observables Confidence band (x,e) (x0,e0) (m, parameters • GOOD: – Conceptually clean and simple. – Guaranteed coverage. – Well-behaved. BAD: – CPU - expensive, especially in large dimensions – Overcoverage – Sensitive to ordering algorithm – Limit for 0 uncertainty problematic (x0,e0)  • mmin mmax m Confidence interval on µ G. Punzi - PHYSTAT 05 - Oxford, UK

4. Past experience with projection method • Some success stories: - Improved solution of the classical Poisson ratio problem (coordinated projections) [See R. Cousins talk] - Hypothesis Testing in Poisson+background with uncertainty (prof LR) [K. Cranmer at PHYSTAT03 and PHYSTAT05] • As far as I know, only one “life size” HEP application (prof LR) – Full-blown analysis of CHOOZ experiment* [Durham‘02, G.P. and Signorelli] • No overwhelming enthusiasm. Most often replaced by some frequentist approximation: – Profile – Smearing (Cousins-Highland) • Wish list: – Minimize overcoverage. – Reasonable CPU requirements – Good behavior when 0 *Eur. Phys.J. C27:331(2003) (arXiv:hep-ex/0301017) G. Punzi - PHYSTAT 05 - Oxford, UK

5. The issue with overcoverage • • Projecting on µ enlarges the CR  overcoverage. BUT: we chose to ignore information on   we may be able to trade a part of it for additional info on µ, but some overcoverage may just be a natural consequence. No basis to assume there is an intrinsic weakness in the projection method: it may well be that any algorithm that covers can’t avoid some overcoverage (as in the discretization effect) Real Question: how can I find the smallest µ interval that does not undercover ? => You want to make the confidence region as close as possible to a rectangle Difficulty: the band is build in the x-e plane, but the results are seen in the µ- plane. Not very intuitive, also considering that the minimum non-trivial dimensionality is 4 ideal shape of conf. region (x0,e0) •  • mmin mmax m • • G. Punzi - PHYSTAT 05 - Oxford, UK

6. Looking for “minimal” overcoverage • We don’t know in general what the optimal band is for a particular problem. However, a necessary (but not sufficient) intuitive requirement for optimality is: For an optimal band, of any two acceptance regions at different values of  , one must be completely included in the other. Non-overlapping regions: BAD (m0) e  If this weren’t true, one might improve limits by “shifting” one towards the other x • One way to achieve the above property is: Use an ordering algorithm in the whole observable space, that does not depend on N.B. also good from the computational viewpoint G. Punzi - PHYSTAT 05 - Oxford, UK

7. Which Ordering ? The requirement of the ordering in (x,e) to be independent of  is a strong constraint: the density p(x,e) will depend on . You will find yourself including in the some values of x,e with very low probability. Must choose ordering wisely. We wish the ordering to converge to the 0-syst ordering, for every . • For each e0, order in x as you would do in the case • • (m0) e Order as per L(0, ˆ (e2)) e2 Order as per L(0, ˆ (e1)) e1 A x This is not yet sufficient, as the ordering must be 2D: need to define how to order in e. But let’s ignore this for a moment and look at an example. G. Punzi - PHYSTAT 05 - Oxford, UK

8. Example: Upper limits, naïve version • Simply use for any e the observed x as ordering function, This is what you normally do in absence of systematics to evaluate upper limits. µ () = 10% 0 uncertainty + region - region G. Punzi - PHYSTAT 05 - Oxford, UK x

9. Upper limits, naïve version (x ordering) Coverage  µ Max coverage Not very exciting… Min coverage G. Punzi - PHYSTAT 05 - Oxford, UK

10. Ordering in the whole observable space • You don’t want to P-order in e. P depends on , and the ordering must not depend on it. Also, we are trying to ignore as much as possible the information relevant to . We want to be as uniform as possible in e. Also, we want the ordering to converge to the “local” ordering for each e, when (m0) e Order as per L(0, ˆ (e2)) e2 Order as per L(0, ˆ (e1)) e1 A •  Order in such a way as to integrate the same conditional probability at each e: 0 p(x |e1,0, ˆ (e1))dx  p(x |e2,0, ˆ (e2))dx A A This gives the ordering function: p(x'|e, 0, ˆ (e))dx' f(x,e;0)  f0(x')f0(x0)  Where f0(x) is the ordering function adopted for the 0-systematics case N.B. I also do some clipping, but mostly for computational reasons G. Punzi - PHYSTAT 05 - Oxford, UK

11. Upper limits, with appropriate 2-D ordering Coverage coverage  Max coverage µ Max coverage Pretty good ! No overcoverage beyond “discretization ripples” Min coverage G. Punzi - PHYSTAT 05 - Oxford, UK

12. Unified limits (F-C) • Let’s say we want Unified limits. The exact procedure, would be to evaluate the probability distribution of LR for each e, and order on it over the space. However, the LR theorem tells you that the distribution of the LR is ~indipendent of the parameters => can use the LR itself as approximation of the ordering function.  Re-discover profile-LR method as an approximation of the current method • 0 uncertainty + region - region () = 10% G. Punzi - PHYSTAT 05 - Oxford, UK

13. Unified limits with systematics Coverage coverage  µ Average coverage Max/Min coverage Pretty good. No need for adjustments, at least in this example G. Punzi - PHYSTAT 05 - Oxford, UK

14. The issue of 0-syst limit The goal was the get the limit to approach the desired 0-syst result when 0 but it doesn’t quite work • µ max  • The problem here is not with the algorithm, but rather due to the transition between discrete and continuous. [G.Feldman, CL workshop@FNAL] G. Punzi - PHYSTAT 05 - Oxford, UK

15. Solution of the 0-syst problem Prevent calculation grid from becoming finer than 0.1 along the “nuisance observable” axis. • µ max  • The problem gets solved in an easy and natural way - can have a perfect match built into the algorithm. G. Punzi - PHYSTAT 05 - Oxford, UK

16. Systematic uncertainty given as a range • Very often, the only information we have about a nuisance parameter is a range: emin< e < emax • Example: theoretical parameters. But many others: all cases where we don’t have an actual experimental measurement of the nuisance parameter. • Automatically handled in the current algorithm (at no point a subsidiary measurement is required, it just enters as part of the pdf). Fast, and free from “transition to continuum” issues • N.B. the result is NOT equivalent to varying the parameter within the range, and take the most conservative interval. Ordering function p(x|2) p(x|1) G. Punzi - PHYSTAT 05 - Oxford, UK x

17. Systematics as a range • Example: 0.85<  <1.15 G. Punzi - PHYSTAT 05 - Oxford, UK

18. Coverage for the “no-distribution” case Coverage  µ G. Punzi - PHYSTAT 05 - Oxford, UK

19. Compare “no distribution” with “flat” Assuming a uniform distribution provides you with more information than just a range => get tighter limits as expected Flat > range Flat< range G. Punzi - PHYSTAT 05 - Oxford, UK

20. Computational Aspects 1) The ordering algorithm being independent of the nuisance parameters makes it much more convenient to perform the calculations • one can order just one time for each value of mu, and then go through the same list every time, just skipping the elements that are clipped off for that particular value of the nuisance parameter. The clipping reduces the amount of space to be searched The chosen ordering naturally provides acceptance regions that are slowly varying with the subsidiary measurement=> need to sample only a very small number of points. (used 10, but 2 would be enough) For the same reasons, when multiple nuisance parameters are present, large combinatorics is unnecessary: sampling the corners of the hyperrectangle is sufficient (plus maybe additional random points). The discreteness limitation further reduces the time needed. Uncertainty given as ranges are extremely fast to calculate - one might willingly use this method for fast evaluation. They are also immune to the 0-limit problem, so need no adjustments. 2) 3) 4) 5) 6) G. Punzi - PHYSTAT 05 - Oxford, UK

21. CONCLUSIONS • Full Neyman construction still alive for problems with nuisance parameters. • It is possible, and practical, to choose any desired ordering algorithm for the 0-systematics case, and extend it. • You can have the 0-systematics limits exactly right. • Fast, accurate algorithms are possible that never undercover and have very limited overcoverage. • Can easily use uncertainties given as ranges - actually a very convenient option whenever possible. G. Punzi - PHYSTAT 05 - Oxford, UK

22. BACKUP G. Punzi - PHYSTAT 05 - Oxford, UK

23. Upper limits, better version G. Punzi - PHYSTAT 05 - Oxford, UK