D Important Probability Distributions Development of stochastic models is facilitated by identifying a few probability distributions that seem to correspond to a variety of data-generating processes, and then studying the properties of these distributions. In the following tables, I list some of the more useful distributions, both discrete distributions and continuous ones. The names listed are the most common names, although some distributions go by different names, especially for specific values of the parameters. In the first column, following the name of the distribution, the parameter space is specified. There are two very special continuous distributions, for which I use special symbols: the uniform over the interval [a, b], designated U(a, b), and the normal (or Gaussian), denoted by N(µ, σ 2 ). Notice that the second parameter in the notation for the normal is the. Sometimes, such as in the functions in R, the second parameter of the normal distribution is the standard deviation instead of the. A normal distribution with µ = 0 and σ 2 = is called the standard normal. I also often use the notation φ(x) for the of a standard normal and Φ(x) for the CDF of a standard normal, and these are generalized in the obvious way as φ(x µ, σ 2 ) and Φ(x µ, σ 2 ). Except for the uniform and the normal, I designate distributions by a name followed by symbols for the parameters, for example, binomial(π, n) or gamma(α, β). Some families of distributions are subfamilies of larger families. For example, the usual gamma family of distributions is a the two-parameter subfamily of the three-parameter gamma. There are other general families of probability distributions that are defined in terms of a differential equation or of a form for the CDF. These include the Pearson, Johnson, Burr, and Tukey s lambda distributions. Most of the common distributions fall naturally into one of two classes. They have either a countable support with positive probability at each point in the support, or a continuous (dense, uncountable) support with zero probability for any subset with zero Lebesgue measure. The distributions listed in the following tables are divided into these two natural classes. Elements of Computational Statistics, Second Edition c 203 James E. Gentle
48 Appendix D. Important Probability Distributions There are situations for which these two distinct classes are not appropriate. For many such situations, however, a mixture distribution provides an appropriate model. We can express a of a mixture distribution as p M (y) = m ω j p j (y θ j ), j= where the m distributions with s p j can be either discrete or continuous. A simple example is a probability model for the amount of rainfall in a given period, say a day. It is likely that a nonzero probability should be associated with zero rainfall, but with no other amount of rainfall. In the model above, m is 2, ω is the probability of no rain, p is a degenerate with a value of at 0, ω 2 = ω, and p 2 is some continuous over IR +, possibly similar to a distribution in the exponential family. A mixture family that is useful in robustness studies is the ɛ-mixture distribution family, which is characterized by a given family with CDF P that is referred to as the reference distribution, together with a point x c and a weight ɛ. The CDF of a ɛ-mixture distribution family is P xc,ɛ(x) = ( ɛ)p(x) + ɛi [xc, [(x), where 0 ɛ. Another example of a mixture distribution is a binomial with constant parameter n, but with a nonconstant parameter π. In many applications, if an identical binomial distribution is assumed (that is, a constant π), it is often the case that over-dispersion will be observed; that is, the sample exceeds what would be expected given an estimate of some other parameter that determines the population. This situation can occur in a model, such as the binomial, in which a single parameter determines both the first and second moments. The mixture model above in which each p j is a binomial with parameters n and π j may be a better model. Of course, we can extend this kind of mixing even further. Instead of ω j p j (y θ j ) with ω j 0 and m j= ω j =, we can take ω(θ)p(y θ) with ω(θ) 0 and ω(θ)dθ =, from which we recognize that ω(θ) is a and θ can be considered to be the realization of a random variable. Extending the example of the mixture of binomial distributions, we may choose some reasonable ω(π). An obvious choice is a beta. This yields the beta-binomial distribution, with ( ) n Γ(α + β) p X,Π (x, π) = x Γ(α)Γ(β) πx+α ( π) n x+β I {0,,...,n} ]0,[ (x, π). This is a standard distribution but I did not include it in the tables below. This distribution may be useful in situations in which a binomial model is appropriate, but the probability parameter is changing more-or-less continuously. Elements of Computational Statistics, Second Edition c 203 James E. Gentle
Appendix D. Important Probability Distributions 49 We recognize a basic property of any mixture distribution: It is a joint distribution factored as a marginal (prior) for a random variable, which is often not observable, and a conditional distribution for another random variable, which is usually the observable variable of interest. In Bayesian analyses, the first two assumptions (a prior distribution for the parameters and a conditional distribution for the observable) lead immediately to a mixture distribution. The beta-binomial above arises in a canonical example of Bayesian analysis. Some distributions are recognized because of their use as conjugate priors and their relationship to sampling distributions. These include the inverted chi-square and the inverted Wishart. General References Evans et al. (2000)give general descriptions of 40 probability distributions. Balakrishnan and Nevzorov (2003) provide an overview of the important characteristics that distinguish different distributions and then describe the important characteristics of many common distributions. Leemis and McQueston (2008) present an interesting compact graph of the relationships among a large number of probability distributions. Currently, the most readily accessible summary of common probability distributions is Wikipedia: http://wikipedia.org/ Search under the name of the distribution. Elements of Computational Statistics, Second Edition c 203 James E. Gentle
420 Appendix D. Important Probability Distributions Table D.. Discrete Distributions (s are wrt counting measure) discrete uniform a,..., a m IR mean m, y = a,..., am P ai/m P (ai ā) 2 /m, where ā = P a i/m Bernoulli π y ( π) y, y = 0, π ]0,[ mean π binomial (n Bernoullis) π( π)! n π y ( π) n y, y n =,2,... ; π ]0,[ CF ( π + πe it ) n mean nπ nπ( π) geometric π( π) y, y=0,,2,... π ]0,[ mean ( π)/π y = 0,,..., n ( π)/π 2! y + n negative binomial (n geometrics) π n ( π) y, y = 0,,2,... n «n π n =,2,... ; π ]0,[ CF ( π)e it multinomial n =,2,..., mean n( π)/π n( π)/π 2 n! dy Q π y i yi! i, yi = 0,,..., n, X y i = n CF i= Pd n i= πieit i for i =,..., d, π i ]0,[, P π i = means nπ i s nπ i( π i) cos nπ iπ j!! M N M hypergeometric y n y! N, n N = 2, 3,...; mean nm/n y = max(0, n N + M),..., min(n, M) M =,..., N; n =,..., N (nm/n)( M/N)(N n)/(n ) continued... Elements of Computational Statistics, Second Edition c 203 James E. Gentle
Appendix D. Important Probability Distributions 42 Table D.. Discrete Distributions (continued) Poisson θ y e θ /y!, y = 0,,2,... θ IR + CF e θ(eit ) power series θ IR + mean CF θ θ h y c(θ) θy, y = 0,,2,... P y hy(θeit ) y /c(θ) {h y} positive constants mean θ d dθ (log(c(θ)) c(θ) = P y hyθy θ d dθ (log(c(θ)) + θ2 d 2 dθ 2 (log(c(θ)) logarithmic, y =, 2,3,... y log( π) π ]0,[ mean π/(( π) log( π)) π y π(π + log( π))/(( π) 2 (log( π)) 2 ) Benford s log b (y + ) log b (y), y =,..., b b integer 3 mean b log b ((b )!) Elements of Computational Statistics, Second Edition c 203 James E. Gentle
422 Appendix D. Important Probability Distributions Table D.2. The Normal Distributions normal; N(µ, σ 2 ) φ(y µ, σ 2 ) def = µ IR; σ IR + CF e iµt σ2 t 2 /2 multivariate normal; N d(µ, Σ) mean µ σ 2 µ IR d ; Σ 0 IR d d CF e iµt t t T Σt/2 mean µ 2πσ e (y µ)2 /2σ 2 T Σ (y µ)/2 (2π) d/2 e (y µ) Σ /2 co Σ matrix normal (Y M) T Σ (Y M))/2 (2π) nm/2 Ψ n/2 e tr(ψ Σ m/2 M IR n m, Ψ 0 IR m m, mean M Σ 0 IR n n co Ψ Σ complex multivariate normal Σ (z µ)/2 (2π) d/2 e (z µ) Σ /2 µ IC d, Σ 0 IC d d mean µ co Σ Elements of Computational Statistics, Second Edition c 203 James E. Gentle
Appendix D. Important Probability Distributions 423 Table D.3. Sampling Distributions from the Normal Distribution chi-squared; χ 2 ν ν IR + mean ν Γ(ν/2)2 ν/2 yν/2 e y/2 I ĪR+ (y) if ν ZZ +, 2ν t Γ((ν + )/2) Γ(ν/2) νπ ( + y2 /ν) (ν+)/2 ν IR + mean 0 F ν/(ν 2), for ν > 2 ν ν /2 ν ν 2/2 2 Γ(ν + ν 2) Γ(ν /2)Γ(ν 2/2) ν, ν 2 IR + mean ν 2/(ν 2 2), for ν 2 > 2 y ν /2 (ν 2 + ν y) (ν +ν 2 )/2 IĪR + (y) 2ν 2 2(ν + ν 2 2)/(ν (ν 2 2) 2 (ν 2 4)), for ν 2 > 4 Wishart W (ν d )/2 2 νd/2 Σ ν/2 Γ exp ` trace(σ W) I d(ν/2) {M M 0 IR d d } (W) d =,2,... ; mean νσ ν > d IR; Σ 0 IR d d noncentral chi-squared ν, λ IR + mean ν + λ noncentral t ν IR +, λ IR noncentral F ν, ν 2, λ IR + co Cov(W ij, W kl) = ν(σ ikσ jl + σ ilσ jk), where Σ = (σ ij) e λ/2 2 ν/2 yν/2 e y/2 X k=0 (λ/2) k k! 2(ν + 2λ) ν ν/2 e λ2 /2 Γ(ν/2)π (ν + /2 y2 ) (ν+)/2 X «ν + k + (λy) k 2 Γ 2 k! ν + y 2 mean k=0 k=0 Γ(ν/2 + k)2 k yk I ĪR+ (y) «k/2 λ(ν/2) /2 Γ((ν )/2), for ν > Γ(ν/2) ν ν 2 ( + λ2 ) ν «2 Γ((ν )/2) 2 λ2, for ν > 2 Γ(ν/2) «ν /2 «ν /2+ν ν e λ/2 y ν /2 ν 2 /2 2 ν 2 ν 2 + ν y X (λ/2) k «k «k Γ(ν 2 + ν + k) ν y k ν 2 I Γ(ν 2)Γ(ν + k)k! ν 2 + ν (y) ĪR+ y mean ν 2(ν + λ)/(ν (ν 2 2)), for ν 2 > 2 «2 ν2 (ν + λ) 2 «+ (ν + 2λ)(ν 2 2) 2, for ν (ν 2 2) 2 2 > 4 (ν 2 4) ν ν 2 Elements of Computational Statistics, Second Edition c 203 James E. Gentle
424 Appendix D. Important Probability Distributions Table D.4. Distributions Useful as Priors for the Normal Parameters «α+ inverted gamma e /βy I Γ(α)β α y (y) ĪR+ α, β IR + mean /β(α ) for α > /(β 2 (α ) 2 (α 2)) for α > 2 «ν/2+ inverted chi-squared e /2y I Γ(ν/2)2 ν/2 y (y) ĪR+ ν IR + mean /(ν 2) for ν > 2 2/((ν 2) 2 (ν 4)) for ν > 4 Table D.5. Distributions Derived from the Univariate Normal lognormal 2πσ y e (log(y) µ)2 /2σ 2 I ĪR+ (y) µ IR; σ IR + mean e µ+σ2 /2 e 2µ+σ2 (e σ2 ) s λ inverse Gaussian 2 2πy 3 e λ(y µ) /2µ 2y I ĪR+ (y) µ, λ IR + mean µ skew normal µ 3 /λ µ, λ IR; σ IR + mean µ + σ Z λ(y µ)/σ 2 πσ e (y µ) /2σ 2 q 2λ π(+λ 2 ) σ 2 ( 2λ 2 /π) e t2 /2 dt Elements of Computational Statistics, Second Edition c 203 James E. Gentle
Appendix D. Important Probability Distributions 425 Table D.6. Other Continuous Distributions (s are wrt Lebesgue measure) beta Γ(α + β) Γ(α)Γ(β) yα ( y) β I [0,] (y) α, β IR + mean α/(α + β) αβ (α + β) 2 (α + β + ) Dirichlet Γ( P d+ i= αi) Q d+ i= Γ(αi) dy i= y α i i! αd+ dx y i I [0,] d(y) α IR d+ + mean α/ α (α d+/ α is the mean of Y d+.) α( α α) α 2 ( α + ) uniform; U(θ, θ 2) I [θ,θ θ 2 θ 2 ](y) θ < θ 2 IR mean (θ 2 + θ )/2 Cauchy (θ2 2 2θ θ 2 + θ)/2 2 «2 y γ πβ + β γ IR; β IR + mean does not exist does not exist logistic e (y µ)/β β( + e (y µ)/β ) 2 µ IR; β IR + mean µ β 2 π 2 /3 Pareto αγ α y I [γ, [(y) α+ α, γ IR + mean αγ/(α ) for α > i= αγ 2 /((α ) 2 (α 2)) for α > 2 power function (y/β) α I [0,β[ (y) α, β IR + mean αβ/(α + ) αβ 2 /((α + 2)(α + ) 2 ) von Mises 2πI 0(κ) eκ cos(x µ) I [µ π,µ+π] (y) µ IR; κ IR + mean µ (I (κ)/i 0(κ)) 2 continued... Elements of Computational Statistics, Second Edition c 203 James E. Gentle
426 Appendix D. Important Probability Distributions gamma Table D.6. Other Continuous Distributions (continued) α, β IR + mean αβ Γ(α)β αyα e y/β I ĪR+ (y) αβ 2 three-parameter gamma Γ(α)β α(y γ)α e (y γ)/β I ]γ, [ (y) α, β IR +; γ IR mean αβ + γ αβ 2 exponential θ e y/θ I ĪR+ (y) θ IR + mean θ double exponential θ 2 µ IR; θ IR + mean µ 2θ e y µ /θ (folded exponential) 2θ 2 α Weibull β yα e y α /β I ĪR+ (y) α, β IR + mean β /α Γ(α + ) β `Γ(2α 2/α + ) (Γ(α + )) 2 extreme value (Type I) α IR; β IR + mean α βγ () β 2 π 2 /6 β e (y α)/β exp(e (y α)/β ) Elements of Computational Statistics, Second Edition c 203 James E. Gentle