Exponential Families

Σχετικά έγγραφα
Solution Series 9. i=1 x i and i=1 x i.

ST5224: Advanced Statistical Theory II

Other Test Constructions: Likelihood Ratio & Bayes Tests

Statistical Inference I Locally most powerful tests

2 Composition. Invertible Mappings

Theorem 8 Let φ be the most powerful size α test of H

SOLUTIONS TO MATH38181 EXTREME VALUES AND FINANCIAL RISK EXAM

Bayesian statistics. DS GA 1002 Probability and Statistics for Data Science.

6. MAXIMUM LIKELIHOOD ESTIMATION

Example Sheet 3 Solutions

3.4 SUM AND DIFFERENCE FORMULAS. NOTE: cos(α+β) cos α + cos β cos(α-β) cos α -cos β

Solutions to Exercise Sheet 5

SOLUTIONS TO MATH38181 EXTREME VALUES AND FINANCIAL RISK EXAM

Homework 8 Model Solution Section

Exercises to Statistics of Material Fatigue No. 5

The Simply Typed Lambda Calculus

EE512: Error Control Coding

CHAPTER 25 SOLVING EQUATIONS BY ITERATIVE METHODS

Inverse trigonometric functions & General Solution of Trigonometric Equations

SCHOOL OF MATHEMATICAL SCIENCES G11LMA Linear Mathematics Examination Solutions

Fourier Series. MATH 211, Calculus II. J. Robert Buchanan. Spring Department of Mathematics

4.6 Autoregressive Moving Average Model ARMA(1,1)

Econ 2110: Fall 2008 Suggested Solutions to Problem Set 8 questions or comments to Dan Fetter 1

5.4 The Poisson Distribution.

6.1. Dirac Equation. Hamiltonian. Dirac Eq.

Second Order Partial Differential Equations

Homework for 1/27 Due 2/5

( y) Partial Differential Equations

Srednicki Chapter 55

Estimation for ARMA Processes with Stable Noise. Matt Calder & Richard A. Davis Colorado State University

Section 8.3 Trigonometric Equations

SOLVING CUBICS AND QUARTICS BY RADICALS

TMA4115 Matematikk 3

Concrete Mathematics Exercises from 30 September 2016

C.S. 430 Assignment 6, Sample Solutions

ω ω ω ω ω ω+2 ω ω+2 + ω ω ω ω+2 + ω ω+1 ω ω+2 2 ω ω ω ω ω ω ω ω+1 ω ω2 ω ω2 + ω ω ω2 + ω ω ω ω2 + ω ω+1 ω ω2 + ω ω+1 + ω ω ω ω2 + ω

ON NEGATIVE MOMENTS OF CERTAIN DISCRETE DISTRIBUTIONS

Reminders: linear functions

Phys460.nb Solution for the t-dependent Schrodinger s equation How did we find the solution? (not required)

6.3 Forecasting ARMA processes

Ordinal Arithmetic: Addition, Multiplication, Exponentiation and Limit

Problem Set 3: Solutions

Homework 3 Solutions

An Inventory of Continuous Distributions

Every set of first-order formulas is equivalent to an independent set

Approximation of distance between locations on earth given by latitude and longitude

forms This gives Remark 1. How to remember the above formulas: Substituting these into the equation we obtain with

D Alembert s Solution to the Wave Equation

Lecture 2. Soundness and completeness of propositional logic

Statistics 104: Quantitative Methods for Economics Formula and Theorem Review

Partial Differential Equations in Biology The boundary element method. March 26, 2013

Finite Field Problems: Solutions

Areas and Lengths in Polar Coordinates

Math221: HW# 1 solutions

CHAPTER 48 APPLICATIONS OF MATRICES AND DETERMINANTS

HOMEWORK 4 = G. In order to plot the stress versus the stretch we define a normalized stretch:

lecture 10: the em algorithm (contd)

Matrices and Determinants

Areas and Lengths in Polar Coordinates

DESIGN OF MACHINERY SOLUTION MANUAL h in h 4 0.

Chapter 6: Systems of Linear Differential. be continuous functions on the interval

PARTIAL NOTES for 6.1 Trigonometric Identities

Instruction Execution Times

Lecture 34 Bootstrap confidence intervals

CRASH COURSE IN PRECALCULUS

Lecture 2: Dirac notation and a review of linear algebra Read Sakurai chapter 1, Baym chatper 3

ΚΥΠΡΙΑΚΗ ΕΤΑΙΡΕΙΑ ΠΛΗΡΟΦΟΡΙΚΗΣ CYPRUS COMPUTER SOCIETY ΠΑΓΚΥΠΡΙΟΣ ΜΑΘΗΤΙΚΟΣ ΔΙΑΓΩΝΙΣΜΟΣ ΠΛΗΡΟΦΟΡΙΚΗΣ 19/5/2007

2. THEORY OF EQUATIONS. PREVIOUS EAMCET Bits.

SCITECH Volume 13, Issue 2 RESEARCH ORGANISATION Published online: March 29, 2018

Μηχανική Μάθηση Hypothesis Testing

Section 9.2 Polar Equations and Graphs

( ) 2 and compare to M.

derivation of the Laplacian from rectangular to spherical coordinates

ESTIMATION OF SYSTEM RELIABILITY IN A TWO COMPONENT STRESS-STRENGTH MODELS DAVID D. HANAGAL

Similarly, we may define hyperbolic functions cosh α and sinh α from the unit hyperbola

b. Use the parametrization from (a) to compute the area of S a as S a ds. Be sure to substitute for ds!

Math 6 SL Probability Distributions Practice Test Mark Scheme

Congruence Classes of Invertible Matrices of Order 3 over F 2

ORDINAL ARITHMETIC JULIAN J. SCHLÖDER

Aquinas College. Edexcel Mathematical formulae and statistics tables DO NOT WRITE ON THIS BOOKLET

Last Lecture. Biostatistics Statistical Inference Lecture 19 Likelihood Ratio Test. Example of Hypothesis Testing.

Exercises 10. Find a fundamental matrix of the given system of equations. Also find the fundamental matrix Φ(t) satisfying Φ(0) = I. 1.

Parametrized Surfaces

ΕΙΣΑΓΩΓΗ ΣΤΗ ΣΤΑΤΙΣΤΙΚΗ ΑΝΑΛΥΣΗ

MA 342N Assignment 1 Due 24 February 2016

k A = [k, k]( )[a 1, a 2 ] = [ka 1,ka 2 ] 4For the division of two intervals of confidence in R +

557: MATHEMATICAL STATISTICS II RESULTS FROM CLASSICAL HYPOTHESIS TESTING

Problem Set 9 Solutions. θ + 1. θ 2 + cotθ ( ) sinθ e iφ is an eigenfunction of the ˆ L 2 operator. / θ 2. φ 2. sin 2 θ φ 2. ( ) = e iφ. = e iφ cosθ.

Web-based supplementary materials for Bayesian Quantile Regression for Ordinal Longitudinal Data

Practice Exam 2. Conceptual Questions. 1. State a Basic identity and then verify it. (a) Identity: Solution: One identity is csc(θ) = 1

Ulf Schepsmeier and. Jakob Stöber

Fractional Colorings and Zykov Products of graphs

Tridiagonal matrices. Gérard MEURANT. October, 2008

Lecture 21: Properties and robustness of LSE

STAT200C: Hypothesis Testing

Gaussian related distributions

If we restrict the domain of y = sin x to [ π, π ], the restrict function. y = sin x, π 2 x π 2

Notes on the Open Economy

ΕΛΛΗΝΙΚΗ ΔΗΜΟΚΡΑΤΙΑ ΠΑΝΕΠΙΣΤΗΜΙΟ ΚΡΗΤΗΣ. Ψηφιακή Οικονομία. Διάλεξη 7η: Consumer Behavior Mαρίνα Μπιτσάκη Τμήμα Επιστήμης Υπολογιστών

Probability and Random Processes (Part II)

Transcript:

Exponential Families Robert L. Wolpert Department of Statistical Science Duke University, Durham, NC, USA Surprisingly many of the distributions we use in statistics for random variables taking value in some space often R or N 0 but sometimes R n, Z, or some other space, indexed by a parameter θ from some parameter set Θ, can be written in exponential family form, with pdf or pmf fx θ = exp [ηθtx Bθ] hx for some statistic t : R, natural parameter η : Θ R, and functions B : Θ R and h : R +. The likelihood function for a random sample of size n from the exponential family is n f n x θ = exp ηθ tx j nbθ hx i, j=1 which is actually of the same form with the same natural parameter η, but now with statistic T n x = tx j and functions B n θ = nbθ and h n x = Πhx j. Examples For example, the pmf for the binomial distribution Bim,p can be written as [ ] m p x 1 p m x p m = exp log x m log1 p x 1 p x and natural sufficient statis- of Exponential Family form with ηp = log p tic tx = x, and the Poisson 1 p θ x x! e θ = exp [log θx θ] 1 1 x!

with η = log θ and again tx = x. The Beta distribution Beα,β with either one of its two parmeters unknown can be written in EF form too: [ ] Γα + β ΓαΓβ xα 1 1 x β 1 Γα 1 x β = exp α log x log Γα + β x1 xγβ [ ] Γβ x α = exp β log1 x log Γα + β x1 xγα with tx = log x or log1 x when η = α or η = β is unknown, respectively. With both parameters unknown the beta distribution can be written as a bivariate Exponential Family with parameter θ = α,β R +: fx θ = exp [ηθ tx Bθ]hx 1 with vector parameter η = α,β and statistic tx = log x,log 1 x and scalar one-dimensional functions Bθ = log Γα + log Γβ logα + β and hx = 1/x1 x. Since this comes up often, we ll let η and T be q-dimensional below; usually in this course q = 1 or. Natural Exponential Families It is often convenient to reparametrize exponential families to the natural parameter η = ηθ R q, leading with Aηθ Bθ to Since any pdf integrates to unity we have e Aη = e η tx hxdx fx η = e η tx Aη hx and hence can calculate the moment generating function MGF for the natural sufficient statistic tx = {t 1 x,,t q x} as M t s = E [e s t] = e s tx e η tx Aη hxdx = e Aη e η+s tx hxdx = e Aη+s Aη,

so log M t s = Aη + s Aη and we can find moments for the natural sufficient statistic by E[t] = log M t 0 = Aη V[t] = log M t 0 = Aη provided that η is an interior point of the natural parameter space E {η R q : 0 < e η tx hxdx < } and that A is twice-differentiable near η. For samples of size n N the sufficient statistic T n x = tx j is a sum of independent random variables, so by the Central Limit Theorem we have approximately No n Aη, n Aη. Note that Aη = log fx θ is both the observed and Fisher expected information matrix I n θ for natural exponential families, and that the score statistic is Z := log fx θ = [ T n x n Aη ]. Conjugate Priors For hyper-parameters α R q and β R such that c α,β := e ηθ α βbθ dθ <, we can define a prior density for θ by πθ α,β = c 1 α,β e ηθ α βbθ dθ. Θ Θ With this prior and with data { i } iid fx θ from the exponential family, the posterior is πθ x e ηθ α βbθ e ηθ Tnx nbθ πθ α = α + T n x, β = β + n, 3

again within the same family but now with parameters α = α + T n and β = β + n. For example, in the binomial example above this conjugate prior family is { } p πθ α,β exp αlog β log1 p = p α 1 p β α, 1 p the Beta family, while for the Poisson example it is πθ α,β exp {α log θ βθ} = θ α e βθ, the Gamma family. Conjugate families for every exponential family are available in the same way. Note not every distribution we consider is from an exponential family. From, for exmple, it is clear set of points where the pdf or pmf is nonzero, the possible values a random variable can take, is just {x : fx θ > 0} = {x : hx > 0}, which does not depend on the parameter θ; thus any family of distributions where the support depends on the parameter uniform distributions are important examples can t be from an exponential family. The next pages show several familiar and some less familiar ones, like the Inverse Gaussian IGµ, λ and Pareto Paα, β distributions in exponential family form. Some of the formulas involve the log gamma function γz = log Γz and its first and second derivatives, the digamma ψz = d/dzγz and trigamma ψ z = d /dz γz, which are built into R, Mathematica, Maple, the gsl library in C, and such, but aren t on pocket calculators or most spreadsheets. In each case Aη is the Information matrix in the natural parametrization, Iθ in the usual parameterization. 4

1 Exponential Family Examples Beα,β fx = Γα+β ΓαΓβ xα 1 1 x β 1, x 0,1 T = log x,log 1 x Bα,β = γα + γβ γα + β η = α,β Aη = γη [ 1 + γη γη 1 ] + η [ ] ψη1 ψη Aη = 1 + η ψα ψα + β ψη ψη 1 + η ψβ ψα + β ψ Aη = η 1 c c c ψ c = ψ η c η 1 + η Bim,p fx = m x p x q m x, x = 0...m T = x Bp = m log q η = logp/q Aη = m log1 + e η p = e η /1 + e η Aη = meη 1+e m p η Aη = Ip = m/pq m e η 1+e η Exλ fx = λe λx, x > 0 T = x Bλ = log λ η = λ Aη = log η Aη = 1/η 1/λ Aη = η Iλ = 1/λ Gaα,λ fx = λα Γα xα 1 e λx, x > 0 T = log x,x Bα,β = γα αlog λ η = α, λ Aη = γη [ 1 η 1 log η ] [ ] ψη1 log η Aη = ψα log λ η 1 /η α/λ ψ Aη = η 1 1/η ψ 1/η η 1 /η Iα,λ = α 1/λ 1/λ α/λ Gep fx = p q x, x = 0,1,,... T = x Bp = log p η = log q Aη = log1 e η p = 1 e η Aη = eη 1 e η q/p Aη = Ip = 1/p q e η 1 e η 5

Exponential Family Examples cont d IGa,b fx = ae a bx /x / πx 3, x > 0 T = 1/x,x Ba,b = ab log a η = a /, b / Aη = η 1 η 1 log η 1 a = η 1, b = η [ ] [ ] η /η Aη = η1 1 1/η 1 b/a + 1/a /η a/b Aη = 1 η η 1 3 + 1 η 1 1 η1 η 1 η1 η η1 η 3 Ia,b = b/a + /a 1 1 a/b NBα,p fx = α x p α q x, x = 0,1,,... T = x Bp = α log p η = log q Aη = α log1 e η p = 1 e η Aη = αeη 1 e η αq/p Aη = Ip = α/p q αe η 1 e η Noµ,σ fx = e x µ /σ / πσ T = x,x Bµ,σ = µ /σ + 1 log σ η = µσ, σ / Aη = η [ 1 /4η 1 log η ] [ η 1 /η µ ] Aη = Aη = η 1 /4η 1/η 1/η η 1 /η η 1 /η η 1 /η 3 + 1/η Ia,b = Poλ fx = λ x e λ /x!, x = 0,1,,... T = x Bλ = λ η = log λ Aη = e η λ = e η Aη = e η λ Aη = e η Iλ = 1/λ µ + σ σ 0 0 σ 4 / Paα,β fx = β α β /x β+1, x > α T = log x Bβ = log β β log α η = β Aη = log η + η log α Aη = log α 1/η log α + 1/β Aη = η Iλ = β 6