STAT 330(Winter ) Mathematical Statistics

Transcript

1 Sprig 3 TABLE OF CONTENTS STAT 33Witer 3-35 Mathematical Statistics Prof. M. Molkaraie Uiversity of Waterloo L A TEXer: W. KONG Last Revisio: April 3, 4 Table of Cotets Review. Probability Spaces Raom Variables The Gamma Fuctio Expectatio a Variace 4. Expectatio Variace Momet Geeratig Fuctios MGFs 6 3. Liear Combiatios Characteristic Fuctio Joit Distributios 8 4. Joit Expectatio a Variace Correlatio Coefficiet Coitioal Distributios 5. Coitioal Expectatio Multivariable Distributios 3 6. Multiomial Distributio Bivariate Normal Distributio Fuctios of Raom Variables to- Bivariate Trasformatios Momet-Geeratig Fuctio Metho Covergece of Raom Variables 8 8. Useful Limit Theorems Delta Metho Poit Estimatio 9. Metho of Momets Maximum Likelihoo Estimatio Notable Fuctios a Matrices Covex Fuctios These otes are curretly a work i progress, a as such may be icomplete or cotai errors. i

2 Sprig 3 ACKNOWLEDGMENTS ACKNOWLEDGMENTS: Special thaks to Michael Baker a his L A TEX formatte otes. They were the ispiratio for the structure of these otes. ii

3 Sprig 3 ABSTRACT Abstract The purpose of these otes is to provie a secoary referece for the material covere i STAT 33. Reaers shoul ote that early /3 of the class is spet reviewig cocepts leare i STAT 3/3 but later material ca prove to be sigificatly more ifficult. The author recommes that stuets who eroll i this shoul have a very goo backgrou i calculus as that is the core of the computatios oe i this course. iii

4 Sprig 3 REVIEW Overview Review Joit, Margial, a Coitioal Distributios 3 Fuctios of RVs 4 Covergece i P & i D 5 Poit Estimatio a Maximum Likelihoo 6 Hypothesis Testig Recommee Reaigs: - Itro to Probability a Mathematical Statistics Bai & Eglehart - Statistical Iferece Casella & Berger Test o Jue th Test o July 7th Review We briefly go over some basic cocepts itrouce i STAT 3 a STAT 3. Probability Spaces Defiitio.. Recall that a probability space is compose of a set S, calle the sample space or the set of all possible outcomes also sometimes give by Ω where E S is calle a evet, a sigma algebra Σ, geerate by S, a a probability fuctio P : Σ R where is usually. Axiom. Here are the properties of the probability fuctio Kolmogorov Axioms: P A, A S P S 3 If A A i are isjoit a I is coutable the P A P A i i I i I Note that 3 is also kow as σ aitivity. Defiitio.. We efie the coitioal probability P A B P A B P B iepeet or A B if P A P A B P A B P AP B. where P B, a we say that A a B are Note. Iepeece Disjoit. Raom Variables Defiitio.3. A raom variable is a fuctio X : S R with the followig properties a otatios: X x} w : w S, Xw x} P X x} P w : w S, Xw x} F X x where the seco form F X : R EXT [, ], is kow as the cf or cumulative istributio fuctio.

5 Sprig 3 REVIEW Propositio.. The cf has the followig properties: F is o-ecreasig F X F X.5 x < x F X x F X x 3 F X is right cotiuous 4 P a < X b P X b P X a F X b F X a, for a < b 5 P X b F X b lim x b F Xx, equal to if X is cotiuous Example.. Cosier T x e x, x R. By observatio it satisfies,, a 3. To check, ote that T x x e x + e x > Defiitio.4. For iscrete raom variables, say X, i aitio to a cf, we efie a probability mass fuctio, calle a pmf: Propositio.. Here are some properties of the pmf: f X x P X x f X x P X X x P X x k P X k Example.. Here is a small list of some iscrete istributios: Uiform [U ifa, b], Geometric [Geop], Poisso [P oissoλ], Biomial [Bi, p] Example.3. Suppose we have a re balls a b black balls a Let X the # of re balls i selectios without replacemet x a+b The PMF is Hyper-Geometric P X x a x b b Let X the # of re balls i selectios with replacemet a The PMF is Bi, a+b. Asie. m+ r r m k k k k k Calle the Vaermoe ietity Example.4. Suppose f X x p x l p. We will show that x N f X x it is clearly o-egative. Observe that x p x x l p l p p x x x a sice we have x l x p x x l p x, x < l p l p

6 Sprig 3 REVIEW Defiitio.5. For cotiuous raom variables, istea of a pmf P X X k we have a pf f X x calle a probability esity fuctio. Cotiuous raom variables still have a cf, eote by F X x P X x. Cotiuous raom variables also have the followig properties: f X x fx x 3 F X x x f Xt t Defiitio.6. The uiform istributio X Uiform[a, b] has cf x x a F X x a b a x x a b a a < x b x > b Example.5. Cosier the pf This is a vali pf provie that f X x θ x θ+ x x < f X x x θ θ θ θ >.3 The Gamma Fuctio Defiitio.7. The gamma fuctio Γα is efie by a has the followig properties. Γα α Γα Γ!, N 3 Γ π Γα y α e y y y α e y y y Defiitio.8. The gamma istributio X Gammaα, β is efie usig the pf f X x xα e x β, x, α >, β > Γαβα 3

7 Sprig 3 EXPECTATION AND VARIANCE a ote that I x α e x β Γαβ α x Γαβ α x α e x β x Γαβ α β α y α e y βy β α Γαβ α Γα Γα y α e y y with x β y x βy βy x. So the gamma istributio is a vali istributio. Defiitio.9. The Weibull istributio X Weibullθ, β is give by the pf f X x β θ β xβ e x θ β β x β θ β xβ exp, x >, θ >, β > θ a ote that for β, we have a Expoetialθ istributio. To see that it is a vali pf, observe that I f X xx β θ β xβ e x θ β x e y y usig y x β θ y βx β x. Suppose that θ. The θ β f X x βx β e xβ, F X x e xβ, x Expectatio a Variace We briefly go over the efiitios a properties of expectatio a variace.. Expectatio Defiitio.. The expectatio of a raom variable X eote as E[X], EX, EX, µ X or µ is efie as EX x Z xp X x i the iscrete case a EX xf X xx for the cotiuous case. We will illustrate examples a properties of expectatio i the cotiuous case from this poit forwar. For geeral fuctios of raom variables, gx, we have EgX 4 gxf X xx

8 Sprig 3 EXPECTATION AND VARIANCE a for joit expectatios EXY we have EXY xyf XY x, yxy Summary. Some properties of expectatio are as follows. Liearity of Expectatio: Ea gx + b gy aex + bey eve if X is epeet o Y X Y EXY EXEY. Variace Defiitio.. We efie the variace of a raom variable X as E[X E[X] ] E[X ] E [X] EX µ X which usually eote as V arx, σ X or σ. Note that EX EX a these are equal whe X is a costat. Defiitio.3. We efie the followig momets arou X k th momet: E[X k ] k th momet arou the mea cetral momet: E [X µ k] 3 k th factorial momet: E [ x k] E[XX...X k + ] Example.. Suppose that X N,, the EX k+, k N sice the itegra is the prouct of a symmetric eve a atisymmetric o fuctio. Summary. Here are some properties of the variace fuctio. V arax + b a V arx V arax + by a V arx + b V arx + abcovx, Y Example.. If X P oisθ the E [ X k] θ k. To see this, we use the efiitio below. usig this result, we ca euce that [ E x k] E[XX...X k + ] e θ x< e θ k x< e θ x< e θ y< θ k e θ θ k y< x k θ k x! x k θ k x! θ k x kx k... θ k y!, y x k EX θ, EXX θ EX EX V arx θ y! 5

9 Sprig 3 3 MOMENT GENERATING FUNCTIONS MGFS Example.3. If X Gammaα, β the E[X p ] β p Γα+p Γα a to see this, we use the efiitio agai E[X p ] β p Γα x p xα e x β Γαβ α x x α+p e x β Γαβ α x y α+p β α+p Γαβ α e y βy, y x β y α+p β p e y y Γα β p Γα + p Γα y α+p e y y We the use this to get E[X] βα a E[X ] β αα with V arx β α. Note that if we kow E[X] a V ar[x], we ca solve for α a β. 3 Momet Geeratig Fuctios MGFs Defiitio 3.. A momet geeratig fuctio of X is create by the followig mappig Example 3.. Let X Bi, p. The E [ e tx] e tx p x p x x x X M X t E [ e tx] M X t x e t p x p x [ pe t + p ] x Example 3.. If X P oisθ the M X t e θet. To see this, we go by efiitio. E [ e tx] x< e tx e θ θ x x! e θ e t θ x x! x< e θ e θet e θet As a sie remark ote that if X Bi, p a, p with p θ the X P oisθ. We also shoul get that their momet geeratig fuctios shoul coverge. Example 3.3. If X Gammaα, β the M X t βt α. To see this, we also go by efiitio M X t E [ e tx] e tx xα e x β Γαβ α x Γαβ α x α e x t β x Γαβ α y α β α β t α β t α e y y α e y y Γα β α β t α βt α y, y β t β t x 6

10 Sprig 3 3 MOMENT GENERATING FUNCTIONS MGFS Example 3.4. Suppose that Z N,. The M Z t e t. As above, we go by efiitio. M Z t e tx π e x x π e x t e t x e t π e x t x e t 3. Liear Combiatios Propositio 3.. Give the MGF of X, we ca compute the MGF of ay liear combiatio of X, say Y ax + b. Proof. We ca o this irectly. M Y t E [ e Y t] [ E e ax+bt] e bt M X at Corollary 3.. If Y Nµ, σ, what is M Y t? Well if X N,, the Y µ + σx. Hece M Y t e µt e σ t e tµ+σ t Summary 3. Recall that Here are some properties of the MGF: M X e tx tx k k! k E [ e tx] k t k k! E [ x k] M X t k ktk k! E [ x k] M X E[X] 3 M X t kk t k k k! E [ x k] M X E[X ] 4 Iuctively, we ca get M X E[X ] 5 M X M Y F X F Y oly i this course; geerally this is ot true 6 If Y X i, the M Y t M X i t Example 3.5. If X Gammaα, β the M Xt αβ βt α E[X] M X αβ M Xt αβα + β βt α E[X ] M X αβ α + V ar[x] αβ α + α β αβ Example 3.6. Suppose that M X t. Fi the MGF of Y X a E[Y ], V ar[y ]. What is the istributio of Y? By observatio, M Y t e t e 4t, E[Y ], V ar[y ] 4 a the istributio of Y is N, 4. 7

11 Sprig 3 4 JOINT DISTRIBUTIONS 3. Characteristic Fuctio Defiitio 3.. The characteristic fuctio of a raom variable X is the Fourier trasform of the pf/pmf: I ω X x e iωx f X xx, e iωx where it always exists a has all of the properties of the MGF. 4 Joit Distributios Example 4.. Cosier rollig two ice, D a D. Let X D + D a Y D D. The P XY X 5, Y 3 P X 5, Y a P X 7, Y 4 4 y P X 7, Y y Summary 4. Here are some basic properties of the margials of a joit istributio i the iscrete case: F F X x P X x lim y F xy x, y F xy x, x y f xyx, y 3 P X X x y P X x, Y y, P Y Y y x P X x, Y y a ow i the cotiuous case: f XY x, y x y F XY x, y x F XY x, y y f XY s, t s t 3 f Y y f XY s, t s, f X x f XY s, t t Example 4.. Suppose that we have ActSc stuets, 9 Stats stuet a 6 Maths stuets. We select 5 stuets without replacemet. Let X # of ActSc stuets a Y # of Stats stuets. The joit PMF of X a Y is the margial of X is P X X x P XY X x, Y y y 9 x y x y 9 6 x y 5 x y 5 5 x, x, y, x + y y 9 6 y 5 x y 5 x 5 x 5 5 a similarly the margial of Y is P Y Y y x 9 x y x y y 5 5 x 6 x 5 x y 9 6 y 5 y 5 5 8

12 Sprig 3 4 JOINT DISTRIBUTIONS Example 4.3. Let Let s check if f XY x, y. I x + y x y Let s try to compute P X 3, Y. I 3 x + y x y y y f XY y y x x + y x, y otherwise x + xy y + xy 3 y y y y y y + y + 3 y + y Note that the cf of this pf is x, y xy F X,Y x, y x + y x, y x, y I the lecture here, we reviewe how to itegrate over arbitrary regios so I will oly give the importat etails Summary 5. If we are aske to compute P fx, Y < c for some costat c a raom variables r.v.s X a Y, isolate Y, raw the regio of itegratio a erive the appropriate itegrals. For example, if x, y, the a P r X + Y < P r Y < X x x y f XY x, y y x P r XY P r Y P Y > f XY x, y y x X X x y x where i the seco example, x y a y x. If X Y the f XY x, yx, y f X xf Y yx, y A A Exercise 4.. Give f XY ke y y, a < x < y <, What is k? As: k What is P X 3, Y 3? As: e 3 e e 3 What is P X < Y? As: 3 What is P X + Y? Hit: P X + Y P X + Y <, As: e 4 Are X a Y iepeet? As: No! Check the margials. Defiitio 4.. We efie the support of a r.v. as x : f X x > }. Propositio 4.. If X Y the gx hy for ay fuctios g a h. Example 4.4. Repeat of a previous example Suppose that we have ActSc stuets, 9 Stats stuet a 6 Maths stuets. We select 5 stuets without replacemet. Let X # of ActSc stuets a Y # of Stats stuets. Are X a Y iepeet? No, they re epeet Example 4.5. Let f XY x, y 3 y x for x, y. Are X a Y iepeet? Yes, check the margials 9

13 Sprig 3 4 JOINT DISTRIBUTIONS Example 4.6. Let f XY x, y θx+y e θ x!y!. This splits ito two iepeet poisso r.v.s. X a Y. Example 4.7. Let f XY x, y π where x y a y. Calculatig the margials gives us f Y y π y a f X x 4 π x. It is clear that X is ot iepeet of Y. Remark 4.. I geeral, X X... X are iepeet if a oly if f XX...X x, x,..., x f Xi x i Remark 4.. Give f XY Z x, y, z gx, yhy, z, we remark that X Z if Y is give. 4. Joit Expectatio a Variace Propositio 4.. If X X... X, the for ay set of equatios h i }. E [h i X i ] E [h i X i ] Defiitio 4.. Defie CovX, Y E[XY ] E[X]E[Y ] E[XY ] µ X µ Y. If X Y the CovX, Y. We also say that if E[XY ] E[X]E[Y ] the X a Y are ucorrelate. However, if for all fuctios f, g we have that the X Y. E[fXgY ] E[fX]E[gY ] Propositio 4.3. Suppose that X is ucorrelate to Y a that Y αx E[αX ] αe[x]e[x] E[X ] E[X] X is a costat so X a Y caot be liearly epeet still caot say that they are iepeet. Propositio 4.4. If X X... X the [ ] V ar a i X i a i V ar[x i ] a i σx i 4. Correlatio Coefficiet Defiitio 4.3. The correlatio coefficiet ρ for two r.v.s is efie as ρ XY CovX, Y σ X σ Y, ρ XY Example 4.8. Recall the pf f XY x, y x + y o x, y a otherwise. We showe that the margials were f X x x +, f Y y y + for x, y a otherwise. It ca be show that E[XY ] 3, E[X] 7 E[Y ], V arx V ary 44 σ X σ Y a so ρ Propositio 4.5. ρ XY for ay r.v.s X a Y.

14 Sprig 3 5 CONDITIONAL DISTRIBUTIONS Proof. Cosier [ X µx E a so CovX,Y σ X σ Y. σ X + Y µ Y σ Y ] [ σx E X µ X ] + [ σy E Y µ Y ] + E[X µ X Y µ Y ] σ X σ Y σ X σx + σ Y σy + CovX, Y σ X σ Y [ ]. A similar metho ca be costructe usig E X µx σ X Y µ Y σ Y i the above to get CovX,Y σ X σ Y 5 Coitioal Distributios Defiitio 5.. For r.v.s X a Y, f X Y x y f XY x, y, p X Y x y p XY x, y a f Y, p Y > f Y y p Y y Example 5.. Cosier f XY x, y π o y, x y a otherwise. We compute the margials to be f X x 4 π x a f Y x π y. It is easy to show that f X Y x y y, f Y Xy x Remark 5.. Prouct Rule We ca express the joit i the followig way x f XY x, y f X Y x y f Y y f Y X y x f X x Example 5.. Suppose that Y P oisµ a X Y y Biy, p. What is the margial of X? The joit istributio is p XY x, y e µ µ y [ ] y! y! x!y x! px p y x e µ µp x µ p y x x!y x! so a X P oisµp. P Hp X x e µ µp x µ p y x yx e µ x! µpx x!y x! yx e µ e µ p µp x x! e µp µp x x! µ p y x y x! Example 5.3. Give P XY x, y θx+y e θ x!y! for x, y,,,... It ca be show that P X x P XY x, y θx e θ, P Y X y x θy e θ x! y! y

15 Sprig 3 5 CONDITIONAL DISTRIBUTIONS Example 5.4. Suppose that Y Gamma α, θ a X Y y Weiy p, p. What is fx? We kow that It ca be show that f Y y Gammaα, β xα e x β β α Γα, Weiθ, β β θ β xβ e x β θ, x θα Γα yα e θy, f X Y x y This gives a equatio for f XY i the form of a itegratig gives us f X x p y p p x p e x y p p f XY x, y pyθα Γα xp y α e θy e yx P f XY x, y y pθ α x p Γαθ + x p α+ t α e t t, t yθ + x p Γα + pθα x p Γαθ + x p α+ pyx p e yxp, x, y 5. Coitioal Expectatio Defiitio 5.. E[gY x] y gyf XY y x E[gY X x] a if X Y the E[gY X x] E[gY ]. Variace is efie i a similar way: V ar[y X x] E[Y X x] E [Y X x]. Example 5.5. Let f Y X y x x, x y x. We wat to compute the variace. First, E[Y X x] sice the term i the itegral is a o fuctio. The, E[Y X x] So V ar[y X x] 3 x. x x x y x x y x Propositio 5.. The ouble expectatio formula states y x y x y y x 3 3 x 3 x E[X] E E[X Y ] V ar[x] E[V arx Y ] + V are[x Y Example 5.6. If P Uif[, ] a Y P p Bi, p the E[E[Y P ]] E[P ] E[P ] 5 a V ar[e[y P ] V ar[p ] Some examples we skip here because they are trivial

16 Sprig 3 6 MULTIVARIABLE DISTRIBUTIONS Example 5.7. Give f XY 6xy x y < x, y < otherwise we ca show that E[XY ] 3, f X Y 6x x y 4 3y with f Y y4 3y, < y < Defiitio 5.3. The joit MGF of XY is efie as a i geeral, with Propositio 5.. As y, F XY x, y F X x. Example 5.8. Give we ca show that M XY t E [ e tx+ty ] M k Xit,.., t k E [e ] k l txi M k Xit, t,..., t k E [ e tx] M X t f XY M XY t, t e y < x < y < otherwise Propositio 5.3. If X k } are a set of iepeet raom variables, t + t t M X k t,..., t M Xk t k Exercise 5.. Show that if X,..., X are ii N, r.v.s, the Y X N,. First remark that V ary V ar [ X ]. We ca further calculate the MGF of Y as a so Y X N,. M Y t ] E [e x t e t e t t e Theorem 5.. If Y,..., Y N, a they are iepeet, the Ȳ µ σ / N, 6 Multivariable Distributios Here, we examie various istributios that comprise of multiple variables. 6. Multiomial Distributio Defiitio 6.. Let X i be the umber of times i comes before total repetitios, a p i be the probability of gettig the item i. The! P X x,..., X k x k x!...x k! px... px k k where x i, p i. We say that X,..., X k Mult, p,..., p k. 3

17 Sprig 3 6 MULTIVARIABLE DISTRIBUTIONS Propositio. Some properties iclue:. M X t p e t p t k k. CovX i, X j p i p j + p k+ 6. Bivariate Normal Distributio Defiitio 6.. If X a X have the followig joit PDF: f XX x, x exp } π Σ / x µt Σ x µ, x x µ σ, µ, Σ ρσ σ x µ ρσ σ σ the X X, X t BivNµ, Σ. Note that the matrix Σ must be positive efiite. Remark 6.. If ρ the f XX exp πσ σ σ σ x µ σ e πσ }} Nµ,σ x µ x µ t σ x µ σ e πσ }} Nµ,σ σ x µ x µ } So X a X are iepeet. This is special to oly the bivariate ormal r.v. Note. I geeral, if X Nµ, σ a X Nµ, σ the if ρ XX it is ot always true that X X. This is oly the case if X a X, collectively, are bivariate ormal. Summary 6. Here are some values that may be useful i the computatio of f X,X : a so f X,X x, x Σ σ σ ρ, Σ / σ σ ρ Σ σ σ ρ πσ σ exp ρ σ σ ρ [ πσ σ exp x µ ρ ρ σ ρσ σ ρσ σ σ [ x µ σ x µ x µ ρσ σ + x µ σ ] } σ + x µ σ ρx ]} µ x µ σ σ Fact 6.. The momet geeratig fuctio is M X t, t E where X Nµ, σ, X Nµ, σ a [ e tt X ] E [ e tx+tx]... e µt t+ tt Σt M X M X t,, M X M X, t Propositio 6.. If C C C t the C t X NC t µ, C t ΣC a Y AX + b Y NAµ + b, AΣA t. Remark 6.. For a coitio istributio X X x with X, X beig joitly bivariate, we have X X x Nµ + ρ σ σ x µ, σ ρ This ca be oe by puttig the joit over of the margial of X. For the sake of saity, I will ot be bashig through the computatio of this. 4

18 Sprig 3 7 FUNCTIONS OF RANDOM VARIABLES Fact 6.. E[X X ] E[E[X X X ]] Example 6.. Suppose that X X are BIV µ, Σ. The a so E[X X X ] X µ + ρ σ σ X µ. Thus, E[X X X x ] x E[X X x ] x µ + ρ σ σ x µ E[X X ] E[E[X X X ]] E[X ]µ + ρ σ σ E[X ] µ µ µ + ρσ σ a so we ca represet the covariace of X a X as CovX, X µ µ + ρσ σ µ µ ρσ σ 7 Fuctios of Raom Variables Example 7.. Suppose that X Z a f Z z π e z /. Remark that So takig erivatives, we have a ote that X Gam,. F X x P X x P Z x P x Z x F Z x F Z x Fact 7.. If Z,..., Z are iepeet N, the f X x x f Zx / πx / e x/ X Z Z χ a E[X]. Example 7.. Suppose that f XY 3y 3y x y otherwise Fi the pf of T XY. Now sice P T t P XY < t P XY > t the we calculate P XY > t as P XY > t y t t y 3y x y + t t 3t What is the pf of T? By irect computatio, this is What is the pf of S Y X. Well, the cf is F S s f T 3 3t, t y y s 3y x y s a so the pf is s F Ss f S s s Example 7.3. Suppose that X,..., X are ii with pf f X a cf F X. Let Y maxx,.., X a T mix,.., X. So F Y y P Y y P X y,..., X y 5 F Xi y FXy f Y y f X yfx y

19 Sprig 3 7 FUNCTIONS OF RANDOM VARIABLES a F T t P T < t P T t F Xi t F X t f T t F X t f X t Example 7.4. If each X i was expλ i the F Xi x e λix a so a T exp λ i. F T t e λit, t F T t e λit, t Example 7.5. Suppose that Z a Z are i.i.. r.v.s that are N,. What is the istributio of X Z Z? Well, ote that Z Z N, so Z Z χ 7. -to- Bivariate Trasformatios If we are give a X, Y bivariate vector r.v.s a f X,Y x, y is kow, the let A x, y, f XY > }, B u, v, u h x, y, v h x, y} If U h X, Y, V h X, Y a X w U, V, Y w U, V the g UV u, v f XY x, y x, y u, v f XY w u, v, w u, v x, y u, v Example 7.6. Suppose that a the U X + Y, V X Y g UV u, v f XY x, y e x +y π π e u +v 4 X U + V π e u, Y U V π e v Example 7.7. Suppose that X Uif[, a Y Uif[,. Usig the Box-Muller trasformatio, if U l X cos πy, V l X si πy it ca be show that U, V are iepeet N,. Also the Jacobia is J x π. Now ote that U + V l x X exp Example 7.8. Suppose that we have U +V f XY x, y so J π e U +V. e y < x < y < o/w If U X + Y a V X. The X V, Y U V a the support is < v < u <. Our Jacobia is J 6

20 Sprig 3 7 FUNCTIONS OF RANDOM VARIABLES a so g UV u, v e u v < v < u < o/w g U u g V v u/ v e u e v v e u/ e u e u e v u e v e v e v e v Example 7.9. Suppose that we have f XY x, y e x y < x, y < o/w a U X + Y, V X the X V a Y U V with J. The support is < v < u < a g UV u, v e v u v e u. Defiitio 7.. If Z N,, X χ, a Y χ X m the Z/ t a X/ Y/m F,m. Remark 7.. If W F,m the V W F m,. X Example 7.. To compute the pf of t let U X a V Z/. The X U a Z V U. We ca use the Jacobia metho above to compute the pf. 7. Momet-Geeratig Fuctio Metho Fact 7.. If X, X,..., X are iepeet a X i has MGF M Xi t the if Y X i we have M Y t M Xi t a if the X i s are i.i.. the M Y t M X t Example 7.. Suppose that X Nµ, σ a Y ax + b where M X t e µt e σ t /. The M Y t E[e ty ] e bt E[e atx ] e bt e aµt+ σ a t e aµ+bt e a σ t Y Naµ + b, a σ Now if X i Nµ i, σ i a Y a ix i the M Y t E[e t aixi ] E[e taixi ] e aiµit e a i σ i t e t aiµi e t Corollary 7.. Suppose that we have X i Nµ, σ where X,..., X are i.i.. the X i Nµ, σ, X a i σ i Y N a i µ i, a i µ i X i Nµ, σ Fact 7.3. We have Example 7.. We kow that Xi µ σ } } χ Y χ m i Xi X χ mi σ }} from other χ Xi + X σ } } χ 7

21 Sprig 3 8 CONVERGENCE OF RANDOM VARIABLES Proof is a exercise. Corollary 7.. X a S Xi X are iepeet. Corcha s Theorem Fact 7.4. We have that if X i Nµ, σ a X a S are efie as above, the X µ s/ t Fact 7.5. If X i Nµ, σ a Y j Nµ, σ are i.i.. for i,..., a j,..., m the S X /σ S Y /σ F 8 Covergece of Raom Variables Covergece ca take place from strogest to weakest: Everywhere Almost surely i L, L,... See PMATH 45 I istributio I probability We will examie the last two efiitios of covergece. Defiitio 8.. The sequece X, X,..., X coverges i probability to X if for ay ɛ > we have We eote this by X p X. lim P X X ɛ lim P X X < ɛ Defiitio 8.. We say that X : Ω A } coverges i istributio to X : Ω B if for ay ɛ > we have lim P X k P X k lim P X P X < ɛ, k A B A example woul be the cetral limit theorem. Alteratively, this is equivalet to lim F X x F X x at all parts where F X x is cotiuous. We the write X X. Propositio 8.. If X p X the X X. Example 8.. Suppose that X k } k are i.i.. Uif[, ]. Let X max k X k a X mi k X k. What is the limitig istributio of X? First remark that the support of X is,. The ote that for < x < we have P X x P X > x P k X k > x x 8

22 Sprig 3 8 CONVERGENCE OF RANDOM VARIABLES So x F x P X x x < x < x lim F x x e x x > What is the limitig istributio of X? Similar to above, the support of X is, a P X x P X > x P a i the limit, we have the same istributio i. That is lim F x X x x e x x > P k X k x x 3 What is the limitig istributio of X? This ca be show to have limitig cf of lim F x x < x X 4 Similarly, what is the limitig istributio of X? This ca be show to have limitig cf of lim F x x < x X Defiitio 8.3. Give a sequece of r.v.s. X }, with correspoig cfs F x} if lim F x < b x x b the X b. Theorem 8.. If X b the X p b. Proof. By irect evaluatio, so takig limits gives us P X b > ɛ P X < b ɛ + P X > b + ɛ P X b ɛ + P X > b + ɛ F b ɛ + F b + ɛ lim b > ɛ lim b ɛ + lim b + ɛ + 9

23 Sprig 3 8 CONVERGENCE OF RANDOM VARIABLES Example 8.. Give X i } i.i.. r.v.s, with f Xi x e x θ x θ o/w Let Y mi i X i a show that Y p θ. It is easier to show that Y as require. P Y x θ. Remark that the support of Y is θ,. We the have x < θ e x θ x θ Fact 8.. Markov s Iequality For ay k N, lim P Y x x < θ x θ Y P X > C E[ X k ] C k P X > C E[ X ] V arx + E[X] C C, k Propositio 8.. A property of the arithmetic mea of raom variables is X p µ. θ Proof. We have P X µ > ɛ ɛ E[ X µ ] ɛ V ar X σ ɛ Remark 8.. If X X the X X but X p X. Theorem 8.. Cetral Limit Theorem σ X µ N, where X } are i.i.. r.v.s. with X µ, σ Proof. No. Observe that for the cf of ay r.v. X we have M X t e t /, f l M X, Now if A σ X µ a Y i Xi µ σ the sice Now from above, a hece lim M Y f M X M X, f M X M X M X M X σ E[e ta ] E [e t ] Yi X i µ t t f Y l M Y t + Ot3 M Y t e t / N,. e t Y i t MY σ X i µ X i µ σ t t e t / M Y e t/ Proof. No. Alteratively, usig otatio from the previous proof, M A t M Yi/ t M Y t

24 Sprig 3 8 CONVERGENCE OF RANDOM VARIABLES Usig a st orer Taylor series, t lim M Y lim + t e t / Corollary 8.. If X i } are i.i.. P oisµ a Y X i the Y µ µ N, If X i } are i.i.. χ mea of χ k is k a variace is k a Y X i the Y N, 8. Useful Limit Theorems p D. If X a the gx p D ga. That is g is cotiuous at a.. Slutsky s Theorem Suppose that X X a Y p b. The, a X + Y b X Y X + b b X c X /Y X/b, b Example 8.3. Suppose X, X,..., X are i.i. X i Uif[,. We showe that X p e X p e a X Z exp. Remark 8.. F X X Uif[, ] Example 8.4. If X i } are P oisµ the Z X µ X µ X X }} N, X N, µ }} p a similarly X µ N, µ 8. Delta Metho Propositio 8.3. Suppose that for X, X,..., X we have X θ N, σ If gx is ifferetiable at θ a g θ the gx gθ N, g θ σ

25 Sprig 3 9 POINT ESTIMATION 9 Poit Estimatio Suppose that we observe X,..., X i.i.. from fx, θ a θ is ukow. The goal is to estimate θ. Defiitio 9.. The t statistic is a fuctio of ata that oes t epe o θ or µ or ay ukow parameter. We eote it by T X T X,..., X as a raom variable a t tx,..., x as its value. The followig are ifferet methos for poit estimatio.. Metho of Momets. Maximum Likelihoo 3. Bayes Estimatio 9. Metho of Momets Here, we wat to set the sample/observe k th momet equal to the theoretical momet. That is we wat M k for l,,..., l}. Example 9.. If X,..., X are i.i.. P oisµ the E[X i ] µ a Xi k E[X l ] ˆµ MM X i Example 9.. If X,..., X are i.i.. a f X θ e x/θ x θ otherwise The E[X i ] θ a Example 9.3. If X,..., X are i.i.. Nµ, σ the ˆθ MM N X i Example 9.4. If X,..., X are i.i.. a The we ca show E[X] ˆµ MM f X X i, ˆσ MM + ˆµ MM θe θ x otherwise, θ > θ θ + θ E[X] E[X] X i ˆθ MM X X Example 9.5. Suppose that X Gamα, β the E[X] αβ a V arx αβ. So ˆ αβ MM X i, αβ ˆ MM + αβmm ˆ X i

26 Sprig 3 9 POINT ESTIMATION Example 9.6. If X,..., X are i.i.. Uif[, θ] the Remark that ˆθ MLE maxx,..., X. ˆθ MM ] X i, E [ˆθMM θ 9. Maximum Likelihoo Estimatio Defiitio 9.. Suppose that X,..., X are i.i.. from fx, θ. We call Lθ, X fx i, θ the likelihoo of θ a l ll the log-likelihoo fuctio. The MLE estimate is Example 9.7. If X,..., X are i.i.. a The it ca be show that If we set lθ θ the ˆθ ML ˆθ MLE argmax Lθ argmax lθ f X Example 9.8. Suppose that X,..., X P oisθ. The θ e x/θ x θ otherwise lθ l θ + l e x+...+x/θ l θ x x θ Lθ e θ θ xi x i! ˆθ ML x i lθ θ + l θ x i l x i! a so l θ + x i θ ˆθ ML x i Example 9.9. Recall that if fx, θ θx θ, < x <, θ > the It ca be show that a so ˆθ MM X X θ Lθ θ x i lθ l θ + θ l x i l θ θ + l x i ˆθ ML l x i Example 9.. Suppose that X,..., X Exp/θ a i.i... The Lθ θ e θ xi lθ lθ θ 3 x i

27 Sprig 3 9 POINT ESTIMATION a hece l θ θ + θ Example 9.. Suppose that X,..., X Berp a i.i... The a so Note that x i ˆθ ML x i x Lp p xi p xi lp l p l p p [ l p x i p + + x i + l p p ˆp ML x i ] x i p < Example 9.. Suppose that X,..., X are i.i.. Nµ, σ. It ca be show that a hece Lθ πσ xi e x i µ σ lθ l π l σ x i µ l µ σ l σ σ + σ 3 x i µ ˆµ ML x i σ x i µ ˆσ ML x i ˆµ ML Example 9.3. Suppose that X,..., X are i.i.. Uif[, θ]. The it ca be show that with f X have a hece ˆθ ML maxx,..., x. Lθ /θ x, x,.., x [, θ] /θ Imaxx o/w,...,x θ Propositio 9.. E[X a ] V ar[x] a equality hols whe a E[X]. /θ x θ o/w we Proof. We have E[X E[X] + E[X] a ] V ar[x] + E[X] a + E [X E[X] E[X] a] V ar[x] }} 9.3 Notable Fuctios a Matrices Defiitio 9.3. The score fuctio is The iformatio fuctio is Iθ θ Sθ l fx, θ θ Sθ l fx, θ θ 4

28 Sprig 3 9 POINT ESTIMATION The Fisher iformatio matrix is Jθ E [Iθ] Summary 7. Some properties iclue:. Sˆθ ML. E [ θ l fx, θ] a This follows from the fact that [ ] E l fx, θ θ θ l fx, θ fx, θx θ θ fx, θ fx, θ fx, θx fx, θx θ [ ] 3. E θ l fx, θ E [ θ l fx, θ ] a To see this, we take the partial with respect to θ of θ l fx, θ fx, θx to get θ l fx, θ fx, θx + [ ] E l fx, θ θ θ l fx, θ fx, θx θ θ l fx, θ l fx, θ fx, θx θ }} [ ] E l fx, θ E θ [ ] [ 4. If X,..., X are i.i.. the E θ l fx, θ E θ l fx, θ ] a This follows from the efiitio of Jθ with Jθ J θ. E[ θ l fx,θ] [ ] l fx, θ θ Propositio 9.. Cramer-Rao Lower Bou Suppose that T X,..., X is a estimator for θ. Remark that if T is ubiase if E[T X] θ. If E[T X] θ the E[T X] is biase. Also, if X,..., X are samples from fx, θ the V art θ E[T ] E [ θ l fx, θ ] [ ] E θ l fx, θ Proof. First remark that CovX, Y V arxv ary. Set X T X a Y θ l fx, θ. The Cov T X, θ [ ] [ ] l fx, θ E T X l fx, θ E l fx, θ E[T X] θ θ }} E[T X] θ T X fx, θx θ 5

29 Sprig 3 9 POINT ESTIMATION Sice V ar [ [ ] θ l fx, θ] E θ l fx, θ because E [ θ l fx, θ] the V ar[t X] θ E[T X] E [ θ l fx, θ ] J θ Example 9.4. Suppose that X P oisµ. The a with the C-R Cramer-Rao bou as Previously, we showe that a so the ML estimator is efficiet. Remark 9.. ˆθ ML ˆθ ML θ N l fx, µ µ µ µ + x l µ l x! + x µ µ l fx, µ x µ Iµ x E[X] Jµ µ µ µ p θ asymptotically This will also imply that ˆθ ML θ N ˆµ ML V art µ X i, Jθ asymptotically ormal µ V ar[µ ML ] µ µ, J θ N, Jθ a ˆθ ML N θ, Jθ. 9.4 Covex Fuctios Defiitio 9.4. We say that a fuctio f is covex if x, x a, b a λ [, ] we have fλx + λx λfx + λfx f > Remark 9.. If f is covex the f is cocave. Propositio 9.3. Jese s iequality If X is a r.v. a f is covex the E[fX] f E[X] Proof. Suppose that the iequality is true for k N. The k k p i p i fx i p k fx k + + p k fx i + p k k p k fx k + p k q i fx i 6

30 Sprig 3 9 POINT ESTIMATION a usig iuctio o the latter term we get k p i fx i p k fx k + + p k f f p k x k + p k k f p i x i k q i fx i k q i fx i Example 9.5. If Y x l x the because Y is cocave i x the l X i l X i X i This shows that the geometric mea is always less tha the arithmetic mea. Note 3. For the fial exam, pay attetio to the tutorial cotet o X i. Metho of Momets for Gamma Q where MM with V armm > V arml. x i, ML maxx,..., X 7

31 Sprig 3 INDEX Iex -to- bivariate trasformatios, 6 bivariate ormal istributio, 4 Box-Muller trasformatio, 6 cetral limit theorem, 8, cetral momet, 5 characteristic fuctio, 8 coitioal istributios, coitioal expectatio, covergece i istributio, 8 covergece i probability, 8 covex fuctios, 6 Corcha s theorem, 8 correlatio coefficiet, Cramer-Rao lower bou, 5 cumulative istributio fuctio, elta metho, expectatio, 4 Fisher iformatio matrix, 5 fuctios of raom variables, 5 gamma istributio, 3 gamma fuctio, 3 iformatio fuctio, 4 Jacobia, 6 Jese s iequality, 6 joit istributios, 8 Kolmogorov axioms, limitig istributio, 9 Markov s iequality, maximum likelihoo estimatio, 3 metho of momets, momet geeratig fuctio, 6 momets, 5 multiomial istributio, 3 poit estimatio, probability mass fuctio, probability space, score fuctio, 4 σ aitivity, Slutsky s theorem, support, 9 uiform istributio, 3 Vaermoe ietity, variace, 4 Weibull istributio, 4 8