Advanced Statistics. Chen, L.-A. Distribution of order statistics: Review : Let X 1,..., X k be random variables with joint p.d.f f(x 1,...

Transcript

1 Avace Statistics Che, L.-A. Distributio of orer statistics: Review : Let X,..., X k be raom variables with joit p..f f(x,..., x k a Y h (X,..., X k, Y h (X,..., X k,..., Y k h k (X,..., X k be - trasformatio with iverse fuctios x w (y,..., y k, x w (y,..., y k,..., x k w k (y,..., y k. The the joit p..f of Y,..., Y k is f Y,...,Y k (y,..., y k f(w (y,..., y k,..., w k (y,..., y k J with Jacobia w w y y k J.. w w y y k Orer statistics are ot - trasformatios. Suppose that we have raom variables X, X.Orer statistics are Y mi{x, X }, Y max{x, X } If Y,..., Y k are ot - trasformatios, we may partitio the space R k ito Y. Y k sets A,..., A m such that A j B is a - trasformatio. There are jacobias J,..., J m such that the the joit p..f of Y,..., Y k is m y f Y,...,Y k (y,..., y k f(x,..., x J j,. B j y k Let X,..., X be a raom sample from a cotiuous istributio with p..f f(x, a < x < b. We cosier orer statistics Y mi{x,..., X },..., Y max{x,..., X } that satisfies Y Y Y. We call Y i the ith orer statistic of raom sample X,..., X. Thm. The joit p..f of Y,..., Y is {!f(y f(y f Y,...,Y (y,..., y f(y if a < y < y < < y < b 0 otherwise.

2 Proof. We cosier case 3 oly. We have X, X, X 3 f(x, a < x < b, the joit p..f of X, X, X 3 is f(x, x, x 3 f(x f(x f(x 3, a < x i < b, i,, 3. X The space of X is X 3 X A { X : a < x i < b, i,, 3} X 3 Cosier the partitios: X A { X : a < x < x < x 3 < b}, x y, x y, x 3 y 3 X 3 X A { X : a < x < x < x 3 < b}, x y, x y, x 3 y 3 X 3 X A 3 { X : a < x < x 3 < x < b}, x y, x y 3, x 3 y X 3 X A 4 { X : a < x < x 3 < x < b}, x y 3, x y, x 3 y X 3 X A 5 { X : a < x 3 < x < x < b}, x y, x y 3, x 3 y X 3 X A 6 { X : a < x 3 < x < x < b}, x y 3, x y, x 3 y X 3 Each set A j forms a - trasformatio, i.e. Y Y Y 3 A j { y y y 3 : a < y < y < y 3 < b}

3 Jacobias: 0 0 J , J , J J , J , J 6 The joit p..f of Y, Y, Y 3 is , f Y,Y,Y 3 (y, y, y 3 f(y f(y f(y 3 + f(y f(y f(y 3 + f(y f(y 3 f(y + f(y 3 f(y f(y + f(y f(y 3 f(y + f(y 3 f(y f(y 6f(y f(y f(y 3 3!f(y f(y f(y 3 Let Y,..., Y be the orer statistics for a raom sample X,..., X from a cotiuous istributio with p..f f(x, a < x < b. The the joit p..f of Y,..., Y k is f Y,...,Y (y,..., y!f(y f(y f(y, a < y < y < < y < b Note: If a < x < b, the { b f(xx F y (x b y F (b F (y F (y y f(xx F a (x y a F (y F (a F (y where F is the.f. of X. Thm. (a The margial p..f of Y is (b The margial p..f of Y is (c The margial p..f of Y j is f Yj (y j f Y (y ( F (y f(y, a < y < b f Y (y (F (y f(y, a < y < b! (j!( j! (F (y j j ( F (y j j f(y j, a < y j < b 3

4 ( For j < k, the joit p..f of Y j a Y k is f Yj,Y k (y j, y k! (j!(k j!( k! (F (y j j (F (y k F (y j k j ( F (y k k f(y j f(y k, a < y j < y k < b Proof. Margial p..f of Y is f Y (y!!!! b b y b b y b b y b b y b y y y y b y f(y f(y f(y y y b y f(y f(y f(y! ( F (y y y b y 3 f(y f(y f(y! ( F (y y y b y 4 f(y f(y f(y 3 3! ( F (y 3 3 y 3 y! f(y f(y y (! ( F (y y! (! f(y ( F (y ( F (y f(y Margial p..f of Y is f Y (y!!! y a y a y a y3 a y3 a y4 a y a f(y f(y f(y y y f(y f(y F (y y y f(y 3 f(y! (F (y 3 y 3 y y! f(y f(y a (! (F (y y!f(y (! (F (y (F (y f(y 4

5 Margial p..f of Y j is b b b yj y f Yj (y j! f(y f(y y y j y y j+ y j y y a a!f(y j (j! (F (y j j ( j! ( F (y j j! (j!( j! (F (y j j ( F (y j j f(y j, a < y j < b Joit p..f of Y j a Y k is yj y yk yk b b b f Yj,Y k (y j, y k! f(y f(y a a y j y k y k y y y y y k+ y k y j+ y y j! (j!(k j!( k! (F (y j j (F (y k F (y j k j ( F (y k k f(y j f(y k, where a < y j < y k < b t-istributio { Z N(0, V χ (r iep. We say that T Z V r Wat the p..f of T. has a t-istributio with.f. r a eote T t(r. Proof. The joit p..f of Z a V is f Z,V (z, v π e z Cosier the trasformatio T ( z A { v ( t For u Γ( r v r r e v, z R, v > 0 Z V r, U V. T ( U t : < z <, v > 0} B { : < t <, u > 0} u B, we have v u, z ut r,uique solutio implies a - trasformatio. Iverse fuctio v u, z ut 5 r.

6 Jacobia is J z t v t z u v u The joit p..f of T a U is ut u f T,U (t, u f Z,V (, u r r u r u t r 0 u r e u r t π Γ( r r The p..f of T is f T (t u r e u u r πγ( r e r u(+ t r u r+, < t <, u > 0 r πγ( r r r ( Let H U( + T 0 r, u πγ( r r r 0 r+ Γ( r+ u r+ e u(+ t r u h + t r h r+ ( + t r+ r, u h + t r h e πγ( r r r ( + t r+ r 0 Γ( r+ πrγ( r ( + t, < t < r+ r + t r Γ( r+ r+ h h r+ e h h X Gamma(α, β if f(x Γ(αβ α xα e x β, x > 0 If β, we let α r, the X χ (r. If X χ (r, the f(x Γ( r x r r e x, x > 0 We ee several covergece theorem for use i costructig C.I. Thm. (a Cetral Limit Theorem ( CLT If X,..., X is a raom sample with mea µ a variace σ <, the X µ σ/ N(0, 6

7 (b Weak Law of Large Numbers ( WLLN If X,..., X are raom variables with mea µ a variace σ <, the X P µ Thm. If Y P P a, the g(y g(a for ay cotiuous fuctio g. Thm. Slutsky s theorem P If X X a Y a, the Cocept of C.I. : X ± Y X Y X Y X ± a ax X, if a 0. a (asuppose that P (X A 0.9. If we observe X may times with x,..., x of large, the there are about umber 0.9 obs. x is such that x i A. (b If (t (X,..., X, t (X,..., X is a 90% C.I. for θ,the we have about umber 0.9 obs.(t, t such that θ (t, t whe we observe (t, t may times (sayig times. Note: (a The ormal approximatio by CLT ca be applie o ay istributio f(x, θ, ormal or ot, whe is large ( 30 (b Why covergece i istributio? If X N(0,,the P (a X b F X (b F X (a F Z (b F Z (a P (a Z b Approximate C.I.: ( Let X,..., X be a raom sample from f(x with mea µ a variace σ, where f ca be ormal or ot. The X µ s/ X µ σ/ σ s N(0, I N(0, by Slutsky s theorem. 7

8 We have Here, (X z α µ. α P ( z α Z z α P ( z α X µ s/ z α P (X z α s, X + z α s µ X + z α s s is a approximate 00( α% C.I. for ( Let Y b(, p. If X,..., X are Beroulli(p, the Y X i. Let ˆp Y. We have The ˆp p ˆp( ˆp We have ˆp p Here, (ˆp z α C.I. for p. ˆp Y ˆp p p( p p( p X i X X p p( p p( p ˆp( ˆp P p by WLLN. N(0, by CLT. α P ( z α Z z α P ( z α ˆp p P (ˆp z α ˆp( ˆp ˆp( ˆp, ˆp + z α N(0, I N(0, by Slutsky s theorem. p ˆp + z α ˆp( ˆp z α ˆp( ˆp ˆp( ˆp is a approximate 00( α% Example: Y b(, p. 00, y 0. Wat a 95.4% for p. z 0.03 A 95.4% C.I for p is approximately as ( 0 z , 0 + z (0.,

9 C.I. for σ or σ: Let X,..., X be a raom sample from N(µ, σ where µ a σ are ukow. We have Let χ α The ( s a χ α Here, ( ( s χ α α P ( C.I for µ : σ χ (, where s, ( s (X i X satisfy α P (χ ( χ α P (χ ( χ. α α P (χ α P (χ α ( s P ( χ α χ ( χ α ( s χ σ α σ ( s χ α is a approximate 00( α% C.I. for σ. χ α ( s σ sice all is positive. χ α ( s χ α (a X,..., X N(µ, σ0 where σ0 is kow. Pivotal quatity: Z X µ σ 0 N(0, (b X,..., X N(µ, σ Pivotal quatity: T X µ s t( (c X,..., X f(x with mea µ, is large ( 30. Pivotal quatity: X µ s N(0, C.I. for σ : X,..., X N(µ, σ Pivotal quatity: ( s χ (. σ Cofiece Iterval for Differece of Meas : Case I : { X,..., X N(µ x, σx iep. a σx, σ Y,..., Y m N(µ y, σy y are kow. 9

10 { X N(µ x, σ x Y N(µ y, σ y m iepeet So, X Y N(µ x µ y, σ x + σ y m Z (X Y (µ x µ y N(0, σ x + σ y m α P ( z α Z z α P ( z α (X Y (µ x µ y σ x + σ y m z α P (X Y z α A 00( α% C.I. for µ x µ y is σ (X Y z α x + σ y, X Y + z σ α x m + σ y m σ x + σ y σ m µ x µ y X Y + z α x + σ y m X,..., X f(x, θ Q h(x,..., X, θ is a pivotal quatity if it has a istributio free of parameter θ a, b s.t. α P θ (a Q b, θ We wat a pivotal quatity with α P (a Q b P (t (x,..., x θ t (x,..., x Case II : Variaces are ukow but they are equal. { X,..., X N(µ x, σ iep. a σ is ukow. Y,..., Y m N(µ y, σ X N(µ x, σ ( s x χ ( σ Y N(µ y, σ iep. where m (m s y χ (m σ { X Y N(µx µ y, σ ( + m ( s x +(m s y σ χ ( + m 0 s x s y m iep. (X i X m (Y i Y

11 The T (X Y (µ x µ y σ + m ( s x +(m s y σ (+m (X Y (µ x µ y ( s x +(m s y +m + m t( + m We have α P ( t α T z α P ( t α P (X Y t α X Y + t α (X Y (µ x µ y ( s x + (m s y + m ( s x + (m s y + m A 00( α% C.I. for µ x µ y is ( s (X Y t x+(m s y α +, X Y + t α +m m ( s x +(m s y +m + m + m µ x µ y + m t α ( s x+(m s y + +m m Example: 0, m 7, X 4., Y 3.4, ( s x 490, (m s y 4, t 0.05 (5.753 A 90% C.I. for µ x µ y is ( ( 5.6, , Chapter Testig Hypothesis Let X,..., X be a raom sample from f(x, θ where θ is ukow. Def. A statistical hypothesis is a cojecture about ukow parameter θ. If it specifies a sigle value for θ, it is calle a simple hypothesis, otherwise, it is a composite hypothesis. Example : H : θ θ 0 is a simple hypothesis. H : θ θ 0, H : θ θ 0, H : θ θ θ are all composite hypothesis. There are two hypothesis, oe calle the ull hypothesis a oe calle the alterative hypothesis.

12 Def. The ull hypothesis, eote by H 0, is a hypothesis that we will reject it oly if the ata reveal strogly that it is ot true. The alterative hypothesis, eote by H, is the hypothesis alterative to the ull hypothesis. We ofte wat to see if a ew prouct (rug, maufacturig process a fertilizer is fuctioig better tha the ol prouct. The hypothesis H 0 a H are both cocerig the quatity of ew prouct as H 0 : the ew prouct is ot better tha the ol prouct. vs. H : the ew prouct is better tha the ol prouct. Oce the hypotheses are set, we ee to o experimet o the ew prouct to geerate a raom sample to efie a test for hypotheses. Def. A test is a rule eciig whether to reject or ot to reject the ull hypothesis. Usually, a test specifies a subset C of the sample space of the raom sample X,..., X that we reject H 0 if the observatio (x,..., x falls i C a o ot reject H 0 if (x,..., x oes ot fall i C. This subset C is calle the critical regio or the rejectio regio. For ay test, there are two possible errors that may occur: Type I error : H 0 is true but we reject H 0. Type II error : H is true but we o ot reject H 0. Def. The power fuctio π c (θ of critical regio C is the probability of rejectig H 0 whe θ is true. Let X,..., X be a raom sample a we cosier a test with critical regio C. The power fuctio is π c (θ P (reject H 0 : θ is true P ( X. X C : θ is true Example : X score of a test N(θ, 00. The past experiece iicates θ 75. Wat to test hypothesis H 0 : θ 75 vs. H : θ > 75 sol : Let X,..., X 5 be a raom sample from N(θ, 00 a we cosier critical regio C {X : X > 75} { X. X 5 : 5 5 X i > 75}.

13 The power fuctio is π C (θ P (X > 75 : θ P ( X θ > 75 θ P (Z > 75 θ P (Z 75 θ where X N(θ, 00 N(θ, 4. 5 π C (75 0.5, π C ( , π C ( If we choose critical regio C {X : X > 78}. Power fuctio is π C (θ P (X > 78 : θ P (Z 78 θ π C ( Def. The size of a test with critical regio C is the maximum of the probability of type I error, i.e. sizemax θ H 0 π c (θ The rule for choosig critical regio is fixig a sigificace level α a fi the test amog class of tests with size α that miimize the probability of type II error. (Usually we let α 0.0 or 0.05 Def. Cosier simple hypothesis H 0 : θ θ 0 vs. H : θ θ. We say that the test with critical regio C is a most powerful (MP test of sigificace level α if, for every set A with P ((X,..., X A : H 0 α, the followigs hol: (a P ((X,..., X C : H 0 π c (θ 0 α (b P ((X,..., X C : H P ((X,..., X A : H, i.e. π c (θ π A (θ We also call C the MP critical regio of sigificace level α. Let X,..., X be a raom sample from a istributio with p..f f(x, θ Joit p..f : f(x,..., x, θ f(x i, θ fuctio of x,..., x Likelihoo fuctio: L(θ, x,..., x f(x i, θ fuctio of θ The likelihoo fuctio is L(θ, x,..., x f(x i, θ 3

14 A ratio of two likelihoo as L(θ 0, x,..., x L(θ, x,..., x is calle the likelihoo ratio. We will erive the MP test through this likelihoo ratio. Thm. Neyma-Pearso Theorem Cosier the simple hypothesis H 0 : θ θ 0 vs. H : θ θ. Let C be the critical regio with k > 0 such that (a L(θ 0, x,..., x L(θ, x,..., x (b L(θ 0, x,..., x L(θ, x,..., x k, for k, for x. x x. x C / C (cα P ((X,..., X C : H 0 α The C is the MP critical regio of sigificace level α. Proof. Note that, for ay set B, P ((X,..., X B : θ sayig B C L(θ 0 f(x,..., x, θx x L(θ, x,..., x x x L(θ Let A be a critical regio with L(θ A 0 α. We wat to show that L(θ C L(θ A Now L(θ L(θ L(θ + L(θ ( C A C A C A c L(θ L(θ C A c A C c A C L(θ + L(θ A C c 4

15 C For For x. x x. x C, L(θ 0 L(θ k L(θ k L(θ 0 C c, L(θ 0 L(θ k L(θ k L(θ 0 L(θ L(θ L(θ 0 L(θ 0 A k C A k c A C c k C A [ L(θ 0 + L(θ 0 ( L(θ 0 + c C A A C c k [ L(θ 0 L(θ 0 ] 0 c A A C L(θ 0 ] Note: MP critical regio is C { x. x : L(θ 0, x,..., x L(θ, x,..., x k} s.t. α P ((X,..., X C : θ θ 0 P ( L(θ 0, x,..., x L(θ, x,..., x k : θ θ 0 You ee to kow istributio of L(θ 0, x,..., x L(θ, x,..., x With Neyma-Pearso theorem, a test with critical regio C { x. x : L(θ 0, x,..., x L(θ, x,..., x k} s.t. α P ( L(θ 0, x,..., x L(θ, x,..., x k : θ θ 0 is a MP test with sigificace level α. Ufortuately, the istributio of L(θ 0, x,..., x is geerally ukow a the costat k is ot available. L(θ, x,..., x Suppose that, for givig k, there exists c such that L(θ 0, x,..., x L(θ, x,..., x k iff either u(x,..., x c or u(x,..., x c 5

16 a we kow the istributio of u(x,..., X, the the test with critical regio C { x. x : u(x,..., x c} s.t. α P (u(x,..., X c : θ θ 0 is a MP test with sigificace level α. Example: X,..., X N(θ, Cosier simple hypothesis H 0 : θ 0 vs. H : θ Wat the MP test with sigificace level α. sol: Likelihoo fuctio is L(θ, x,..., x e (x i θ (π e π (π e [ L(θ 0, x,..., x L(θ, x,..., x (π e MP critical regio is C {x c}. s.t. x i θ x i +θ ] (π e [ (x i θ e [ x i θ 0 x i +θ0 ] x i + x i θ x i +θ ] k x i + l k x i l k x ( l k c α P (type I error P (reject H 0 : H 0 P (X c : θ 0 P ( X 0 c 0 : θ 0 P (Z c 6

17 c z α c z α The MP critical regio with sigificace level α is C {x zα } Example : X,..., X Poisso(λ H 0 : λ 0 vs. H : λ sol: The likelihoo fuctio is L(λ, x,..., x L(λ 0, x,..., x L(λ, x,..., x 0 x i e 9 k λ x i e λ ( x i l 0 9 l k x i! 0 x ie 0 x i! x ie x i! λ x i e λ x i! 0 x i e 9 k x i (l k + 9 c l 0 The MP critical regio with sigificace level α is C { x i c} where c satisfies α P (Y x i c : λ 0 c (0 y e 0 y0 y! Hypothesis H 0 : θ θ 0 vs. H : θ θ is equivalet to the hypothesis H 0 : X f(x, θ 0 vs. H : X f(x, θ The we ca solve MP critical regio for hypothesis H 0 H : X f (x : X f (x vs. Example: X,..., X with p..f f(x, where f is ukow. Cosier hypothesis { e x 0,,, H 0 : X f (x x! 0 elsewhere. { ( vs. H : X f (x x x 0,,, 0 elsewhere. 7

18 sol: L(f (x, x,..., x L(f (x, x,..., x e x i! ( x i x i e xi! x i e k x i! ( x i l l( x i! c The MP critical regio is C {( x i l l( x i! c} where c satisfies α P (( x i l l( x i! c : f f (x Cosier testig the hypothesis H 0 : θ θ 0 vs. the composite hypothesis H : θ {θ, θ }. Cosierig sigificace level α, we ca costruct two MP critical regios with sigificace level α for simple hypothesis as follows: Hypothesis MP critical regio with sigificace level α H 0 : θ θ 0 vs. H : θ θ C H 0 : θ θ 0 vs. H : θ θ C If C C, we call C C C the uiformly most powerful (UMP critical regio with sigificace level α for simple hypothesis H 0 : θ θ 0 vs. composite hypothesis H : θ {θ, θ }. Suppose that C C, we say that the UMP test or UMP critical regio oes t exists. Def. A test with critical regio C is calle a UMP test with sigificace level α for testig simple hypothesis H 0 : θ θ 0 vs. composite hypothesis H : θ Θ, if, for θ Θ, C is MP critical regio with sigificace level α for H 0 : θ θ 0 vs. H : θ θ Usually UMP test exists for the followig hypothesis: (ah 0 : θ θ 0 vs. H : θ > θ 0 (Oe sie hypothesis 8

19 (bh 0 : θ θ 0 vs. H : θ < θ 0 (Oe sie hypothesis Usually UMP test oes t exist for hypothesis H 0 : θ θ 0 vs. H : θ θ 0 (Two sie hypothesis Example: X,..., X N(µ,, H 0 : µ 0 vs. H : µ > 0. Let µ > 0. Cosier simple hypothesis H 0 : µ 0 vs. H : µ µ. sol: By Nayma-Pearso theorem, L(µ 0, x,..., x L(µ µ, x,..., x π e x i π e (x i µ e µ xi + µ k µ x i + µ l k µ e x i e [ x i µ x i +µ ] x i l k µ where µ < 0 for µ > 0 x i (l k µ µ x c uer H 0 α P ( x c : H 0 P (Z c c z α MP critical regio for H 0 : µ 0 vs. H : µ µ is C { x z α } This hols for every µ µ. Hece C { x z α } is UMP critical regio with sigificace level α for H 0 : µ 0 vs. H : µ > 0. Example: X,..., X N(0, σ, H 0 : σ vs. H : σ <. Let θ <. Cosier simple hypothesis H 0 : σ vs. H : σ θ. 9

20 sol: L(σ, x,..., x L(σ θ, x,..., x π e x i πθ e x i θ e x i x i θ e θ θ e ( θ x i k ( θ x i l(θ k where ( θ > 0 for θ < x i ( l(θ k c θ Xi H 0 χ ( MP critical regio for hypothesis H 0 : σ vs. H : σ θ with sigificace level α is C { x i χ α(} This hols for every σ θ <. Hece C is the UMP critical regio for hypothesis H 0 : σ vs. H : σ <. Example: X,..., X Poisso(λ. Hypothesis H 0 : λ vs. H : λ. Show that UMP test with sigificace level α oes t exist. You ee to fi λ a λ such that MP tests for H 0 : λ vs. H : λ λ a H 0 : λ vs. H : λ λ are ifferet. sol: Let λ 0. Cosier hypothesis H 0 : λ vs. H : λ λ 0. The likelihoo fuctio is L(λ, x,..., x λ x i e λ x i! λ x i e λ x i! 0

21 By Neyma-Pearso theorem, L(λ, x,..., x L(λ λ 0, x,..., x λ e xi 0 e λ 0 x i λ0 ke λ 0 x i l λ 0 l(ke λ 0 x i ( l λ 0 l(ke λ 0 k If λ 0 >, the l λ 0 < 0. x i l λ 0 l(ke λ 0 k MP critical regio for H 0 : λ vs. H : λ λ 0 is C { x i k } s.t. α P ( X i k : λ If λ 0 <, the l λ 0 < 0. x i l λ 0 l(ke λ 0 k MP critical regio for H 0 : λ vs. H : λ λ 0 is C { x i k } s.t. α P ( X i k : λ C C UMP test oes t exist. There are several types of hypothesis that we ee to treat them i ifferet ways: A : H 0 : θ θ 0 vs.h : θ θ (simple hypothesis the MP test always exists by Neyma-Pearso theorem. A : H 0 : θ θ 0 vs.h : θ > θ 0 or H 0 : θ θ 0 vs.h : θ < θ 0 the UMP test exists for some istributio. A3 : H 0 : θ θ 0 vs.h : θ θ 0 the UMP test geeral oes t exist.

22 A4 : Suppose that we have composite hypothesis H 0 : θ θ 0 vs. H : θ > θ 0 or H 0 : θ θ 0 vs. H : θ < θ 0 (Oe sie test the UMP test exists for some special istributios. A5 : Geeral hypothesis H 0 : θ Θ 0 vs. H : θ / Θ 0 If there is o UMP test, we will cosier likelihoo ratio test. We ow cosier UMP tests for oe sie hypothesis: A test with critical regio C has power fuctio. π c (θ P ((X,..., X C : θ is true If the hypothesis is H 0 : θ Θ 0 vs. H : θ / Θ 0, the size of the test is sup θ Θ 0 π c (θ Def. Cosier composite hypothesis H 0 : θ Θ 0 vs. H : θ / Θ 0. A test with critical regio C is calle a UMP test with sigificace level α if it satisfies (a sup θ Θ 0 π c (θ α. (b Ay critical regio C with size α satisfies π c (θ π c (θ, θ / Θ 0. Deote L(θ, x,..., x the likelihoo fuctio of a raom sample X,..., X from p..f f(x, θ. Def. A family of esities {f(x, θ : θ Θ} is sai to have a mootoe likelihoo ratios if, for θ < θ, there exists a statistic T t(x,..., X such that the likelihoo ratio L(θ, x,..., x L(θ, x,..., x h(t is either oicreasig or oecreasig i T. If X with a istributio has a mootoe likelihoo ratio a hypothesis is oe sie, the it has UMP test. Iea of UMP test:

23 L(θ L(θ h(t i T i T Larger T θ (small is more reliable. Smaller T θ (large is more reliable. If H : θ < θ 0, c {T t 0 } If H : θ > θ 0, c {T t 0 } Larger T θ (large is more reliable. Smaller T θ (small is more reliable. If H : θ < θ 0, c {T t 0 } If H : θ > θ 0, c {T t 0 } We will say that the test followig this rule is UMP test if mootoe likelihoo ratio exists. Note: Whe there is mootoe likelihoo ratio (MLR, the UMP test has mootoe power fuctio. If H 0 : θ θ 0, π c (θ is. If H 0 : θ θ 0, π c (θ is. Thm. Suppose that the family of esities has a MLR i a statistic T t(x,..., X, (acosier hypothesis H 0 : θ θ 0 vs. H : θ > θ 0 (oe sie hypothesis If the LR L(θ h(t is oecreasig i T, the the UMP critical regio L(θ with sigificace level α is C {T t 0 } s.t. α sup θ θ 0 P (T t 0 : θ P (T t 0 : θ 0 If the LR L(θ h(t is oicreasig i T, the the UMP critical regio L(θ sigificace level α is C {T t 0 } s.t. α sup θ θ 0 P (T t 0 : θ P (T t 0 : θ 0 (b Cosier hypothesis H 0 : θ θ 0 vs. H : θ < θ 0 If the LR L(θ h(t is oecreasig i T, the the UMP critical regio L(θ with sigificace level α is C {T t 0 } s.t. α sup θ θ 0 P (T t 0 : θ P (T t 0 : θ 0 If the LR L(θ h(t is oicreasig i T, the the UMP critical regio L(θ with sigificace level α is C {T t 0 } s.t. α sup θ θ 0 P (T t 0 : θ P (T t 0 : θ 0 3

24 Example: X,..., X U(0, θ, θ > 0. H 0 : θ θ 0 vs. H : θ > θ 0 sol: f(x, θ I(0 < x < θ. The likelihoo fuctio is θ L(θ, x,..., x θ I(0 < x i < θ θ I(0 < x i < θ θ I(0 < y < θ where Y max{x,..., X } Let θ < θ. The LR is L(θ, x,..., x L(θ, x,..., x I(0 < y (θ < θ The UMP critical regio is C {y c} s.t. α P (Y c : θ 0 The p..f of Y is (θ I(0 < y (θ < θ θ I(0 < y < θ I(0 < y < θ f Y (y (F (y f(y ( y θ y θ y θ, 0 < y < θ α P (Y c : θ 0 θ0 c y y c θ0 θ0 c θ 0 ( α The UMP critical regio with sigificace level α is C {y θ 0 ( α } Example: X,..., X N(µ,.H 0 : µ 0 vs. H : µ < 0. Wat the UMP test. sol: Let µ < µ. L(µ, x,..., x L(µ, x,..., x (π (π e (x i µ e (x i µ e [ x i µ x i +µ ] e [ x i µ x i +µ ] e [(µ µ x i +(µ µ ], i x i The UMP critical regio is C { x i c} s.t. α P ( X i c : µ 0 P ( X i c : µ 0 P (Z c 4

25 c z α c z α. So, the UMP critical regio sigificace level α is C { x i z α } {x zα } X,..., X N(µ, σ 0 where σ 0 is kow. Hypotheses UMP critical regio H 0 : µ µ 0 vs. H : µ > µ 0 x µ 0 σ 0 / z α H 0 : µ µ 0 vs. H : µ < µ 0 x µ 0 σ 0 / z α Likelihoo Ratio Test: Let X,..., X be a raom sample from p..f f(x, θ, θ Θ. Deote by L(θ, x,..., x as the likelihoo fuctio of X,..., X. So L(θ, x,..., x f(x i, θ If we have hypothesis H 0 : θ Θ 0 vs. H : θ / Θ 0, cosier two maximum likelihoo as max L(θ, x,..., x a max L(θ, x,..., x L(ˆθ θ Θ 0 θ Θ mle, x,..., x where ˆθ mle is the m.l.e. of θ. Obviously, max θ Θ 0 L(θ, x,..., x max θ Θ L(θ, x,..., x If θ Θ 0 is true, they must be very close. If θ Θ 0 is ot true, they are o loger close. Def. The geeralize likelihoo ratio is λ λ(x,..., x max L(θ, x,..., x θ Θ 0 max L(θ, x,..., x θ Θ max L(θ, x,..., x θ Θ 0 L(ˆθ mle, x,..., x The likelihoo ratio test for testig hypothesis H 0 : θ Θ 0 vs. H : θ / Θ 0 is rejectig H 0 if λ λ 0 where λ 0 satisfies α sup θ Θ 0 P (λ(x,..., X λ 0 : θ Note: The power fuctio P (λ(x,..., X λ 0 : θ is ot geerally mootoe, 5

26 but it ofte is. H 0 : θ θ 0 vs. H : θ θ MP test exists by Neyma-Pearso Theorem. H 0 : θ θ 0 vs. H : θ > θ 0 or H 0 : θ θ 0 vs. H : θ < θ 0 UMP test ofte exists by Neyma-Pearso Theorem. H 0 : θ θ 0 vs. H : θ > θ 0 or H 0 : θ θ 0 vs. H : θ < θ 0 Oe sie hypothesis + MLR UMP test exists. Critical regio of likelihoo ratio test is C {λ λ(x,..., x s.t. α sup θ Θ 0 P (λ(x,..., X λ 0 : θ max L(θ, x,..., x θ Θ 0 max L(θ, x,..., x λ 0} θ Θ I may cases, there is a statistic T t(x,..., X such that λ(x,..., x λ 0 if a oly if t(x,..., x k a the C {t(x,..., x k} Example: X,..., X with p..f f(x, θ θe θx, x > 0, θ > 0 H 0 : θ θ 0 vs. H : θ > θ 0 sol: L(θ, x,..., x θe θx i θ e θ l L(θ, x,..., x l θ θ l L(θ θ m.l.e ˆθ θ x i 0 x x i x i x i So, max θ Θ L(θ, x,..., x L(ˆθ, x,..., x ( x e Sice L(θ, x,..., x achieves maximum at θ x { So, max L(θ, x,..., x max θ e θ xi ( θ θ 0 θ θ 0 θ0 e θ 0 x e xi if θ 0 > x if θ 0 < x 6

27 So, λ λ(x,..., x max L(θ 0<θ θ 0 max L(θ θ>0 { if θ0 > x θ0 e θ 0 xi (θ ( x e 0 x e (θ 0x if θ 0 < x { if x > θ 0 (θ 0 x e (θ 0x if x θ 0 i x The critical regio of LRT is C {x c} s.t. α P (X c : θ 0 χ α θ 0 c c χ α θ 0 α P (X c : θ 0 P ( P ( θ 0 X i X i c : θ 0 c θ 0 : θ 0 P (χ ( θ 0 c Example: X,..., X N(θ,, H 0 : θ 0 vs. H : θ 0. This is o UMP test for this hypothesis. We wat LRT. sol: L(θ, x,..., x e (x i θ (π e π Likelihoo ratio λ l L(θ, x,..., x θ 0 m.l.e. ˆθ x (x i θ max L(θ, x,..., x θ0 max L(θ, x,..., x L(0, x,..., x L(ˆθ, x θ R,..., x e [ x i x i +x x i x ] e x λ 0 x l λ 0 x c 0 x c e x i e (xi x X H 0 N(0, X X H 0 N(0, 7

28 α P ( x c : θ 0 P ( Z c P ( Z c P ( Z c α c z α The critical regio with sigificace level α for LRT is C {x : x z α } Example: X,..., X N(θ, θ, < θ <, θ > 0 H 0 : θ 0 vs. H : θ 0. i.e.h 0 : θ 0, θ > 0 vs. H : θ 0, θ > 0. 8

29 sol: L(θ, θ, x,..., x (π θ e (xi θ max L(θ, θ, x,..., x max L(0, θ, x,..., x max θ 0,θ >0 θ >0 θ >0 (π θ l L(0, θ, x,..., x l π x l θ i θ l L(0, θ, x,..., x x + i 0 θ θ ˆθ x i θ max θ 0,θ >0 L(θ, θ, x,..., x L(0, θ e x i θ x i, x,..., x (π ( x i e (xi θ max L(θ, θ, x,..., x max θ R,θ >0 θ R,θ >0 (π θ e θ l L(θ, θ, x,..., x l π l θ (xi θ θ l L(θ, θ, x,..., x (xi θ 0 θ θ ˆθ x l L(x, θ, x,..., x + (xi x θ θ θ ˆθ max L(θ, θ, x,..., x (π ( θ R,θ >0 max L(θ, θ, x,..., x θ 0,θ >0 λ max L(θ, θ, x,..., x θ R,θ >0 (xi x l x i l λ 0 (xi x x i (xi x x i x ( s > ( c 0 x s > c t α (xi x e ( x i e ( (xi x e (xi x (xi x + x e l λ 0 + x (xi x x ( s x ( s > c 0 (xi x (xi x ( x λ0 i 9

30 α P ( x s > c : θ 0 P ( T > c, c t α The critical regio with sigificace level α for LRT is F-istributio: C { x s > t α } Def. A raom variable F is sai to have F-istributio with egrees of freeom (r, r if there are iepeet chi-square r.v.s χ (r a χ (r such that χ (r r F χ (r, F > 0 r We eote by F f(r, r We eote f α (r, r with P (F f α (r, r α. I the table we ca fi f α (r, r with α 0.95, 0.975, 0.99 oly. How ca we fi f α (r, r with α 0.05, 0.05, 0.0? Thm. f α (r, r f α (r,r Proof. If F f(r, r, the there exists χ (r a χ (r, iepeet, χ (r r with F χ (r, the χ (r F r χ (r f(r, r r r Let F f(r, r. We wat f α (r, r with α 0.05, 0.05, 0.0. α P (F f α (r, r P ( F f α (r, r P (f α(r, r P (f α (r, r f α (r, r α P (f α (r, r f α (r, r f α (r, r f α(r, r f α (r, r f α (r, r f α (r, r 30

31 Ratio of Variaces σ x σ y { ( s x σx (m s y σy C.I. for σ x : σy Let a,b satisfy So, { or σ y : σx X,..., X N(µ x, σx Y,..., Y m N(µ y, σy χ ( iep. where χ (m F (m s y σ y (m ( s x σ x ( σ xs y σys x α P (f(m, a a α a iep. s x s y m f(m, f α (, m, b f α (x i x m (y i y P (f(m, b (m, α P (f α (m, F f α (m, P ( P ( f α f α xs y σ (, m σys x sx σ x (, m s y σy Hece, a 00( α% C.I. for σ x σ y ( f α is: Hypothesis Testig for Equal Variace: Cosier the hypothesis H 0 : σx σy vs. H : σx σy or H 0 : σ x vs. H σy : σ x σy f α (m, f α (m, s x sx, f α (, m (m, s x s y s y s y 3

32 Sice F σ x s y σ ys x α P (σ xs y σ ys x a α P (σ xs y σ ys x We have α P ( s y s x f(m,, we have f α (m, P (s y s x f α (m, P (s y s x f α sy or (, m The a critical regio with sigificace level α for H 0 is C { s y s x f α sy or (, m What is this test? A UMP test or LRT? We will show that it is a LRT. Likelihoo Ratio Test for Ratio of variaces: { X,..., X N(µ, σ Y,..., Y m N(µ, σ s x s x f α (, m : H 0 f α (m, : H 0 f α (m, : H 0 f α (m, } iep. Θ {(µ, µ, σ, σ : µ, µ R, σ, σ > 0} Hypothesis H 0 : σ σ( σ vs. H σ : σ σ Θ 0 {(µ, µ, σ, σ : µ, µ R, σ σ > 0} Likelihoo fuctio is the joit p..f of X,..., X a Y,..., Y m which is L(µ, µ, σ, σ (π +m The likelihoo ratio is λ l L(µ, µ, σ, σ + m Uer H 0 : l L(µ, µ, σ σ σ + m (x i µ m (y i µ e σ sup L(µ, µ, σ, σ µ,µ R,σ σ sup L(µ, µ, σ, σ µ,µ R,σ,σ >0 σ (σ (σ m m l(π l(σ m (x i µ (y i µ l(σ l(π + m 3 σ σ l(σ m σ [ (x i µ + (y i µ ]

33 l L(µ x, µ y, σ σ σ σ l L µ σ (xi µ 0 ˆµ x l L µ σ (yi µ 0 ˆµ y ˆσ + m [ (x i x + sup L(µ, µ, σ σ σ (π +m Θ 0 We wat sup L(µ, µ, σ, σ Θ + m + m σ σ [ (x 4 i x + (y i y ] 0 ( m (y i y ] m + m [ (x i x + L(µ, µ, σ, σ µ 0 ˆµ x, L( ˆµ, µ, σ, σ σ L(µ, µ, σ, σ µ 0 ˆµ y, L(µ, ˆµ, σ, σ σ 0 ˆσ 0 ˆσ m (y i y ] +m e +m (x i x m (y i y sup L(µ, µ, σ, σ (π +m ( Θ Likelihoo ratio: λ (x i x ( m m ( + m +m ( (x i x + m (y i y +m m m ( (x i x ( m (y i y m (y i y m e +m λ 0 ( ( ( + (x i x ( m (y i y m (x i x + m m (y i y (x i x (y i y +m ( + (x i x m m (y i y λ λ 33

34 F ( + x ( + 0 as x or x 0 m x m (y i y c or c (x i x m (Y i Y (m (X i X ( The LR test with sigificace level α is f α (m, or f α (m, F m (Y i Y (m (X i X ( f α (m, or f α (m, Testig for Hypothesis of Equal Meas: { X,..., X N(µ, σ Y,..., Y m N(µ, σ iep. H 0 : µ µ vs. H : µ µ ( Suppose that it is kow that σ σ σ. We have { X,..., X N(µ, σ iep. Y,..., Y m N(µ, σ T X N(µ, σ ( s x χ ( σ Y N(µ, σ iep. where m (m s y χ (m σ { X Y N(µ µ, σ ( + m X Y (µ µ σ + m ( s x +(m s y σ (+m ( s x+(m s y σ χ ( + m X Y ( s x +(m s y (+m 34 s x s y m + m (x i x m (y i y iep. t( + m

35 A Reject H 0 if T T X Y ( s x +(m s y (+m X Y ( s x +(m s y σ (+m + m + m H 0 t( + m > t α ( + m ( Suppose that we o t kow if σ σ is true. Two step to test H 0 : (i Test H 0 : σ σ Reject H 0 if F m (y i y (m (x i x ( f α (m, or f α (m, (ii If we accept H 0, we test H 0 : µ µ by rejectig H 0 if X Y T > t α ( + m ( s x +(m s y +m + m (iii If we reject H 0, we ca o othig. Hypothesis Testig for p: Y b(, p. Wat to test hypothesis H 0 : p p 0 vs. H : p p 0 Let Y,..., Y Beroulli(p The Y vy i b(, p ˆp Y ˆp p p( p Y i ˆp p ˆp( ˆp Uer H 0, Y i P p by WLLN. p p( p p( p ˆp( ˆp ˆp p 0 ˆp( ˆp N(0, by CLT. ˆp p p( p N(0, α P ( Z z α P ( ˆp p 0 ˆp( ˆp N(0, N(0, by Slutsky s theorem z α : H 0 35

36 The, a approximate test with sigificace level α is rejectig H 0 if z α or z α ˆp p 0 ˆp( ˆp Table of approximate tests for p. Hypothesis H 0 : p p 0 vs. H : p p 0 ˆp p 0 ˆp( ˆp H 0 : p p 0 vs. H : p > p 0 ˆp p 0 H 0 : p p 0 vs. H : p < p 0 ˆp p 0 Critical regio z α or z α ˆp( ˆp ˆp( ˆp Hypothesis for ifferece of p s: { X b(, p Y b(m, p z α z α iep. Wat to test hypothesis H 0 : p p vs. H : p p. Let ˆp X, ˆp Y. We have, by CLT. m N(0, { or Sice ˆp ˆp p p ( p ˆp p p ( p m N(0, ˆp ˆp N(p p, p ( p ( ˆp ˆp (p p p ( p + p ( p m P P p, ˆp p, we have We further have Sice α P ( ( ˆp ˆp (p p the rule for testig H 0 is rejectig H 0 if ˆp ( ˆp + ˆp ( ˆp m ˆp ˆp ˆp ( ˆp + ˆp ( ˆp m ˆp ˆp ˆp ˆp p ˆ ( pˆ + pˆ ( pˆ m ˆp ( ˆp + ˆp ( ˆp m ˆp N(p, p ( p ˆp N(p, p ( p m iep. z α or z α 36 + p ( p m N(0, N(0, H 0 N(0, z α or z α : H 0

37 Bivariate Normal Distributio: Note: (a S: Sample space, P: Probability set fuctio Raom variable X : S R, P (X A P ({w S : X(w A} If there exists f 0, such that { f(x, iscrete X P (X A x A f(xx,, cotiuous X the we call f the p..f of r.v. X. The p..f f satisfies f(x 0 a f(xx A (bfor each fuctio f, f(x 0 a f(xx, there exists a r.v. X such that f is the p..f r.v. X. O the other ha, if f(x, y satisfies f(x, y 0 a f(x, yxy,there exists r.v. s X a Y such that f(x, y is joit p..f of X, Y Cosier the fuctio f(x, y πσ σ ρ e for some µ, µ R, σ, σ > 0, < ρ < ρ [ (x µ σ ρ (x µ (y µ + (y µ σ σ σ ], < x <, < y < We wat to o the followigs: (a Show that f(x, yxy. There exists X a Y such that f(x, y is the joit p..f of X, Y. We call f the bivariate ormal istributio. (b Fi margial p..f s of X a Y. (c Fi coitioal mea E(Y X x a variace Var(Y X x. This is the basis for liear regressio. Now, f(x, y 0 for x, y R To show f beig a joit p..f ees to show that f(x, yxy. 37

38 We wat to show that f(x, yxy ( f(x, yyx f(x, yy πσ σ ρ e (x µ σ πσ e (x µ σ πσ e ρ [ (x µ σ ρ (x µ (y µ + (y µ σ σ σ σ (y (µ+ρ σ (x µ σ e ( ρ y πσ ρ ] y f(x, yxy e (x µ σ x πσ f(x, y is a joit p..f of two r.v. s X a Y. We have show that a bivariate ormal istributio has p..f f(x, y πσ σ ρ e a it may be writte as ρ [ (x µ σ ρ (x µ (y µ + (y µ σ σ σ ], < x <, < y < f(x, y e (x µ σ πσ πσ ρ σ (y (µ+ρ σ (x µ σ e ( ρ We ow erive the margial p..f s. The margial p..f of X is f X (x f(x, yy e (x µ σ πσ e (x µ σ, x R πσ πσ ρ σ (y (µ+ρ σ (x µ σ e ( ρ y The X N(µ, σ We ca also show that Y N(µ, σ Coitioal istributio : The istributio of Y give X x is calle the coitioal istributio. The coitioal mea is E(Y x a coitioal variace Var(Y x E[(Y 38

39 E(Y x x]. The coitioal p..f of Y give X x is f Y x (y f(x, y f X (x πσ e (x µ (y (µ +ρ σ σ (x µ σ πσ ρ e σ ( ρ πσ e (x µ σ σ (y (µ+ρ σ (x µ σ e ( ρ, y R πσ ρ A So, Y x N(µ + ρ σ σ (x µ, σ ( ρ X y N(µ + ρ σ σ (y µ, σ ( ρ Y x µ + ρ σ σ (x µ + ɛ, ɛ N(0, σ ( ρ µ ρ σ µ + ρ σ x + ɛ σ σ β 0 + β x + ɛ, ɛ N(0, σ This is the liear regressio moel. We have observatios ( x ( y,..., x y, the the liear regressio moel is y i β 0 + β x i + ɛ i, i,..., where ɛ,..., ɛ are N(0, σ The ( problem i liear regressio is that we have a sequece of raom vectors X ( Y,..., X ( Y with observatios x ( y,..., x y a picture. But we believe that the observatios obey the followig liear regressio moel. y i β 0 + β x i + ɛ i, i,..., where ɛ,..., ɛ are raom variables with mea 0 a variace σ The aim i liear regressio aalysis is to ifluece (estimatio a hypothesis testig the parameter β 0, β. If we kow β 0 a β, the the preictio of a future y 0 where x is x 0 is ŷ 0 β 0 + β x 0 39

40 Chi-square Test(Gooess of Fit Test I evelopig of C.I. for θ a hypothesis testig, most methos are erive assumig that raom sample X,..., X is raw from ormal istributio. A importat questio is: how o you kow that it is really raw from ormal istributio? So, we may try to test the followig hypothesis: H 0 : X,..., X N(µ, σ This is a gooess of fit problem that ca be solve be chi-square test by Karl Pearso (Father of Pearso by Neyma-Pearso Theorem We first cosier the hypothesis H 0 : X,..., X f 0 (x where f 0 is kow p..f. Let A,..., A k be a partitio (mutually exclusive sets for space of X. Defie P j P (x A j H 0 f 0 (xx, j,..., k A j k P j j Let N j eotes the umber of X,..., X fallig i set A j, j,..., k. k So, N j. We have the followig theorem. j Thm. Let Q k k j (N j P j P j k j (pratical # theoretical # theoretical # i A j We have Q k χ (k if H 0 is true. Proof. We cosier k oly. Q (N P P + (N P P (N P (N P ( + P ( P (N P P ( P N P ( P ( P χ ( as + (( N ( P P ( P 40

41 N (# of X,..., X fallig A H 0 b(, P N P P ( P N(0, by CLT Def. The Pearso s chi-square test for hypothesis H 0 : X,..., X f 0 (x is rejectig H 0 if Q k χ α(k, where α P (χ (k χ α(k Note: P (Q k χ α(k : H 0 P (χ (k χ α(k α Example: Meelia theory: Shape a color of a pea ought to be groupe ito four groups with probabilities as follows: Groups probabilities obs Rou a yellow P 9 N 6 35 Rou a gree P 3 N 6 08 Agular a yellow P 3 3 N Agular a gree P 4 N With sample 556(x,..., x 556, the umbers groupe are isplayig above. We wat to test hypothesis at sigificace level α 0.05 H 0 : P 9 6, P 3 6, P 3 3 6, P 4 6. Q 4 4 (N j P j j P j 9 ( ( ( χ 0.05(3 7.8 > Q 4 We o ot reject H 0 a coclue that Meelia s theorem is correct. How ca we test the hypothesis H 0 : X,..., X f(x, θ,..., θ p 4 (

42 where f is kow p..f but θ,..., θ p are ukow? Let A,..., A be a partitio of space of X a ˆθ,..., ˆθ p be mle s of θ,..., θ p. Defie ˆP j f(x, ˆθ,..., ˆθ p x, j,..., k A j Agai, we eote ˆN j the # of X,..., X fallig i A j. So, Thm. Let We have Q k Q k k ( ˆN j ˆP j j ˆP j χ (k p if H 0 is true. k j ˆN j Def. The chi-square test for H 0 : X f(x, θ,..., θ p is rejectig H 0 if Q k χ α(k p Example: X,..., X f(x, θ. Cosier hypothesis H 0 : X N(µ, σ. sol: mle s ˆµ x, ˆσ (x i x Let A (, a, A (a, a,..., A k (a k, partitio of space of N(x, s.defie.. ˆP P N(x,s (X a P N(x,s ( X x s ˆP P N(x,s (a X a P N(x,s ( a x s P (Z a x s P (Z a x s a x s ˆP j P N(x,s (a j X a j P (Z a j x s ˆP k P N(x,s (X a k P (Z a k x s Q k k ( ˆN j ˆP j j Reject H 0 if Q k χ (k 3 ˆP j 4 P (Z a x s X x s χ (k 3 a x s P (Z a j x s