ECE 830 Fall 2011 Statistical Sigal Processig istructor: R. Nowak, scribe: Iseok Heo Lecture 17: Miimum Variace Ubiased (MVUB Estimators Ultimately, we would like to be able to argue that a give estimator is(or is ot optimal i some sese. Usually this is very difficult, but i certai cases it is possible to make precise statemets about optimality. MVUB estimators are oe class where this is sometimes the case. First let s quickly review some key estimatio-theoretic cocepts. 1 Review of key estimatio cocepts Observatio model Loss Example 1 X p(x θ, x X, θ Θ l : Θ Θ R + Risk : expected loss of estimator ˆθ(x l 2 : l(θ 1, θ 2 θ 1 θ 2 2 2 l 1 : l(θ 1, θ 2 θ 1 θ 2 1 log-likelihood : l(θ 1, θ 2 log p(x θ 2, where x p(x θ 1 R(θ, ˆθ E[l(θ, ˆθ MSE : the l 2 risk is usually called the mea square error : MSE(ˆθ E[ θ ˆθ 2 2 Recall that the MSE ca be decomposed ito the bias ad variace MSE(ˆθ E[ θ ˆθ 2 2 E[ θ Eˆθ + Eˆθ ˆθ 2 2 θ Eˆθ 2 2 + 2E[(θ Eˆθ T (Eˆθ ˆθ + E[ Eˆθ ˆθ 2 2 θ Eˆθ 2 2 + E[ ˆθ Eˆθ 2 2 bias 2 (ˆθ + var(ˆθ It is usually impossible to desig ˆθ to miimize the MSE because the bias depeds o θ, which is of course ukow. But suppose we restrict our attetio to ubiased estimators; i.e., ˆθ satisfyig Eˆθ θ. The MSE(ˆθ var(ˆθ ad var(ˆθ does ot deped o θ. So a realizable approach is to optimize the MSE with respect to the class of ubiased estimators. The Miimum Variace UBiased (MVUB estimator is defied as θ arg mi E[ ˆθ Eˆθ 2 2 ˆθ : E[ˆθθ 1
Lecture 17: Miimum Variace Ubiased (MVUB Estimators 2 Example 2 X 1, X 2,..., X iid N (θ, 1 Is this the MVUB estimator? ˆθ 1 xi E[ˆθ θ [ 1 2 MSE(ˆθ E xi Eˆθ var xi 2 Fidig the MVUB estimator 1 2 var(xi 1 Fidig the MVUB estimator ca be difficult, but sometimes it is easy to verify that a particular estimator is MVUB. Theorem 1 (Cramér-Rao Lower Boud (CRLB Let x deote a -dimesioal radom vector with desity p(x θ, θ R k Assume that the first ad secod derivatives of log p(x θ exist. Let ˆθ ˆθ(x be a ubiased estimator of θ. The the error covariace satisfies the matrix iequality E[(ˆθ Eˆθ(ˆθ Eˆθ T I 1 (θ where I(θ is the Fisher-Iformatio matrix with i,jth elemet [ I ij (θ 2 log p(x θ E i j Remark : The meaig of the iequality is that the eigevalues of the symmetric matrix are o-egative. As a cosequece C : E[(ˆθ Eˆθ(ˆθ Eˆθ T I 1 (θ C I 1 (θ var(ˆθ tr(e[(ˆθ Eˆθ(ˆθ Eˆθ T tr(c tr(i 1 (θ Proof : We will prove the scalar case (θ R. The geeral case follows i a similar fashio. The Fisher- Iformatio is scalar i this case : [ I(θ 2 log p(x θ E 2 Before proceedig we will show first that [( 2 log p(x θ I(θ E
Lecture 17: Miimum Variace Ubiased (MVUB Estimators 3 To this ed, first observe that 2 log p(x θ 2 ( log p(x θ 1 p 2 (x θ p(x θ p(x θ p(x θ p(x θ p(x θ p(x θ 2 + 1 2 p(x θ p(x θ 2 2 + 1 2 p(x θ p(x θ 2 Cosider the expectatio of the secod term : [ 1 2 p(x θ 1 2 p(x θ E p(x θ 2 p(x θ 2 p(x θdx 2 p(x θ 2 dx 2 2 p(x θdx 2 2 (1 0 Thus, [ 2 log p(x θ E 2 [( 2 1 p(x θ E p(x θ [( 2 log p(x θ E The gradiet of the log-likelihood is called the score fuctio. Let s deote it S(θ, x : log p(x θ Observe that the MLE satisfies S(ˆθ, x 0. Also ote that E[S(θ, x log p(x θ p(x θ dx p(x θdx 0 ad therefore the Fisher-Iformatio is the variace of the score fuctio [( 2 log p(x θ I(θ E So we see that the Fisher-Iformatio measures the variability of the score fuctio at θ θ. We will also show that E[S(θ, x(ˆθ θ 1 To verify this, ote that E[ˆθ θ (ˆθ θ p(x θ dx 0
Lecture 17: Miimum Variace Ubiased (MVUB Estimators 4 sice ˆθ is ubiased. Take the derivative 0 (ˆθ θp(x θdx ( p(x θdx + (ˆθ θ p(x θ dx 1 + (ˆθ θ log p(x θ p(x θ dx 1 + E[S(θ, x(ˆθ θ Now we apply the Cauchy-Schwarz iequality. (i.e., f(xg(xdx f 2 (xdx g 2 (xdx var(ˆθ Example 3 X 1, X 2,..., X iid N (θ, 1 E[S(θ, x(ˆθ θ 1 θθ 1 E[S(θ, x(ˆθ θ E[S 2 (θ, x E[(ˆθ θ 2 var(s(θ, x var(ˆθ 1 var(s(θ, x I 1 (θ log p(x θ log p(x i θ log p(x θ log p(x i θ log (x i θ [( 2 log p(x θ I(θ E But recall the ubiased estimator e (x i θ2 2 2π E[(x i θ 2 MVUB estimator variace 1 ˆθ 1 x i var( θ 2 1 ˆθ is the MVUB estimator!
Lecture 17: Miimum Variace Ubiased (MVUB Estimators 5 3 Efficiecy A ubiased estimator that achieved the CRLB is said to be efficiet. Efficiet estimators are MVUB, but ot all MVUB estimators are ecessarily efficiet. A estimator ˆθ is said to be asymptotically efficiet if it achieves the CRLB, as. Recall that uder mild regularity coditios, the MLE has a asymptotic distributio ˆθ asymp N (θ, 1 I(θ ad so ˆθ is asymptotically ubiased ad so it is asymptotically efficiet. var(ˆθ 1 I 1 (θ