Statstcs et Y ~ p( ; ( ; ( ; ( ( ; ( p Y t p g t h Y suffcet statstc (for the parametrc faml p( ; s depedet of t Nema-Fsher factorzato: where g ad h are o-egatve fucto p( ; Smple bar hpotheses: {, } : t( p( ; p( ; p( ; For {,, }, t (,, p( ; p( ; Suffcet a (measurable fucto h ( such that t( ht ( s a suffcet statstc s a suffcet statstc t s a mmal suffcet statstc f, for a other suffcet t, there s Suppose there exsts a fucto p( x; the rato p( ; he t s complete f A statstc ( Y such that for two sample pots x ad, s a costat as a fucto of f ad ol f ( x ( Y s a mmal suffcet statstc for Prg t Y Λ ( Eg t Y g t p ; d ( Y complete ad suffcet mmal suffcet ad uque Ex X ~ (, t( x max{ x} d U, K-parameter expoetal faml: A faml of dstrbutos s sad to be a K- parameter expoetal faml f ( ; exp K c t + d + s IA K p c( t( e f ( h( IA ( If c {( c,, c K, Λ} has a teror pot, the t ( ( t (,, tk ( s complete ad suffcet mmal suffcet ad uque Setup Parameter space { } Pr where IA ( s the dcator fucto ot related to Λ Λ P( Θ Λ Λ Pr [ Θ ] Θ Θ Λ pothess: : ~ ( ; Observato space Γ d Y p Λ,,,, -
A determstc detector : {,, -} Parttos the observato space Γ to K dsjot subsets Γ wth Θ Whe Θ { }, : Λ { } ( : pdf/pmf o {,,, } ( ( Γ ad detf A radomzed detector ( D Y Pr ( he detecto D d s a realzato accordg to D ( Cost: C(, Cost C(, j Cost j Uform cost: [ ], Assume Cj > C for all, j he Baesa Detector Baesa / Baes Rs: R E p R R p R d ( [ Cost] ( ( ( Codtoal rs: Cost R E Y (, Pr Θ (, ( ; R C D C p d Γ C, C j, Λ j C, Pr Γ Θ C, p ; d for determstc detector R R d Λ Baesa Detector B argm R o the detector, the Baesa detector for a gve : B ( argmr( argme Cost Y Smple potheses: { } he Baesa detector s determstc: Because the dstrbuto of does t depeds,,, Pr D Y, o/ w d argme Cost D Y, argm Cj jp ; j where E Cost D Y, C p Θ Y C p ; j j j j j j j p j (, Pr Θ (, ( ; R C D C p d Γ,
R( j Cj p( ; j d Wth uform cost, t s the probablt of j Γ error, Baesa detector for pror p(, τ ( C C d ( p( where τ ( C C, otherwse ( C p ( C p ( C p ( C p ( + + c Γ : p τp Γ Bar Smple potheses: Pr[ ] B, { ( } + + + ( ( ; ( ; ( ; ( ; R C p d C p d C p d C p d Γ Γ Γ Γ Ex : Y ~ N (, σ, : Y ~ (, σ N < + σ C C he, Γ : γ + l For uform cost, ( CC γ γ R( Q + Q σ σ a b a τ b + a + b Uform cost wth dett cost matrx Composte Bar pothess estg Pr[ Θ ] p( Θ Λ p( Pr Θ Θ Λ d p( d Λ p( Pr[ ] d Θ Λ Pr[ Θ ] Λ Pr Θ Θ Λ Λ, ( τ ( C C B τ where, ( < τ ( C C P( Θ Λ Pr [ Θ ] d Λ Λ
( p p ( Θ Λ ( Θ Λ Λ Λ p d Pr[ Θ ] ( C C τ ( Pr[ Θ ] p d R Pr D Θ Λ + Pr D Θ Λ C C p dp Λ d + p dp Λ d Λ Γ Λ Γ p p dd+ p p dd Γ Λ Γ Λ p p dd+ p p dd Λ Γ Λ Γ max Detecto max detector/rule/crtero: If we ow Θ ~ Λ mmax R Λ R R d Smple bar hpothess testg: the mmax detector s mmax R Rs for gve pror : r(, R ( ( R ( r(, r(, B, max detector s mmax { R, R} mum Baesa rs V( r(, B, ( ( V r, B, + (ear wrt R ( B, V ( r (, B, R ( ( ( V R R ( B, Cocave ad cotuous [,]
argmaxv ( least favorable pror If V ( exsts at ( ( B, B,, the r (, B, V R R V at s a taget le of ( Cost (, ( ; + (, ( ( ; B, R E C p d C p d ( Cost (, ( ; + (, ( ( ; R E C p d C p d the Baesa detector desged at ( B, argm r, mum Baesa rs gve pror: r r ν,,, Solvg for max detector for bar smple hpotheses: Equalzer rule: If there exsts a pror s the least favorable pror V( V ( B, If ot the, If or, the B, B, such that R ( R ( B, B,, the, ad the mmax detector Otherwse, (f V ( s lear the small eghborhood of ( wth probablt q ad B, B, R +, ( ( R ( B, + B, ( ( + ( +,, ( R + R +,, ( R( B, R(, B B B, cosder wth q from wth probablt q + + ( V ( R R B B V q V ( (, B, V r (, (, B, r r ( (, B, V r r (, B, r (, r (, B, +
he mmax rs V( R ( B, R ( B, A detector there exsts a pror Radomzed Baesa Detector: ( ( ( for all cases s mmax f t satsfes the equalzer rule R ( R such that V( ( wth probablt q wth probablt q R qr + q R R r (, R( R( R R( R( ad he equalzer rule remas vald for composte hpotheses If the m Baesa rs s dfferetable at the least favorable pror, the the Baesa detector s the mmax detector Nema-Pearso Detector m ( subject to R K R α, < K Bar potheses: : Y p( ; alteratve, Λ,, : ull hpothess : the Smple bar hpotheses: : Y p( ; ; : Y p( ; rego; Γ : rejecto rego Γ : acceptace False alarm (I P ( ; Pr[ D ; ] ( p( ; F d E ( ( he sze / level of a detector α( sup P F ( ; Λ Λ ss detecto (II P ( ; Pr[ D ; ] ( ( p( ; d E ( ( Λ he power of a detector P ( ; Pr[ D ; ] ( p( ; d E ( ( Λ P ( ; P ( ; D D F D has the same formula but usg from dfferet sets Uforml most powerful (UP: A sze α detector UP s UP f of sze α, P ( ; P ( ; D UP D Λ For smple bar hpotheses, (If Λ P ( ; ad P ( ; Λ ( α( α( P ( ; P ( ; UP ( ; UP D UP D ( arg max P ; E Pr D ; α UP UP s a sze α NP detector for : Y p ( ; ; : Y p ; for a Λ? For smple bar hpotheses, NP detector s arg max P arg m P arg m R wth uform cost PF ( α D P PF α R α α D, the
Nema-Pearso emma for smple bar hpotheses: Optmalt A p( > ηp(, ( ; ( ; p( ηp(, ; ; γ p ηp, ; < ; γ, s the best of ts sze [ ] ( ( η ( ( D D F F P P > P P for all Exstece α [,] η, there exsts a detector of the form above m η γ ( Pr ( > η; α for some η ad γ, Pr η ; arbtrar, otherwse Pr ( > η ; s a complmetar dstrbuto fucto, rght cotuous, ad mootocall decreasg γ ( α Pr > η ; Pr η ; ( α P Pr ( η ; γ Pr ( η ; F > + 3 Uqueess If s a sze α NP detector, the ( has the form above except perhaps for a set of wth zero probablt uder both ad P Pr D ; Pr η ; γ Pr η ; D > + Sce Note: [ ] > η Pr ; s also mootocall decreasg, we wat low η to get hgh P elpful to plot Pr D ( > η ; stead of ( whe the trasformato s :, creasg vs η 3 Ca wor wth t ( UP detector et be a real parameter he real-parameter faml ( ; lelhood rato ( ( f <, p( ; ad ( ; p( ; ( ;, s a odecreasg fucto of some real valued p( ; Ex Ce µ σ ( u ( Q ( c h e d Beroull, ( Oe-sded potheses estg: Y p( > : ; p has mootoe p are dstct ad whe Q ( s mootoe 3 Y p( : ; > (or he Kal Rub heorem: et be a real parameter ad let p( ; have mootoe lelhood rato ( For testg the oe-sded hpotheses, there
exsts a sze α UP detector of the form ( are determed b the sze costrat τ ( ;, > τ γ, τ, < τ m τ, Pr > τ ; α E p d α τ ad γ are fuctos of Gve a <, ad P ; P ; where τ ad γ wth NP form he F F E ( Pr D s a odecreasg ( fucto of So, for Λ (, s α( sup PF( ; PF( ;, the sze (false alarm of wo-sded pothess: ( ; Λ a ( b wth odecreasg p h e UP detector for : or, : (, of the form, c < < c ( γ, c where c < c ad γ are determed b, otherwse E ( ( α E If s surrouded b, the suspect o UP detector Baesa Estmao Estmate radom ~ p( a Θ from Y ~ p( he cost s EC( ΘΘ mmze EC( ( Y Θ Baesa ( argm R ( R( ( Θ ( C p( d estmator E C Y SE: ( argm Baesa SE p d EΘ Y SE, µ +Λ Y Λ YY µ N Θ Θ Y I addto, f µ µ are zero, the ( Λ Λ, lear For For jotl Gaussa Θ ad Y,, Θ Y example, let Y a N SE, N, Θ Y YY d Θ+,,, Θ ~ N (,σ Θ N ~ N (, σn σ Θ SE, N, ( a where γ If a, the σ N aa+ γ γ SE, N, ( + γ ear observato model: et Y S + W, S W, S ~ N ( µ, Σ S S, W ~ N (, ΣW, the the SE estmator s he
E SY µ +Σ S S ( Σ +Σ S W ( Y µ S Error covarace matrx: CovSY Σ Σ ( Σ +Σ Σ S S S W S C C x C x, ad covex, Gve, assume s smmetrcal, e Θ, the p( +, the ( ( let E m ( ( m ( m p p s smmetrcal wth respect to m, e Baesa SE m ( argm p( d argm p AE p d p d Θ Y For C u, ( argmax + +, >, (, ( p d et d + + u, argm p( d, the AP ( argmax p( SE For E X X ubased, Cov( X X E ( X X( X X SE( X X X X trace( ( X E Cov X X E ear SE: EΘ? Gve zero mea radom varables Y,,,, Θ f Y fy f EΘY ( YY E mmzes SE ΘΘ E Θ E ΘY E YY E YΘ E to Θ s the orthogoal projecto of Θ oto { Y Y } E ΘΘ For Y C, X C, m F XY E E YY spa,, X s the lear SE estmate of X X s also the optmal lear estmate usg the weghted cost fucto m E ( X F Y Λ( X F Y for a Λ F
ear observato model: Y S + W S, E, W S S Σ SS Cov, W W Σ, the the lear SE estmate of S s gve b WW S ( Y ΣSS Σ +Σ SS WW Affe SE estmator: Gve radom vector Y If E Y ad E X are ow, the the SE affe estmator of X b Y s gve b X µ +Σ Σ X XY Y ( Y µ Y ( Cov X X Σ Σ Σ Σ E X X X XY Y YX (, (, Cov X X Cov X X Σ Σ Σ XY Y YX If s osgular ad u, the the SE affe estmator of x usg s the same as that usg u If W X ad are ucorrelated, ad V BX + W, the the SE affe Y V Y BX Bµ + BΣ Σ Y µ, estmator of V usg Y s gve b X XY Y ( Y Cov V V Pot Estmato ( ( BCov X X B affe,se +Σ g E gy g for a fxed E gy g gy gy + gy g E E E ĝ of g( s ubased f E g ( Y g( he, ( g gy gy E E ; For ubased, ( trace{ Cov( ( Y } Cov Y Cov Y Cov Y + a alwas Crtero: mze SE (rs ( A estmator Note that ( ( UVU A estmator ĝ( of g( s UVU (uforml mmum varace ubased f ubased For all ubased ĝ, ( g ( g Rao-Blacwell heorem: Suppose that estmator for g( wth Eg ( W Y s suffcet for ad that < for all ĝ s a
et g ( Eg( ( Y ( ( ( YY he g g p d, g ( Y g E E gy g( 3 If compoets of ĝ have fte varaces, the the strct equalt holds uless Prg ( Y g( Y E gy g gy g ( Y + g ( Y g( E E Furthermore, f ĝ( s ubased, the 4 ĝ ( s ubased for g( 5 ( ca be wrtte as Varg ( Y Varg( Y ĝ h g g If, the ehma-scheffé heorem: If ( Y s complete suffcet, ad ubased estmator of g( he g ( ( g( Y ( Y ( estmator Shortcut: Kowg ĝy s a E s a UVU Y s complete suffcet, tr fdg ( Y E For oe-parameter expoetal faml, (Y s complete ad suffcet, f t s ubased, the t s UVU CRB: Cramér-Rao ower Boud he score fucto s( l p( ; ; l p( ; E sy; l p( ; K I Es Y; s Y; Cov s Y; Fsher Iformato atrx: Ij E l py ( ; l py ( ; E l p( Y; j j ( l ( ; ( l ( ; l ( ; I E py py py E For scalar, I E l p( ; l p( ; E l p ( ; he Cramér-Rao Boud: et be a scalar ubased estmator of he, CRB: ( Var( Var( ( Y E ( Y I wth equalt ff ( ; l ( ; ( s p I
Iformato lower boud: For based estmator, E ( Y Φ(, the ( Φ ( Var ( Y wth equalt ff s( x; I I ( ( Φ If ( Y acheves formato lower boud, the t has mmum varace amog all s effcet ad UVU Y s estmators ( Y satsfg E ( Y E ( Y Furthermore, f ubased, the ( Y Oe parameter expoetal faml: et Λ be a ope terval, ad p( ; g ( C( e h( ( Y f a ol f ( Y ( Y Wth regulart, the formato lower boud s acheved b Also, (Y s complete ad suffcet 3 If (Y s ubased, the t s UVU ad effcet A ubased estmator s effcet f t acheves CRB A effcet estmator s UVU but a UVU estmator ma ot be effcet (whe CRB s ot achevable Y acheves CRB, the t s the soluto to the lelhood equato If l p( ; effcet estmator dstrbuto of the observato must belog to the expoetal faml he effcet estmator ca be foud b the estmator CRB ubased estmator of, the E ff l p( ; I ( et E wth equalt ( Y ( ( Y I ( g ĝ be a ubased estmator of ( gy g( gy g( ( (, the ( dg I ( ( dg(, wth equalt ff g g dg I l p ; µ Y ~ N µ, Σ µ Σ Σ he, I( Σ + tr Σ Σ j j µ µ ( µ Σ Σ j where,,, et Y ~ N( µ, Σ, the I( ( d µ Σ d µ l p ; dµ Σ µ et Also,
ear model: X + W, W ~ N (, Σ he X ~ N (, Σ, ad I Σ l p ; Σ Σ Σ Σ Σ s UVU, effcet, Gaussa,, east-square Need full colum ra for detfablt ~, ( Σ et Y ~ CN ( µ, Σ, real he I ( j Σ Σ tr Σ Σ j State Estmato N µ µ Re Σ + State Estmato: states: S AS + + U observato: Y S + W Kow dstrbuto of S, put sequece { U }, observato ose { W } E S s, VAR S Σ Fd the SE estmator of S gve Y, Y,, e, s E S,, Dscrete-me Kalma Buc X FX GU Y X + V Q Cov( U, R Cov( V +, + t t t t X E [ X ], t Σ Σ Cov( X Σ Cov tt ( Xt Y Kalma ga matrx K t Σ t tt tσ tt t + R t t X E X ( tt t Y X + K tt t Yt X t tt Σ Σ K tt tt t tσ t X E X tt t t t+ Y FX + t Σ F tt tσ F t t tt t + GQG + t t t Kalma Flterg t Notato: {,, } t t the past samples Σ ( ( tt t tt t tt s the SE predcto of S from t t E S tt t t E S s S s t s E S tt t (the SE flter t Σ ( ( tt E St stt St stt u, { } Λ U Euu Λ W Eww [ ] s ~ N( s, Λ depedet of { u }, { w } Italzato: ŝ ES, Σ VARS Σ, s SS Gaussa odel: { } w are zero mea, depedet, Gaussa [ ] easuremet Update: flterg: K Σ Σ +Σ W s s + K ( Σ Σ K Σ
me Update: predcto: s As + Σ A Σ A + +Σ U s + + + Same formula for lear SE of o Gaussa Example Normal d: µ ; σ ( µ based Expoetal d: p( ; e I( > e I > ( d X ~ P ( λ ( λ s UVU ( based x λ x λ e λ ; λ ; λ p x e x h x ( xj! ( xj! j j p x; I max x < I m x > d X ~ U (, ( { } g ( max { x }, + max complete ad suffcet ( { } Bar d: p( x; UVU p ; g ( Bomal: Estmator ( { } t( x max{ x } s UVU he estmator of parameter from Y ~ ( ; argmax p( ; argmaxl p( ; Θ Θ s complete ad suffcet ; s complete No ubased estmator for p he best lear ubased estmator (BUE s BUE ABUE E AY subject to [ AY ] Θ s ( ABUE where argm E + wth zero mea ose, For lear model X W BUE s
For the K-parameter expoetal faml, let C be the teror of the rage of {( c,, c, } K Λ If E t( Y t(,,, K have a soluto ( for whch c (,, c ( K C, the s the uque estmator of oto Ivarace: et g ( : Θ Φ, g ( φ : Φ { A: A } mage Defe ( ; φ sup p( ; g ( φ Θ be the verse l If s the estmate of, the φ argsup l ; φ ( φ Φ ( g ( py ( ; p( ; D E l ( ; p l d p( Y; p( ; p( ; p( ; ae If s detfable, the s the global mmum of D wth equalt ff D D, ad m ( max l py ( ; Θ E For d Y, argmax l p( ; argmax l py ( ; o solve for : Θ N N E s ; l p ; ( + ( Newto-Raphso: J ; s ; J ; l p ; Scorg ethod: et the complete data ( + s( ; + I S Z Y ~ p ( z ; Ol Y ~ ( ; ( ( ; E ( l ( ;, + Q, > Q, l p ; l p ; p s observed E: Q pz Y argmax Q ; If dstrbuto of S does ot deped o, ( Q( ; E ( l pys ( ; Y + costat lme Y Asmptotcall ubased Cosstec: (d dstrbuto lm p ( p (, (p wea lmpr( Y >, (wp strog ( Y Pr lm, (ms mea square lme ( Y (wp (p (d (ms p p ad bouded Θ (ms ~ N, Σ Asmptotcall Normal:
Var ( Y Asmptotcall effcet (BAN: best asmptotcall ormal lm CRB ( ~ (, N I ; I lm I Y d Uder regulart codtos, (wea ( s asmptotcall Gaussa ad asmptotcall effcet Sequetal Detecto Fxed sample sze (FSS detector Example: Cosder the -sample smple bar d hpotheses vs : Y ~ ( µ, σ µ µ µ µ l Y ( Y σ σ ( µ µ ( µ µ N,, σ σ l Y ( ~ ( µ µ ( µ µ N,, σ σ N,,,,,, µ > µ he he mmum such that the σ µ µ optmal detector has sze < α, ad power > β s Q ( α Q ( β d ~ ; Smple bar hpotheses vs Y p(,,, A sequetal detector ( φ, s defed b stoppg rule sequece [ ] φ ( R { } φ : stop data collecto & mae decso :, termal decso cotue data collecto : ( Pr D Y Stop N φ m : φ Y rule sequece [ ] { } tme: ( he sequetal probablt rato test: p( ; SPR(A,B: (, p( ; ( B, stop, φ( or ( A,, ( ( AB,, ( B (, ( A ( B B A A Choose Choose
β SPR s optmal the Baesa problem SPR satsfes A ad B β α α he Wald-Wolfowtz heorem: he SPR(A,B detector ( φ, has the mmum stop tme amog all detectors (cludg FSS detector wth sze o larger ad power PF ( PF ( o less tha those of ( φ, E N N(,, φ φ PD( PD( E py ( ; p ( ; Z l z l l ( z py; p ; Wald s Approxmatos : Gve α ad β, the optmal SPR(A,B ca be β β approxmated b SPR ( A, B wth A ad B α α β β? β β A A < B B α < β α α α α AB, A, B the approxmato requres more samples P ( + P ( α + ( β α + ( β P ( + P ( F F he Wald s Equato: et Z be depedet ad d wth EZ et N be a E Z + + Z E Z E N stoppg tme, the, [ ] [ ] [ ] E [ N ] [ Z] N β β αlb+ α l A αl + ( α l α α E E [ Z] β βlb + ( β l A βl + ( β l E [ N ] α α E [ Z] E [ Z] E p( ; [ Z] ( ; p l d p( ; [ ] p( ; E Z ; p l d p( ; Revew b( x ( + ( + ( +,, f ( x g( x, d, f ( x g b ( x g( xb, ( x a ( x g( xax, + ( xd,, x! N, N + a λx xe dx + λx e λ 6 ax e dx a λx β bx a x a x x e ax b 4ac ( ax + bx+ c 4a e dx e α ( Γ + dx + a, for
( f x ( df ( x f f x df ( (,, ( d x x x x d( Ax + b A ( d a x a d ( f ( x g ( x f ( x dg ( x + g ( x df ( x d( f ( x Qf ( x f ( x Qdf ( x ( d f x f ( x df ( x x x x, x ( Ax + b A, x ( a x a, x ( f ( x g( x g x ( x f x + x ( f ( x g( x x ( f ( x Qf ( x x ( f ( x Qf ( x ( x Qx Qx x f x f x f x x ( x f XY, Y f x, pxy xdx E f XY, Y E f ( X g( Y g ( Y f ( X Y g (, f(x x, the have E [ X] XY X X µ X Λ Λ XX XY Z s jotl Gaussa ~ N, Y Y he, µ Λ Λ Y YX YY p( x ~ ( EX Λ, X ( EX µ µ +Λ Λ, X XY YY Y Λ Λ Λ Λ Λ X XX XY YY YX Defe K Λ Λ he E XY YY X µ + K X ( µ Y, Λ Λ K X XX ΛYX ( XYW,, jotl Gaussa W ( XY, BX + W he V ~ N( E V, Λ V where EV BEX + E W, E a costat ad Λ BΛ B +Λ ( V X WW aa + ci I aa c c a a c ( + Schur complemet of A A AA A I A A I A A A A A I A A ; I A AA A A ( A A ad ( A A A A A A ( + ( + A BCD A A B C DA B DA ( + ( + A BCD A A B I CDA B CDA
N(m,σ : X ( det( Λ ( σ ( x m σ f x e σ e, Λ ( x m ( x m E e jvx e jmv v σ x µ σ ; d exp σ µ µ exp x + x σ σ σ det ( Λ e ( x m ( x m Λ ; CN ( µ, Σ N ( µ, Σ Q (, Q( z Q( z xm Q ( Q( z z P[ X > x] Q σ xm xm P[ X < x] Q Q σ σ λ λ ux Posso P(λ, e ; Ω N, λ, EX λ, VAR(X λ, Φ X ( u Ee ( e e λ u! Bomal p ( p ; p, p(-p, (pe u +-p a+ b Uform U(a,b,, b a b+ a s u ( b a u, e Expoetal E(α,, b a u α α, α aplaca α u αx α x (α, e x < e α ;α >,,, αx e x α, α α + u q x Gamma fucto: Γ ( q x e dx ; q > Γ ( (! for N ( N! Γ λ x Γ q, λ p( x Γ f { } dstrbuto: E ( λ s (,λ Γ Beta dstrbuto: p( z β q, q ( x x ( x q q λx ( q Γ + Γ Gamma e Γ +! q > x Expoetal dstrbuto ( q q ( q Γ( q Γ + q ( q z z z Γ ; z (,
et X ~ p( x λ x Γ e q q λ x ( q X X, depedet he Z X + X are depedet Z β ( z Z X X ~ ( q q, λ ~ Γ q, λ X ~ q, q X + X Cetral ch-square dstrbuto: X ~ (, σ σ e, ( u p X d σ ( σ ~ N, Y ( Φ + Γ + N Y juσ X he ad Z X + X ch-square (or gamma: X he σ p e, σ ( σ Φ u j uσ E [ Y] σ, Var[ Y] Decorrelato If A s the covarace matrx ( E[ xx ] Γ, Γ 4 σ x of a zero mea radom vector x x he vector x x ca be decorrelated va trasform x x AA x wth A covarace Cov( E[ ] A A A A Cov ( x A A x wth equalt ff Pr x A A x