Higher Order Properties of Bootstrap and Jackknife Bias Corrected Maximum Likelihood Estimators

Transcript

1 Higer Order Properties of Bootstrap ad Jackkife Bias Corrected Maximum Likeliood Estimators Jiyog Ha Brow Uiversity Guido Kuersteier MIT Marc 5, 2002 Witey Newey MIT Abstract Pfazagl ad Wefelmeyer (978) sow tat bias corrected ML estimators are iger order efficiet. Teir procedure owever is computatioally complicated because it requires itegratig complicated fuctios over te distributio of te MLE estimator. Te purpose of tis paper is to sow tat tese itegrals ca be replaced by sample averages witout affectig te iger-order variace. We focus o bootstrap ad jackkife based bias correctio as a way to implemet bias correctios i a oparametric way. We fid tat our bootstrap ad jackkife bias corrected ML estimators ave te same iger order variace as te efficiet estimator of Pfazagl ad Wefelmeyer. Bias corrected ML estimators are terefore iger order efficiet eve if te bias fuctio is estimated from te data rater ta computed aalytically. Departmet of Ecoomics, Brow Uiversity, Box B, Providece, RI Jiyog_Ha@brow.edu MIT, Departmet of Ecoomics, E52-37A, 50 Memorial Drive, Cambridge, MA gkuerste@mit.edu. Fiacial support from NSF grat SES is gratefully ackowledged. MIT, Departmet of Ecoomics, E52-262, 50 Memorial Drive, Cambridge, MA wewey@mit.edu.

2 Itroductio Optimality properties of oterwise asymptotically efficiet estimators are ofte distorted by small sample biases. Pfazagl ad Wefelmeyer sow tat bias corrected imum likeliood (ML) estimators are iger order asymptotically efficiet. Teir bias correctio ivolves aalytical computatio of ofte complicated itegrals ivolvig te true data desity. From a applied poit of view tis procedure is terefore uattractive if ot ofte ifeasible. As a result te statistical ad ecoometric literature as see a variety of alterative bias correctio teciques tat acieve ubiasedess up to stocastic orders of i iid settigs, were istesamplesize. Adrews (993) cosiders media ubiased estimatio of a autoregressive model wit ormal errors. His procedure is based o te iversio of te quatile fuctio tat is i priciple kow or ca be umerically evaluated. Sice tis procedure depeds o complicated fuctios of te uderlyig desity geeralizatios are ot straigt forward. Taylor series expasios of te estimator ca be used to obtai approximate formulae for te bias. Tese formulas te eed to be estimated, ofte by meas of oparametric teciques requirig te coice of furter uisace parameters suc as badwidt selectios. I te cotext of simultaeous equatios models Nagar (959) proposed a similar correctio based o iger order expasios of te two stage least squares estimator. A more geerally applicable, ad less parametric procedure is te Jackkife procedure ad more geeral resamplig algoritms. Te Jackkife bias estimator goes back to Queouille (949). Bootstrap bias estimatio was discussed by Parr (983), Sao (988a,b) ad Horowitz (998) i te cotext of oliear trasformatios of OLS estimators of liear models ad oliear fuctios of te mea. Bootstrap bias correctio is also metioed i Hall (992). We exted te literature o te bootstrap ad jackkife bias corrected estimator i two directios. First, we aalyze geuiely oliear estimators rater ta oliear trasformatios of liear estimators as i Sao. Tis requires a careful treatmet of te potetial oexistece of momets of te actual estimator. We address tis problem by itroducig trucatig parameters tat become less restrictive as te sample expads. Secodly, te literature o bootstrap bias corrected estimators as bee focused o aalyzig bias properties witout ivestigatig te effects of bias correctio o te iger order variace. We are istead workig wit tird rater ta secod order expasios of te bias corrected estimators. Tis allows us to aalyze te effect bias correctio as o te iger order variace of te estimator. We are carryig out our aalysis i a likeliood framework. Tis allows us to aalyze iger order efficiecy properties of bias corrected estimators. We sow tat te bias correctio, eve toug based o sample averages, does ot icrease te iger order variace of te estimator compared to te bias corrected estimators of Pfazagl ad Wefelmeyer (978) ad tus leads to a iger order efficiet procedure. 2

3 2 Higer Order Compariso of Bootstrap ad Jackkife Bias Corrected MLE I tis sectio, we first cosider a simple parametric model, ad derive iger order expasios of te imum likeliood estimator (MLE). We te derive iger order expasios of te bootstrap ad jackkife bias corrected MLE, ad argue tat tey are iger order equivalet. We argue tat suc bias corrected estimators sould ave te same iger order variace as te bias corrected MLE developed by Pfazagl ad Wefelmeyer (978), wic was sow to be tird order optimal. 2. Higer Order Expasio of MLE Cosider a stadard parametric model i f (z, 0 ),wicsatisfies sufficiet smootess coditios. For simplicity, it is assumed tat dim ( 0 )=. We cosider properties of te MLE b were b X i= log f ( i, ). We later impose coditios tat guaratee tat te MLE ca be uderstood as a solutio to X log f i, b X 0= ` i, b. i= i= Tis coditio is certaily satisfied for parametric models tat ave aalytical solutios for b,atleast wit probability tedig to oe if te true parameter is i te iterior of a compact parameter space. More geerally, tis coditio could also be satisfied, at least up to a error tat vaises at a sufficietly fast rate if te model does ot posses a aalytical solutio but te imum of te log likeliood lies witi te iterior of te parameter space wit probability tedig to oe. We are abstractig from tese tecical details to keep te expositio of our mai results as clear as possible. It is coveiet to uderstad b b, were (²) deotes te solutio to te estimatig equatio Here, 0= ` (, ) df ² (z). F ² F ² F ² bf F, ² 0, ad F ad F b deote te uderlyig cumulative distributio fuctio ad te empirical distributio fuctio F b (z) P i= { i z}. We obtai bootstrapped estimates b by samplig,..., idetically ad idepedetly from te empirical distributio bf. We deote te distributio of,..., by bf (z) = P i= { i z}. 3

4 Usig previous otatio it terefore follows tat b b is te solutio to were 0= ` (, ) d bf ² (z), bf ² bf ² b bf ² F b bf, ² 0,. Here, b is te bootstrap empirical process b F b bf. We are imposig te followig tecical coditios to guaratee te validity of our stocastic expasios. Coditio (i) Te fuctio log f (,, α) is cotiuous i Υ; (ii) Te parameter space Υ is a compact, separable metric space, 0 it (Υ); (iii) Tere exists a fuctio M (z) suc tat m log f (z, ) m M (z) 0 m 7 ad E M ( i ) Qi < for some Q > 6; (iv) Lettig G () E [log f ( i, )], we ave G ( 0 ) sup {: 0 >η} G () > 0 for eac η > 0. Coditio 2 Let φ (F, ) R ` (, (²)) df (z) be a fuctioal of F idexed by Υ, adletφ 0 (F, ) R `(,(²)) df (z). For all D [, ] wit kk ad lim z (z) = lim z (z) =0ad all ² [0, / ] fixed, tere exist b (²), b (²) Υ suc tat φ F ², b (²) =0ad φ F b ², b (²) =0 except for some bf wit probability zero. Also, assume φ 0 F ², b (²) 6= 0ad φ 0 bf ², b (²) 6= 0 except for some bf wit probability zero. Coditio 3 For eac Υ ad for m 7 let m log f (z, ) / m be a F -measurable fuctio of z. Coditio 4 Let F be te class of fuctios m log f (z,) / m idexed by Υ wit evelope M (z). Te evelope fuctio M (z) satisfies Pollard s etropy coditio v u µ /2 t log N ε M dq 2, F,L 2 (Q) dε <, () 0 sup Q P were P is te class of probability measures o R tat cocetrate o a fiite set ad N is te cover umber defied i va der Vaart ad Weller (996, p.90). Coditio is a stadard coditio guarateeig idetificatio of te model ad imposig sufficiet smootess coditios as well as existece of iger momets to allow for a iger order stocastic expasio of te estimator. Coditio 2 formalizes te fact tat te first order coditios are satisfied exactly at te imum for observatios tat are geerated from distributios tat are cotaied witi a eigborood of te true data desity. Tis latter requiremet is importat for bootstrap based 4

5 samples tat are ot derived directly from te origial data desity. Coditio 3 togeter wit separability of te parameter space guaratees measurability of suprema of our empirical processes. As is well kow from te probability literature, measurability coditios could be relaxed somewat at te expese of more refied covergece argumets. We are abstractig from suc refiemets for te purpose of tis paper. Te idea beid usig te bootstrap is rougly speakig tat uder certai regularity coditios it ca be sow tat b b as te same properties as b 0 ad ca terefore be used to obtai bias approximatios ad oter correctios to te small sample distributio. From Gie ad i (990, Teorem 2.4) ad Coditios,2,3 ad 4 it follows tat, almost surely, /2 ˆF ˆF T weakly i l (F). We use te result o te covergece of te empirical processes to obtai a expasio of te estimators b ad b. Let ` (, ) log f (, )/, ` (, ) 2 log f (, ) ± 2, ` (, ) 3 log f (, ) ± 3,etc. Defie I E `( i, 0 ), Q () E ` ( i, ) ad Q 2 () E ` ( i, ). It is coveiet to express te resultig expasio i terms of U ad V-statistics. We defie U i () ` ( i, ), V i () ` ( i, ) E ` ( i, ), W i ` ( i ) E ` ( i ) ad let U () /2 P i= U i (), V () /2 P i= V i (), ad W () /2 P i= W i (). i Propositio Assume tat dim () =. Uder Coditio, tere exists some ² 0, suc tat b satisfies te expasio b 0 = ² (0) ²² (0) 2 6 ²²² (0) (2) 24 ²²²² (0) 20 2 ²²²²² (0) ²²²²²² (e²) (3) were ² (0) = I U ( 0 ), (4) ²² (0) = I 3 Q ( 0 ) U ( 0 ) 2 2I 2 U ( 0 ) V ( 0 ) (5) ad ²²² (0) = I 4 Q 2 ( 0 ) U ( 0 ) 3 3I 5 Q ( 0 ) 2 U ( 0 ) 3 9I 4 Q () U ( 0 ) 2 V ( 0 ) (6) 3I 3 U ( 0 ) 2 W ( 0 )6I 3 U ( 0 ) V ( 0 ) 2. Moreover, ² (0) = O p (), ²² (0) = O p (), ²²² (0) = O p () ad ² 0, i ²²²² (²) =O p (). Proof. See Appedix A.4. Usig expasio (2), we defie te approximate estimator e 0 /2 ² (0) 2 ²² (0) 6 3/2 ²²² (0). We ca ow defie te expected value of tis approximate estimate as i E e = b () c () 3/2 5

6 were b () 2 E [²² ]= 2I 2 E ` I 2 E ``. i i i If E b exists, te E e will be approximately equal to E b at least up to order / except possibly for some patological cases. Terefore, we ca uderstad b() as te iger order bias of b. Likewise, we ave " µ 2 E e b () 0 I υ were υ 4 Var (²² ) 3 E [²²² ² ] Terefore, we ca uderstad I υ as te iger order variace of b. Fially, we ca defie te iger order MSE as E e 0 2 I υ b ()2. I order to approximate te bias of te bootstrapped estimate b we eed a similar iger order expasio as i te case of te ML estimator. Here, owever, te referece poit aroud wic we develop our approximatio is te empirical distributio bf rater ta te origial distributio F. Te covergece of bf to F te guaratees tat bootstrapped statistics are close to te origial statistics. We replace I, Q ad Q 2 wit bi= P `( i= i, b ), bq = P i= ` ( i, b ) ad bq 2 = P i= ` ( i, b ) ad defie bootstrapped U ad V-statistics as Ui () ` ( i, ), V i () ` (i, ) P i= ` ( i, ), Wi ` (i ) P i= ` ( i, ) ad let U () = /2 P i= U i (), V () = /2 P W () = /2 P i= W i () we obtai for te followig result for te bootstraped estimate b. i= V i () ad Propositio 2 Assume tat dim () =. Uder Coditios,2,3 ad 4 ² 0, /2 suc tat b satisfies te expasio b b = b ² (0) ²² b (0) 2 6 b ²²² (0) 24 b ²²²² (0) 20 b 2 ²²²²² ( ²) a.s. were b ² (0) = I b U b, b ²² (0) = I b 3 Q b b U b 2 2I b 2 U b V b etc. Moreover, b ² (0) = O p (), b ²² (0) = O p (), b ²²² (0) = O p (), b ²²²² (0) = O p () ad ² [0, /2 ] b ²²²²² (²) =O p () were all orders of probability old P N a.s., were P N is defiedipropositio6iteappedix. 2.2 Bootstrap Bias Correctio Bootstrap Bias estimatio ad Bias correctio was aalyzed i te cotext of liear models by Sao (988a,b). Let E be te expectatio operator wit respect to bf. Te idea beid te Bootstrap bias 6

7 i correctio is to estimate E b 0, if it exists, by E b i b. Sice i our case we ca ot guaratee i existece of E b we sow istead tat E b i is close to b (). Tis i tur will allow us to costruct te bias corrected estimate 2 b E b i. Sice is a potetially igly oliear fuctio of te uderlyig desity we itroduce a trucated versio of te bootstrap bias estimator. A similar form of trucatio was suggested by Sao (988b). However, Sao required tat te trucatio poit was bouded i probability as te sample size grows. We allow for growig trucatio poits i order to guaratee te asymptotically iger order ubiasedess of our procedure. We itroduce te statistic C (), werec () =O p () ad {,..., }. Te trucated bootstrap bias estimate is te defied as b E b b i were (x) = α C () x α C () if x< α C () if x < α C () if x> α C () (7) Possible coices for C () are estimates suc as te stadard deviatio of b. We first establis tat E b b i estimates te iger order bias b () cosistetly. Propositio 3 Assume tat dim () =.AlsoassumeCoditios,2,3ad4old.Let be defied i 7witα 0, 4 30.Te b = b ( 0) o p. Proof. See Appedix A.4. Note tat te restrictios imposed o α deped o te smootess of te uderlyig model ad could be relaxed at te cost of additioal restrictios o te desities f(z,). Wile tis result establises tat we ca cosistetly estimate te iger order bias it is ot sufficiet to guaratee good iger order properties of te bias corrected estimator. For tis reaso we establis te ext result. Propositio 4 Assume tat dim () =.Let be defied as i 7. Te b E b i 0 were B is defied i (44) i te Appedix. = I U( 0) µ 2 ²² (0) b( 0 ) 6 ²²² (0) 2 B o p µ, 7

8 Proof. See Appedix A.4. Because E ²² (0) 2 b( 0 ) =0, we ca see tat te bootstrap successfully removes bias. It terefore follows tat E e E b 0 2 Var e 0 2 E [B² ]. 2.3 Jackkife Bias Correctio A alterative to te bootstrap is a jackkife bias corrected estimator. We develop te iger order teory for suc a estimator i tis sectio ad compare its iger order properties to te properties of te bootstrap bias corrected procedure. Let b (i) deote te MLE based o delete-i sample. Te jackkife bias corrected estimator is give by J = b X i= b (i) Te followig propositio establises te iger order properties of te Jackkife bias corrected ML estimator. Propositio 5 Assume tat dim () =. Also assume Coditio olds. Te te jackkife bias corrected ML estimator as a iger order expasio as i (J 0 ) = ² µ 2 ²² (0) b( 0 ) 6 ²²² µ 2 J o p were J is defied i (52) i te Appedix. Proof. See Appedix A.4. It follows at oce tat J = B, (8) wic meas tat te Jackkife ad Bootstrap bias corrected versios of te ML estimator are iger order equivalet. Tey do ot oly ave te same iger order variace but agree more geerally i terms of teir iger order distributio at least as far as te stocastic approximatio allows to make suc comparisos. 8

9 3 Higer Order Efficiecy I tis sectio we obtai te iger order asymptotic properties of te bias corrected estimator of Pfazagl ad Wefelmeyer (978). Sice tat estimator was sow to be iger order efficiet we will coclude tat our bias corrected estimator is iger order efficiet if it as te same iger order variace as te Pfazagl ad Wefelmeyer estimator. I geeral, we dim (), we still ave a expasio similar to Propositio 2: b 0 = ² (0) ²² (0) µ 2 6 ²²² (0) O p, Te asymptotic bias of MLE is te equal to b ( 0 ) ²² (0) = E 2 Suppose tat tere is some vector of fuctios m (z, ) ad aoter vector of fuctios τ (m) wit µ b () τ m (z, ) f (z,) dz. Tis leads to a bias corrected estimator b c b b b Implemetatio of te Pfazagl ad Wefelmeyer procedure is potetially complicated if te itegral R m (z, ) f (z,) dz is difficult to compute. A alterative is terefore to replace te itegral by sample averages. For b () τ X i m ( i, ), a alterative bias correctio is te b b b a b We ca sow tat b a ad b c ave te same mea squared error up to order O by aalyzig teir iger order variace. Let m () E [m ( i, )] = m (z, ) f (z, 0 ) dz (9) wit j-t elemet m j () ad write m = m ( 0 ), τ m τ (m) / m 0, M E ad Λ E m ( i, 0 ) ` ( i, 0 ) 0. i m(i, 0) 0 = R m(z, 0) f (z, 0 0 ) dz, We dim () =, we ave see tat b () = 2I 2 E ` I 2 E ``. Terefore, we ave b () τ R m (z, ) f (z,) dz, were τ (t,t 2,t 3 ) 2t 2 t 2 t 2 t 3, t R ` (z,) 2 f (z,) dz, t 2 R ` (z,) f (z,) dz, t 3 R ` (z, ) ` (z,) f (z, ) dz, ad m (z, ) ` (z,) 2,` (z,),`(z,) ` (z,) 0. 9

10 Teorem Assume Coditios,3 ad 4 old. Let be defied i 7 wit α 0, 4 30.Te, bc 0 = ² (0) µ 2 ²² (0) b ( 0 ) µ 6 ²²² (0) C o p, ba 0 = ² (0) µ 2 ²² (0) b ( 0 ) µ 6 ²²² (0) A o p were A = τ m M I U ( 0 ) /2 P i (m (z i, 0 ) m), C = τ m (M Λ) I U ( 0 ),ad E C ² (0) 0 = E A ² (0) 0 = τ m (M Λ) I. Proof. See Appedix A.4. Tis result implies tat te four bias corrected estimators all ave te same iger order variace term. Assume for simplicity tat dim () =. Because bc 0 = ² (0) µ 2 ²² (0) b ( 0 ) µ 6 ²²² (0) C o p, we ca defie te approximate MSE of bc 0 tobetemeasquareofterhsigorigte o p. It is easy to see tat te approximate MSE of bc 0 is equal to " µ E ( ² (0)) 2i 2 µ E 2 ²² (0) b ( 0 ) E =Var( ² (0)) µ Var 2 ²² (0) ² (0) Likewise, we ca see tat te approximate MSE of ba 0 is equal to 6 ²²² (0) C µ Cov 6 ²²² (0), ² (0) Cov (C, ² (0)) Var ( ² (0)) µ Var 2 ²² (0) µ Cov 6 ²²² (0), ² (0) Cov (A, ² (0)) Teorem idicates tat Cov (C, ² (0)) = Cov (A, ² (0)), ad terefore, te approximate MSEs are idetical. I order to uderstad te ituitio beid tis result, it is useful to uderstad b b as a efficiet estimator of b ( 0 ). Tis is because b is a efficiet estimator of 0. Assume tat b b b ( 0 ) is asymptotically a osigular liear combiatio of b 0, i.e., b b b ( 0 ) = Υ /2 P i= ψ ( i, 0 ) o p () for some osigular Υ. Here, ψ ( i, 0 ) I ` ( i, 0 ) deotes te efficiet ifluece fuctio. Suppose tat b is ay oter regular 2 estimator of b ( 0 ) suc tat (b b ( 0 )) = /2 P i= % ( i, 0 ) o p () for some % ( i, 0 ) suc tat E [% ( i, 0 )]=0. Asymptotic covariace betwee b 0 ad b b b is te equal to Cov (% ( i, 0 ) Υψ ( i, 0 ), ψ ( i, 0 )) = Υ Cov Υ % ( i, 0 ) ψ ( i, 0 ), ψ ( i, 0 ). 2 See, e.g., Bickle, Klaasse, Ritov, ad Weller (993, p. 2). 0

11 Because Cov Υ % ( i, 0 ) ψ ( i, 0 ), ψ ( i, 0 ) = 0 by te ituitio uderlyig Hausma-test, we obtai te coclusio tat te asymptotic covariace is zero. I oter words, we ave Cov (% ( i, 0 ), ψ ( i, 0 )) = Cov (Υψ ( i, 0 ), ψ ( i, 0 )) (0) Now, ote tat we ave te expasios b b b 0 = ² (0) µ 2 ²² (0) b ( 0 ) 6 ²²² (0) Υ /2 X ψ ( i, 0 ) i= µ O p ad µ b b 0 = ² (0) µ 2 ²² (0) b ( 0 ) 6 ²²² (0) /2 X % ( i, 0 ) i= µ O p Because ² (0) = /2 P i= ψ ( i, 0 ), equatio (0) implies tat covariaces of te adjustmet terms of order wit ² (0) are equal to eac oter. It terefore follows tat te approximate MSE of te bias corrected MLE is ivariat to te metod of bias correctio. Teorem turs out to be just a special case. Give te precedig discussio, it is peraps ot surprisig tat te Bootstrap ad Jackkife bias corrected Maximum Likeliood estimators ave te same approximate MSE as b c : Teorem 2 Assume tat dim () =. Assume Coditios,3 ad 4 old. Let be defied i 7 wit α 0, 4 30.Te, 2 E [B² (0)] = 2 E [J² (0)] = τ m (M Λ) I. Proof. See Appedix A.4. Remark Teorems ad 2 are irrelevat we relevat loss fuctio is ot approximate MSE. O te oter ad, equatio (8) idicates tat te iger order equivalece of Bootstrap ad Jackkife goes beyod te MSE compariso. 4 Coclusios We ave sow tat oparametric bootstrap ad jackkife procedures ca be used to remove bias terms of stocastic order from a ML estimator. Te bootstrap bias corrected ML estimator acieves te same iger order variace as te efficiet estimators of Pfazagl ad Wefelmeyer (978). Tis idicates tat tere is o eed to derive te aalytic bias formula. Estimated versios of te bias correctios do ot icrease te iger order variace ad are terefore iger order efficiet i a mea squared sese.

12 A Proofs A. Some Prelimiary Lemmas Lemma Assume tat W i are iid wit E [W i ]=0ad E Wi 2k <. Te, E ( P i= W i) 2ki = C(k) k o( k ) for some costat C(k). Proof. By adoptig a argumet i te proof of Lemma 5. i Lairi (992), we ave E ( P i= W i) 2ki P = 2k P C(α,..., α j ) P jq E W α s i s, () α I s= were for eac fixed j {,..., 2k}, P α exteds over all j-tuples of positive itegers (α,..., α j ) suc tat α... α j =2k ad P I exteds over all ordered j-tuples (i,..., i j ) of itegers suc tat i j. Also, C(α,..., α j ) stads for a bouded costat. Note, tat if j>kte at least oe of te idices α j =. By idepedece ad te fact tat EW i =0it follows tat E Q j s= W α s i s =0weever j>k. Tis sows tat E ( P i= W i) 2k = C(k) k o( k ) for some costat C(k). Lemma 2 Suppose tat {ξ i,i=, 2,...} is a sequece of zero mea i.i.d. radom variables. We also assume tat E ξ i 6i <. We te ave Pr " X ξ > η = O 8 i i= for every η > 0. Proof. Usig Lemma, we obtai E X ξ i6 C 8 o( 8 ), i= were C>0is a costat. Terefore, we ave 8 X Pr " ξ > η O( i 8 C8 6 )=O(). η6 i= Lemma 3 Suppose tat, for eac i, {ξ i (φ),i=, 2,...} is a sequece of zero mea i.i.d. radom variables idexed by some parameter φ Φ. We also assume tat sup φ Φ ξ i (φ) B i for some sequece of radom variables B i tat is i.i.d. Fially, we assume tat E B i 6i <. Weteave X Pr " ξ i (φ ) > 2 υ = o 6v i= 2

13 for every υ suc tat υ < 6.Forυ < 48 we ave X Pr " ξ i (φ ) > 2 υ = o( ). i= Here, φ is a arbitrary sequece i Φ. Proof. By Markov s iequality, we ave " " X Pr sup ξ φ Φ i (φ) > 2 υ X = Pr sup ξ i= φ Φ i (φ ) > 2 7 υ i= E sup φ Φ ( P i= ξ i (φ)) 6i sup φ Φ E = υ η 6 were te last equality is based o domiated covergece. By Lemma, we ave 6 X E ξ i (φ) C 8, i= were C>0is a costat. Terefore, we ave " X Pr sup ξ i (φ) > 2 υ C 8 4/36υ 28/36υ η = O. 6 φ Φ i= ( P i= ξ i (φ)) 6i υ η 6, Lemma 4 Let bg () P i= log f ( i, ). Suppose tat Coditio old. We te ave for all η > 0 tat Pr sup bg () G () η = o 23 3 Proof. Note tat Pr sup bg () G () η =Pr sup were υ = 5 2. Te te result follows by Lemma 3. Lemma 5 Uder Coditio, we ave " Pr (²) 0 η = o ² for every η > 0. bg () G () η 2 υ Proof. Let η be give, ad let ε G ( 0 ) sup {: 0 >η} G () > 0. Lettig g (z, ) log f (z, ), we ave g (z, ) df ² (z) = ² G ()² bg () 3

14 ad g (z, ) df ² (z) G () ² bg () G () bg () G (). Here, te last iequality is based o te fact tat 0 ². By Lemma 4, we ave " Pr 0 ² sup g (z,) df ² (z) G () η = o 23 3 Terefore, for every 0 ² wit probability equal to o 23 3,weave 0 >η 0 >η g (z, ) df ² (z) 0 >η G () 3 ε < G( 0 ) 2 3 ε < g (z, 0 ) df ² (z) 3 ε. We also ave g (z,) df ² (z) g (z, 0 ) df ² (z) by defiitio. It follows tat g (z, ) df ² (z) < g (z, ) df ² (z) 3 ε for every 0 ². We terefore obtai tat Pr i 0 ² (²) 0 η = o Lemma 6 Assume tat Coditio olds. Suppose tat K (z; (²)) is equal to m log f (z; (²)) m for some m 6. Te, for ay η > 0, weave " Pr K (z; (²)) df ² (z) E [K ( i ; 0 )] > η 0 ² = o Also, " Pr 0 ² K ( ; (²)) d >C 2 υ = o 6υ for some costat C>0 ad for every υ suc tat υ < 6.Ifυ < 48 te te above order is o. Proof. Note tat we may write K (z; (²)) df ² (z) E [K ( i ; 0 )] = K (z; (²)) df ² (z) K (z; 0 ) df (z) = K (z; (²)) df ² (z) K (z; 0 ) df ² (z) = K (z; ) ( (²) 0 ) df ² (z)² K (z; 0 ) df ² (z) K (z; 0 ) d F b F (z) K (z; (²)) df (z) 4

15 were is betwee 0 ad (²). Terefore,weave K (z; (²)) df ² (z) E [K ( i ; 0 )] (²) 0 E [M ( i )] X M ( i ) i= X (M ( i ) E [M ( i )]) were M ( ) is defied i Coditio. Usig Lemma 5, we ca boud K (z; (²)) df ² (z) E [K ( i ; 0 )] 0 ² T i absolute value by some η > 0 wit probability o Usig Coditio ad Lemmas 3, we ca also sow tat R K ( ; (²)) d ca be bouded by C 2 υ for some costat C>0ad υ suc tat υ < 6 wit probability o 6υ. Similarly, if υ < 48, te te statemet olds wit probability o( ). i= 23 3 Lemma 7 Suppose tat Coditio olds. Te, we ave " Pr ² (²) >C2 υ = o 6υ 0 ² Pr " Pr " 0 ² ²² (²) >C 0 ² ²²²²²² (²) >C υ υ 6 = o 6υ. = o 6υ for some costat C>0 ad for every υ suc tat υ < 6.Ifυ < 48 te te above orders are o. Proof. From (30), we ave ² (²) = ` (z, ²) df ² (z) ` (,²) d. Usig Lemma 6, we ca boud te deomiator by some C>0, ad te umerator by some C 2 υ wit probability o 6υ,fromwictefirst coclusio follows. As for te secod coclusio, we ote from (3) tat we ave 0=E ` ² ( i,²) ( ² (²)) 2 ` E ² ( i,²) µ ²² (²)2 ` (z, ²) d (z) ² (²) Te secod coclusio follows by usig Lemmas 6 alog wit te first coclusio. Te rest of te Lemmas ca be establised similarly. Note tat if υ < 48 te we ca apply te specialized result of Lemma 6 i tesamewayasbefore. 5

16 Lemma 8 Suppose tat Coditio olds. Let m j ().be as defied i 9. Te I b I = V ( 0 ) Q ( 0 ) I U ( 0 )o p (), Q b b Q ( 0 ) = W ( 0 )Q 2 ( 0 ) I U ( 0 )o p (), m b m ( 0 ) =2E [U i ( 0 ) V i ( 0 )] I U ( 0 )o p (), m 3 b m 3 ( 0 ) = E V i ( 0 ) 2i E ` ( i, 0 ) 2 E [Ui ( 0 ) W i ( 0 )] I U ( 0 ) Proof. Let m 0 () R ` (z,) f (z, 0 ) dz. Note tat bi I = P ` i, b E ` ( i, 0 ) i= = P i= ` ( i, 0 ) m 0 ( 0 ) o p /2 m 0 b m 0 ( 0 ), were te last equality is based o te usual stocastic equicotiuity. Also ote tat m 0 ()/ = R ` (z, ) f (z, 0 ) dz by domiated covergece. We terefore obtai b I I = /2 P i= ` ( i, 0 ) E ` ( i, 0 ) E ` ( i, 0 ) b 0 o p () = V ( 0 ) Q ( 0 ) I U ( 0 )o p (), Likewise, we obtai Q b b Q ( 0 ) m b m ( 0 ) = /2 P i= ` ( i, 0 ) E ` ( i, 0 ) E ` ( i, 0 ) b 0 o p () = W ( 0 )Q 2 ( 0 ) I U ( 0 )o p (), = 2E ` ( i, 0 ) ` ( i, 0 ) b 0 o p () = 2E [U i ( 0 ) V i ( 0 )] I U ( 0 )o p (), m 3 b m 3 ( 0 ) = = ` E ( i, 0 ) 2i E ` ( i, 0 ) ` ( i, 0 ) b 0 o p () E V i ( 0 ) 2i E ` ( i, 0 ) 2 E [Ui ( 0 ) W i ( 0 )] I U ( 0 ) A.2 Lemmas for Bootstrapped Statistics Propositio 6 Assume tat Coditios,2,3 ad 4 old. Let F be te class of measurable fuctios defied i Coditio 4. Let deote weak covergece. Let (Ω, F, P) be a probability space suc tat i : Ω N, F N,P N (Ω, F,P) are coordiate projectios. Te, for f F, F b F f Tf were 6

17 T is a tigt Browia bridge wit variace covariace fuctio F (t s) F (s) F (t). LetBL be te set of all fuctio : l (F) 7 [0, ] suc tat (z ) (z 2 ) kz z 2 k F for every z ad z 2 were l (F) is te set of uiformly bouded real fuctios o F ad k.k F is te uiform orm for maps from F to R. i Te sup BL E F b bf f E[Tf] 0, P N -a.s. Proof. We first sow tat for f F, F b F f Tf or i oter words tat F is a Dosker class. Defie F δ = f g : f,g F,E kf gk 2i o < δ, F = {f g : f,g F} ad F 2 = ª f 2 : f F. I ligt of va der Vaart ad Weller (996, Teorem 2.5.2), it is eoug to sow tat F δ ad F 2 are F measurable classes for every δ > 0 ad E M (z) 2i <. Te secod requiremet is satisfied by Coditio. Sice F δ F te first coditio olds if for f F 2 ad ay vector a R ad ay te fuctio s(,..., )=sup, 2 Υ P i a `(k) i ( i, ) `(k) ( i, 2 ) 2 is measurable. Let Υk be a icreasig sequece of coutable subsets of Υ wose limit is dese i Υ. Te X `(k) s k (,..., )= sup a i ( i, ) `(k) 2 ( i, 2 ), 2 Υ k i is measurable by Coditio 3. By cotiuity of `(k) ( i, ) i it follows tat lim if k s k (,..., )= s(,..., ) suc tat measurability of s follows from Royde (988, Teorem 20, p.68). Coditioal weak covergece of b follows from Gie ad i (990, Teorem 2.4). Note tat by measurability of i ˆF F f ad Gie ad i (990, p86) te covergece of sup BL E F b bf f E[Tf] is a.s. Lemma 9 Assume tat Coditio is satisfied. Suppose tat, for eac i, ξ i (φ) =τ ( i, φ) P i= τ ( i, φ), i = {, 2,...} is a sequece of bootstrapped trasformatios of radom variables idexed by some parameter φ Φ wit E [ξ i (φ)]=0. We also assume tat sup φ Φ τ ( i, φ) B i for some sequece of radom variables B i tat is i.i.d. Fially, we assume tat E B i 6i <. Weteave P " X ξ i (φ ) > 2 υ = o p 6υ i= for every υ suc tat υ < 6.Moreover, P " X ξ i (φ ) > 2 υ = o p 23 3 i=. Here, φ is a arbitrary sequece i Φ ad P is te coditioal probability measure of i give i. Proof. Note tat P i= ξ i (φ) = P i= (N i ) τ( i, φ) were N,..., N is multiomially distributed wit parameters (, /,..., /) =(k, p,..., p ) ad idepedet of i suc tat Pr ( i= {N i = i })= / ( Q i i) Q i i were P i i =, i 0. Let κ r r 2...r be te mixed iger order cumulat of N,..., N of order r = r... r for r i 0, r i iteger. Mixed iger order cumulats ca be obtaied 7

18 from Guldberg s (935) recurrece relatio κ r r 2..r i...r = a i (κ r r 2..r i...r ) / a i were a i = p i /p.let b be te umber of o zero idices r i. Te argumets i Wisart (949) imply tat for p i = we ave κ r r 2...r c b for some costat c. For otatioal coveiece we will represet cumulats wit zero idices as lower order cumulats of te variables wit o-zero idices, i.e. write κ...r.. = κ rr 2...r were r j =0. Cosider P "sup X ξ φ Φ i (φ) > 2 υ = P "sup X ξ i (φ i= φ Φ ) > 7 2 υ i= E sup φ Φ ( P i= ξ i (φ)) 6i υ η 6 sup φ Φ E ( P i= ξ i (φ)) 6i =, υ η 6 were te last equality uses te fact tat sup φ Φ does ot ivolve N,..., N. By adoptig a argumet i te proof of Lemma 5. i Lairi (992), we ave E ( P i= ξ i (φ))2k = 2k P α P C(α,..., α j ) P I jq τ( it, φ) αt E t= j Q s= (N is ) α s, (2) were for eac fixed j {,...,2k}, P α exteds over all j-tuples of positive itegers (α,..., α j ) suc tat α... α j =2k ad P I exteds over all ordered j-tuples (i,..., i j ) of itegers suc tat i j. Also, C(α,..., α j ) stads for a bouded costat. Next we cosider te mixed cetral momets µ(α,..., α j )=E Q j s= (N i s ) αs. From Siryaev (989, Teorem 6, p.290) we obtai a relatiosip betwee cumulats ad mixed momets. Let α =(α,..., α j ) 0,r (p) = r (p),..., r(p) j, r (p) = r (p)...r(p) j ad r (p) =r (p)...r(p) suc tat j µ(α,...,α j )= X r ()...r (q) =α α q r ()...r (q) qy p= κ r (p) r(p) 2...r(p) j were P r ()...r (q) =α idicates te sum over all ordered sets of oegative itegral vectors r(p), r (p) > 0,wose sum is α. Sice te order of 2 depeds bot o te umber of ozero terms i P I ad te size of µ(α,...,α j ) for eac j, we aalyze te term S(, j) = X jy jy τ( it, φ) αs E (N is ) α s I t= s= for eac j. Note tat Qj t= τ( i t, φ) αs is bouded almost surely ad terefore does ot affect te aalysis. Also, P I is a sum over j terms ad tus is O( j ) if all tese terms are ozero. Te crucial factor i determiig te overall order is terefore E Q j s= (N i s ) α s. We start wit j =. Te α =2k, q =...2k ad r (p) are scalars. Cosequetly, κ (p) r = c were c is some costat ad S(, ) P c 2 i= τ( i t, φ) 2k for some oter costat c 2. If j k te for q =...2q, r (p) are vectors wit possibly 8

19 oly oe elemet differet from zero. Agai, S(, j) c 2 PI Q j t= τ( i t, φ) α s for j k. If j k te α cotais at least 2(j k) elemets α i =. Now assume tat for some p, r (p) i i 6= j. Te κ r (p) i ad r (p) j = E (N is ) = 0 ad tus Q q p= κ r (p) r(p) 2...r(p) j 6= 0for at least oe j 6= i te κ (p) r r(p) 2...r(p) =ad r (p) j =0for =0. O te oter ad if r (p) i = c. Sice tere must exists p 0 correspodig to te oter α i 0 =suc tat eiter r (p0 ) =ad ) i 0 r(p0 j =0for i 0 6= j or r (p0 ) i =ad ) 0 r(p0 j 6= 0 for at least oe j 6= i 0, it follows tat Q q p= κ = c r (p) 3 2(jk), at most. It ow follows tat r(p) 2...r(p) j S(, j) c 2 2(jk) P Q j I t= τ( i t, φ) αs for all j>k.te, X jy E S(, j) c 2 E τ( it, φ) αs c 2 j E τ( it, φ) 2k for j k ad I t= E S(, j) c 2 2(jk) X I jy E τ( it, φ) α s c 2 2kj E τ( it, φ) 2k c 2 k E τ( it, φ) 2k t= for j>k.togeter tese results imply tat E E ( P i= ξ i (φ)) 2k C(k) k E τ( it, φ) 2k were C(k) is a costat tat depeds o k. By te Markov iequality it follows tat E ( P i= ξ i (φ)) 2k = O p ( k ). We coclude tat P " X ξ i (φ ) > 2 υ O p( 8 ) i= υ η = O 6 p 6υ 4 3. Te secod result follows immediately from P " X ξ i (φ ) > η = P " X ξ i (φ ) > η/2 o p 23 3 i= by te previous result. i= Lemma 0 Uder Coditio, we ave " P b (²) b η = o p ². Proof. For ay η > 0, tere exists some δ > 0 suc tat 0 > η/2 implies G () G ( 0 ) > δ. Let G b () R g(z, )df b (z) ad G b ² () R g (z, ) df b ² (z). Te, " " P b (²) b η P G( b (²)) G b > δ. 0 ² 0 ² Because G b (²) G b = G b (²) b G ² bg b (²) b (²) bg ² G b b (²) b b G ² b G bg b ² (²) b 9

20 ad bg ² () bg () bg () bg (), we obtai 0 ² G b (²) G b sup bg () bg () sup bg () G () Υ Υ bg b 0 ² ² (²) bg ² b bg 0 ² ² sup bg () bg () sup bg () G () Υ Υ bg b 0 ² ² (²) bg ² b bg 0 ² ² sup bg () bg () sup bg () G () Υ Υ bg b 0 ² ² (²) bg ² 2sup bg () bg () 2sup bg () G () Υ Υ b G b b bg b bg b bg b bg b bg b G b bg ² 0 ² b G b b (²) bg ² b (3) By Lemma 9, we ave P sup bg () bg () > δ = o p 23 3 Υ 6 Coditioal o data, sup Υ bg () G () > δ is a o-stocastic evet. Terefore, we ca write P sup bg () G () > δ Υ ½ = sup bg ¾ () G () > δ, Υ (4) were { } deotes a idicator fuctio. For every σ > 0, weave Pr P sup bg () G () > δ ½ > σ 23 3 = Pr sup Υ 6 bg () G () > δ ¾ > 0 (5) Υ 6 = Pr sup bg () G () > δ = o() Υ 6 were te last equality is implied by Lemma 4. It terefore follows tat P sup bg () G () > δ = o p (6) 6 Υ 20

21 Fially, bg b 0 ² ² (²) bg ² b bg b 0 ² ² (²) b bg b bg b bg 0 ² ² sup bg ² () sup bg () bg 0 ² ² sup bg ² () sup bg () bg b bg b sup bg ² () bg () sup bg () bg () sup bg () bg () sup bg () bg () bg () bg () sup bg () bg () bg () bg () = 0 ² 0 ² 0 ² 0 ² sup = 2sup Here, te first equality is based o te defiitios of b (²) ad b. Because P sup bg () bg () > δ = o p 23 3 Υ we ca coclude tat P " 0 ² bg ² b (²) bg ² Te coclusio follows by combiig (3) - (7). b bg b δ b > = o p (7) 3 Lemma Assume tat Coditio is satisfied. Let K ( ; (²)) be defied as i Lemma 6. Te, for ay η > 0, weave " P K (z; (²)) d bf ² (z) E [K (z; 0 )] > η = o p ² Also, P " 0 ² K ; b (²) d b >C 2 υ = o p 6υ 23 3, for some costat C>0 ad for every υ suc tat υ < 6. Proof. ItesamewayasiteproofofLemma6 K z; b (²) df b ² (z) K (z; 0 ) df b (z) K (z; ) = b (²) 0 d bf ² (z)² K (z; 0 ) d F b bf (z) 2

22 were is betwee 0 ad b (²). Terefore, we ave K (z; (²)) df ² (z) K (z; 0 ) d bf (z) b X (²) 0 M ( i ) i= X M (i ) X M ( i ) i= i= X M (i ) were M ( ) is defied i Coditio. Let M = P i= M ( i) ad M = P i= M ( i ). Te, for ay η ad some c b i P (²) 0 M b i >η P (²) 0 > η/c P M E [M ( i )] >c = o p 23 3 sice P M E [M ( i )] >c =wit probability equal to P M E [M ( i )] >c = o 23 3 by Lemma2adzerooterwiseforsomec. Te, P M E [M ( i )] >c = o p 23 3 by te same argumet as i 5 Moreover, b i P (²) 0 M M b i > η P (²) 0 > η/c P M M >c = o p 23 3 i= by Lemmas 9 ad 0. It tus follows tat for ay η > 0, P µ K (z; (²)) df ² (z) K (z; 0 ) d bf (z) > η = o p R Fially ote tat P K (z; 0 ) d bf (z) EK (z; 0 ) > η =wit probability P µ K (z; 0 ) d bf (z) E [K (z; 0 )] > η = o( 23 3 ) R by Lemma 2. Tus, by te same argumet as i 5 P K (z; 0 ) d bf (z) EK (z; 0 ) > η = o p For te secod result fix δ > 0 arbitrary. Te " P K ; b (²) db >C 2 υ P sup K ( ; ) d b >C 2 υ 0 ² b <δ " P b (²) b δ 0 ² were P sup b <δ K ( ; ) d b >C 2 υ = o p 6υ follows directly from Lemma 9 ad " P b (²) b δ = o p ² follows from Lemma 0. 22

23 Lemma 2 Suppose tat Coditio olds. Te, we ave " P ² b (²) >C2 υ = o p 23 3, 6υ 0 ² " P ²² b (²) >C υ 2 2 = o p 23 3, 6υ 0 ² " P ²²²²²² b (²) >C υ ². = o p 23 3, 6υ for some costat C>0 ad for every υ suc tat υ < 6. Proof. Let M ² = R ` (z, ²) d bf ² (z) suc tat ² b (²) = M ² ` (,²) d b ad for ay δ > 0 some C>0ad for every υ suc tat υ < b 6 P ² (²) >C υi 2 P sup υ ` (,²) db > δc 2 ² = o p 23 3, 6υ P sup M ² E ` (z, 0 ) δ ² by Lemma. Te rest of te Lemma ca be establised similarly. Lemma 3 Let Xi = τ i, b be some trasformatio of i,wereτ possibly depeds o te sample { i } i= troug b.te E P P were X i = τ Xi i= i, b. Proof. Note tat E [X i ]= P = X j wic i tur implies µ P E Xi i= = X i i= P i= P X j = P X j. Lemma 4 Let Xk,i = τ k i, b for k =, 2 be some trasformatio of i,wereτ k possibly depeds o te sample { i } i= troug b.te µ µ E P P P = i= were X k,i = τ k ( i, b ). X,i i= X,i X 2,i µ X2,i i= µ P X,i i= i= P X 2,i 23

24 Proof. Tis is because E X,iX 2,i = P X,j X 2,j, wic i tur implies tat µ E P = i= = i= = = X,i i= µ P X2,i i= P E X,iX 2,i P E X,i E X 2,i 0 i6=i 0 P P P X,j X 2,j P P X,j X 2,j i6=i 0 P ( ) P P X,j X 2,j X,j X 2,j P X,j X 2,j P P X,j X 2,j. Te coclusio follows easily from tis result. Lemma 5 Let Xk,i = τ k i, b for k =, 2 be some trasformatio of i,wereτ k possibly depeds o te sample { i } i= troug b. Te µ µ µ E P P P = X,i i= X2,i i= X3,i i= P X,j X 2,j X 3,j P P X,j X 2,j X 3,j P P X 3,j X,j X 2,j P X 2,j X 3,j P P P X,j X 2,j X 3,j. P X,j Proof. Note tat µ P µ P µ P = P X,i i= Terefore, we ave P i= X,iX 2,iX 3,i i= X2,i i= X3,i i= P X,iX 2,iX 3,i P X,iX 0 2,i 0X 3,i P i6=i 0 i6=i 0 P X,i X 2,i 0X 3,i 00 i6=i 0 6=i 00 6=i E X,iX 2,iX 3,i = P i= i6=i 0 X,i 0X 2,iX 3,i P X,j X 2,j X 3,j P = X,j X 2,j X 3,j 24

25 ad P E X,iX 2,iX 3,i 0 i6=i 0 P E X,iX 2,i 0X 3,i i6=i 0 P E X,i 0X 2,iX 3,i i6=i 0 P i6=i 0 6=i 00 6=i E X,iX 2,i 0X 3,i 00 = P E X,iX 2,i E X 3,i 0 i6=i 0 = P P P X,j X 2,j X 3,j i6=i 0 = P P X,j X 2,j X 3,j = = = = P P X 3,j X,j X 2,j P P X 2,j X 3,j X,j P i6=i 0 6=i 00 6=i P i6=i 0 6=i 00 6=i E X,i E X2,i E X 0 3,i 00 = P X,j P X,j P P X 2,j P X 2,j from wic we obtai µ P µ P µ P E X,i X2,i X3,i i= i= i= P = X,j X 2,j X 3,j P P X,j X 2,j X 3,j P P X 3,j X,j X 2,j P P X 2,j X 3,j X,j P P P X,j X 2,j X 3,j. P X 3,j X 3,j Lemma 6 Let Ui () ` (i, ), Vi () ` (i, ) ` (, ) ` (i, ) P i= ` ( i, ), Wi (i ) ` (, ) ` (i ) P i= ` ( i, ) ad let U () = /2 P i= U i (), V () = /2 P i= V i () = /2 P i= W i (). Te(a) E U b i = 0, E V b i = 0, E W b i = 0, (b) E U b 2 E U b V b i P = i= P = i= ` i, b 2 ` i, b ` i, b 25

26 (c) E U () 3i = E U b 2 V b E U b 2 W b E U b V b 2 = = = P P P P ` i, b 3, µ ` i, b 2 ` i, b `, b µ ` i, b 2 ` i, b `, b ` i, b µ ` i, b 2 `, b Proof. For part (a) it follows from Lemma 3 tat E U b i i = P ` i, b =0, i= were te last iequality follows from te defiitio of b. Te remaiig results ca be establised i te same way. For part (b) we ote tat it follows from Lemma 4 tat E U b P µ 2 ` i, b 2 P 2 ` i, b = i= P = i= ` i, b 2, were te last equality is based o P i= ` E U b V b i = i= i= i, b =0. For te secod part we use P ` i, b µ ` i, b µ P i= P ` i, b ` i, b. = i= For part (c) we use Lemma 5 to obtai E U b 3 = E U b 2 V b P = ` i, b 3 3 µ P 2 P i= µ µ ` i, b P 3 ` i, b, P ` i, b µ P ` i, b µ P i= i= U b 2 j Vj b P ` i, b 2 U j b V j b P 2 U j b P P U b 2 j P U j b P V j b, i= ` i, b ` j, b P V j b 26

27 ad E U b 2 W b E U b V b 2 suc tat te result follows. = P 2 = U b 2 j Wj b P 2 P U j b W j b P U b 2 j U j b V b 2 j P U j b V j b P U j b P P U b 2 j P U j b P W j b, V b 2 j P V j b P V b 2 j P W j b P U j b A.3 Lemmas for Jackkifed Statistics Lemma 7 Let W = X X X i, W (j) = Te, we ave i= W X W (j) = W Proof. Note tat X W (j) = It terefore follows tat W X X X X i X i = X X i = i= W W (j) = W W = W Lemma 8 Let X X W = X,i X 2,i, W (j) = X X,i X X 2,i Te, W X i= W (j) = i= X X,i X 2,j 27

28 Proof. We first prove tat jx W (j) = 2 W ( ) For tis purpose, ote tat X W (j) = = X X,i X 2,i i= X X X,i X X 2,i X µ X X,j µ X2 X 2,j = 3 X X X X 2 P i= X,iX 2,i ( ) 2 X = 2 2 ( ) 2 X X 2 = ( ) 2 2 ( ) 2 W ( ) 2 i= X,i X 2,i X X,i X 2,i, were X deotes te sample average of X,i.Terefore,weave X W (j) = 2 W X X,i X 2,i ( ) i= i= ad X W W (j) = = = = µ ( 2) W X X,i X 2,i i= W X X,i X 2,i i= X X X,i X 2,i i= i= X X,i X 2,j X X,i X 2,i i= Lemma 9 Let X X X W = X,i X 2,i X 3,i, i= i= i= W (j) = X X,i X X 2,i X X 3,i 28

29 Te, r W = 2 ( ) 2 W X 2 ( ) 2 X 2 ( ) 2 X3 W (j) X X 2,i X 3,i i= X X,i X 2,i i= 2 ( ) 2 X2 ( ) 2 Proof. Observe tat X X X,i X X 2,i X X 3,i = X µ X X,j = ( ) 3 X X 2 X 3 µ X2 X 2,j X X 2,i X 3,i ( ) 3 X 2 ( ) 3 X i= TX ( ) 3 X 3 X,i X 2,i t= ( ) 3 µ X3 X 3,j TX X 3,i X,i t= X X 3,i X,i i= TX X,i X 2,i X 3,i t= TX X,i X 2,i X 3,i It terefore follows tat X 3 W (j) = X X2 X3 ( ) 3/2 X X ( ) 3/2 X 2,i X 3,i X3 ( ) 3/2 ad terefore, r X W W (j) = 2 ( ) 2 W 2 ( ) 2 X 2 ( ) 2 X3 i= t= X X,i X 2,i ( ) 3/2 i= X X 2,i X 3,i i= X X,i X 2,i i= 2 ( ) 2 X2 ( ) 2 X2 ( ) 3/2 TX t= X X 3,i X,i i= TX X,i X 2,i X 3,i t= X X 3,i X,i i= X,i X 2,i X 3,i 29

30 Lemma 20 Let X X X X W = X,i X 2,i X 3,i X 4,i, i= i= i= i= W (j) = X X,i X X 2,i X X 3,i X X 4,i Te, W 2 X = 2 3 ( ) 3 W W (j) 3 ( ) 3 X X2 3 ( ) 3 X X4 3 ( ) 3 X2 X4 3 ( ) 3 3 ( ) 3 2 ( ) 3 X i= X X 3,i X 4,i i= X X 2,i X 3,i i= X X,i X 3,i i= X3 i= X X,i X 2,i X 3,i X 4,i i= 3 ( ) 3 X X3 3 ( ) 3 X2 X3 3 ( ) 3 X3 X4 X 3 X 2,i X 3,i X 4,i ( ) 3 X2 X 3 X,i X 2,i X 4,i ( ) 3 X4 X X 2,i X 4,i i= X X,i X 4,i i= X X,i X 2,i i= X X,i X 3,i X 4,i i= X X,i X 2,i X 3,i i= 30

31 Proof. Observe tat X X X,i X X 2,i X X 3,i X X 4,i = X µ X X,j = ( ) 4 X X 2 X 3 X 4 ( ) 4 X X 2 ( ) 4 X 2X 3 ( ) 4 X ( ) 4 X 3 ( ) 4 It terefore follows tat X i= i= µ X2 X 2,j µ X3 X 3,j X X 3,i X 4,i ( ) 4 X X 3 X X,i X 4,i i= ( ) 4 X 2X 4 X X 2,i X 3,i X 4,i ( ) 4 X 2 X X,i X 2,i X 4,i i= X X,i X 2,i X 3,i X 4,i i= ( ) 4 X 4 µ X4 X 4,j X X 2,i X 4,i i= ( ) 4 X X 4 X X,i X 3,i i= ( ) 4 X 3X 4 X X,i X 3,i X 4,i i= X X,i X 2,i X 3,i i= W (j) = 2 4 ( ) 2 X X2 X3 X4 ( ) 2 X X2 ( ) 2 X X4 ( ) 2 X2 X4 2 ( ) ( ) 2 ( ) 2 from wic te coclusio follows. X i= X X 3,i X 4,i i= X X 2,i X 3,i i= X X,i X 3,i i= X3 i= X X,i X 2,i X 3,i X 4,i, i= X X 2,i X 3,i i= X X,i X 2,i i= ( ) 2 X X3 ( ) 2 X2 X3 ( ) 2 X3 X4 X X 2,i X 3,i X 4,i 2 X2 ( ) X X,i X 2,i X 4,i ( ) 2 X4 X X 2,i X 4,i i= X X,i X 4,i i= X X,i X 2,i i= X X,i X 3,i X 4,i i= X X,i X 2,i X 3,i i= 3

32 Lemma 2 Let X X X X X W = X,i X 2,i X 3,i X 4,i X 5,i, i= i= i= i= i= W (j) = X X,i X X 2,i X X 3,i X X 4,i X X 5,i i= Te, W ( ) X W (j) = ( ) 4 X X2 X3 X4 X5 2 ( ) 4 X X2 X5 2 ( ) 4 X X2 X3 2 ( ) 4 X X3 X5 2 ( ) 4 X X4 X5 2 ( ) 4 X X3 X4 2 ( ) 4 X3 X4 X5 2 ( ) 4 X2 X3 X5 2 ( ) 4 X2 X4 X5 2 ( ) 4 X2 X3 X4 2 ( ) 4 X X2 X4 X X X X X X X X X X X 3,j X 4,j X 4,j X 5,j X 2,j X 4,j X 2,j X 3,j X 2,j X 5,j X,j X 2,j X,j X 4,j X,j X 3,j X,j X 5,j X 3,j X 5,j 32

33 (cotiued) ( ) 4 X3 X5 X X,j X 2,j X 4,j ( ) 4 X X5 X X 2,j X 3,j X 4,j ( ) 4 X X3 X X 2,j X 4,j X 5,j ( ) 4 X X4 X X 2,j X 3,j X 5,j ( ) 4 X X2 X X 3,j X 4,j X 5,j ( ) 4 X2 X5 X X,j X 3,j X 4,j ( ) 4 X2 X3 X X,j X 4,j X 5,j ( ) 4 X2 X4 X X,j X 3,j X 5,j ( ) 4 X4 X5 X X,j X 2,j X 3,j ( ) 4 X3 X4 X X,j X 2,j X 5,j ( ) 4 X X X 2,j X 3,j X 4,j X 5,j ( ) 4 X2 X X,j X 3,j X 4,j X 5,j ( ) 4 X3 X X,j X 2,j X 4,j X 5,j ( ) 4 X4 X X,j X 2,j X 3,j X 5,j ( ) 4 X5 X X,j X 2,j X 3,j X 4,j ( ) 4 X X,j X 2,j X 3,j X 4,j X 5,j 33

34 Proof. Observe tat X X X,i X X 2,i X X 3,i X X 4,i X X 5,i = X µ X X,j µ X2 X 2,j µ X3 X 3,j µ X4 X 4,j µ X5 X 5,j = ( ) 5 X X 2 X 3 X 4 X 5 2 ( ) 5 X X 2 X 5 X X 3,j X 4,j 2 ( ) 5 X X 2 X 3 X X 4,j X 5,j 2 ( ) 5 X X 3 X 5 X X 2,j X 4,j 2 ( ) 5 X X 4 X 5 X X 2,j X 3,j 2 ( ) 5 X X 3 X 4 X X 2,j X 5,j 2 ( ) 5 X 3X 4 X 5 X X,j X 2,j 2 ( ) 5 X 2X 3 X 5 X X,j X 4,j 2 ( ) 5 X 2X 4 X 5 X X,j X 3,j 2 ( ) 5 X 2X 3 X 4 X X,j X 5,j 2 ( ) 5 X X 2 X 4 X X 3,j X 5,j ( ) 5 X 3X 5 X X,j X 2,j X 4,j ( ) 5 X X 5 X X 2,j X 3,j X 4,j ( ) 5 X X 3 X X 2,j X 4,j X 5,j ( ) 5 X X 4 X X 2,j X 3,j X 5,j ( ) 5 X X 2 X X 3,j X 4,j X 5,j ( ) 5 X 2X 5 X X,j X 3,j X 4,j ( ) 5 X 2X 3 X X,j X 4,j X 5,j ( ) 5 X 2X 4 X X,j X 3,j X 5,j ( ) 5 X 4X 5 X X,j X 2,j X 3,j ( ) 5 X 3X 4 X X,j X 2,j X 5,j ( ) 5 X X X 2,j X 3,j X 4,j X 5,j ( ) 5 X X 2 X,j X 3,j X 4,j X 5,j ( ) 5 X X 3 X,j X 2,j X 4,j X 5,j ( ) 5 X X 4 X,j X 2,j X 3,j X 5,j ( ) 5 X X 5 X,j X 2,j X 3,j X 4,j ( ) 5 X X,j X 2,j X 3,j X 4,j X 5,j 34

35 Terefore, we ave X = W (j) X X2 X3 X4 X5 ( ) 2 ( ) 2 X X2 X5 X ( ) 2 X X2 X3 X ( ) 2 X X3 X5 X ( ) 2 X X4 X5 X ( ) 2 X X3 X4 X ( ) 2 X3 X4 X5 X ( ) 2 X2 X3 X5 X ( ) 2 X2 X4 X5 X ( ) 2 X2 X3 X4 X ( ) 2 X X2 X4 X X 3,j X 4,j X 4,j X 5,j X 2,j X 4,j X 2,j X 3,j X 2,j X 5,j X,j X 2,j X,j X 4,j X,j X 3,j X,j X 5,j X 3,j X 5,j 35

36 (cotiued.) ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 ( ) 2 X3 X5 X X5 X X3 X X4 X X2 X2 X5 X2 X3 X2 X4 X4 X5 X3 X4 X X2 X3 X4 X5 X X X X X X X,j X 2,j X 4,j X 2,j X 3,j X 4,j X 2,j X 4,j X 5,j X 2,j X 3,j X 5,j X 3,j X 4,j X 5,j X,j X 3,j X 4,j X X,j X 4,j X 5,j X X,j X 3,j X 5,j X X,j X 2,j X 3,j X X,j X 2,j X 5,j X X 2,j X 3,j X 4,j X 5,j X X,j X 3,j X 4,j X 5,j X X,j X 2,j X 4,j X 5,j X X,j X 2,j X 3,j X 5,j X X,j X 2,j X 3,j X 4,j X X,j X 2,j X 3,j X 4,j X 5,j 36

37 from wic te coclusio follows. A.4 Proofs of Mai Results Proof of Propositio. Cosider te fuctio G (², ) R ` (, ) df ² (z). Note tat G(.,.) :R Υ R ad is cotiuously differetiable o R 2. By Coditio 2 tere exists (²) Υ suc tat G(², (²)) = 0 for all ² [0, / ]. Sice G(², )/ 6= 0almost surely by assumptio it follows by te implicit fuctio teorem tat tere exists a cotiuously differetiable fuctio (²) :U R suc tat G (², (²)) = 0 a.s. Sice G(², ) is i fact m times cotiuously differetiable (²) ierits tis property. Measurability of (²) follows from cotiuity of f(z, ) ad separability of Υ. By Taylor s teorem tere exists some ² [0, / ] suc tat b /2 = (0) P m k= (k) (0) k k/2 (m) ( ²). By Lemmas 5 ad 6 it follows tat m m/2 0 ² /2 (k) (²) =O p () suc tat te remaider term (m) ( ²) =O m m/2 p m/2 for m 6. Tofid te derivatives (k),let (z, ²) ` (z, (²)), ad rewrite te first order coditio as 0= (z, ²) df ² (z) Differetiatig repeatedly wit respect to ², weobtai d (z, ²) 0 = df ² (z) (z, ²) d (z) (8) d² d 2 (z, ²) d (z, ²) 0 = d² 2 df ² (z)2 d (z) (9) d² d 3 (z, ²) d 2 (z, ²) 0 = d² 3 df ² (z)3 d² 2 d (z) (20) d 4 (z, ²) d 3 (z, ²) 0 = d² 4 df ² (z)4 d² 3 d (z) (2) d 5 (z, ²) d 4 (z, ²) 0 = d² 5 df ² (z)5 d² 4 d (z) (22) d 6 (z, ²) d 5 (z, ²) 0 = d² 6 df ² (z)6 d² 5 d (z) (23) Note tat d (²) d² = ` ² (24) d 2 (²) d² 2 = ` ( ² ) 2 ` ²² (25) d 3 (²) d² 3 = ` ( ² ) 3 3` ² ²² ` ²²² (26) 37

38 d 4 (²) d² 4 = ` ( ² ) 4 6` ( ² ) 2 ²² 3` ( ²² ) 2 4` ² ²²² ` ²²²² (27) d 5 (²) d² 5 = ` ( ² ) 5 0` ( ² ) 3 ²² 5` ² ( ²² ) 2 (28) 0` ( ² ) 2 ²²² 0` ²² ²²² 5` ² ²²²² ` ²²²²² d 6 (²) d² 6 = ` ( ² ) 6 5` ( ² ) 4 ²² 45` ( ² ) 2 ( ²² ) 2 (29) 20` ( ² ) 3 ²²² 5` ( ²² ) 3 60` ² ²² ²²² 5` ( ² ) 2 ²²²² 0` ( ²²² ) 2 5` ²² ²²²² 6` ² ²²²²² ` ²²²²²² Here, ² deotes te derivative of wit respect to ². Combiig (8) - (2) wit (24) - (27), we obtai ` 0=E ² ( i,²) ² (²) ` (z,²) d (z) (30) 0=E ` ² ( i,²) ( ² (²)) 2 ` E ² ( i,²) µ ²² (²)2 ` (z, ²) d (z) ² (²) (3) 0 = E ` ² ( i,²) ( ² (²)) 3 3E ` ² ( i,²) ² (²) ²² ` (²)E ² ( i,²) ²²² (²) µ µ 3 ` (z,²) d (z) ( ² (²)) 2 3 ` (z,²) d (z) ²² (²) (32) 0 = E ` ² ( i,²) ( ² (²)) 4 6E ` ² ( i,²) ( ² (²)) 2 ²² (²)3E ` ² ( i,²) ( ²² (²)) 2 4E ` ² ( i,²) ² (²) ²²² ` (²)E ² ( i,²) µ ²²²² (²)4( ² (²)) 3 ` (z, ²) d (z) µ 2 ² (²) ²² (²) µ ` (z, ²) d (z) 4 ²²² (²) ` (z,²) d (z) (33) 0 = E ² ` ( i,²) ( ² (²)) 5 0E ² ` ( i,²) ( ² (²)) 3 ²² (²)5E ² ` ( i,²) ² (²)( ²² (²)) 2 0E ² ` ( i,²) ( ² (²)) 2 ²²² (²)0E ² ` ( i,²) ²² (²) ²²² (²) 5E ² ` ( i,²) ² (²) ²²²² (²)E ² ` ( i,²) ²²²²² (²)5( ² (²)) 4 µ µ 30 ( ² (²)) 2 ²² (²) µ 20 ² (²) ²²² (²) µ ` (z, ²) d (z) 5( ²² (²)) 2 µ ` (z,²) d (z) 5 ²²²² (²) ` (z,²) d (z) ` (z, ²) d (z) ` (z, ²) d (z) (34) 38

39 ad 0 = E ` ² ( i,²) ( ² (²)) 6 5E ` ² ( i,²) ( ² (²)) 4 ²² (²) 45E ` ² ( i,²) ( ² (²)) 2 ( ²² (²)) 2 20E ` ² ( i,²) ( ² (²)) 3 ²²² (²) 5E ` ² ( i,²) ( ²² (²)) 3 60E ` ² ( i,²) ² (²) ²² (²) ²²² (²) 5E ` ² ( i,²) ( ² (²)) 2 ²²²² (²)0E ` ² ( i,²) ( ²²² (²)) 2 5E ` ² ( i,²) ²² (²) ²²²² (²)6E ` ² ( i,²) ² (²) ²²²²² (²) ` E ² ( i,²) µ ²²²²²² (²)6( ² (²)) 5 (z, ²) d (z) µ µ 60 ( ² (²)) 3 ²² (²) (z, ²) d (z) 90 ² ( ²² (²)) 2 (z,²) d (z) µ µ 60 ( ² (²)) 2 ²²² (²) ` (z,²) d (z) 60 ²² (²) ²²² (²) ` (z, ²) d (z) µ µ 30 ² (²) ²²²² (²) ` (z, ²) d (z) 6 ²²²²² (²) ` (z, ²) d (z) (35) Here, E ² [ ] is defied suc tat E ² [g ( i,²)] g (z, ²) df ² (z) Evaluatig expressios (30) - (33) at ² =0, we obtai µ ² = `d = `d, (36) E [`] I ²² = E [`] = E ` µ I 3 µ E ` µ ( ² ) `d 2 µ I 2 `d ² = E ` µ `d µ E [`] (² ) 2 2 `d E [`] ² `d, (37) ²²² = E ` E [`] (² ) 3 3 E ` µ µ E [`] ² ²² 3 E [`] ` d ( ² ) 2 3 `d E [`] ²² E ` = I 4 3 E ` 2 µ 3 9E ` µ 2 µ `d I 5 I 4 `d `d 3 µ 2 µ I 3 `d ` d 6 µ µ 2 I 3 `d `d (38) ²²²² = E ` ( E [`] ² ) 4 6 E ` E [`] (² ) 2 ²² 3 E ` E [`] (²² ) 2 4 E ` µ E [`] ² ²²² 4 E [`] (² ) 3 ` d µ µ 2 E [`] ² ²² ` d 4 `d E [`] ²²² (39) 39

40 ²²²²² = E ` ( E [`] ² ) 5 0 E ` ( E [`] ² ) 3 ²² 5 E ` E [`] ² ( ²² ) 2 0 E ` E [`] (² ) 2 ²²² 0 E ` E [`] ²² ²²² 5 E ` µ E [`] ² ²²²² 5 E [`] (² ) 4 ` d µ µ 30 E [`] (² ) 2 ²² ` d 5 E [`] (²² ) 2 ` d µ µ 20 E [`] ² ²²² ` d 5 `d E [`] ²²²² (40) ProofofPropositio2. By te same argumets as i te proof of propositio it follows tat tere exists some ² 0, /2 suc tat b b = b ² (0) P m b k= (k) (0) b (m) ( ²) k k/2 m m/2 a.s., were b ² (0) is obtaied from evaluatig R d(z,²) d² d bf ² (z) R (z, ²) d b (z) at ² =0. We obtai 0= `(z,b)d bf (z) b ² (0) ` z,b d b (z), were R `(z, b )d bf (z) P `( i= i, b ) ad b (z) F b (z) bf (z). Similar expressios ca be foud for iger order derivatives of b (²). Tese expressios deped o P i= `(k) i, b ad R `(k) z, b d b (z) for k =0,,..., 6. By Coditio ad Lemma 5, it follows tat P i= `(k) i, b p E `(k) ( i, 0 ) by a uiform law of large umbers. By Propositio 6 te class F is Dosker. By te proof of Teorem 2.4 i Gie ad i (990) it follows tat te followig coditioal stocastic equicotiuity property P N a.s., lim lim supp sup δ 0 `(k) (z, t) `(k) (z, s) d b (z) > η =0 ts <δ olds. Te P µ `(k) z, b `(k) (z, 0 ) db (z) > η P sup `(k) (z, t) `(k) (z,s) d b (z) b > η/2 P 0 η/2 or 0 <δ `(k) z, b d b (z) = `(k) (z, 0 ) d b (z)o p () P N a.s. (4) It ow follows from Propositio 6 ad Teorem 2.4 of Gie ad i (990) tat R `(k) (z, 0 ) d b (z) R `(k) (z, 0 ) dt (z) almost surely, were T (z) is a Browia Bridge process. We fially ave to aalyze te term b (m) ( ²) wic cotais expressios of te form R `(k) (z, b (²))dF b ² (z) ad R z, `(k) b (²) d b (z). For R z, `(k) b (²) d b (z) we use te same iequality as i (4) togeter wit Lemma 0 to sow tat `(k) z, b (²) d b (z) = `(k) (z, 0 ) d b (z)o p () P N a.s. 40

41 Next cosider `(k) (z, b (²))d bf ² (z) `(k) (z, 0 )df (z) ² `(k) (z, b (²))d b (z) `(k) (z, 0 )d F (z) bf (z) `(k) (z, b i (²)) `(k) (z, 0 ) d bf (z) were R `(k) (z, b (²))d b (z) =O p () P N a.s. by Propositio 6 ad sup ² = O( /2 ). Te secod term is o p () by a law of large umbers. Fially, P µsup `(k) (z, b i (²)) `(k) (z, 0 ) d bf (z) > η ² i P sup `(k) (z, ) `(k) (z, 0 ) d bf (z) > η P sup b (²) 0 δ 0 <δ 0 ² / were te first probability goes to zero because for ay ε > 0, δ > 0 suc tat i E sup `(k) (z,) `(k) (z, 0 ) d bf (z) = sup E`(k) ( i, ) E`(k) ( i, 0 ) ε 0 <δ 0 <δ ad te secod probability goes to zero by Lemma 0. It follows tat R `(k) (z, b (²))d bf ² (z) p E`(k) (z, 0 ) P N (k) a.s. Togeter, tese results imply tat sup b ² (²) = O p () P N a.s. for k 6. Tis establises te validity of te expasio. ProofofTeorem3. Let b a /2 ² b (0) 2 b ²² (0) 6 b ²²² (0) 3/2 24 b 2 ²²²² (0). Because (x) (y) 2 α C () kx yk, weave b b b a mi 2 α C (), 96 5/2 sup 0 ² / b ²²²²² (²). Fix ε > 0 ad 7 96 < δ < 2 arbitrary. Takig expectatios wit respect to te measure bf leads to E b b i " E b a i ε/ 2δ 2 α C () P sup 96 5/2 b ²²²²² (²) > ε/ 2δ. 0 ² / i Use te fact tat P sup ²²²²² b 24 5/2 0 ² / (²) > ε/ 2δ = o p 76/60(6/5)δ by settig v = /60 δ/5 i Lemma 2. Coose δ (7/96 (5/6) α, /2). It follows tat E b b i E b a i ε/ 2δ 2o p 76/60(6/5)δα C () = O p ( δ2 )=o p ( 3/2 ) 4