Sequential Bayesian Search Appendices

Squnial Baysian Sarch Appndics Zhn Wn Branislav Kvon Brian Eriksson Sandilya Bhamidipai A Proof of Thorm Assum ha a h binnin of am, h sysm s blif in h usr s prfrnc is P crainy-quival usr prfrnc durin am is Thn, h π i = E π P πi] i I Rcall w dfin = min π i, Lmma A- formalizs h rsul ha if π is clos o π, hn for any dcision r T, E i π NT, i] is clos o E i π NT, i]: Lmma A-: For any dcision r T, w hav ha E i π NT, i] E i π NT, i] π π E i π NT, i] Proof: Noic ha E i π NT, i] E i π NT, i] = π i π i] NT, i π i π i NT, i = π i π i π π int, i i π i π ] i max π π int, i i π i π ] i = max π E i π NT, i] i max π i π i E i π NT, i] = π π E i π NT, i], whr h firs inqualiy follows from h rianular inqualiy and h scond inqualiy follows from h Höldr s inqualiy QED

No ha h bound in Lmma A- is ih in h followin xampl Assum I =, and NT, = 0 NT, = π = ε π = ε π = ε π = ε Thn E i π NT, i] = ε, E i π NT, i] = ε, and hrfor Ei π NT, i] E i π NT, i] = ε On h ohr hand, w hav π π = ε Furhrmor, for ε, w hav ha Thus w hav = π = ε π π E i π NT, i] = ε ε ε = ε Hnc, h bound in Lmma A- is ih in his xampl Throuhou his scion, w assum h crainy-quivaln CE opimizaion problm is solvd xacly, and us T o dno h soluion of h CE opimizaion problm in am, = 0,, Lmma A- sas ha if π π is small, h on-am rr condiionin on π is also small : Lmma A-: If π π <, hn w hav π π π π E i π NT, i] E i π NT, i] E i π NT, i] 0 Proof: By dfiniion of T, w hav ha E i π NT, i] E i π NT, i] On h ohr hand, from h inqualiy, w hav ha Similarly, w hav ha E i π NT, i] E i π NT, i] π π E i π NT, i] = π min π π E i π NT, i] E i π NT, i] E i π NT, i] + π π E i π NT, i] = π min + π π E i π NT, i]

Combinin h abov hr inqualiis, w hav ha Tha is So w hav + π π E i π NT, i] π min π π E i π NT, i] + π π π π E i π NT, i] E i π NT, i] E i π NT, i] E i π NT, i] π π π π E i π NT, i] Finally, noic ha by dfiniion of T i T ar min T E i π NT, i], w hav ha Thus, w hav provd Lmma A- QED E i π NT, i] E i π NT, i] Now w considr h cas whn h prior blif P 0 of h sysm is modld as a Dirichl disribuion wih paramr α R M + hncforh dnod as Dirα Spcifically, is probabiliy dnsiy funcion PDF ovr h probabiliy simplx M is ivn by f P0 π = Bα πi, π M, whr πi is h probabiliy mass a im i, and is h associad paramr Bα is a normalizin consan ivn by Bα = Γ Γ, whr Γ is h classical amma funcion Th main advana of Dirichl prior is ha i rsuls in a simpl posrior disribuion, sinc i is h conjua prior of h mulinomial disribuion Spcifically, = 0,,, w dfin h indicaor vcor Z R M as follows: if i = i Z i = 0 ohrwis Thn, basd on h Bays rul, h posrior blif a h binnin of am is P = Dir α + Z τ From h propris of Dirichl disribuion, w hav ha Noic ha i I Thus, w hav π i = E π P πi] = i I τ=0 τ=0 Z τ i = τ=0 i I Z τ i = + τ=0 Z τ i αi + τ=0 Z τ i ] τ=0 = Furhrmor, w dfin = i I αi π i = + τ=0 Z τ i = + τ=0 Z τ i + + + Throuhou his papr, w us h convnion ha 0 0 = 0, so for = 0, w hav π 0 i = Th abov quaion has a vry nic inrpraion: noic ha is h sima of π i basd on h prior blif, 3

Zτ i τ=0 whil is h sima of π i basd on obsrvaions, h abov quaion sas ha π i is a convx combinaion wihd avra of hs wo simas Furhrmor, h wihs dpnd on, h indx of h currn inraciv am or quivalnly, h numbr of pas obsrvaions From Hoffdin s inqualiy, ɛ > 0, w hav ha τ=0 P Z τ i π i ɛ xp ɛ Tha is, for any i I, a h binnin of am, wih probabiliy a las xp ɛ, w hav ha τ=0 Z τ i π i ɛ τ=0 Zτ i L E i dno h vn ha π i > ɛ Thn w hav provd ha PE i xp ɛ for any i I From h union bound of h probabiliy, w hav ha P E i P E i M xp ɛ Thus, wih probabiliy a las M xp ɛ, w hav ha τ=0 max Z τ i π i ɛ Finally, noic ha i I, w hav ha π i π i = + + π i + Thus w hav ha max π i π i max + + max π i + + + π i + π i + τ=0 Z τ i τ=0 Z τ i + + max π i π i τ=0 Z τ i τ=0 Z τ i ] π i π i Noic ha max π i is h maximum simaion rror basd on h prior blif, which is indpndn of h obsrvaions On h ohr hand, max τ=0 Zτ i π i is h maximum simaion rror basd on obsrvaions, which is a random variabl Lmma A-3 uppr bounds h rr in am : Lmma A-3: > 0 and 0 < η 3, if + max π i < ηπ min, 4

hn w hav ha E i π NT, i] E i π NT, i] < 3ηE i π NT, i] + M Q xp 4ηα 0 max Proof: Sinc + max π i < η, hus, on sufficin condiion o nsur ha is max τ=0 Z τ i max π i π i η π i ηπ min + max π i ηπ min] π i 3 Dfin ɛ = η + max π i, from h abov discussion, w know ha inqualiy 3 holds wih probabiliy a las M xp ɛ Furhrmor, from Lmma A-, max π i π i η implis ha η η E i π NT, i] E i π NT, i] E i π NT, i] 4 Thus, w hav provd ha wih probabiliy a las M xp ɛ, inqualiy 4 holds In ohr words, if w dfin E as h vn ha inqualiy 4 holds, hn w hav ha PE M xp ɛ On h ohr hand, noic ha a naiv bound on h rr is E i π NT, i] E i π NT, i] Q Wih E dfind as h vn ha inqualiy 4 holds and Ē dfind as h complmn of E, w hav ha: E i π NT, i] E i π NT, i] PE E i π NT, i] E i π NT, i] E + PE] Ei π NT, i] E i π NT, i] Ē PE η η E i π NT, i] + PE] Q On h ohr hand, noic ha E i π NT, i] Q by dfiniion, and η 3 implis ha η w hav η η E i π NT, i] Q Tohr wih PE M xp ɛ, w hav ha η, hus E i π NT, i] E i π NT, i] M xp ɛ ] η η E i π NT, i] + M xp ɛ Q Noic ha 0 < η 3 < implis ha 0 < η 3 η, hus 0 < η η η E i π NT, i] + M Q xp ɛ 3η Hnc w hav ha E i π NT, i] E i π NT, i] < 3ηE i π NT, i] + M Q xp ɛ From h dfiniion of ɛ, w hav ] ɛ = η + ηπ min max π i = η] + ηα 0 ηπ min max π i > η] ηα 0 max π i, 5 + α 0 η max π i

whr h las inqualiy follows from h fac ha α 0 ηπ min max 0 So w hav ɛ < 4η max Thus, w hav provd Lmma A-3 QED π i ηπ min] π i 0 and ηπ min ] > W dfin τ E as τ E = min 4 : π min and 4 6 3 max π i, 5 whr is h loarihm funcion wih bas Noic ha for 3, is monoonically dcrasin Noic ha τ E dpnds on and h qualiy of h prior Lmma A-4 drivs a mor usful on-am rr bound basd on Lmma A-3 and h dfiniion of τ E : Lmma A-4: τ E, w hav E i π NT, i] E i π NT, i] < 6 whr τ E is dfind in Eqn5 Proof: For τ E, w choos η = Lmma A-3 Sinc Ei π NT, i] + M Q, W firs show ha his paricular η saisfis h condiions of is monoonically dcrasin for 3 and τ E 4, w hav ha η = τe τ E 6 = 3 On h ohr hand, sinc is monoonically incrasin, hus, τ E implis ha 4 3 max π i Thus + max π i < max π i 3 ] 4 = 3 8 Thus, h condiions of Lmma A-3 ar saisfid and w hav ha E i π NT, i] E i π NT, i] < 3ηE i π NT, i] + M Q xp 4ηα 0 max Noic ha and 4η max π i = 8 η] = α0 max π i 8 ] = 6 = 3π min η < ηπ 8 min 3 4 = 6, ] = 8 π i ηπ min]

Thus w hav xp 4ηα 0 max On h ohr hand, w hav ha 3η = 6 π i ηπ min] xp 6 8 = xp =, hus w hav E i π NT, i] E i π NT, i] < 3ηE i π NT, i] + M Q xp QED 6 In his rmaindr of his scion, w prov Thorm : Proof of Thorm : Noic ha a naiv bound on 4η max Ei π NT, i] + M Q E i π NT, i] E i π NT, i] is π i ηπ min] E i π NT, i] E i π NT, i] Q Thus, for 0 τ < τ E, w hav ha E i π NT, i] E i π NT, i] Q τ + =0 On h ohr hand, from Lmma A-4, for τ τ E, w hav ha E i π NT, i] E i π NT, i] = =0 + E =0 =τ E Q τ E + Q τ E + E i π NT, i] E i π NT, i] E i π NT, i] E i π NT, i] =τ E =τ E 6 E i π NT, i] E i π NT, i] Ei π NT, i] + M Q whr h firs inqualiy follows from h naiv bound and h scond inqualiy follows from Lmma A-4 Sinc τ E >, w hav ha < < =τ E =τ E =τ E = =τ E ] = τ E On h ohr hand, noic ha is monoonically dcrasin on inrval τ E, Sinc τ E 3, ] and h drivaiv of h funcion is +, w hav ha τ < =τ E τ E d < τ τ E + 7 d = τ τ τ E τ E ],

Thus, for τ τ E, w hav ha QED E i π NT, i] E i π NT, i] Q τ E + =0 < Q τ E + E i π NT, i] =τ E 6 Ei π NT, i] + M Q τ τ τ E τ E + M Q τ E = Oτ τ ] 8

B Proof of Thorm Throuhou his scion, w assum ha h crainy-quivaln CE opimizaion problm is solvd by h rdy alorihm, and us T o dno h soluion basd on h rdy alorihm of h CE opimizaion problm in am, = 0,, No ha in h proof, w sill us T o dno h xac soluion of h CE opimizaion problm in am Lmma A-5 is h counrpar of Lmma A- in his cas: Lmma A-5: If π π <, hn w hav E i π NT π, i] E i π NT, i] min + π π ] π π π π Proof: Bfor procdin, noic ha from Thorm 0 of ], w hav ha E i π NT, i] E i π NT, i] min i π i +, whr T is h xac soluion of h CE opimizaion problm in am, and T is h approximaion soluion basd on h rdy alorihm From Lmma A-, w know ha and Thus w hav ha E i π E i π NT, i] π min π π E i π NT, i], NT, i] π min + π π E i π NT, i] π π E i π NT, i] π min + π π E i π NT, i] min i π i + W dfin c = π min + π π π π, hus w hav ha E i π NT, i] ce i π NT, i] Combinin wih Lmma A-, w hav ha E i π NT, i] c E i π NT, i] Finally, assum ha min i π i = π i, w hav ha min i π i min i π i + + 6 So w hav Hnc min i π i = π i = π i + π i π i ] π π min i π i min i π i + π π π π + = π π 9

Plu h abov inqualiy ino Eqn6, w hav provd Lmma A-5 QED Lmma A-6 uppr bounds h scald rr in am : Lmma A-6: > 0 and 0 < η <, if ] + η 4η + η η η hn w hav ha E i π NT, i] ] + η < η η + M Q xp + 4η max and + max E i π NT, i] 4η η E i π NT, i] π i ηπ min] Proof: Sinc + max π i < η, hus, on sufficin condiion o nsur ha is max τ=0 Z τ i max π i π i η π i ηπ min + max W dfin ɛ = η + max π i ha inqualiy 7 holds wih probabiliy a las M xp ɛ π π η implis ha Thus w hav ha E i π NT, i] E i π NT, i] = E i π NT, i] ] + η η + η η ] η η + π i < ηπ min, π i 7 From h discussion bfor Lmma A-3, w know + η η E i π NT, i] 4η η Furhrmor, from Lmma A-5, η E i π NT, i] E i π NT, i] 8 Thus, w hav provd ha wih probabiliy a las M xp ɛ, inqualiy 8 holds In ohr words, if w dfin E as h vn ha inqualiy 8 holds, hn w hav ha PE M xp ɛ On h ohr hand, noic ha a naiv bound on h scald rr is E i π NT, i] E i π NT, i] Q 0

Wih E dfind as h vn ha inqualiy 8 holds and Ē dfind as h complmn of E, w hav ha: E i π NT, i] E i π NT, i] PE + PE] PE E i π NT, i] E i π NT, i] ] + η + η η E i π NT, i] E E i π NT, i] Ē 4η η On h ohr hand, noic ha E i π NT, i] Q by dfiniion, and E i π NT, i] + PE] Q, hus w hav ] + η 4η + η η η E i π NT, i] Q Tohr wih PE M xp ɛ, w hav ha E i π NT, i] ] + η + η η 4η η E i π NT, i] From h dfiniion of ɛ, w hav ] ɛ = η + ηπ min max π i = η] + ηα 0 ηπ min max π i > η] ηα 0 max π i, ] +η η η + 4η η E i π NT, i] + M Q xp ɛ + α 0 whr h las inqualiy follows from h fac ha α 0 ηπ min max 0 So w hav ɛ < 4η max Thus, w hav provd Lmma A-6 QED η max π i ηπ min] π i π i 0 and ηπ min ] > Bfor procdin, w driv a sufficin condiion for + η + 4η η η η ] ha is asy o vrify Noic ha fη = +η η η + 4η η is an incrasin and coninuous funcion of η on inrval 0,, and f0 = 0, lim η fη =, hus, hr xiss an η 0, such ha fη = Similarly, w can show ha η = funcion of η, and 0378 = +η η ] η + 4η η is an incrasin and coninuous

W now show ha if η 0378 0378 π min 0378 Thus w hav, for η 0378, Thus, on sufficin condiion for W dfin τ G as τ G = min 4 :, hn fη Noic ha sinc, hn w hav fη f 0378 0378 = 9 ] +η η η + 4η η 00689π min and 4 3 max is ha η 0378 π i 0 Lmma A-7 drivs a mor usful on-am rr bound basd on Lmma A-5 and dfiniion of τ G : Lmma A-7: τ G, w hav ha Proof: < For τ G, w choos η = Lmma A-6 Sinc E i π NT, i] ] 8 + E i π NT, i] Ei π NT, i] + M Q W firs show ha his paricular η saisfis h condiions of is monoonically dcrasin for 3 and τ G 4, w hav ha η = τg τ G 0378 ] From h discussion abov, w hav +η η η + 4η η for η = On h ohr hand, sinc is monoonically incrasin, hus, τ G implis ha 4 3 max Similarly as h proof for Lmma A-4, w hav ha + max π i < max π i π i 3 ] 4 = 3 8 = 3π min η < ηπ 8 min Thus, h condiions of Lmma A-6 ar saisfid Furhrmor, similarly as h proof for Lmma A-4, w hav ha xp 4ηα 0 max π i ηπ min] xp 6 8 = xp =

for τ G W now bound h rm Thus w hav for 0 η ] +η η η + 4η η Noic ha for τ G, w hav ha η = 0378 0378 ] +η η 745 <, and 345 < 5 On h ohr hand, noic ha η η, hus, for η 0378, w hav ha ] + η 4η + η η η < 4η + 6η = 4 + 6 Combinin h abov inqualiis and h rsul of Lmma A-6, w hav ha E i π NT, i] E i π NT, i] for τ G QED < Finally, w prov Thorm Proof of Thorm : ] 4 + 6 Ei π NT, i] + M Q, Th proof is similar o Thorm Spcifically, for 0 τ < τ G, w hav ha E i π NT, i] E i π NT, i] Q τ + =0 =0 On h ohr hand, from Lmma A-7, for τ τ G, w hav ha E i π NT, i] = + G =0 =τ G Q τ G + Q τ G + E i π NT, i] E i π NT, i] =τ G =τ G E i π NT, i] ] 8 + E i π NT, i] E i π NT, i] E i π NT, i] E i π NT, i] Ei π NT, i] + M Q ], η whr h firs inqualiy follows from h naiv bound and h scond inqualiy follows from Lmma A-7 Sinc τ G >, w hav ha < < = ] = τ =τ G =τ G =τ G G 3 =τ G

On h ohr hand, noic ha is monoonically dcrasin on inrval τ G, Sinc τ G 3, ] and h drivaiv of h funcion is +, w hav ha τ < =τ G τ G Thus, for τ τ G, w hav ha =0 Q τ G + d < τ τ G E i π NT, i] =τ G 8 + < Q τ G + 6 + 4 = O τ τ Noic ha, so w hav QED Rfrncs =0 < Q τ G + 40 = O τ τ + E i π NT, i] ] Ei π NT, i] E i π NT, i] Ei π NT, i] d = τ τ τ G τ G ] Ei π NT, i] + M Q τ τ τ G τ G + M Q τ G E i π NT, i] τ τ τ G τ G + M Q τ G ] Danil Golovin and Andras Kraus Adapiv submodulariy: Thory and applicaions in aciv larnin and sochasic opimizaion Journal of Arificial Inllinc Rsarch, 4:47 486, 0 4