Markov Processes ad Alcatos Dscrete-Tme Markov Chas Cotuous-Tme Markov Chas Alcatos Queug theory Performace aalyss
Dscrete-Tme Markov Chas Books - Itroducto to Stochastc Processes (Erha Clar), Cha. 5, 6 - Itroducto to Probablty Models (Sheldo Ross), Cha. 4 - Performace Aalyss of Commucatos Networks ad Systems (Pet Va Meghem), Cha. 9, - Markov Chas (J.R. Norrs), Cha. - Dscrete Stochastc Processes (R. Gallager), Cha. 4 - Elemetary Probablty for Alcatos (Rck Durrett), Cha. 5 - Itroducto to Probablty, D. Bertsekas & J. Tstskls, Cha. 6 2
INTRODUCTION : th order df of some stoc. roc. { X 2 2 3 2 } s gve by f ( x, x,..., x ) f ( x x, x,..., x ) f ( x x, x,..., x )... f ( x x ) f ( x ) very dffcult to have t geeral If { X } s t t t t t t t t t t t t t a de. rocess: t t t f ( x, x,..., x ) f ( x ) f ( x )... f ( x ) If { X t t t t t t t 2 } s a rocess wth de. cremets: f ( x, x,..., x ) f ( x ) f ( x x )... f ( x x ) t t t t t t t t 2 2 Note : Frst order df's are suffcet for above secal cases If { X } s a rocess whose evoluto beyod t s (robablstcally) t comletely determed by x ad s de. of x, t t, gve x, the: f ( x, x,..., x ) f ( x x )... f ( x t t t t t t 2 2 t t t x ) f ( x ) Ths s a Markov rocess ( th order df smlfed) t t 3
Defto of a Markov Process (MP) A stoch.roc.{ X a MarkovProcess (MP)ff : f ( x or f ( x for all x t t t x x t t t ad all t ; t I} that takes values from a set E s called,..., x,..., x t t t ) ) 2 P( x t f ( x t... t x x t t ) ( E ) ( E ad all. coutable) ucoutable) Notce: The"ext" state x t s de.of rovded that the"reset"s kow. the"ast"{ x t,..., x t 2 } 4
Defto of a Markov Cha (MC) (Dscrete- tme & dscrete - value MP) If I s coutable ad E s coutable the a MPs called ad s descrbed by thetrasto robabltes : (, ) P{ X X }, E (de.of for a tme - homogeeou s MC).Assume E a MC {,,2,...}(state-saceof the MC) Trastomatrx : P(,) P(,) P P(,) P s o - egatve, For a gve P(,) P(,) P(,)......... P(, ) P (stoch.matrx) P(, ) P(, ) P(, ) a,......... (stochastc matrx) MCmay be costructed 5
6 ), ( ), ( 2 2 3 2 2 3 2 2 ), ( ), ( ), ( } { through teratos) (geeral 3 : For Proof. the trasto matrx of th ower the of etry ), the ( s ), ( ;,, ), ( } {,,...,,, ), ( )..., ( ) ( },...,,, { the, }, { ) ( s.t. PMFo a s If P E l l P E l k k k l P l l P l P X X P k P k P k E P X X P k E N P P X X X X P E X P E N N k - stetrastos : Charule:
Chama Kolmogorov Equatos : From revous, P m (, ) P m (, k) P ( k, ), E ke I order for { X } to be after m stes ad startg from, t wll have to be some k after m stes ad move the to the remag stes. 7
Bertsekas & Tstskls 8
Examle : # of successes Beroull rocess { N ; }, N # of successes trals N Y,, Y de. Beroull, P{ Y } Notce: N N Y evoluto of { N } beyod does ot deed o { N } (gve N ) ad thus { N } s a M.C. P{ N N, N,..., N } P{ Y N N, N,..., N } q... f N... f ad q q N P q... otherwse Notce: { N } s a secal M.C. whose cremet s de. both from reset ad ast (rocess wth de. cremets) 9
[Ross]
[Ross]
[Norrs] 2
3
[Ross] 4
[Ross] 5
Examle : Sum of..d. RV's wth PMF { ; k,, 2,...} X Y Y2... Y k X X Y P{ X X,..., X } P{ Y X X,..., X } X Thus { X } s a M.C. wth P(, ) P{ X X } 2 3... 2... P...... 6
Examle X P{ X { X Notce that rows are detcal (If, X }s P,...d. wth ( k) a () () P () has : Ideedet trals M.C. all X () () (),..., X } rows detcal, P{ X k the ad X,,2,... P, } ( ) m X P m,... are..d.) 7
8 (double - stochastc matrx) (here) colums (stoch.matrx) rows M.C. a }s {, (modulo 5) },,,, { wth {,,2,3,4} }are..d. :{ 4 3 2 4 3 2 2 4 3 3 2 4 4 3 2 4 3 2 P X Y X X Y Y Examle
Examle : Remag lfetme A equmet s relaced by a detcal as soo as t fals Pr{a ew equ. lasts for k tme uts} k,2,... k X X Z remag lfetme of equ. at tme X( ) f X( ) ( ) Z( ) f X( ) ( ) s the lfetme of equ. stalled at tme It s deedet of X, X,..., X X s a M.C. 9
: P(, ) P{ X X } P{ X X } f P{ X X } f : P(, ) P{ X X } P{ Z X } P{ Z } P 2 3 4 2
Theorem : (codtoal de. of future from ast gve reset) Let Y be a bouded fucto of X, X,.... The E{Y X, X,..., X } E{Y X } Proosto : E{ f(x,x,...)x } E{ f(x,x,...)x } Corollary : f a bouded fucto o E E... Let g() E{ f(x,x,...)x }. The N E{ f(x,x,...)x, X,..., X } g(x ) 2
Theorem : (codtoal de. of future from ast gve reset) Let Y be a bouded fucto of X, X,.... The E{Y X, X,..., X } E{Y X } 22
Proosto : E{ f(x,x,... ) X } E{ f(x,x,... ) X } 23
Corollary : f a bouded fucto o E E... Let g() E{ f(x,x,...)x }. The N E{ f(x,x,...)x, X,..., X } g(x ) 24
S tog Tmes: Prevous results derved for fxed What f tme s a RV stead? tme N are codtoally roerty s f If If for a RV T T s a stogtme, the above hold true( T s the evet { T, the ast { X m de.gve reset sad to hold at T. ; m } ca be determed T}ad the future{ X X T, the thestrog M arkov by lookg at m ; m a stogtme X, X T},..., X ) 25
For ay stog tme T : E{ f ( X, X,...) X, T} E{ f ( X, X,...) X } T T T T T For g( ) E{ f ( X, X,...) X } E{ f ( X, X,...) X ; T} g( X ) T T T f am e.g., f f ( a, a,...) E, m N f am E{ f ( X, X m,...) X } P{ X X } P (, ) m E{ f ( X, X,...) X, T} P{ X X ; T} T T T m Strog Markov roerty at T: m P{ X X ; T} P ( X, ) T m T 26
(art of)[norrs] 27
Vsts to a state X { X N } MC, State sace E, Trasto matrx P. Notato: P { A} P{ A X } ad E[ Y] E[ Y X ] Let E, ad Defe: N ( ) = total umber of tmes state aears X( ) X ( ). N ( ), X evetually leaves state ever to retur. N ( ), X vsts aga ad aga. Let T( ) T 2( ) the successve dces for whch X( ). If the T ( ) T2 ( ) T ( ) If aears a fte umber of tmes m, the Tm ( ) Tm ( ) Tm 2( ) T m( ) N, { Tm ( ) } s equvalet to aears { X( ) X ( )} at least m tmes. T m s a stog tme. 28
Examle T ( ) 4 T ( ) 6 T ( ) 7 T ( ) 9 2 3 4 E 29
Proosto: E km { Tm } P Tm Tm k T Tm P{ T k} { Tm } Comutato of P{ T k }. Let Fk( ) P{ T k } k Fk ( ) P{ T } P{ X } P( ) k 2 Fk ( ) P { X X k X k } P{ X b} P{ X X X X b } be {} 2 k k P{ {} } { 2 } X b P b E b X X k X k Thus, P( ) k F ( ) k P( b) Fk ( b ) k 2 be { } 3
Examle: Let 3 a the trasto matrx Fd f ( ) F ( ) 23 k k P / 2 /6 /3 /3 3/5 /5 k. I ths case f s the 3rd colum of matrx P. Hece, f() F( ), f(2) F(2 ) 3, f(3) F(3 ) 5 k 2. I ths case F (, ) k be { } k k k be { } be { } P( b) F ( b ) f F (2, ) P(2 b) F ( b ) Q f where k F (3, ) k P(3 b) F ( b ) After some algebra f / 3 f2 /8 f3 /8 f 4 / 648 /5 / 5 / 3 /8 ad geeral k k 5 Fk ( 3) Fk (23) Fk (3 3) k 2 36 3 k 2 5 6 3 k k Q / 2 / 6 /3 3/5 3
k k 5 Fk ( 3) Fk (23) Fk (3 3) k 2 36 3 k 2 5 6 3 Now we ca state: Startg at state, X ever vsts 3 wth robablty: PT { } Startg at state 2, X frst vsts 3 at k wth robablty: () k Startg at state 2, X ever vsts 3 wth robablty: { } { } ( ) k 3 2 2 k 3 6 5 P T P T Startg at state 3, X ever vsts 3 aga wth robablty: 52 P{ T } P{ T } 3 3 75 3 6 32
Now, for every we defe F( ) P{ T } F ( ) k k F( ) exresses the robablty: startg at the MC wll ever vst state. F( ) P( ) P( b) F( b ) E be { } If by N we deote the total umber of vsts to state, the m P { N m} F( ) F( ) ad for, P{ N m} F( ) m m F( ) F( ) F( ) m 2 >From the revous we obta the Corollary: F( ) P{ N } F( ) 33
Now, for every we defe m P { N m} m F m ( ) F( ) F( ) P{ T } F ( ) F( ) exresses the robablty: startg at the MC wll ever vst state. F( ) F( ) be { } k F( ) P( ) P( b) F( b ) E m x m k, x x If by N we deote the total umber of vsts to state, the m P { N m} F( ) F( ) ad for, P{ N m} F( ) m m F( ) F( ) F( ) m 2 >From the revous we obta the Corollary: F( ) P{ N } F( ) 34
m m Now, for every we defe m P { N m} m F m ( ) F( ) F( ) F( ) P{ T } F ( ) F( ) exresses the robablty: startg at the MC wll ever 2 vst state. 2 F( ) F( ) be { } k F( ) P( ) P( b) F( b ) E If by N we deote the total umber of vsts to state, the m P{ N m} F( ) F( ) If F( ) N w... Therefore, f X E [ N ] If F( ) the N follows geometrc dstrbuto F( wth ) robablty mof success ad for, P{ N m} m F( ). Hece, E ( ) F( ) F( ) m 2 [ N ] F ( ) m m x m k ( x), x >From the revous we obta the Corollary: F( ) P{ N } F( ) 35
Let R( ) E [ N ] ( R s called the otetal matrx of X ) The, R( ) F ( ) R( ) F(, ) R( ), ( ) R(, ) F(, ) R(, ) ( F(, )) Comutato of R( ) frst ad the F( ) Defe: k X( ) ( k) ( X( )) k X( ) The, N ( ) R( ) ( X ( )) E ( ) ( ) { } ( ) X E X P X P I matrx otato: 2 2 R I P P RP PR P P R I from whch we obta R( I P) ( I P) R I 36
Classfcato of states X: MC, wth state sace E, trasto matrx P T : The tme of frst vst to state N : The total umber of vsts to state Defto State s called recurret f P{ T } State s called traset f P{ T } A recurret state s called ull f E[ T ] A recurret state s called o-ull f E[ T ] A recurret state s called erodc wth erod, f 2 s the greatest teger for whch P { T for some } 37
[Gallager] (a) δ=2 (b) δ=2 38
If s recurret the startg at the robablty of returg to s. F( ) R( ) E [ N ] P { N } If s traset the there exsts a ostve robablty F( ) of ever returg to. F( ) R( ) E [ N ] P { N } I ths case R( ) F( ) R( ) R( ) ad sce R( ) P ( ) we coclude that lm P ( ) 39
Theorem: If traset or recurret ull the E lm P ( ) If recurret o-ull the ( ) lm P ( ) ad E lm P ( ) F( ) ( ) If erodc wth erod, the a retutr to s ossble oly at stes umbered, 2, 3,... P ( ) P { X } oly f { 2 } 4
Recurret o-ull Recurret ull Traset P{ T } P{ T } E [ ] T E[ T ] F( ) R( ) E [ N ] P{ N } ( ) lm P ( ) ad E lm P ( ) F( ) ( ) F( ) R( ) E [ N ] P{ N } E lm P ( ) A recurret state s called erodc wth erod, f 2 s the greatest teger for whch P { T for some } If erodc wth erod, the a retur to s ossble oly at stes umbered, 2, 3,... P ( ) P { X } oly f { 2 } 4
We say that state ca be reached from state f P ( ) ff F( ) Defto: A set of states s closed f o state outsde t ca be reached from ay state t. A state formg a closed set by tself s called a absorbg state A closed set s called rreducble f o roer subset of t s closed. A MC s called rreducble f ts oly closed set s the set of all states Commets: If s absorbg the P( ). If MC s rreducble the all states ca be reached from each other. If C { c c2 } E s a closed set ad Q( ) P( c c ), cc C, the Q s a Markov matrx. If ad k the k. To fd the closed set C that cotas we work as follows: Startg wth we clude C all states that ca be reached from : P( ). We ext clude C all states k that ca be reached from : P( k ). We reeat the revous ste 42
Examle: MC wth state sace E { ab cde } ad trasto matrx P 2 2 3 4 4 2 3 3 4 2 4 3 3 3 Commets: Closed sets: { ace } ad { ab cde } There are two closed sets. Thus, the MC s ot rreducble. c e 43
Examle: MC wth state sace E { ab cde } ad trasto matrx Commets: 2 2 Closed sets: { ace } ad { ab cde } 3 4 4 2 P 3 3 2 2 4 2 4 2 Q 3 3 3 3 3 3 3 3 There are two closed sets. Thus, the MC s ot rreducble. If we delete the 2 d ad 4 th rows we obta the Markov matrx: If we relabel the states a, 2 c, 3 e, 4 b ad 5 d we get 2 2 2 3 3 P 3 3 3 3 4 4 4 2 4 44
Lemma If recurret ad k k. Thus, F( k ). Proof: If k the k s reached wthout returg to wth robablty a. Oce k s reached, the robablty that s ever vsted aga s F( k ). Hece, F( ) a( F( k )) But s recurret, so that F( ) F( k ) As a result: If k but k, the must be traset. Theorem: From recurret states oly recurret states ca be reached. Theorem: I a Marcov cha the recurret states ca be dvded a uque maer, to rreducble closed sets C, C 2,, ad after a arorate arragemet: P P2 P P3 Q Q2 Q3 Q 45
Theorem: Let X a rreducble MC. The, oe of the followg holds: All states are traset. All states are recurret ull All states are recurret o-ull Ether all aerodc or f oe s erodc wth erod, all are erodc wth the same erod. Proof: Sce X s rreducble the k ad k, whch meas that rs r s P ( k ) ad P ( k ). Pck the smallest rs ad let r s P ( k) P ( k ). If k recurret recurret. If k traset traset. (If t was recurret the k would be recurret) m If k recurret ull the P ( kk ) as m. But rs P ( kk) P ( ) P ( ) 46
Corollary: If C rreducble closed set of ftely may states, the recurret ull states. Proof: If oe s recurret ull the all states are recurret ull. Thus, lm P ( ) C. But, C P ( ) lm P ( ) C C Because, we have fte umber of states lm P ( ) lm P ( ) C C Corollary: If C s a rreducble closed set wth ftely may states the there are o traset states 47
Algorthm - Fte umber of states Idetfy rreducble closed sets. All states belogg to a rreducble closed set are recurret ostve The rest of the states are traset Perodcty s checked to each rreducble set 48
Examle: The rreducble closed sets are { 3}, {27 9} ad {6}. The states {4 58 } are traset. If we relabel the states we obta 2 2 2 3 3 3 4 4 P 3 3 3 4 4 4 4 3 3 3 49
Examle: Let see N the umber of successes the frst Beroull trals. As we have P( ) P{ N N } q otherwse Thus, P q q q we have but => s traset.. Ths meas that s ot recurret 5
Examle: Remag lfetme X( ) X( ) Remember: X ( ) Z( ) X( ) from whch we obta: P( ) P{ X X } P{ X X } P( ) P{ X X } P{ Z X } P{ Z } P 2 3 5
2 3 P >From state we reach state oe ste. From we ca reach, 2,...,,. Thus, all states ca be reached from each other, whch meas that the MC s rreducble. Sce, P () the MC s aerodc. Retur to state occurs f the lfetme s fte: F() Sce state s recurret, all states are recurret. If the exected lfetme: the state s ull ad all states are recurret ull. If the exected lfetme: the state s o-ull ad all states are recurret o-ull. 52
Algorthm - Ifte umber of states Theorem: Let X a rreducble MC, ad cosder the system of lear equatos: ( ) ( ) P( ) E The all states are recurret o-ull ff there exsts a soluto wth ( ) E E Theorem: Let X a rreducble MC wth trasto matrx P, ad let Q be the matrx obtaed from P by deletg the k -row ad k -colum for some k E. The all states are recurret f ad oly f the oly soluto of h( ) Q( ) h( ) h( ) E E s h ( ) for all E. E E {} k. Use frst theorem to determe whether all states are recurret o-ull or ot. I the latter case, use the secod theorem to determe whether the states are traset or ot. 53
Examle: Radom walks. P q q All states ca be reached from each other, ad thus the cha s rreducble. A retur to state ca occur oly at stes umbered 2,4,6,... Therefore, state s erodc wth erod 2. Sce X s rreducble all states are erodc wth erod 2. Ether all states are recurret ull, or all are recurret o-ull, or all the states are traset. Check for a soluto of P. q q 2 2 q 3 3 2 q 4 54
Hece, Ay soluto s of the form If q, the q ad q 2 2 q q q 3 2 2 3 q q q q 2 q q 2q q q q If we choose q the 2q ad 2 q ( ) 2q q q I ths case all states are recurret o ull 55
If q ether all states are recurret ull or all states are traset. Cosder the matrx q Q q The equato h Qh gves ( h h() ) h q q q h If q the h h for all ad the oly way to have h for all s by choosg h whch mles h that s all states are recurret ull. If q, the choosg h ( q ), we get q h whch also satsfes h. I ths case all states are traset. 56
Calculato of R ad F R( ) E [ N ] Exected umber of vsts to state. F( ) The robablty of ever reachg state startg at. Recurret state: F( ) R( ) R( ) F( ) R( ) R( ) F( ) F( ) Traset / Recurret state: F( ) R( ) Traset Let D { the traset states }, Q( ) P( ), S( ) R( ), D. The P m K m K P L Q Lm Q m Hece, m K m m 2 m L m m m Q R P S Q I Q Q 57
Comutato of S S 2 I Q Q 2 SQ QS Q Q S I ( I Q) S I, S( I Q) I Proosto: If there are ftely may traset states S ( I Q) Whe the set D of traset states s fte, t s ossble to have more tha oe soluto to the system. Theorem: S s the mmal soluto of ( I Q) Y I, Y Theorem: S s the uque soluto of ( I Q) Y I f ad oly f the oly bouded soluto of h Qh s h, or equvaletly h Qh h h 58
Examle: Let X a MC wth state sace E { 23 45 6 7 8} P 4 3 3 6 4 5 5 8 2 4 6 4 4 2 3 6 { 2 3} are recurret ostve aerodc. {4 5} are recurret ostve aerodc. {67 8} are traset 4 6 6 6 Q 2 S ( I Q) 2 6 6 59
recurret, ca be reached from traset, recurret recurret, caot be reached from, traset recurret traset recurret traset R 25 75 5 66 66 66 5 75 5 66 66 66 75 45 75 66 66 66 S 6
Comutato of F( ) recurret belogg to the same rreducble closed set F( ) recurret belogg to dfferet rreducble closed sets F( ) traset The R( ) ad R( ) F( ) F( ) R( ) R( ) traset, recurret???? Lemma: If C s rreducble closed set of recurret states, the for ay traset state : F( ) F( k) for all, k C. Proof: For k C F( k) F( k ). Thus, oce the cha reaches ay oe of the states of C, t also vsts all the other states. Hece, F( ) F( k) s the robablty of eterg the set C from. 6
Let P P2 P P 3 Q Q2 Q3 Q Lum all states of C together to make oe absorbg state: ˆ P b ( ) P( k) D kc b b2 b3 bm Q The robablty of ever reachg the absorbg state from the traset state by the cha wth the trasto matrx ˆP s the same as that of ever reachg ˆ I P B b bm B( ) P( k) ı D B Q kc C from. I 2 ˆ ( ) B Q P B I Q Q Q B B ( ) s the robablty that startg from, the cha eters the recurret class C 62
Defe: k k G lm B Q B SB G( ) s the robablty of ever reachg the set C from the traset state : ( F( ) ) Proosto: Let Q the matrx obtaed from P by deletg all the rows ad colums corresodg to the recurret states, ad let B be defed as revously, for each traset ad recurret class C. Comute S Comute G SB G( ) F( k), k C. If there s oly oe recurret class ad ftely may traset states, the thgs are dfferet. I ths case, t ca be roved that: G F( ) C 63
Examle: Let X a MC wth state sace E { 23 45 6 7 8} P 4 3 3 6 4 5 5 8 2 4 6 4 4 2 3 6 64
, recurret belogg to the same rreducble closed set, recurret belogg to dfferet rreducble closed sets recurret traset recurret traset F 472 2 2 2 2 6 6 2 traset, recurret, traset F ( ) R( ) R( ) F ( ) R( ) oe (reachable) recurret class ad ftely may traset states 65
Examle: 5 5 8 2 4 6 ˆ P P 5 3 2 2 2 4 2 2 3 2 4 Thus, 7 5 25 S ( I Q) ad F 2 6 5 75 2 2 8 8 8 5 25 5.2 8 3 7 G S B 5 75 2 2 4.6 3 4 4 6 6 6 3 7 66
Recurret states ad Lmtg robabltes Cosder oly a rreducble set of states. Theorem: Suose X s rreducble ad aerodc. The all states are recurret oull f ad oly f ( ) ( ) P( ) E ( ) E has a soluto. If there exsts a soluto, the t s strctly ostve, there are o other solutos, ad we have E ( ) lm P ( ) E Corollary: If X a rreducble aerodc MC wth ftely may states (o-ull states, o traset states), the P has a uque soluto. The soluto s strctly ostve, ad ( ) lm P ( ),. 67
A robablty dstrbuto whch satsfes dstrbuto for X. P, s called a varat If s the tal dstrbuto of X, that s, P{ X } ( ), E the P{ X } ( ) P ( ) ( ), for ay E Proof: P P P 2 Algorthm: for fdg lm P ( ) Cosder the rreducble closed set cotag Solve for ( ). Thus, we fd lm P ( ) For every (ot ecessarly E) lm P ( ) F( )lm P ( ) Comute F( ) frst. The, fd lm P ( ) 68
Examle: E={, 2, 3}, P 3 5 2 6 4 4 () () 3 (2)6 P (2) () 5 (3)4 (3) () 2 (2)4 (3)6 System s Soluto: 6 7 23 23 23 6 7 6 7 P lm P ( ) 23 23 23 23 23 23 6 7 23 23 23 69
Examle: E={, 2, 3, 4, 5, 6, 7}, P 2 8 7 3 3 5 2 6 4 4 6 2 2 3 2 4 3 5 2 2 8 7 8 6 7 P P2 6 4 2 7 3 5 5 23 23 23 4 6 F(6) F(65) 2 2 8 8 8 F(7) F(75) 4 4 6 6 6 7
Thus, P 7 8 5 5 7 8 5 5 6 7 23 23 23 6 7 lm P 23 23 23 6 7 23 23 23 4 6 48 56 8 5 5 23 23 23 28 32 36 42 6 5 5 23 23 23 7
Examle: Radom walks: q q q q P q ( X rreducble aerodc (sce state s aerodc)) 2 2 2q 2 q 2 q q 2 2 3q q q 2 2 3 q q q q 3 2 3 q If If q: o soluto of P, q: lm P ( ) q q 72
Examle: Remag lfetme P 2 3 Thus, 2 2 2 3 3 2 2 2 3 2 3 3 2 2 3 3 m ( ) ( ) ( ) m E[ Z ] s the exected lfetme. If m the all states are recurret ull ad lm P ( ) 73
Iterretato of Lmtg Probabltes Proosto: Let be a aerodc recurret o-ull state, ad let m( ) be the exected tme betwee two returs to. The, ( ) lm P ( ) m( ) The lmtg robablty ( ) of beg state s equal to the rate at whch s vsted. Proosto: Let be a aerodc recurret o-ull ad let ( ) defed as revously. The, for almost all lm ( Xm( )) ( ). If f s a bouded fucto o E, the m f ( X ) f ( ) ( X ) m m m E m Corollary: X rreducble recurret MC, wth lmtg robablty. The, for ay bouded fucto f o E : lm f ( X m) f f ( ) f ( ) m E 74
Smlar results hold for exectatos Corollary: Suose X s a rreducble recurret MC wth lmtg dstrbuto. The for ay bouded fucto f o E lm E[ f ( X m)] f m deedet of. If f( ) s the reward receved wheever X s, the both the exected average reward the log ru ad the actual average reward the log ru coverge to the costat f. The rato of the total reward receved durg the stes by usg fucto f to the corresodg amout by usg fucto g s lm m m f( X ) m gx ( ) m f g The same holds eve the case that X s oly recurret (ca be ull or erodc or both) 75
Theorem: Let X be a rreducble recurret cha wth trasto matrx P. The, the system P has a strctly ostve soluto; ay other soluto s a costat multle of that oe. Theorem: Suose X s rreducble recurret, ad let be a soluto of P. The for ay two fuctos f ad g o E for whch the two sums f ( ) f ( ) g ( ) g( ) E coverge absolutely ad at least oe s ot zero we have lm m m deedetly of E. Moreover we also have for almost all lm m m E [ f ( X )] m E [ g( X )] m f( X ( )) m gx ( ( )) m E f g f g 76
Ay o-egatve soluto of P s called a varat measure of X. Commets: Ay rreducble recurret cha X has a varat measure, ad ths s uque u to a multlcato by a costat. Furthermore, f X s also o-ull, the ( ) s fte, ad s a costat multle of the lmtg dstrbuto satsfyg P, The exstece of a varat measure for X does ot mly that X s recurret. For f, g ad k lm E m k X m k E m m ( ) ( ) ( X ) ( ) ( k ) s the rato of the exected umber of vsts to k durg the frst stes to ( ) the exected umber of returs to durg the same erod as ( k ) s the exected umber of vsts to k betwee two vsts to state ( ) 77
Perodc States It s suffcet to cosder oly a rreducble MC wth erodc recurret states. Lemma: Let X be a rreducble MC wth recurret erodc states wth erod. The, the states ca be dvded to dsot sets B, B 2 B such that P( ) uless B B or B B or B B. 2 2 3 78
Examle: X MC wth E { 23 45 6 7} 2 4 4 2 3 3 2 3 3 P 2 2 3 4 4 2 2 3 4 4 All states are erodc wth erod 3. The sets are B { 2}, B2 {3 4 5} ad B3 {6 7}. >From B oe ste the MC reaches B 2, two stes B 3 ad three stes B. 79
P Note: 23 25 7 2 48 48 92 92 7 29 43 8 8 72 72 2 4 3 9 3 3 36 36 36 3 5 9 3 49 P 8 8 48 32 96 7 9 3 7 3 6 6 32 64 64 5 57 3 2 8 24 288 288 3 9 8 8 6 6 92 92 P 2 3 3 P P P P2 Cha corresodg to P has three closed sets B, B 2, B 3 ad each oe of these s rreducble, recurret ad aerodc. The revous lmtg theory ales to comute lm P m, m lm P m, m 2 lm P m searately. m 3 P 3 8
Theorem: Let P the trasto matrx of a rreducble MC wth recurret erodc states of erod, ad let B, B 2 B be as revously. The, the MC wth trasto matrx P P, the classes B, B 2 B are rreducble closed sets of aerodc states. P Commets: If a P 2 P Pa( ) P ( ) Ba P B, the P{ X B } b a m(mod ) m b P ( ) does ot have a lmt as excet whe all the states are ull ( whch case P ( ) ) m The lmts P ( ) exst as, but are deedet o the tal state. 8
Theorem: Let P ad B a as revously ad suose that the cha s o-ull. The, for ay m { } m ( ) Ba Bb b a m(mod ) lm P ( ) otherwse The robabltes ( ), E form the uque soluto of ( ) ( ) P( ) ( ) E E 82
5 5 4 6 Examle: Let X be a MC wth state sace E { 23 4 5}, P 8 2 The cha s rreducble, recurret o-ull erodc wth erod 2. P 2 4 5 32 6 8 P 4 6 4 6 32 6 8 4 6 2 32 6 8 32 6 8 2 lm P 32 6 8 4 6 4 6 32 6 8 32 6 8 4 6 4 6 2 lm P 4 6 83
Examle: Radom Walks ( q) q P q Cyclc Classes B { 2 4 }, B 2 { 3 5 } Ivarat soluto P 2 2 4 2 q q 2 4 3 3 5 5 q q q Normalze: Multly each term by. ( 2 2 2 2 q q q q q q q ) 2 4 3 q 2 4 q q 2 4 3 5 q q q q 3 5 84
Hece, 3 2 4 q q 2 3 q q 2 3 lm P q 2 4 q q 2 3 q q 2 3 q q 3 2 4 q q 2 2 lm P q 3 q q q q 3 2 4 85
Traset States If a MC has oly ftely may traset states, the t wll evetually leave the set of traset states ever to retur. If there are ftely may traset states, t s ossble for the cha to rema the set of traset states forever. Examle: P 2 3 2 All states are traset If tal state s, the the cha stays forever the set { 2 }. As, X ( ) 86
Let A E, Q the matrx obtaed from P by deletg all the rows ad colums corresodg to states whch are ot A. The, for A Q ( ) Q ( ) Q ( 2) Q ( ) A A PX A X A X Q ( ) PX A X A X A A The evet { X A X A} s a subset of { X A X A}, therefore Q ( ) Q ( ) A A Let f ( ) lm Q ( ) A A f() s the robablty that startg at A, the cha stays the set A forever. 87
Proosto: The fucto f s the maxmal soluto of the system h Qh h Ether f or su f ( A ) A alcato of the revous roosto was gve a theorem o the classfcato of states: Theorem: Let X a rreducble MC wth trasto matrx P, ad let Q be the matrx obtaed from P by deletg the k -row ad k -colum for some k E. The all states are recurret f ad oly f the oly soluto of h( ) Q( ) h( ) h( ) E E s h ( ) for all E. E E {} k. Proof: Fx a ertcular state ad ame t. Sce X s rreducble t s ossble to go from to some A E {}. If the robablty f() of remag A forever s f( ) for all A, the wth robablty, the cha wll leave A ad eter aga. Hece, f the oly soluto of the system s h, the state s recurret, ad that tur mles that all states are recurret. Coversely, f all states are recurret, the the robablty of remag the set A forever must be zero, sce wll be reached wth robablty oe from ay state A 88
Examle: (Radom Walk) Q q q If q all states are traset. q f ( ) 23 Ths s the maxmal soluto sce su f( ). Iterretato: Startg at a state k (e.g. k 7 ) the robablty of stayg forever wth the set q s equal to 7 { 2 3 }. If k k, the robablty of remag { 23 } s greater. From the shae of P : the restrcto of P to the set { kk } s the same as the matrx Q. Hece, for all k { 23 } q Pk X k X 2 k 89
For ay subset A of E, let fa() the robablty of remag forever A gve the tal state A. The, If A s a rreducble recurret class, f A. If A s a roer subset of a rreducble recurret class, f A. If A s a fte set of traset states, f A. If A s a fte set of traset states, the ether f A or f A. I the latter case the cha travels through a sequece of sets ( A A2 A3 ) to fte. 9