Markov Processes and Applications

Transcript

1 Markov rocesses ad Applcatos Dscrete-Tme Markov Chas Cotuous-Tme Markov Chas Applcatos Queug theory erformace aalyss ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ)

2 Dscrete-Tme Markov Chas Books - Itroducto to Stochastc rocesses (Erha Clar), Chap. 5, 6 - Itroducto to robablty Models (Sheldo Ross), Chap. 4 - erformace Aalyss of Commucatos Networks ad Systems (et Va Meghem), Chap. 9, - Elemetary robablty for Applcatos (Rck Durrett), Chap. 5 ( - Itroducto to robablty, D. Bertsekas & J. Tstskls, Chap. 6 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 2

3 INTRODUCTION : th order pdf of some stoc. proc. { X } s gve by f( x, x,..., x ) f( x x, x,..., x ) f( x x, x,..., x )... f( x x ) f( x ) very dffcult to have t geeral If { X If { X } s t t t t t t t t t t t t t a dep. process: t t t f( x, x,..., x ) f( x ) f( x )... f( x ) t t t t t t t 2 } s a process wth dep. cremets: f( x, x,..., x ) f( x ) f( x x )... f( x x ) t t t t t t t t 2 2 Note : Frst order pdf's are suffcet for above specal cases If { X t } s a process whose evoluto beyod t s (probablstcally) completely determed by x ad s dep. of x, t < t, gve x, the: f( x, x,..., x ) f( x x )... f( x t t t t t t 2 2 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 3 t t t x ) f( x ) Ths s a Markov process ( th order pdf smplfed) t t

4 for all Defto of a Markov rocess (M) A stoch. proc.{ X a Markov rocess (M) ff : f ( xt or f ( x x t t x x t t t ad all t ; t I} that takes values from a set E s called,..., x,..., x < t t t ) ) 2 ( x t f ( x t <... < t x x t t ) ( E ) ( E ad all >. coutable) ucoutable) Notce :The "ext"state x t s dep.of provded that the "preset"s kow. the "past"{ x t,..., x t 2 } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 4

5 Defto of a Markov Cha (MC) (Dscrete - tme & dscrete - value M) If I s coutable ad E s coutable the a M s called a MC ad s descrbed by the trasto probabltes : p(, ) { X + X }, E (dep. of for a tme - homogeeous MC). Assume E {,,2,...}(state -space of the MC) Trasto matrx : (,) (,) M (,) M (,) (,) (,) s o - egatve, For a gve M M (, ) (stoch. matrx) a (, ) (, ) M (, ) M, (stochastc matrx) MC may be costructed ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 5

6 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) ), ( ), ( ), ( ), ( ), ( } { through teratos) (geeral 3 : For roof. the trasto matrx of th power the etry of ), the ( s ), ( ;,, ), ( } {,,...,,, ), ( )..., ( ) ( },...,,, { the, }, { ) ( s.t. MF o s a If E l l E l k k k l l l l X X k k k E X X k E N X X X X E X E + + N N k - step trastos : Cha rule : π π π

7 Chapma Kolmogorov Equatos : From prevous, m+ m (, ) (, k) ( k, ), E k E I order for { X } to be after m+ steps ad startg from, t wll have to be some k after m steps ad move the to the remag steps. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 7

8 Example : # of successes Beroull process { N ; }, N # of successes trals N Y,, Y dep. Beroull, { Y } p Notce: N N + Y evoluto of { N } beyod + + does ot deped o { N } (gve N ) ad thus { N } s a M.C. N { N, N,..., N} Y { N N, N,..., N} + + q p... p f N +... f ad q p q p N q p... otherwse M Notce: { N } s a specal M.C. whose cremet s dep. both from preset ad past (process wth dep. cremets) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 8

9 Example : Sum of..d. RV's wth MF { p ; k,,2,...} X Y + Y Y k X X + Y + + X { X,..., X} Y { X X,..., X} p + + X Thus { X } s a M.C. wth (, ) { X X } p + p p p2 p3... p p p2... p p... p... M M M M O ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 9

10 Example : Idepedet trals X { X { X (If, X + }s a π () π () M π () M,...d. wth π ( k) X M.C. π () π () M π () M,..., }, { X + Notce that rows are detcal ad X has all rows detcal the L L L O k X,,2,..., } π ( ) m X m,... are..d.) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ)

11 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) (double - stochastc matrx) (here) colums matrx) (stoch. rows M.C. }s a {, (modulo 5) },,,, wth { {,,2,3,4} }are..d. :{ p p p p p p p p p p p p p p p p p p p p p p p p p X Y X X p p p p p Y Y Example

12 Example : Remag lfetme A equpmet s replaced by a detcal as soo as t fals p r{a ew equp. lasts for k tme uts} k,2,... k X X Z + + remag lfetme of equp. at tme X( ω) f X( ω) ( ω) Z+ ( ω) f X( ω) ( ω) s the lfetme of equp. stalled at tme It s depedet of X, X,..., X X s a M.C. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 2

13 : (, ) { X X } { X X } + f X { + X } f : (, ) { X X } { Z X } + + Z { + } p + + p p2 p3 p4 L L L L M M M M O ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 3

14 Theorem : (codtoal dep. of future from past gve preset) Let Y be a bouded fucto of X, X,.... The E{Y X, X,..., X } E{Y X } + roposto : E{ f(x,x,...)x } E{ f(x,x,...)x } + Corollary : f a bouded fucto o E E... Let g() E{ f(x,x,...)x }. The N E{ f(x,x,...)x, X,..., X } g(x ) + ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 4

15 Stoppg Tmes : revous results derved for fxed tme What f tme s a RV stead? N If are codtoally dep.gve preset property s sad to hold at T. If T f for a RV T s a stoppg tme, the above hold true ( T the evet { T, the past { X m ; m T}ad the future{ X }ca be determed by lookg at X T, m ; m the the strog Markov s a stoppg tme X, X T},..., X ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 5

16 For ay stoppg tme T : E{ f( X, X,...) X, T} E{ f( X, X,...) X } T T+ T T+ T For g ( ) E{ f( X, X,...) X } E{ f( X, X,...) X ; T} g( X ) T T+ T f am e.g., f f ( a, a,...) E, m N f am E{ f( X, X m,...) X } { X X } (, ) m E{ f( X, X,...) X, T} { X X ; T} T T+ T+ m Strog Markov property at T: m X { X; T} ( X, ) T+ m T ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 6

17 Vsts to a state X { X ; N } MC, State space E, Trasto matrx. Notato: { A} { A X } ad EY [ ] EY [ X ] Let E, ω Ω ad Defe: N ( ω) total umber of tmes state appears X( ω), X ( ω),. N ( ω ) <, X evetually leaves state ever to retur. N ( ω ), X vsts aga ad aga. Let T( ω), T 2( ω), the successve dces for whch X( ω ). If / the T( ω) T2( ω) T ( ω) L If appears a fte umber of tmes m, the T ( ω) T ( ω) T 2( ω) T ( ω) L m+ m m+ m N, { T ( ω) } s equvalet to appears { X( ω), L, X ( ω)} at least m tmes. T m s a stoppg tme. m ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 7

18 Example T( ω) 4, T ( ω) 6, T ( ω) 7, T ( ω) 9, E ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 8

19 roposto: E, k, m { Tm } T { m+ Tm kt T,, m} { T k} { Tm < } Computato of { T k }. Let Fk(, ) T { k } k Fk (, ) { T } { X } (, ) k 2 Fk (, ) { X, L, X k, X k } { X b} { X, L, X, X X b } b E { } 2 k k { { } } {, L, 2, } X b b E b X X k X k Thus, (, ) k F (, ) k bf (, ) k ( b, ) k 2 b E { } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 9

20 Example: Let 3 a the trasto matrx Fd f () F (, ), 23,, k k /2 /6 /3 /3 3/5 /5 k. I ths case f s the 3rd colum of matrx. Hece, f() F(, ), f(2) F(2, ) / 3, f(3) F(3, ) / 5 k 2. I ths case F (, ) k b E { } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 2 k k k b E { } b E { } (, b) F ( b, ) f F (2, ) (2, b) F ( b, ) Q f where k F (3, ) k (3, b) F ( b, ) k k Q /2 /6 /3 3/5 After some algebra f /3 f2 /8 f3 /8 f 4 /648 /5 /5 /3 /8 L ad geeral k k 5 Fk(, 3), Fk(2, 3), Fk(3, 3) k k

21 k k 5 Fk(, 3), Fk(2, 3), Fk(3, 3) k k Now we ca state: Startg at state, X ever vsts 3 wth probablty: T { + } Startg at state 2, X frst vsts 3 at k wth probablty: () k Startg at state 2, X ever vsts 3 wth probablty: { } { } ( ) k <+ k T T Startg at state 3, X ever vsts 3 aga wth probablty: 52 { T + } { T <+ } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 2

22 Now, for every, we defe F(, ) T { <+ } F(, ) k k F(, ) expresses the probablty: startg at the MC wll ever vst state. F(, ) (, ) + bfb (, ) (, ), E b E { } If by N we deote the total umber of vsts to state, the m { } F(, ) F(, ) N m ( ) ad for, { N m} F (, ) m ( ) m F ( F ) ( ) F( ) m 2,,,,, >From the prevous we obta the Corollary: F(, ) < { N <+ } F(, ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 22

23 Now, for every, we defe F(, ) T { <+ } F(, ) k k F(, ) expresses the probablty: startg at the MC wll ever vst state. F(, ) (, ) + bfb (, ) (, ), E b E { } If by N we deote the total umber of vsts to state, the m { } F(, ) F(, ) N m ( ) ad for, { N m} F (, ) m m F(, ) F(, ) ( F(, ) ) m, 2, >From the prevous we obta the Corollary: F(, ) < { N <+ } F(, ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 23

24 Now, for every, we defe F(, ) T { <+ } F(, ) k k F(, ) expresses the probablty: startg at the MC wll ever vst state. F(, ) (, ) + bfb (, ) (, ), E b E { } If by N we deote the total umber of vsts to state, the m { } F(, ) F(, ) N m ( ) ad for, { N m} F (, ) m m F(, ) F(, ) ( F(, ) ) m, 2, >From the prevous we obta the Corollary: F(, ) < { N <+ } F(, ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 24

25 Let R (, ) E [ N ] ( R s called the potetal matrx of X ) The, R(, ) F (, ) R (, ) F(, ) R( ),, ( ) R(, ) F (, ) R(, ) + ( F (, )) Computato of R (, ) frst ad the F(, ) Defe:, k, X( ω) ( k) ( X( ω)), k, X( ω) The, N ( ω ) R (, ) ( ( ω )) X E ( X ) E ( X ) { X } (, ) I matrx otato: from whch we obta 2 2 R I L R R + + L R I RI ( ) ( I R ) I ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 25

26 Classfcato of states X: MC, wth state space E, trasto matrx T : The tme of frst vst to state N : The total umber of vsts to state Defto State s called recurret f { T < } State s called traset f { T } > A recurret state s called ull f E[ T ] A recurret state s called o-ull f E[ T ]< A recurret state s called perodc wth perod δ, f δ 2 teger for whch { T δ forsome } s the greatest ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 26

27 If s recurret the startg at the probablty of returg to s. F(, ) R(, ) E [ N ] + { N + } If s traset the there exsts a postve probablty F(, ) returg to. of ever F(, ) < R(, ) E [ N ] < { N < } I ths case R (, ) F (, R ) (, ) < R(, ) < ad sce R (, ) (, ) we coclude that lm (, ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 27

28 Theorem: If traset or recurret ull the E, lm (, ) If recurret o-ull the π( ) lm (, ) > ad E, lm (, ) F (, ) π( ) If perodc wth perod δ, the a retutr to s possble oly at steps umbered δ, 2δ, 3δ,... (, ) { X } > olyf {, δ, 2 δ, } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 28

29 Recurret o-ull Recurret ull Traset { T < } { T } > E [ ]< T E[ T ] F(, ) R(, ) E [ N ] + { N + } π ( ) lm (, ) > ad E, lm (, ) F(, ) π ( ) F(, ) < R(, ) E [ N ] < { N < } E, lm (, ) A recurret state s called perodc wth perodδ, f δ 2 s the greatest teger for whch { T δ for some } If perodc wth perod δ, the a retur to s possble oly at steps umbered δ, 2δ, 3δ,... (, ) { X } > oly f {, δ, 2 δ, } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 29

30 We say that state ca be reached from state, f : (, ) >, ff F (, ) > Defto: A set of states s closed f o state outsde t ca be reached from ay state t. A state formg a closed set by tself s called a absorbg state A closed set s called rreducble f o proper subset of t s closed. A MC s called rreducble f ts oly closed set s the set of all states Commets: If s absorbg the (, ). If MC s rreducble the all states ca be reached from each other. If C { c, c2, L} E s a closed set ad Q (, ) c (, c ), c, c C, the Q s a Markov matrx. If ad k the k. To fd the closed set C that cotas we work as follows: Startg wth we clude C all states that ca be reached from : (, ) >. We ext clude C all states k that ca be reached from : (, k ) >. We repeat the prevous step ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 3

31 Example: MC wth state space E { abcde,,,, } ad trasto matrx Commets: Closed sets: { ace,, } ad { abcde,,,, } There are two closed sets. Thus, the MC s ot rreducble. c e ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 3

32 Example: MC wth state space E { a, b, c, d, e } ad trasto matrx Commets: 2 2 Closed sets: { ace,, } ad { abcde,,,, } Q There are two closed sets. Thus, the MC s ot rreducble. If we delete the 2 d ad 4 th rows we obta the Markov matrx: If we r elabel the states a, 2 c, 3 e, 4 b ad 5 d we get ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 32

33 Lemma If recurret ad k k. Thus, Fk (, ). roof: If k the k s reached wthout returg to wth probablty a. Oce k s reached, the probablty that s ever vsted aga s F( k, ). Hece, F(, ) a( F( k, )) But s recurret, so that F(, ) F( k, ) As a result: If k but k /, the must be traset. Theorem: From recurret states oly recurret states ca be reached. Theorem: I a Marcov cha the recurret states ca be dvded a uque maer, to rreducble closed sets C, C 2,, ad after a approprate arragemet: L 2 L L 3 L L L O M Q Q2 Q3 L Q ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 33

34 Theorem: Let X a rreducble MC. The, oe of the followg holds: All states are traset. All states are recurret ull All states are recurret o-ull Ether all aperodc or f oe s perodc wth perod δ, all are perodc wth the same perod. roof: Sce X s rreducble the k ad k, whch meas that rs, : r s (, k ) > ad ( k, ) >. ck the smallest rs, ad let β r s (, k ) ( k, ). If k recurret recurret. If k traset traset. (If t was recurret the k would be recurret) m If k recurret ull the ( k, k ) as m. But + r+ s ( k, k) β (, ) (, ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 34

35 Corollary: If C rreducble closed set of ftely may states, the / recurret ull states. roof: If oe s recurret ull the all states are recurret ull. Thus, lm (, ),, C. But, C,, (, ) lm (, ) C Because, we have fte umber of states lm (, ) lm (, ) C Corollary: If C s a rreducble closed set wth ftely may states the there are o traset states C C ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 35

36 Algorthm - Fte umber of states Idetfy rreducble closed sets. All states belogg to a rreducble closed set are recurret postve The rest of the states are traset erodcty s checked to each rreducble set ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 36

37 Example: The rreducble closed sets are {, 3}, {2, 7, 9} ad {6}. The states {4, 5, 8, } are traset. If we relabel the states we obta ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 37

38 Example: Let see N the umber of successes the frst Beroull trals. As we have p + (, ) { N+ N } q otherwse Thus, q p L q p L q L M M M O we have + but + /. Ths meas that s ot recurret. Sce the MC s rreducble all states are traset. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 38

39 Example: Remag lfetme X( ω) X( ω) Remember: X + ( ω) Z+ ( ω) X( ω) from whch we obta: (, ) { X+ X } { X X } (, ) { X X } { Z X } + + { Z + } p + + p p2 p3 L L L L M M M O ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 39

40 p p2 p3 L L L L M M M O >From state we reach state oe step. From we ca reach, 2,...,,. Thus, all states ca be reached from each other, whch meas that the MC s rreducble. Sce, (, ) > the MC s aperodc. Retur to state occurs f the lfetme s fte: p F(, ) p Sce state s recurret, all states are recurret. If the expected lfetme: p + the state s ull ad all states are recurret ull. If the expected lfetme: p < the state s o-ull ad all states are recurret o-ull. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 4

41 Algorthm - Ifte umber of states Theorem: Let X a rreducble MC, ad cosder the system of lear equatos: ν ( ) ν ( ) (, ), E The all states are recurret o-ull ff there exsts a soluto ν wth ν ( ) E E Theorem: Let X a rreducble MC wth trasto matrx, ad let Q be the matrx obtaed from by deletg the k -row ad k -colum for some k E. The all states are recurret f ad oly f the oly soluto of h ( ) Q (, h ) ( ), h ( ), E E s h () for all E. E E {} k. Use frst theorem to determe whether all states are recurret o-ull or ot. I the latter case, use the secod theorem to determe whether the states are traset or ot. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 4

42 Example: Radom walks. L q p L q p L M M M M O All states ca be reached from each other, ad thus the cha s rreducble. A retur to state ca occur oly at steps umbered 2,4,6,... Therefore, state s perodc wth perod δ 2. Sce X s rreducble all states are perodc wth perod 2. Ether all states are recurret ull, or all are recurret o-ull, or all the states are traset. Check for a soluto of ν ν. ν q ν ν ν + q ν 2 ν 2 pν + q ν 3 ν 3 pν 2 + q ν 4 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 42

43 Hece, Ay soluto s of the form p q, the / < If < pq ad ν ν ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 43 ν ν ν p 2 ν 2 q q q ν 3 ( ) 2 p 2 p ν p 3ν q q q q p ν ν, 2,, q q p 2q ν + ν ν q q q p If we choose ν q p the ν 2q ad p, 2 q ν ( ) p p, 2q q q I ths case all states are recurret o ull q

44 If p > q ether all states are recurret ull or all states are traset. Cosder the matrx p L q p L Q q p L M M M M O The equato h Qh gves ( h h() ) h q q q + + L + + h p p p + If p q the h h for all ad the oly way to have h for all s by choosg h whch mples h that s all states are recurret ull. If p > q, the choosg h ( q/ p ), we get q h p whch also satsfes h. I ths case all states are traset. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 44

45 Calculato of R ad F R (, ) E[ N] Expected umber of vsts to state. F(, ) The probablty of ever reachg state startg at. Recurret state: F(, ) R(, ) R (, ) F(, ) R(, ) R (, ) F (, ) + F (, ) > Traset / Recurret state: F (, ) R (, ), Traset Let D { the traset states }, Q (, ) (, ), S (, ) R (, ),, D. The m K m K L Q Lm Q m Hece, m K m m m L m m m Q R S Q I Q Q ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 45 L

46 Computato of S S 2 I + Q+ Q + L 2 SQ QS Q+ Q + L S I ( I Q) S I, SI ( Q) I roposto: If there are ftely may traset states S ( I Q) Whe the set D of traset states s fte, t s possble to have more tha oe soluto to the system. Theorem: S s the mmal soluto of ( I Q) Y I, Y Theorem: S s the uque soluto of ( I Q) Y I f ad oly f the oly bouded soluto of h Qh s h, or equvaletly h Qh, h h ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 46

47 Example: Let X a MC wth state space E { },,,,,,, {23},, are recurret postve aperodc. {4, 5} are recurret postve aperodc. {6, 78}, are traset Q.. 2. S ( I Q) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 47

48 recurret, ca be reached from traset, recurret recurret, caot be reached from, traset recurret traset recurret traset R S ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 48

49 Computato of F(, ), recurret belogg to the same rreducble closed set F (, ), recurret belogg to dfferet rreducble closed sets F (, ), traset The R (, ) < ad R (, ) F(, ), F(, ) R(, ) R(, ) traset, recurret???? Lemma: If C s rreducble closed set of recurret states, the for ay traset state : F(, ) F(, k) for all, k C. roof: For k, C F( k, ) Fk (, ). Thus, oce the cha reaches ay oe of the states of C, t also vsts all the other states. Hece, F(, ) F(, k) s the probablty of eterg the set C from. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 49

50 Let 2 3 O Q Q2 Q3 Q Lump all states of C together to make oe absorbg state: ˆ, b () (, k), D O k C b b2 b3 bm Q L The probablty of ever reachg the absorbg state from the traset state by the cha wth the trasto matrx ˆ s the same as that of ever reachg ˆ I, B b L bm, B(, ) ( k) ı D,, B Q k C C from. I 2 ˆ, ( L + ) B Q B I Q Q Q B B ( ), s the probablty that startg from, the cha eters the recurret class C ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 5

51 Defe: k k G lm B Q B SB G (, ) s the probablty of ever reachg the set C from the traset state : ( F(, ) ) roposto: Let Q the matrx obtaed from by deletg all the rows ad colums correspodg to the recurret states, ad let B be defed as prevously, for each traset ad recurret class C. Compute S Compute G SB G (, ) Fk (, ), k C. If there s oly oe recurret class ad ftely may traset states, the thgs are dfferet. I ths case, t ca be proved that: G F(, ), C ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 5

52 Example: Let X a MC wth state space E { },,,,,,, ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 52

53 , recurret belogg to the same rreducble closed set, recurret belogg to dfferet rreducble closed sets recurret traset recurret traset F traset, recurret, traset F(, ), R(, ) R (, ) F(, ) R(, ) oe (reachable) recurret class ad ftely may traset states ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 53

54 Example: ˆ Thus, S ( I Q) ad F G S B ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 54

55 Recurret states ad Lmtg probabltes Cosder oly a rreducble set of states. Theorem: Suppose X s rreducble ad aperodc. The all states are recurret oull f ad oly f π( ) π( ) (, ), E, π( ) E E has a soluto π. If there exsts a soluto π, the t s strctly postve, there are o other solutos, ad we have π ( ) lm (, ),, E Corollary: If X a rreducble aperodc MC wth ftely may states (o-ull states, o traset states), the π π, π has a uque soluto. The soluto π s strctly postve, ad π ( ) lm (, ),,. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 55

56 A probablty dstrbuto π whch satsfes dstrbuto for X. π π, s called a varat If π s the tal dstrbuto of X, that s, { X } π ( ), E the { X } π() (, ) π( ), for ay E roof: π π π L π 2 Algorthm: for fdg lm (, ) Cosder the rreducble closed set cotag Solve for π ( ). Thus, we fd lm (, ) For every (ot ecessarly E) lm (, ) F (, ) lm (, ) Compute F(, ) frst. The, fd lm ( ), ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 56

57 Example: E{, 2, 3}, π () π(). 3 + π(2). 6 π π π(2) π() π(3). 4 π π(3) π(). 2 + π(2). 4 + π(3). 6 System s Soluto: π lm ( ), ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 57

58 Example: E{, 2, 3, 4, 5, 6, 7}, π π2 7 3, F(6), L F(65), F(7), F(75), L ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 58

59 Thus, lm ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 59

60 Example: q p L ( X rreducble aperodc (sce state s q p Radom walks: L aperodc)) q p L M M M M O p π q π πq+ πq 2 p p π 2 p 2q π 2 p q π π π + π / 2 p p q q π 2 π2 πp+ π3q L q q p p p M M π 3 / q 2 3 q q q M M If p If p q: o soluto of π π, π p p < q : lm (, ) ( q)( q) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 6

61 Example: Remag lfetme p p2 p3 L L L M M M O Thus, π πp+ π ν π π πp2 + + π 2 ν p π π p + + π ν p p M M M M ν p p2 p3 p2 p3 p3 p + 2p 2 + 3p 3 + L m ( L) + ( + + L) + ( + L) + L m E[ Z ] s the expected lfetme. If m the all states are recurret ull ad lm (, ) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 6

62 Iterpretato of Lmtg robabltes roposto: Let be a aperodc recurret o-ull state, ad let m( ) be the expected tme betwee two returs to. The, π ( ) lm (, ) m( ) The lmtg probablty π ( ) of beg state s equal to the rate at whch s vsted. roposto: Let be a aperodc recurret o-ull ad let π ( ) defed as prevously. The, for almost all ω Ω lm ( X m( ω)) π( ) +. If f s a bouded fucto o E, the m f ( X ) f( ) ( X ) m m m E m Corollary: X rreducble recurret MC, wth lmtg probablty π. The, for ay bouded fucto f o E : lm f ( Xm) π f, π f π( ) f( ) + m E ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 62

63 Smlar results hold for expectatos Corollary: Suppose X s a rreducble recurret MC wth lmtg dstrbuto π. The for ay bouded fucto f o E lm E[ f ( X m)] π f + m depedet of. If f( ) s the reward receved wheever X s, the both the expected average reward the log ru ad the actual average reward the log ru coverge to the costat π f. The rato of the total reward receved durg the steps,,, by usg fucto f to the correspodg amout by usg fucto g s lm m m f( X ) m g( X ) m π f π g The same holds eve the case that X s oly recurret (ca be ull or perodc or both) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 63

64 Theorem: Let X be a rreducble recurret cha wth trasto matrx. The, the system ν ν has a strctly postve soluto; ay other soluto s a costat multple of that oe. Theorem: Suppose X s rreducble recurret, ad let ν be a soluto of ν ν. The for ay two fuctos f ad g o E for whch the two sums ν f ν() f(), ν g ν() g() E coverge absolutely ad at least oe s ot zero we have lm m m depedetly of, E. Moreover we also have for almost all ω Ω lm m m E[ f( X )] m E[ g( X )] m f( X ( ω)) m g( X ( ω)) m E ν f ν g ν f ν g ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 64

65 Ay o-egatve soluto of ν ν s called a varat measure of X. Commets: Ay rreducble recurret cha X has a varat measure, ad ths s uque up to a multplcato by a costat. Furthermore, f X s also o-ull, the ν ν ( ) s fte, ad ν s a costat multple of the lmtg dstrbuto π satsfyg π π, π The exstece of a varat measure ν for X does ot mply that X recurret. For f, g ad k lm ( ) ( ) ( X ) ν ( ) E m k X m ν k E m m ν ( k ) s the rato of the expected umber of vsts to k durg the frst steps to ν ( ) the expected umber of returs to durg the same perod as ν ( k ) s the expected umber of vsts to k betwee two vsts to state ν ( ) s ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 65

66 erodc States It s suffcet to cosder oly a rreducble MC wth perodc recurret states. Lemma: Let X be a rreducble MC wth recurret perodc states wth perod δ. The, the states ca be dvded to δ dsot sets B, B 2,, B δ such that (, ) uless B, B, or B, B, or L B, B δ ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 66

67 Example: X MC wth E {234567},,,,,, All states are perodc wth perod 3. The sets are B {, 2}, B 2 {345},, ad B 3 {6, 7}. >From B oe step the MC reaches B 2, two steps B 3 ad three steps B. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 67

68 Note: , , 2 Cha correspodg to has three closed sets B, B 2, B 3 ad each oe of these s rreducble, recurret ad aperodc. The prevous lmtg theory apples to compute lm m, m lm m, m 2 lm m separately. m 3 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 68 3

69 Theorem: Let the trasto matrx of a rreducble MC wth recurret perodc states of perod δ, ad let B, B 2, B δ be as prevously. The, the MC wth trasto matrx δ, the classes B, B 2, B δ are rreducble closed sets of aperodc states. Commets: If a 2 δ, a(, ) (, ),, Ba O δ B, the { X B }, b a+ m(mod δ ) m b (, ) does ot have a lmt as except whe all the states are ull ( whch case (, ),,, ) δ + m The lmts (, ) exst as, but are depedet o the tal state. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 69

70 Theorem: Let ad B a as prevously ad suppose that the cha s o-ull. The, for ay m {,, δ, } δ + m π ( ) Ba, Bb, b a+ m(mod δ ) lm (, ) otherwse The probabltes π ( ), E form the uque soluto of π ( ) π() (, ), π() δ E E ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 7

71 Example: Let X be a MC wth state space E {2345},,,,, The cha s rreducble, recurret o-ull perodc wth perod δ ( ) π ( 4 6) π..., lm lm ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 7

72 Example: Radom Walks ( p < q ) q p q p O O Cyclc Classes B {24,,, }, B 2 {, 3, 5, } Ivarat soluto ν ν p p ν, ν2, ν 2 4, 2 q q 2 4 p p ν, ν3, ν 3 5, 5 q q q Normalze: ν L Multply each term by. ( ν 2 2 p p q q q q p p q q p q 3 p p p,,, ( )(, q 2, 4, q q ) 2 4 p p p,,, ( )(, q q 3, 5, q q ) ( π ) π2 π4 ) ( π π π ) 3 5 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 72

73 Hece, 3 p p L 2 4 q q 2 p L 3 q q 2 p 3 lm p p q L 2 4 q q 2 p L 3 q q M M M M M O 2 p L 3 q q 3 p p L 2 4 q q 2+ p 2 lm p q L 3 q q 3 p p L 2 4 q q M M M M M O ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 73

74 Traset States If a MC has oly ftely may traset states, the t wll evetually leave the set of traset states ever to retur. If there are ftely may traset states, t s possble for the cha to rema the set of traset states forever. Example: p p p p p p p p p 2 3 L L 2 L p L M M M M O All states are traset If tal state s, the the cha stays forever the set {, +, + 2, }. As, X ( ω) ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 74

75 Let A E, Q the matrx obtaed from by deletg all the rows ad colums correspodg to states whch are ot A. The, for, A Q (, ) L Q (, ) Q (, ) L Q (, ) { X A, L, X A, X } A A 2 Q (, ) { X A, L, X A X A}, A The evet { X A X,, + A} s a subset of { X A X,, A}, therefore Q (, ) Q + (, ) A A Let f () lm Q (, ), A A f () s the probablty that startg at A, the cha stays the set A forever. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 75

76 roposto: The fucto f s the maxmal soluto of the system h Qh, h Ether f or sup f ( A ) A applcato of the prevous proposto was gve a theorem o the classfcato of states: Theorem: Let X a rreducble MC wth trasto matrx, ad let Q be the matrx obtaed from by deletg the k -row ad k -colum for some k E. The all states are recurret f ad oly f the oly soluto of h () Q (, h ) ( ), h (), E E s h () for all E. E E {} k. roof: Fx a pertcular state ad ame t. Sce X s rreducble t s possble to go from to some A E {}. If the probablty f () of remag A forever s f() for all A, the wth probablty, the cha wll leave A ad eter aga. Hece, f the oly soluto of the system s h, the state s recurret, ad that tur mples that all states are recurret. Coversely, f all states are recurret, the the probablty of remag the set A forever must be zero, sce wll be reached wth probablty oe from ay state A ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 76

77 Example: (Radom Walk) Q p q p q p O If p > q all states are traset. q f (), 23,,, p Ths s the maxmal soluto sce sup f( ). Iterpretato: Startg at a state k (e.g. k 7 ) the probablty of stayg forever wth the set q,,, s equal to ( ) 7 {23 }. p If k > k, the probablty of remag {23,,, } s greater. From the shape of : the restrcto of to the set { kk, +, } s the same as the matrx Q. Hece, for all k {23,,, } ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 77 q k+ { X k, X2 k, } p +

78 For ay subset A of E, let f () the probablty of remag forever A gve the A tal state A. The, If A s a rreducble recurret class, f A. If A s a proper subset of a rreducble recurret class, f A. If A s a fte set of traset states, f A. If A s a fte set of traset states, the ether f A or f A. I the latter case the cha travels through a sequece of sets ( A A2 A3L) to fte. ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης - ΕΚΠΑ) 78