Discrete-Time Markov Chains

Markov Processes ad Alicatios Discrete-Time Markov Chais Cotiuous-Time Markov Chais Alicatios Queuig theory Performace aalysis ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ Books Discrete-Time Markov Chais - Itroductio to Stochastic Processes (Erha Cilar, Cha. 5, 6 - Itroductio to Probability Models (Sheldo Ross, Cha. 4 - Performace Aalysis of Commuicatios Networks ad Systems (Piet Va Mieghem, Cha. 9, - Markov Chais (J.R. Norris, Cha. - Discrete Stochastic Processes (R. Gallager, Cha. 4 - Elemetary Probability for Alicatios (Rick Durrett, Cha. 5 - Itroductio to Probability, D. Bertsekas & J. Tsitsiklis, Cha. 6 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

INTRODUCTION : th order df of some stoc. roc. { X t } is give by f( x, x,..., x f( x x, x,..., x f( x x, x,..., x t t t t t t t t t t t... f( x x f( x t t t very difficult to have it i geeral If { } is a ide. rocess: X t f ( x, x,..., x f ( x f ( x... f ( x t t t t t t If { X } is a rocess with ide. icremets: t f( x, x,..., x ( (... ( t t t f x f x t t x f x t t x t Note : First order df's are sufficiet for above secial cases If { X } is a rocess whose evolutio beyod t is (robabilistically t comletely determied by x ad is ide. of x, t < t, give x, the: f( x, x,..., x f( x x... f( x t t t t t t t t t x f( x t t This is a Markov rocess ( th order df simlified ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ Defiitio of a Markov Process (MP A stoch. roc.{ X ; t I } that takes values from a set E is called a Markov Process (MP iff or t f ( xt x,..., ( t x t P x t x t : ( E coutable f ( xt x,..., ( ( ucoutable t x t f x t x t E for all x ad all t < t <... < t ad all >. t Notice : The "ext" state xt is ide.of the "ast"{ x,..., } t x t rovided that the "reset"is kow. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4

Defiitio of a Markov Chai (MC (Discrete - time & discrete - value MP If I is coutable ad E is coutable the a MP is called a MC ad is described dby the trasitio robabilities biliti : ( i, P{ X + X i} i, E (ide. of for a time - homogeeous MC. Assume E {,,,...}(state,, -sace of the MC Trasitio matrix : P(, P(,... P(, P(, P(,... P(, P M M M P(, P(,... P(, M M M P is o - egative, P( i,........., i (stochastic matrix For a give gve P (stoch. matrix a MC may be costructed ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5 Chai rule : If π is a PMF o E st s.t. π ( i P { X P{ X i, X i, X N, i, i,..., i k - ste trasitios : k N, P{ X + k X i} P i, E, k N ; of the trasitio matrix P. Proof : For k P{ X P k i,..., X k E ( i, i }, i E, the i } π ( i P( i, i... P( i ( i, is the ( i, etry of the kth ower (geeral through iteratios + X i} P( i, l P( l, l P( l, l E l E 44 444 P ( l, 4444 44444 P ( i,, i ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6

Chama Kolmogorov Equatios : From revious, m+ m P (, i P (, i k P ( k, i, E k E I order for { X } to be i after m+ stes ad startig from i, it will have to be i some k after m stes ad move the to i the remaiig stes. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 siklis ekas & Tsits Bertse ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8

Examle : # of successes i Beroulli rocess { N ; }, N # of successes i trials N Y,, Y ide. Beroulli, P{ Y } i i i i Notice: N N + Y evolutio of { N } beyod + + does ot dee d o { Ni} i (give N ad thus { N} is a M.C. PN { N, N,..., N } PY { N N, N,..., N } + + q... if N +... if ad q q N P q... otherwise M Notice: { N } is a secial M.C. whose icremet is ide. both from reset ad ast (rocess with ide. icremets ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 9 Examle : Sum of i.i.d. RV's with PMF { k; k,,,...} X Y + Y +... + Y X+ X + Y+ PX { + X,..., X} PY { + X X,..., X} Thus { X } is a M.C. with P( i, PX { X i} X + i...... P...... M M M M O ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

Examle : Ideedet trials X, X,...i.i.d. iid with π ( k, k,,,... P{ X X,..., X } P{ X + } {X }is a M.C. π ( π ( L π ( π ( L P M M π ( π ( L M M O + π Notice that rows are idetical ad P m ( P m (If P has all rows idetical the X, X,...are iid i.i.d. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ Examle :{ Y }are iid i.i.d. Y X {,,,,4} + X + Y + (modulo 5, { X with { }is a M.C.,,,, } 4 P 4 4 4 4 4 rows (stoch. matrix colums (here (double - stochastic matrix ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

Examle : Remaiig lifetime A equimet is relaced by a idetical as soo as it fails Pr{a ew equi. lasts for k time uits} k,,... k X remaiig lifetime of equi. at time X ( ω if X( ω X + ( ω Z+ ( ω if X( ω Z+ ( ω is the lifetime of equi. istalled at time It is ideedet of X, X,...,, X is a M.C. X ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ i : Pi (, P { X X i } P { X X i } + if i PX { + X i} if i i : P(, P{ X X } P{ Z X } + + PZ { + } + + P 4 L L L L M M M M O ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4

Theorem : (coditioal ide. of future from ast give reset Let Y be a bouded fuctio of X, X +,.... The E{Y X, X,..., X } E{Y X } Proositio : E{ f(x,x +,... X i} E{ f(x,x,... X i} Corol lary : f a bouded fuctio o E E... Let g(i E{ f(x,x,...x i}. The N E{ f(x,x,...x, X,..., X } g(x + ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5 Theorem : (coditioal ide. of future from ast give reset Let Y be a bouded fuctio of X, X,.... The E{Y X, X,...,, X } E{Y X } + ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6

Proositio : E{ f(x,x +,... X i} E{ f(x,x,... X i} ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 Corollary : f a bouded fuctio o E E... Let g(i E{ f(x,x,... X i}. The N E{ f(x,x,...x, X,..., X } g(x + ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8

Stoig Times : Previous results derived for fixed time N What if time is a RV istead? If for a RV T, the ast { X are coditioally ide.give reset X roerty is said to hold at T. If T m ; m T}ad the future{ X is a stoig time, the above hold true ( T T m ; m T}, the the strog Markov is a stoig time if the evet { T }ca be determied by lookig at X, X,..., X ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 9 For ay stoig time T : E{ f( X, X,... X, T} E{ f( X, X,... X } T T+ T T+ T For gi ( E { f ( X, X,... X i } E{ f( X, X,... X ; T} g( X T T+ T if am eg e.g., if f( a, a,... E, m N if am E { f ( X, X,... X i } P { X X i } P m (, i f m E{ f( X, X,... X, T} P{ X X ; T} T T+ T+ m Strog Markov roerty at T: m PX { T+ m X; T } P ( X T, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

Visits to a state X { X ; N } MC, State sace E, Trasitio matrix P. Notatio: Pi { A} P{ A X i } ad EY i[ ] EY [ X i ] Let E, ω Ω ad Defie: N ( ω total umber of times state aears i X ( ω, X ( ω,. N ( ω <, X evetually leaves state ever to retur. N ( ω, X visits agai ad agai. Let T( ω, T ( ω, the successive idices for which X ( ω. If / the T ( ω T ( ω T ( ω L If aears a fiite umber of times m, the T + ( ω T ( ω T + ( ω T ( ω L m m m m N, { Tm ( ω } is equivalet to aears i { X ( ω, L, X ( ω } at least m times. T m is a stoig time. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ Examle T ( ω 4, T ( ω 6, T ( ω 7, T ( ω 9, 4 E ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

Proositio: i E, k, m { Tm } PT i { m+ Tm kt T,, m} P{ T k} { Tm < } Comutatio of P{ T k }. Let Fk( i, PT i{ k } k Fk ( i, Pi{ T } Pi{ X } Pi (, k Fk ( i, Pi{ X, L, X k, X k } Pi { X b } Pi { X, L, Xk, Xk X b } b E { } L b E { } i b k k P{ X b} P{ X,, X, X } Thus, Pi (, k F (, k i PibF (, k ( b, k b E { } ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ Examle: Let a the trasitio matrix Fid f ( i F ( i,, i,, k k P / /6 / / /5 /5 k. I this case f is the rd colum of matrix P. Hece, f( F(,, f( F(, /, f( F(, / 5 (, (, P b F b k F (, b E { } k k. I this case fk F (, (, (, P b F b Q f k k k where Q b E { } / /6 F (, / /5 k P(, b F ( b, k b E { } After some algebra f / f /8 f /8 f 4 / 648 /5 / 5 / /8 L ad i geeral k k 5 Fk(,, Fk(,, Fk(, k 6 k 5 6 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4

k k 5 Fk(,, Fk(,, F k(, k 6 k 5 6 Now we ca state: Startig at state, X ever visits with robability: PT { + } Startig at state, X first visits at k with robability: ( k Startig at state, X ever visits with robability: { } { } ( k P T + P T <+ k 6 5 Startig at state, X ever visits agai with robability: 5 P{ T + } P{ T <+ } 75 6 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5 Now, for every i, we defie F( i, P{ T <+ } F ( i, i k k F( i, exresses the robability: startig at i the MC will ever visit state. If by F ( i, P ( i, + P ( i, b F ( b,, i E b E { } N we deote the total umber of visits to state, the m { } F(, F(, P N m ( ad for F ( i, m i, Pi{ N m} m F( i, F(, ( F(, m,, >From the revious we obtai the Corollary: F(, < P{ N <+ } F (, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6

Now, Pfor { Nevery mi}, we defie m F( i, Pi{ T <+ } Fk( i, m F( ( F( k,, m m F( i, exresses the robability: startig at i the MC x will ever, visit x < state. m ( F(, x F(, F( i, P( i, + P( i, b F( b,, i E b E { } If by N we deote the total umber of visits to state, the m P{ N m } F(, ( F(, ad for F ( i, m i, Pi{ N m} m F( i, F(, ( F(, m,, >From the revious we obtai the Corollary: F(, < P{ N <+ } F (, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 Now, for every i, we defie mp { N m } m F( i, Pi{ T <+ } Fk( i, m mf(, ( F(, k m F( i, exresses the robability: startig at i the MC will ever visit state. ( ( m F(, mx, x < F(, ( F(, m ( x F( i, P( i, + P( i, b F( b,, i E b E { } If by N we deote the total umber of visits to state, the m P{ N m } F(, ( F(, If F(, N + w... Therefore, if X E[ N ] + If F (, < the N follows geometric distributio F ( i, with robability mof success ad for i, Pi{ N m} m F(,. Hece, E ( i, F(, ( F(, m,, [ N ] F(, >From the revious we obtai the Corollary: F(, < P{ N <+ } F (, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8

Let Ri (, E i [ N ] ( R is called the otetial matrix of X The, R(, F (, Ri (, F(, i R(,, (i Ri (, Fi (, R(, + ( Fi (, Comutatio of Ri (, first ad the F( i, Defie:, k, X( ω ( k ( X( ω, k, X( ω The, N ( ω Ri (, ( ( ω X i i i E ( X E ( X P{ X } P ( i, I matrix otatio: from which we obtai R I + P + P + L RP PR P + P + L R I RI ( P ( I PR I ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 9 Classificatio of states X: MC, with state sace E, trasitio matrix P T : The time of first visit to state N : The total umber of visits to state Defiitio State is called recurret if P { T < } State is called trasiet if P{ T } > A recurret state is called ull if E[ T ] A recurret state is called o-ull if E [ T ]< A recurret state is called eriodic with eriod δ, if δ is the greatest iteger for which P{ T δ forsome } ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

If is recurret the startig at the robability of returig to is. F(, R(, E [ N ] + P{ N + } If is trasiet the there exists a ositive robability F(, returig to. of ever F(, < R(, E [ N ] < P{ N < } I this case Ri (, Fi (, R (, < R (, < ad sice Ri (, P ( i, we coclude that lim P ( i, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ Theorem: If trasiet or recurret ull the i E, lim P ( i, If recurret o-ull the π( lim P (, > ad i E, lim P ( i, Fi (, π( If eriodic with eriod δ, the a retutr to is ossible oly at stes umbered δ, δ, δ,... P (, P{ X } > olyif {, δ, δ, } ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ

Recurret o-ull Recurret ull Trasiet P{ T < } P{ T } > E [ ]< T E[ T ] F(, R(, E [ N ] + P{ N + } π ( lim P (, > ad i E, lim P ( i, F( i, π ( F(, < R(, E [ N ] < P{ N < } i E, P i, lim ( A recurret state is called eriodic with eriodδ, if δ is the greatest iteger for which P{ T δ forsome } If eriodic with eriod δ, the a retur to is ossible oly at stes umbered δ, δ, δ,... P (, P { X } > olyif {, δ, δ, } ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ We say that state ca be reached from state i i, if : P ( i, > i, iff F( i, > Defiitio: A set of states is closed if o state outside it ca be reached from ay state i it. A state formig a closed set by itself is called a absorbig state A closed set is called irreducible if o roer subset of it is closed. A MC is called irreducible if its oly closed set is the set of all states Commets: If is absorbig the P(,. If MC is irreducible the all states ca be reached from each other. If C { c, c, L} E is a closed set ad Qi (, Pc ( i, c, ci, c C, the Q is a Markov matrix. If i ad k the i k. To fid the closed set C that cotais i we work as follows: Startig with i we iclude i C all states that ca be reached from i : Pi (, >. We ext iclude i C all states k that ca be reached from : P(, k >. We reeat the revious ste ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4

Examle: MC with state sace E { abcde,,,, } ad trasitio matrix 4 4 P 4 4 Commets: Closed sets: { ace,, } ad { abcde,,,, } There are two closed sets. Thus, the MC is ot irreducible. c e ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5 Examle: MC with state sace E { a, b, c, d, e } ad trasitio matrix Commets: Closed sets: { ace,, } ad { abcde,,,, } There are two closed sets. Thus, the MC is ot irreducible. 4 4 If we delete the d ad 4 th rows we obtai the P Markov matrix: 4 4 Q If we relabel the states a, c, e, 4 b ad 5 d we get P 4 4 4 4 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6

Lemma If recurret ad k k. Thus, Fk (,. Proof: If k the k is reached without returig to with robability a. Oce k is reached, the robability that is ever visited agai is F( k,. Hece, F(, a( F( k, But is recurret, so that F (, F ( k, As a result: If k but k /, the must be trasiet. Theorem: From recurret states oly recurret states ca be reached. Theorem: I a Marcov chai the recurret states ca be divided i a uique maer, ito irreducible closed sets C, C,,, ad after a aroriate arragemet: P L P L P P L L L L O M Q Q Q L Q ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 Theorem: Let X a irreducible MC. The, oe of the followig holds: All states are trasiet. All states are recurret ull All states are recurret o-ull Either all aeriodic or if oe is eriodic with eriod δ, all are eriodic with the same eriod. Proof: Sice X is irreducible the k ad k, which meas that, rs: r s P (, k > ad P ( k, >. Pick the smallest rs, ad let β r s P (, kp ( k,. If k recurret recurret. If k trasiet trasiet. (If it was recurret the k would be recurret m If k recurret ull the P ( k, k as m. But + r+ s P ( k, k β P (, P (, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8

Corollary: If C irreducible closed set of fiitely i may states, the / recurret ull states. Proof: If oe is recurret ull the all states are recurret ull. Thus, lim P ( i,, i, C. But, i C,, P ( i, lim P ( i, C C Because, we have fiite umber of states lim P ( i, lim P ( i, C C Corollary: If C is a irreducible closed set with fiitely may states the there are o trasiet states ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 9 Algorithm - Fiite umber of states Idetify irreducible closed sets. All states belogig to a irreducible closed set are recurret ositive The rest of the states are trasiet Periodicity is checked to each irreducible set ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4

Examle: The irreducible closed sets are {, }, {, 7, 9} ad {6}. The states {4, 5, 8, } are trasiet. t If we relabel lthe states t we obtai 4 4 P 4 4 4 4 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4 Examle: Let see Thus, N the umber of successes i the first Beroulli trials. As we have i+ P( i, P{ N+ N i} q i otherwise q L q L P q L M M M O we have + but + /. This meas that is ot recurret. Sice the MC is irreducible all states are trasiet. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4

Examle: Remaiig lifetime X( ω X( ω Remember: X + ( ω Z ( ω ( ω + X from which we obtai: i i P( i, PX { + X } PX { X } i P(, P{ X X } P{ Z X } i + + P{ Z + } + + L L P L L M M M O ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 4 L L P L L M M M O >From state we reach state i oe ste. From we ca reach,,...,,.thus,allstates states ca be reached from each other, which meas that the MC is irreducible. Sice, P (, > the MC is aeriodic. Retur to state occurs if the lifetime is fiite: F (, Sice state is recurret, all states are recurret. If the exected lifetime: + the state is ull ad all states are recurret ull. If the exected lifetime: < the state is o-ull ull ad all states are recurret o-ull. ull ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 44

Algorithm - Ifiite umber of states Theorem: Let X a irreducible MC, ad cosider the system of liear equatios: ν ( ν ( i P ( i,, E The all states are recurret o-ull iff there exists a solutio ν with ν ( i E E Theorem: Let X a irreducible MC with trasitio matrix P, ad let Q be the matrix obtaied from P by deletig the k -row ad k -colum for some k E. The all states are recurret if ad oly if the oly solutio of hi ( Qi (, h (, hi (, i E E is hi ( for all i E. E E {} k. Use first theorem to determie whether all states are recurret o-ull or ot. I the latter case, use the secod theorem to determie whether the states t are trasiet or ot. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 45 Examle: Radom walks. L q L P q L M M M M O All states ca be reached from each other, ad thus the chai is irreducible. A retur to state ca occur oly at stes umbered,4,6,... Therefore, state is eriodic with eriod δ. Sice X is irreducible all states are eriodic with eriod. Either all states are recurret ull, or all are recurret o-ull, or all the states are trasiet. Check for a solutio of ν νpν P. ν q ν ν ν + qν ν ν + q ν ν ν + q ν4 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 46

Hece, Ay solutio is of the form q, the / < If < q ad ν ν q ν ν ν ν q q q ν ν ν q( q q q ν ν,,, q q q ν + ν ν q q q If we choose ν q the ν q ad, q ν (, q q q I this case all states are recurret o ull ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 47 If > q either all states are recurret ull or all states are trasiet. Cosider the matrix L q L Q q L M M M M O The equatio h Qh gives ( hi h( i i i q q q hi + + + L + + h If q the hi ih for all i ad the oly way to have h i for all i is by choosig h which imlies h that is all states are recurret ull. If > q, the choosig h ( q/, we get i q h i which also satisfies h. I this case all states are trasiet. i i ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 48

Calculatio of R ad F Ri (, E[ N] Exected umber of visits to state. i F( i, The robability of ever reachig state startig at i. Recurret state: F(, R(, Fi (, Ri (, FiR (, (, Ri (, + Fi (, > Trasiet / i Recurret state: Fi (, Ri (, i, Trasiet Let D { the trasiet states }, Qi (, Pi (,, Si (, Ri (,, i, D. The m K m K P P L Q Lm Q m Hece, L m K m m + + + m L m m m Q R P S Q I Q Q ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 49 Comutatio of S S I + Q+ Q + L SQ QS Q + Q + L S I ( I Q S I, SI ( Q I Proositio: If there are fiitely may trasiet states S ( I Q Whe the set D of trasiet states is ifiite, it is ossible to have more tha oe solutio to the system. Theorem: S is the miimal solutio of ( I Q Y I, Y Theorem: S is the uique solutio of ( I Q Y I if ad oly if the oly bouded solutio of h Qh is h, or equivaletly h Qh, h h ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5

Examle: Let X a MC with state sace E {45678},,,,,,, 4..... 6. 4 5. 5.... P 8..... 4.. 6. 4 4......... 6... {},, are recurret ositive aeriodic. {4, 5} are recurret ositive aeriodic. {6, 78}, are trasiet 4. 6.. 6. 6.. Q... S ( I Q... 6... 6... ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5 recurret, ca be reached from i trasiet, i recurret recurret, caot be reached from i, i trasiet recurret trasiet i recurret R 5 75 5 66 66 66 i trasiet ta se t 5 75 5 S 66 66 66 75 45 75 66 66 66 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5

Comutatio of F( i, i, recurret belogig gto the same irreducible closed set Fi (, i, recurret belogig gto differet irreducible closed sets Fi (, i, trasiet The Ri (, < ad Ri (, F(,, F( i, R(, R(, i trasiet, recurret???? Lemma: If C is irreducible closed set of recurret states, the for ay trasiet state i : F( i, F( i, k for all, k C. Proof: For k, C F( k, Fk (,. Thus, oce the chai reaches ay oe of the states of C, it also visits all the other states. Hece, F( i, F( i, k is the robability of eterig the set C from i. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 5 Let Lum all states of C together to make oe absorbig state: P P P P Pˆ, b ( i ( P i, k, i D k C O O Q Q Q Q b b b bm Q L The robability bili of ever reachig the absorbig bi state from the trasiet state i by the chai with the trasitio matrix ˆP is the same as that of ever reachig ˆ I P, B b L bm, B ( i, P ( i k ı D,, B Q k C C from i. I Pˆ, B ( I + Q+ Q + + Q B L B Q B ( i, is the robability that startig from i, the chai eters the recurret class C ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 54

Defie: k k G lim B Q B SB Gi (, is the robability of ever reachig the set C from the trasiet state i : ( F( i, Proositio: Let Q the matrix obtaied from P by deletig all the rows ad colums corresodig to the recurret states, ad let B be defied as reviously, for each trasiet t i ad recurret class C. Comute S Comute G SB Gi (, Fik (,, k C. If there is oly oe recurret class ad fiitely may trasiet states, the thigs are differet. I this case, it ca be roved that: G F( i,, C ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 55 Examle: Let X a MC with state sace E {45678},,,,,,, P 4.... 6. 4. 5. 5.... 8..... 4. 6.. 4. 4........ 6... ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 56

i, recurret belogig to the same irreducible closed set i, recurret belogig to differet irreducible closed sets recurret trasiet i recurret i trasiet F trasiet, i recurret, i trasiet 47... F(,,... R(, 6. 6.. Ri (, F ( i, R(, oe (reachable recurret class ad fiitely it may trasiet t states t ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 57 Examle: 5. 5. 8... 4. 6. ˆ P... P. 5........ 4.............. 4. Thus, 7.. 5. 5. S ( I Q ad F. 6. 5. 75... 8. 8. 8. 5. 5.. 5.. 8. 7 G S B 5. 75... 4..6 4. 4. 6. 6. 6. 7 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 58

Recurret states ad Limitig robabilities Cosider oly a irreducible set of states. Theorem: Suose X is irreducible ad aeriodic. The all states are recurret oull if ad oly if π( π( i P( i,, E, π( i E E has a solutio π. If there exists a solutio π, the it is strictly ositive, there are o other solutios, ad we have π ( lim P ( i,, i, E Corollary: If X i a irreducible aeriodic MC with fiitely may states (o-ull ull states, o trasiet states, the π P π, π has a uique solutio. The solutio π is strictly ositive, ad π ( lim P ( i,,, i. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 59 A robability distributio π which satisfies distributio for X. π π P, is called a ivariat If π is the iitial distributio of X, that is, P{ X } π (, E the P{ X } π( ip( i, π(, for ay E Proof: π π P π P L π P Algorithm: for fidig lim P ( i, Cosider the irreducible closed set cotaiig Solve for π (. Thus, we fid lim P (, For every i (ot ecessarily i E i lim P ( i, Fi (, lim P (, Comute F( i, first. The, fid lim P ( i, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6

Examle: E{,, },. 5.. P 6.. 4.. 4.. π ( π(. + π(. 6 π P π π ( π (. 5 + + π (. 4 π( π(. + π(. 4 + π(. 6 π System s Solutio: 6 7 6 7 6 7 π P lim P ( i, 6 7 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6 Examle:. 8. 7... 5.. E{,,, 4, 5, 6, 7}, P 6.. 4.. 4. 6.............. 4.. 5... 8. 7 8 6 7 P π P 6 4 π 7, 5 5...... 4.. 6 F(6, L F(65,.. 8. 8. 8. F(7 F(75 4 4 6 6 6, L,..... ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6

Thus, P 7 8 5 5 7 8 5 5 6 7 6 7 lim P 6 7 4. 6. 48. 56. 8.. 5 5 8.. 6. 4. 6.. 5 5 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 6 Examle: q L ( X irreducible aeriodic (sice state is q Radom walks: P L aeriodic q L M M M M O π q π πq+ πq π q π q π π π + π / q q π L π π+ πq q q M M π / q q q q M M If q : o solutio of π π P, π If < q : lim P ( i, ( q( q ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 64

Examle: Remaiig lifetime L P L L M M M O Thus, π π + π ν π π π + + π ν π π + + π ν M M M M ν ( + + + L + ( + + L + ( + L + L + + + L m m E[ Z ] is the exected lifetime. If m the all states are recurret ull ad lim P ( i, ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 65 Iterretatio of Limitig Probabilities Proositio: Let be a aeriodic recurret o-ull state, ad let m( be the exected time betwee two returs to. The, π ( lim P (, m( The limitig robability π ( of beig i state is equal to the rate at which is visited. Proositio: Let be a aeriodic recurret o-ull ad let π ( defied as reviously. The, for almost all ω Ω lim ( X m( ω π( + m. If f is a bouded fuctio o E, the f ( X f( ( X m m m E m Corollary: X irreducible ibl recurret MC, with limitig iti robability bilit π. The, for ay bouded fuctio f o E : lim f ( Xm π f, π f π ( f ( + m E ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 66

Similar results hold for exectatios Corollary: Suose X is a irreducible recurret MC with limitig distributio π. The for ay bouded fuctio f o E lim Ei[ f( Xm] π f + m ideedet of i. If f ( is the reward received wheever X is i, the both the exected average reward i the log ru ad the actual average reward i the log ru coverge to the costat π f. The ratio of the total reward received durig the stes,,, by usig fuctio f to the corresodig amout by usig fuctio g is lim f( X m m π g( X π m m The same holds eve i the case that X is oly recurret (ca be ull or eriodic or both f g ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 67 Theorem: Lt Let X be a irreducible ibl recurret chai with trasitio matrix P. The, the system ν ν P has a strictly ositive solutio; ay other solutio is a costat multile of that oe. Theorem: Suose X is irreducible recurret, ad let ν be a solutio of ν ν P. The for ay two fuctios f ad g o E for which the two sums ν f ν ( i f ( i, ν g ν ( i g ( i i E coverge absolutely ad at least oe is ot zero we have m lim m ideedetly of i, E. Moreover we also have for almost all ω Ω lim m m i E Ei [ f ( Xm ] f E[ g( X ] ν ν g i m f( Xm( ω ν g( X ( ω ν m f g ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 68

Ay o-egative solutio of ν ν P is called a ivariat measure of X. Commets: Ay irreducible recurret chai X has a ivariat measure, ad this is uique u to a multilicatio by a costat. Furthermore, if X is also o-ull, the ν ν ( is fiite, ad ν is a costat multile of the limitig distributio π satisfyig π P π, π The existece of a ivariat measure ν for X does ot imly that X is recurret. For f k, g ad i E ( k X m m ( lim ν k E ( ( Xm ν m ν ( k is the ratio of the exected umber of visits to k durig the first stes to ν ( the exected umber of returs to durig the same eriod as ν ( k is the exected umber of visits to k betwee two visits to state ν ( ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 69 Periodic States It is sufficiet to cosider oly a irreducible MC with eriodic recurret states. Lemma: Let X be a irreducible MC with recurret eriodic states with eriod δ. The, the states ca be divided ito δ disoit sets B, B,, B δ such that P( i, uless i B, B, or i B, B, or L i B, B. δ ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7

Examle: X MC with E {4567},,,,,, 4 4 P 4 4 4 4 All states are eriodic with eriod. The sets are B {, }, B {45},, ad B {6, 7}. >From B i oe ste the MC reaches B, i two stes B ad di three stes B. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 P Note: 5 7 48 48 9 9 7 9 4 8 8 7 7 4 9 6 6 6 5 9 49, P 8 8 48 96 7 9 7 6 6 64 64 5 57 8 4 88 88 9 8 8 6 6 9 9 P P P, P P Chai corresodig to P has three closed sets B, B, B ad each oe of these is irreducible, recurret ad aeriodic. The revious limitig theory alies to comute lim P m, m lim P m, m lim P m searately. m ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 P

Theorem: LtP Let P the trasitio matrix of a irreducible ibl MC with recurret eriodic states of eriod δ, ad let B, B, B δ be as reviously. The, i the MC with trasitio matrix P P δ, the classes B, B, B δ are irreducible closed sets of aeriodic states. P P δ P, Pa( i, P ( i,, i, Ba O P Commets: If a δ i B, the P{ X B }, b a+ m(mod δ i m b P ( i, does ot have a limit as excet whe all the states are ull (i which case P ( i,, i,, δ + m The limits P ( i, exist as, but are deedet o the iitial state i. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 7 Theorem: Let P ad Ba as reviously ad suose that the chai is o-ull. The, for ay m {,, δ, } δ + m π ( i Ba, Bb, b a+ m(mod δ lim P ( i, otherwise The robabilities π (, E form the uique solutio of π ( π ( i P ( i,, π ( i δ i E i E ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 74

5. 5. 4. 6. Examle: Let X be a MC with state sace E {45},,,,, P.. 8...... The chai is irreducible, recurret o-ull eriodic with eriod δ. P 4. 5... 6. 8. P... 4. 6. 4. 6. ( 6 8 π ( 4 6 π...,... 6. 8. 4. 6.. 6. 8. 4. 6. lim P. 6. 8. lim P + 4. 6. 4. 6.. 6. 8. 4. 6.. 6. 8. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 75 Examle: Radom Walks ( < q q P q O O Cyclic Classes B {4,,, }, B {,, 5, } Ivariat solutio ν ν P ν, ν, ν 4, q q 4 ν, ν, ν 5, 5 q q q Normalize: ν + q + q + q + L + q q Multily l each term by. ( ν i q q,,, ( (,, 4, q q q 4,,, ( (, q q, 5, q q ( π i π π 4 ( π π π 5 ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 76

Hece, L 4 q q L q q lim P q L 4 q q L q q M M M M M O L q q L 4 q q + lim P q L q q L 4 q q M M M M M O ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 77 Trasiet States If a MC has oly fiitely may trasiet states, the it will evetually leave the set of trasiet states ever to retur. If there are ifiitely may trasiet states, it is ossible for the chai to remai i the set of trasiet states forever. Examle: L L P L L M M M M O All states are trasiet If iitial state is i, the the chai stays forever i the set { ii, +, i +, }. As, X ( ω ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 78

Let A E, Q the matrix obtaied from P by deletig all the rows ad colums corresodig to states which are ot i A. The, for i, A Q ( i, L Qii (, Qi (, i L Qi (, P { X A, L, X A, X } i A i A Q ( i, P { X A, L, X A X A } i, A i The evet { X A X,, + A} is a subset of { X A X,, A}, therefore Q ( i, Q + ( i, A A Let f ( i lim Q ( i,, i A A f ( i is the robability that startig at i A, the chai stays i the set A forever. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 79 Proositio: The fuctio f is the maximal solutio of the system h Qh, h Either f or su f ( i i A A alicatio of the revious roositio was give i a theorem o the classificatio of states: Theorem: Let X a irreducible MC with trasitio matrix P, ad let Q be the matrix obtaied from P by deletig the k -row ad k -colum for some k E. The all states are recurret if ad oly if the oly solutio of hi ( Qi (, h (, hi (, i E E is hi ( for all i E. E E {} k. Proof: Fix a erticular state ad ame it. Sice X is irreducible it is ossible to go from to some i A E {}. If the robability f ( i of remaiig i A forever is f( i for all i A, the with robability, the chai will leave A ad eter agai. Hece, if the oly solutio of the system is h, the state is recurret, ad that i tur imlies that all states are recurret. Coversely, if all states are recurret, the the robability of remaiig i the set A forever must be zero, sice will be reached with robability oe from ay state i A ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8

Examle: (Radom Walk q Q q O If > q all states are trasiet. q f ( i, i,,, This is the maximal solutio sice su i f ( i. Iterretatio: Startig at a state k (e.g. k 7 the robability of stayig forever withi the set q 7 {,,, } is equal to (. If k > k, the robability of remaiig i {,,, } is greater. From the shae of P : the restrictio of P to the set { kk, +, } is the same as the matrix Q. Hece, for all k {,,, } i q P k + i { X k, X k, } ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8 i+ For ay subset A of E, let f A ( i the robability of remaiig forever i A give the iitial state i A. The, If A is a irreducible recurret class, f A. If A is a roer subset of a irreducible recurret class, f A. If A is a fiite set of trasiet states, f A. If A is a ifiite set of trasiet states, the either f A or f A. I the latter case the chai travels through a sequece of sets ( A A AL to ifiite. ΠΜΣ54: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ 8