Stationary ARMA Processes

University of Pavia 2007 Stationary ARMA Processes Eduardo Rossi University of Pavia

Moving Average of order 1 (MA(1)) Y t = µ + ǫ t + θǫ t 1 t = 1,...,T ǫ t WN(0, σ 2 ) E(ǫ t ) = 0 E(ǫ 2 t) = σ 2 E(ǫ t ǫ t j ) = 0 j 0 E(Y t ) = E(µ + ǫ t + θǫ t 1 ) E(Y t ) = µ + E(ǫ t ) + θe(ǫ t 1 ) E(Y t ) = µ Eduardo Rossi c - Macroeconometria 2

Moving Average of order 1 (MA(1)) E(Y t µ) 2 = E(ǫ t + θǫ t 1 ) 2 = E(ǫ 2 t + θ 2 ǫ 2 t 1 + 2θǫ t 1 ǫ t ) = σ 2 + θ 2 σ 2 + 0 = (1 + θ 2 )σ 2 First Autocovariance γ(1) = E(Y t µ)(y t 1 µ) = E(ǫ + θǫ t 1 )(ǫ t 1 + θǫ t 2 ) = E(ǫ t ǫ t 1 + θǫ 2 t 1 + θǫ t ǫ t 2 + θ 2 ǫ t 1 ǫ t 2 ) = 0 + θσ 2 + 0 + 0 = θσ 2 Higher Autocovariances are all zero γ(j) = E[(Y t µ)(y t j µ)] = 0 Eduardo Rossi c - Macroeconometria 3

Moving Average of order 1 (MA(1)) MA(1) is covariance stationary regardless the value of θ. γ(j) = (1 + θ 2 )σ 2 + θσ 2 < j=0 If ǫ t is a Gaussian White Noise, then MA(1) is ergodic for all moments. Autocorrelation function ρ(j) γ(j) γ(0) ρ(j) 1 ρ(1) = ρ(j) = 0 j > 0 θσ 2 (1 + θ 2 )σ 2 = θ 1 + θ 2 Eduardo Rossi c - Macroeconometria 4

Moving Average of order 1 (MA(1)) The largest possible value for ρ(1) is 0.5. This occurs if θ = 1. The smallest is 0.5, θ = 1. For 0.5 < ρ(1) < 0.5 there are two different values of θ that could produce that autocorrelation. θ 1+θ 2 is unchanged if θ is replaced by 1/θ. Eduardo Rossi c - Macroeconometria 5

Invertibility MA(1): Y t = µ + ǫ t + θ 1 ǫ t 1 Y t = µ + (1 + θ 1 L)ǫ t Autocorrelation function: ρ(1) γ(1) γ(0) = θ 1 + θ 2 Replacing θ by 1/θ and assuming that the (unobserved) shock process has a variance of θ 2 σ 2 instead of σ 2 yields a process with the same autocovariance structure as the original process. The invertibility of (1 + θ 1 z) depends on the roots of 1 + θ 1 z = 0 Eduardo Rossi c - Macroeconometria 6

Invertibility invertibility requires θ < 1; if θ 1 the infinite sequence (1 θl + θ 2 L 2 θ 3 L 3 +...) would not be well defined. For a MA(q) there are 2 q representations of the process having the same correlogram. Identification problem. To overcome this problem we impose the invertibility condition. AR( ) representation: θ(l) 1 Y t = θ(1) 1 µ + ǫ t Eduardo Rossi c - Macroeconometria 7

Moving average of order q (MA(q)) Y t = µ + θ(l)ǫ t t = 1,...,T ǫ t WN(0, σ 2 ) θ(l) = 1 + θ 1 L +... + θ q L q E(Y t ) = µ E[(Y t µ) 2 ] = E[(θ(L)ǫ t ) 2 ] = E(ǫ 2 t + θ1ǫ 2 2 t 1 +... + θqǫ 2 2 t q) = (1 + θ1 2 +... + θq)σ 2 2 Eduardo Rossi c - Macroeconometria 8

Moving average of order q (MA(q)) because E(θ i θ j ǫ t i ǫ t j ) = 0 i j i, j = 0,...,q θ 0 = 1 The autocovariance function γ(j) = E[(θ(L)ǫ t )(θ(l)ǫ t j )] = E[(ǫ t +... + θ j ǫ t j +... + θ q ǫ t q )(ǫ t j + θ 1 ǫ t j 1 +... + θ q ǫ t q j )] = E(θ j ǫ 2 t j + θ 1 θ j+1 ǫ 2 t j 1 +... + θ q θ q j ǫ 2 t q) = (θ j + θ 1 θ j+1 +... + θ q θ q j )σ 2 j = 1,...,q γ(j) = 0 j > q Eduardo Rossi c - Macroeconometria 9

Example: MA(2) Y t = µ + ǫ t + θ 1 ǫ t 1 + θ 2 ǫ t 2 γ(0) = (1 + θ1 2 + θ2)σ 2 2 γ(1) = E[(ǫ t + θ 1 ǫ t 1 + θ 2 ǫ t 2 )(ǫ t 1 + θ 1 ǫ t 2 + θ 2 ǫ t 3 )] = θ 1 E(ǫ 2 t 1) + θ 1 θ 2 E(ǫ 2 t 2) = (θ 1 + θ 1 θ 2 )σ 2 γ(j) = 0 j = 3, 4,... Eduardo Rossi c - Macroeconometria 10

The Infinite-Order Moving Average Process Y t = µ + ψ j ǫ t j j=0 ǫ t WN(0, σ 2 ) The infinite sequence generates a c.s. process provided that square summability holds ψj 2 < j=0 a slightly stronger condition is the absolute summability ψ j < j=0 Eduardo Rossi c - Macroeconometria 11

The Infinite-Order Moving Average Process E(Y t ) = µ γ(0) = E(Y t µ) 2 = lim T E(ψ 0ǫ t + ψǫ t 1 +... + ψ T ǫ t T ) = lim T (ψ2 0 + ψ 2 1 +... + ψ 2 T)σ 2 γ(j) = E[(Y t µ)(y t j µ)] = σ 2 (ψ j ψ 0 + ψ j+1 ψ 1 +...) Eduardo Rossi c - Macroeconometria 12

The Infinite-Order Moving Average Process An MA( ) with absolutely summable coefficients has absolutely summable covariances γ(j) < j=0 An MA( ) absolutely summable is ergodic for the mean. If ǫ t i.i.d.n(0, σ 2 ) then the process is ergodic for all moments. Eduardo Rossi c - Macroeconometria 13

The Autoregressive process of order 1 (AR(1)) Y t = c + φy t 1 + ǫ t ǫ t WN(0, σ 2 ) Y t = c T 1 j=0 Y t is c.s. if φ < 1. φ j + φ T Y t T + T 1 j=0 φ j ǫ t j Eduardo Rossi c - Macroeconometria 14

The Autoregressive process of order 1 (AR(1)) First Order Difference Equation (1 φl)y t = w t If φ < 1 then backward solution: y t = w t + φw t 1 +... If φ > 1 forward solution based on (1 φl) 1 = φ 1 L 1 1 φ 1 L 1 = φ 1 L 1 [1+φ 1 L 1 +φ 2 L 2 +...] (1 φl)(1 φl) 1 = 1 when it is applied to a bounded sequence {w t } t= the result is another bounded sequence. Applying (1 φl) 1 we are implicitly imposing a boundedness assumption. Eduardo Rossi c - Macroeconometria 15

The Autoregressive process of order 1 (AR(1)) Premultiplying by [1 + φ 1 L 1 +... + φ (T 1) L (T 1) ][ φ 1 L 1 ] the limit of this operator exists and is (1 φl) 1 when φ > 1 (1 φl) 1 = [ φ 1 L 1 ][1 + φ 1 L 1 +...] Applying this operator amounts to solving the difference equation forward. Eduardo Rossi c - Macroeconometria 16

The Autoregressive process of order 1 (AR(1)) For a AR(1) process with φ > 1: Y t = (1 φl) 1 ǫ t = [ φ 1 L 1 ][1 + φ 1 L 1 +...]ǫ t Y t = [ φ 1 L 1 ][ǫ t + φ 1 ǫ t+1 +...] = φ j ǫ t+j this is the unique stationary solution. This is regarded as unnatural since Y t is correlated with {ǫ s, s > t} a property not shared by the solution obtained when φ < 1. It is customary when modelling stationary time series to restrict attention to AR(1) processes with φ < 1 for which Y t has the representation in terms of {ǫ s, s t}. If φ = 1 there is no stationary solution. j=1 Eduardo Rossi c - Macroeconometria 17

The Autoregressive process of order 1 (AR(1)) When Y t is c.s. we can write: Y t = (c + ǫ t ) + φ(c + ǫ t 1 ) + φ 2 (c + ǫ t 2 ) +... when φ < 1 = c + cφ + cφ 2 +... + ǫ t + φǫ t 1 +... c = 1 φ + ǫ t + φǫ t 1 + φ 2 ǫ 2 t 2 +... }{{} MA( ) ψj =φ j ψ j = j=0 φ j = j=0 1 1 φ this ensures that the MA( ) representation exists. The AR(1) process is ergodic for the mean. Eduardo Rossi c - Macroeconometria 18

The Autoregressive process of order 1 (AR(1)) µ E(Y t ) = c 1 φ γ(0) = E[(Y t µ) 2 ] = E[(ǫ t + φǫ t 1 + φǫ t 2 +...) 2 ] = (1 + φ 2 + φ 4 +...)σ 2 = σ 2 /(1 φ 2 ) γ(j) = E[(Y t µ)(y t j µ)] = E[(ǫ t + φǫ t 1 + φ 2 ǫ t 2 +...)(ǫ t j + φǫ t j 1 + φ 2 ǫ t j 2 +...)] = (φ j + φ j+2 + φ j+4 +...)σ 2 = φ j (1 + φ 2 + φ 4 +...)σ 2 = σ 2 (φ j /(1 φ 2 )) Eduardo Rossi c - Macroeconometria 19

The Autoregressive process of order 1 (AR(1)) Autocorrelation function ρ(j) = γ(j) γ(0) = φj geometric decay. solution: γ(j) = φγ(j 1) γ(j) = φ j γ(0) Eduardo Rossi c - Macroeconometria 20

The AR(2) process Y t = c + φ 1 Y t 1 + φ 2 Y t 2 + ǫ t ǫ t WN(0, σ 2 ) (1 φ 1 L φ 2 L 2 )Y t = c + ǫ t The difference equation is stable provided that the roots of 1 φ 1 z φ 2 z 2 = 0 lie outside the unit circle. ψ(l) = (1 φ 1 L φ 2 L 2 ) 1 = ψ 0 + ψ 1 L + ψ 2 L 2 +... Eduardo Rossi c - Macroeconometria 21

The AR(2) process c µ = 1 φ 1 φ 2 The autocovariances Y t = µ(1 φ 1 φ 2 ) + φ 1 Y t 1 + φ 2 Y t 2 + ǫ t Y t µ = φ 1 (Y t 1 µ) + φ 2 (Y t 2 µ) + ǫ t multiplying both sides by (Y t j µ) and taking expectations produces E[(Y t µ)(y t j µ)] = φ 1 E[(Y t 1 µ)(y t j µ)] +φ 2 E[(Y t 2 µ)(y t j µ)] + E[ǫ t (Y t j µ)] γ(j) = φ 1 γ(j 1) + φ 2 γ(j 2) j = 1, 2,... Eduardo Rossi c - Macroeconometria 22

The AR(2) process The autocovariances follow the same the second-order difference equation as does the process for Y t. The autocorrelations ρ(j) = φ 1 ρ(j 1) + φ 2 ρ(j 2) j = 1, 2,... Setting j = 1 For j = 2 ρ(1) = φ 1 + φ 2 ρ(1) ρ(1) = φ 1 1 φ 2 ρ(2) = φ 1 ρ(1) + φ 2 The variance of a c.s. AR(2) E[(Y t µ) 2 ] = φ 1 E[(Y t 1 µ)(y t µ)]+φ 2 E[(Y t 2 µ)(y t µ)]+e[(ǫ t )(Y t µ)] Eduardo Rossi c - Macroeconometria 23

The AR(2) process E(ǫ t )(Y t µ) = E(ǫ t )[φ 1 (Y t 1 µ) + φ 2 (Y t 2 µ) + ǫ t ] = φ 1 0 + φ 2 0 + σ 2 γ(0) = φ 1 γ(1) + φ 2 γ(2) + σ 2 γ(0) = φ 1 ρ(1)γ(0) + φ 2 ρ(2)γ(0) + σ 2 Substituting ρ(1) and ρ(2) [ ] φ 2 γ(0) = 1 + φ 2 (φ 1 ρ(1) + φ 2 ) γ(0) + σ 2 1 φ 2 [ φ 2 = 1 + φ 2φ 2 ] 1 + φ 2 2 γ(0) + σ 2 1 φ 2 1 φ 2 Eduardo Rossi c - Macroeconometria 24

The AR(2) process γ(0) = = = = = [ 1 φ2 1 φ 2φ 2 ] 1 1 φ 2 2 σ 2 1 φ 2 1 φ 2 [ 1 φ2 φ 2 1 φ 2 φ 2 1 φ 2 2(1 φ 2 ) 1 φ 2 (1 φ 2 )σ 2 1 φ 2 φ 2 1 φ 2φ 2 1 φ2 2 (1 φ 2) (1 φ 2 )σ 2 1 φ 2 φ 2 1 φ 2φ 2 1 φ2 2 (1 φ 2) (1 φ 2 )σ 2 (1 + φ 2 )[(1 φ 2 ) 2 φ 2 1 ] ] 1 σ 2 Eduardo Rossi c - Macroeconometria 25

The AR(p) process Y t = c + φ 1 Y t 1 + φ 2 Y t 2 +... + φ p Y t p + ǫ t ǫ t WN(0, σ 2 ) provided that the roots of φ(z) = 1 φ 1 z... φ p z p = 0 all lie the unit circle. Covariance-stationary representation: Y t = µ + ψ(l)ǫ t = c + 1 φ 1 φ 2... φ p 1 1 φ 1 L... φ p L p ǫ t Eduardo Rossi c - Macroeconometria 26

The AR(p) process where ψ(z) = (1 φ 1 z... φ p z p ) 1 = φ(z) 1 and ψ j < j=0 The mean is µ = E(Y t ) = c 1 φ 1... φ p Y t µ = φ 1 (Y t 1 µ) + φ 2 (Y t 2 µ) +... + φ p (Y t p µ) + ǫ t Autocovariances are found by multiplying both sides by (Y t j µ) and taking expectations Eduardo Rossi c - Macroeconometria 27

The AR(p) process The autocovariance function φ 1 γ(j 1) + φ 2 γ(j 2) +... + φ p γ(j p) j = 1, 2,... γ(j) = φ 1 γ(1) +... + φ p γ p + σ 2 j = 0 Eduardo Rossi c - Macroeconometria 28

The AR(p) process Dividing the autocovariance function by γ 0 we obtain the Yule-Walker equations: ρ j = φ 1 ρ j 1 + φ 2 ρ j 2 +... + φ p ρ j p j = 1, 2,... Thus the autocovariances and autocorrelations follow the same p-th order difference equation as does the process itself. For distinct roots, their solutions take the form γ(j) = g 1 λ j 1 + g 2λ j 2 +... + g pλ j p where the eigenvalues (λ 1,...,λ p ) are the solutions to λ p φ 1 λ p 1... φ p = 0 Eduardo Rossi c - Macroeconometria 29

The Autoregressive Moving Average process (ARMA(p,q)) Y t = c + φ 1 Y t 1 +... + φ p Y t p + ǫ t + θ 1 ǫ t 1 +... + θ q ǫ t q (1 φ 1 L... φ p L p )Y t = c + (1 + θ 1 +... + θ q L q )ǫ t φ(l)y t = c + θ(l)ǫ t where φ(l) = 1 φ 1 L... φ p L p θ(l) = 1 + θ 1 +... + θ q L q the stationarity depends on the roots of 1 φ 1 z... φ p z p = 0 Eduardo Rossi c - Macroeconometria 30

The Autoregressive Moving Average process (ARMA(p,q)) If the roots are outside the unit circle then the inverse of φ(z) exists, then dividing by φ(l) both sides Y t = µ + ψ(l)ǫ t ψ(l) = θ(l) φ(l) c µ = 1 φ 1... φ p ψ j < j=0 Eduardo Rossi c - Macroeconometria 31

The Autoregressive Moving Average process (ARMA(p,q)) c = µ(1 φ 1... φ p ) Y t = µ(1 φ 1... φ p ) + φ 1 Y t 1 +... + φ p Y t p + ǫ t +... + θ q ǫ t q Y t µ = φ 1 (Y t 1 µ) +... + φ p (Y t p µ) + ǫ t + θ 1 ǫ t 1 +... + θ q ǫ t q The variance E[(Y t µ) 2 ] = φ 1 E[(Y t 1 µ)(y t µ)] +... + φ p E[(Y t p µ)(y t µ)] +E[ǫ t (Y t µ)] + θ 1 E[ǫ t 1 (Y t µ)] +... + θ q E[ǫ t q (Y t µ)] E[(Y t µ) 2 ] = φ 1 [σ 2 (ψ 1 ψ 0 + ψ 2 ψ 1 +...)] +... + φ p [σ 2 (ψ p ψ 0 + ψ p+1 ψ 1 +...)] +E[ψ 0 ǫ 2 t] + θ 1 E[ψ 1 ǫ 2 t 1] +... + θ q E[ψ q ǫ 2 t q] Eduardo Rossi c - Macroeconometria 32

The Autoregressive Moving Average process (ARMA(p,q)) E[(Y t µ) 2 ] = φ 1 [σ 2 (ψ 1 ψ 0 + ψ 2 ψ 1 +...)] +... + φ p [σ 2 (ψ p ψ 0 + ψ p+1 ψ 1 +...)] +ψ 0 σ 2 + θ 1 ψ 1 σ 2 +... + θ q ψ q σ 2 An ARMA(p,q) process will have more complicated autocovariances for lags 1 through q than would the corresponding AR(p) process γ(j) = E[(Y t µ)(y t j µ)] = φ 1 E[(Y t 1 µ)(y t j µ)] +... + φ p E[(Y t p µ)(y t j µ)] +E[ǫ t (Y t j µ)] + θ 1 E[ǫ t 1 (Y t j µ)] +... + θ q E[ǫ t q (Y t j µ)] Eduardo Rossi c - Macroeconometria 33

The Autoregressive Moving Average process (ARMA(p,q)) For j > q autocovariances are given by E[(Y t µ)(y t j µ)] = φ 1 E[(Y t 1 µ)(y t j µ)] +... + φ p E[(Y t p µ)(y t j µ)] +E[ǫ t (Y t j µ)] + θ 1 E[ǫ t 1 (Y t j µ)] +... + θ q E[ǫ t q (Y t j µ)] E[ǫ t q (Y t j µ)] = E[ǫ t q (ψ(l)ǫ t j )] = E[ǫ t q (ψ 0 ǫ t j + ψ 1 ǫ t j 1 +...)] = ψ 0 E[ǫ t q ǫ t j ] + ψ 1 E[ǫ t q ǫ t j 1 ] +... = 0 then E[(Y t µ)(y t j µ)] = φ 1 E[(Y t 1 µ)(y t j µ)] +... + φ p E[(Y t p µ)(y t j µ)] Eduardo Rossi c - Macroeconometria 34

The Autoregressive Moving Average process (ARMA(p,q)) γ(j) = φ 1 γ(j 1)+φ 2 γ(j 2)+...+φ p γ(j p) j = q+1, q+2,... Thus after q lags the autocovariance function follow the p-th order difference equation governed by the autoregressive parameters. Eduardo Rossi c - Macroeconometria 35

The Autoregressive Moving Average process (ARMA(p,q)) There is a potential for redundant parameterization with ARMA processes. Consider a simple white noise process Y t = ǫ t Suppose both sides are multiplyied by (1 ρl): (1 ρl)y t = (1 ρl)ǫ t Both are valid representations, thus the latter might be described as an ARMA(1,1) process, with φ 1 = ρ and θ 1 = ρ. Since any value of ρ describes the data equally well, we will get into trouble trying to estimate the parameter ρ by maximum likelihood. Eduardo Rossi c - Macroeconometria 36

The Autoregressive Moving Average process (ARMA(p,q)) A related overparameterization can arise with an ARMA(p,q) model. Consider the factorization of the lag polynomial operators: (1 λ 1 L)(1 λ 2 L)...(1 λ p L)(Y t µ) = (1 η 1 L)...(1 η q L)ǫ t Assume that λ i < 1 for all i, so that the process is c.s.. If φ(l) and θ(l) have any roots in common, λ i = η j for some i and j, then both sides can be divided by (1 λ i L) p k=1,k i (1 λ k L)(Y t µ) = q k=1,k j (1 η k L)ǫ t Eduardo Rossi c - Macroeconometria 37

The Autoregressive Moving Average process (ARMA(p,q)) (1 φ 1L... φ p 1L p 1 )(Y t µ) = (1+θ 1L+...+θ q 1L q 1 )ǫ t where (1 φ 1L... φ p 1L p 1 ) p k=1,k i (1 λ k L) (1 + θ 1L +... + θ q 1L q 1 ) q k=1,k j (1 η k L) The stationary process ARMA(p,q) process is clearly identical to the stationary ARMA(p-1,q-1) process. Eduardo Rossi c - Macroeconometria 38

ARMA(1,1) Y t = c + φ 1 Y t 1 + ǫ t + θ 1 ǫ t 1 (1 φ 1 L)Y t = c + (1 + θ 1 L)ǫ t φ 1 < 1 ψ(l) = 1 + θ 1L 1 φ 1 L = (1 + θ 1L)(1 + φ 1 L + φ 2 1L 2 +...) ψ 0 + ψ 1 L + ψ 2 L 2 +... = (1 + θ 1 L)(1 + φ 1 L + φ 2 1L 2 +...) Eduardo Rossi c - Macroeconometria 39

ARMA(1,1) ψ 0 = 1 ψ 1 = θ 1 + ψ 1 ψ 2 = φ 2 1 + φ 1 θ 1... =... γ(0) = φ 1 E[(Y t 1 µ)(y t µ)]+e[ǫ t (Y t µ)]+θ 1 E[ǫ t 1 (Y t µ)] γ(0) = φ 1 [σ 2 (ψ 1 ψ 0 + ψ 2 ψ 1 +...)] + σ 2 + θ 1 ψ 1 σ 2 γ(1) = φ 1 γ(0) + θ 1 σ 2 γ(2) = φ 1 γ(1) Eduardo Rossi c - Macroeconometria 40