ARMA Models: I autoregressive moving-average (ARMA) processes play a key role in time series analysis for any positive integer p & any purely nondeterministic process {X t } with ACVF {γ X (h)}, there is an AR(p) process {Y t } with ACVF {γ Y (h)} such that γ Y (h) = γ X (h) for h p corresponding statement does not hold for MA(q) processes (cf. AR(1) and MA(1) processes), but adding MA component to form ARMA processes increases flexibility by defining potentially useful models with small number of parameters will now extend notions introduced for ARMA(1,1) model to higher order ARMA models VIII 1
ARMA Models: II {X t } is said to be an ARMA(p, q) process if it is stationary and if, for t Z, X t φ 1 X t 1 φ p X t p = Z t + θ 1 Z t 1 + + θ q Z t q, where {Z t } WN(0, σ 2 ), and the polynomials 1 φ 1 z φ p z p and 1 + θ 1 z + + θ q z q have no common roots (factors) in above z is a complex-valued variable note: ARMA model sometimes written in 3 other ways: X t φ 1 X t 1 φ p X t p = Z t θ 1 Z t 1 θ q Z t q X t + φ 1 X t 1 + + φ p X t p = Z t + θ 1 Z t 1 + + θ q Z t q X t + φ 1 X t 1 + + φ p X t p = Z t θ 1 Z t 1 θ q Z t q BD 83, CC 77, SS 92 VIII 2
ARMA Models: III polynomial condition is sometimes stated in terms of 1 φ 1 z 1 φ p z p and 1 + θ 1 z 1 + + θ q z q having no common roots (as will be noted later, this formulation has one distinct advantage) to see why no common root is stipulated, recall ARMA(1,1) process X t = φx t 1 + Z t + θz t 1, for which φ + θ 0 was stipulated reason for this stipulation became clear when we considered causal stationary solution X t = Z t + (φ + θ) φ j 1 Z t j ; j=1 note that {X t } degenerates into WN model when φ + θ = 0 BD 83, CC 78, SS 93 VIII 3
ARMA Models: IV ARMA(1,1) polynomial condition says 1 φz & 1 + θz should not have a common root 1 φz = 0 & 1 + θz = 0 yield roots of 1/φ & 1/θ, and 1/φ 1/θ is equivalent to φ + θ 0 can write ARMA(p, q) model more compactly as φ(b)x t = θ(b)z t, with φ(z) = 1 φ 1 z φ p z p & θ(z) = 1+θ 1 z + +θ q z q (as before, B is the backward shift operator) needed conditions φ < 1 and θ < 1 on ARMA(1,1) parameters for process to be stationary, causal and invertible similarly, need conditions on φ j s and θ k s for ARMA(p,q) process to be such these can be stated as conditions on polynomials φ(z) and θ(z) BD 84, CC 78, 80, SS 94, 95 VIII 4
ARMA Models: V 1. there is a (unique) stationary solution to φ(b)x t = θ(b)z t if and only if φ(z) 0 for all z = 1 2. ARMA(p, q) process is causal, meaning that, for t Z, X t = ψ j Z t j = ψ(b)z t with ψ(b) = ψ j B j & ψ j <, j=0 if φ(z) 0 for all z 1 3. ARMA(p, q) process is invertible, meaning that, for t Z, Z t = π j X t j = π(b)x t with π(b) = π j B j & π j <, j=0 if θ(z) 0 for all z 1 j=0 j=0 j=0 j=0 BD 84, 85, 86, CC 78, 80, SS 94, 95 VIII 5
ARMA Models: VI for complex variable z = x + iy, where i 1, unit circle defined to be set of all z s such that z 2 x 2 + y 2 = 1 unit circle handily described by e iω cos (ω) + i sin (ω) as ω varies from 0 to 2π (note that e iω 2 = cos 2 (ω) + sin 2 (ω) = 1) conditions can be restated in terms of roots of φ(z) and θ(z), i.e., values z l and z m such that φ(z l ) = 0 and θ(z m ) = 0 1. stationarity: requires all roots z l of φ(z) be off the unit circle; i.e., must have z l 1 2. causality: requires all roots z l of φ(z) to be outside the unit circle; i.e., must have z l > 1 3. invertibility: requires all roots z m of θ(z) to be outside the unit circle; i.e., must have z m > 1 BD 84, 85, 86, CC 78, 80, SS 94, 95 VIII 6
Unit Circle, Some Roots z and Their Reciprocals 1/z y 2 1 0 1 2 * * 2 1 0 1 2 x VIII 7
ARMA Models: VII causality condition on φ(z) implies existence of inverse of filter φ(b), namely, 1 φ(b) = χ j B j = χ(b), where χ j < j=0 j=0 likewise, invertibility condition on θ(z) implies existence of inverse of filter θ(b), namely, 1 θ(b) = π j B j = π(b), where π j < j=0 j=0 BD 85, 86, CC 78, 80, SS 94, 95 VIII 8
ARMA Models: VIII 1. definition of ARMA process says φ(b)x t = θ(b)z t 2. causality of ARMA process says X t = ψ(b)z t 3. multiplication of above by φ(b) says φ(b)x t = φ(b)ψ(b)z t comparison of 3 & 1 says φ(b)ψ(b) = θ(b) and hence (1 φ 1 B φ p B p )(ψ 0 + ψ 1 B + ) = 1 + θ 1 B + + θ q B q ( ) expanding out left-hand side (LHS) of ( ) yields ψ 0 + ψ 1 B + ψ 2 B 2 + ψ 3 B 3 + φ 1 ψ 0 B φ 1 ψ 1 B 2 φ 1 ψ 2 B 3 φ 2 ψ 0 B 2 φ 2 ψ 1 B 3 φ 3 ψ 0 B 3 BD 85, CC 79, SS 101 VIII 9
now take ARMA Models: IX ψ 0 + ψ 1 B + ψ 2 B 2 + ψ 3 B 3 + φ 1 ψ 0 B φ 1 ψ 1 B 2 φ 1 ψ 2 B 3 φ 2 ψ 0 B 2 φ 2 ψ 1 B 3 φ 3 ψ 0 B 3 collect together coefficients for B, B 2, B 3,... to get ψ 0 + (ψ 1 φ 1 ψ 0 )B + (ψ 2 φ 1 ψ 1 φ 2 ψ 0 )B 2 + (ψ 3 φ 1 ψ 2 φ 2 ψ 1 φ 3 ψ 0 )B 3 + and equate with 1 + θ 1 B + θ 2 B 2 + θ 3 B 3 + (RHS of ( )): 1 = ψ 0 θ 1 = ψ 1 φ 1 ψ 0 θ 2 = ψ 2 φ 1 ψ 1 φ 2 ψ 0 θ 3 = ψ 3 φ 1 ψ 2 φ 2 ψ 1 φ 3 ψ 0 BD 85, CC 79, SS 101 VIII 10
ARMA Models: X rewrite as 1 = ψ 0 θ 1 = ψ 1 φ 1 ψ 0 θ 2 = ψ 2 φ 1 ψ 1 φ 2 ψ 0 θ 3 = ψ 3 φ 1 ψ 2 φ 2 ψ 1 φ 3 ψ 0. ψ 0 = 1 ψ 1 = φ 1 ψ 0 + θ 1 ψ 2 = φ 1 ψ 1 + φ 2 ψ 0 + θ 2 ψ 3 = φ 1 ψ 2 + φ 2 ψ 1 + φ 3 ψ 0 + θ 3. BD 85, CC 79, SS 101 VIII 11
stare at ARMA Models: XI ψ 0 = 1 ψ 1 = φ 1 ψ 0 + θ 1 ψ 2 = φ 1 ψ 1 + φ 2 ψ 0 + θ 2 ψ 3 = φ 1 ψ 2 + φ 2 ψ 1 + φ 3 ψ 0 + θ 3. to see recursive scheme for computing ψ j s: p ψ j = φ k ψ j k + θ j, j = 0, 1, 2,..., k=1 for which we need to define θ 0 = 1, θ j = 0 for j > q and ψ j = 0 for j < 0 (also take p k=1 φ kψ j k to be 0 if p = 0) BD 85, CC 79, SS 101 VIII 12
ARMA Models: XII now start with 1. definition of ARMA process: θ(b)z t = φ(b)x t 2. invertibility of ARMA process: Z t = π(b)x t 3. multiplication of above by θ(b): θ(b)z t = θ(b)π(b)x t comparison of 3 & 1 says θ(b)π(b) = φ(b) and hence (1 + θ 1 B + + θ q B q )(π 0 + π 1 B + ) = 1 φ 1 B φ p B p same argument as before (with φ k replaced by θ k and with θ j replaced by φ j ) leads to scheme for computing π j s: q π j = θ k π j k φ j, j = 0, 1, 2,..., k=1 where φ 0 1, φ j 0 for j > p and π j 0 for j < 0 BD 86, CC 79 VIII 13
Example ARMA(1,1) Process: I note: already considered in overheads VII 21 to VII 28 process takes the form X t φx t 1 = Z t + θz t 1 here φ(z) = 1 φz and θ(z) = 1 + θz roots of φ(z) = 0 and θ(z) = 0 are 1/φ and 1/θ stationary, causal & invertible if 1/φ > 1 & 1/θ > 1, i.e., φ < 1 and θ < 1 (easily checked!) have already noted ψ 0 = 1 and ψ j = (φ + θ)φ j 1 for j 1 also have π 0 = 1 and π j = (φ + θ)( θ) j 1 for j 1 next overheads show (1) plot of roots and their reciprocals and (2) one realization for specific ARMA(1,1) model X t 0.5X t 1 = Z t + 0.4Z t 1, {Z t } Gaussian WN(0, 1) BD 86, 87, CC 77, 78, SS 95, 96 VIII 14
Root Plot ( / = AR/MA; red/blue = root/root 1 ) y 2 1 0 1 2 * * 2 1 0 1 2 x VIII 15
Realization of ARMA(1,1) Process 0 20 40 60 80 100 4 2 0 2 4 t x t VIII 16
Example B&D s AR(2) Process: I AR(2) process takes form X t φ 1 X t 1 φ 2 X t 2 = Z t invertibility trivially true: Z t = X t φ 1 X t 1 φ 2 X t 2 here φ(z) = 1 φ 1 z φ 2 z 2 (note: π 1 = φ 1 and π 2 = φ 2 ) need to find roots z 1 and z 2 to see if {X t } is stationary & causal B&D consider X t = 0.7X t 1 0.1X t 1 + Z t, for which φ(z) = 1 0.7z + 0.1z 2 = (1 0.5z)(1 0.2z) roots are thus z 1 = 2 and z 2 = 5 both z 1 and z 2 are outside the unit circle process is thus stationary & causal next overheads show plots of roots and one realization, for which {Z t } Gaussian WN(0, 1) BD 87 VIII 17
Root Plot (red/blue = root/root 1 ) y 4 2 0 2 4 4 2 0 2 4 x VIII 18
Realization of B&D s AR(2) Process x t 4 2 0 2 0 20 40 60 80 100 t VIII 19
Example B&D s AR(2) Process: II for AR(2) processes, recursive scheme for computing ψ j s, namely, 2 ψ j = φ k ψ j k + θ j, j = 0, 1, 2,..., k=1 leads to ψ 0 = 1, ψ 1 = φ 1 & ( ) ψ j = φ 1 ψ j 1 + φ 2 ψ j 2, j 2 theory of homogeneous linear difference equations says that, if roots z 1 and z 2 are distinct, have ψ j = α 1 z j 1 + α 2 z j 2, j 2 since ( ) says ψ 2 = φ 2 1 + φ 2 and ψ 3 = φ 3 1 + 2φ 1φ 2, can solve for α l s using ψ 2 = α 1 z1 2 + α 2 z2 2 and ψ 3 = α 1 z1 3 + α 2 z2 3 for B&D AR(2) process, get ψ j = 5 3 2 j 2 3 5 j, j 2 BD 87 VIII 20
ψ j s for B&D s AR(2) Process ψ j 1.0 0.5 0.0 0.5 1.0 0 5 10 15 20 j VIII 21
Example Second AR(2) Process: I now consider X t = 0.75X t 1 0.5X t 2 + Z t, for which φ(z) = 1 0.75z+0.5z 2 z = 1 z 1 3 4 + 23 4 i 3 4 23 4 i roots z 1 and z 2 are 3 4 ± 23 4 i (complex conjugates) here z 1 = z 2 = 2, so roots are outside the unit circle process is thus stationary & causal can reexpress roots as 2e ±iω, where ω =. 1.01 radians (58.0 ) realizations will tend to fluctuate roughly with period 2π. ω = 6.2 next overheads show plots of roots and one realization, for which {Z t } Gaussian WN(0, 1) VIII 22
Root Plot (red/blue = root/root 1 ) y 2 1 0 1 2 2 1 0 1 2 x VIII 23
Realization of Second AR(2) Process 0 20 40 60 80 100 4 2 0 2 t x t VIII 24
Example Second AR(2) Process: II as before, ψ 0 = 1 & ψ 1 = φ 1, while φ j s for j 2 satisfy ψ j = αz j 1 + α z j 2 = α e iϕ z 1 j e iωj + α e iϕ z 1 j e iωj = 2 α cos (ωj ϕ)/ z 1 j (note: α is complex conjugate of complex variable α α e iϕ ) letting α x + iy, can also write ψ j = 2[x cos (ωj) + y sin (ωj)]/ z 1 j as before, can get ψ 2 and ψ 3 from recursive scheme, yielding two equations to solve to get x and y (and hence α and ϕ): ψ 2 = 2[x cos (2ω)+y sin (2ω)]/ z 1 2 & ψ 3 = 2[x cos (3ω)+y sin (3ω)]/ z 1 3 here get x = 0.5 & y. = 0.313 and hence α = 0.59 & ϕ. = 0.559 VIII 25
ψ j s for Second AR(2) Process ψ j 1.0 0.5 0.0 0.5 1.0 0 5 10 15 20 j VIII 26
Example AR(4) Process now consider AR(4) process X t = 2.7607X t 1 3.8106X t 2 +2.6535X t 3 0.9238X t 4 +Z t, where {Z t } Gaussian WN(0, 1) thus φ(z) = 1 2.7607z + 3.8106z 2 2.6535z 3 + 0.9238z 4 polyroot function in R calculates roots as 0.650 ± 0.786i and 0.786 ± 0.650i, with corresponding magnitudes 1.0199 and 1.0201 thus {X t } is stationary and causal getting closed form expression for ψ j s is tedious, so opt to just compute them using recursive scheme VIII 27
Root Plot (red/blue = root/root 1 ) y 1.5 0.5 0.0 0.5 1.0 1.5 1.5 0.5 0.0 0.5 1.0 1.5 x VIII 28
Realization of AR(4) Process 0 20 40 60 80 100 60 40 20 0 20 40 60 t x t VIII 29
ψ j s for AR(4) Process ψ j 10 5 0 5 10 0 20 40 60 80 100 j VIII 30
Aside Harmonic Processes: I reconsider stationary process of Problem 2(b): X t = Z 1 cos (ωt) + Z 2 sin (ωt), where Z 1 and Z 2 are independent N (0, 1) RVs above is an example of a harmonic process realizations of harmonic processes are qualitatively very different from those for ARMA processes (see next overhead) homework exercise: given X 1 and X 2, can write X t = 2 cos (ω)x t 1 X t 2, above resembles AR(2) process t Z Y t = φ 1 Y t 1 + φ 2 Y t 2 + Z t if we set φ 1 = 2 cos (ω), φ 2 = 1 and Z t = 0 (can achieve by stipulating {Z t } WN(0, 0)) VIII 31
Three Realizations of Harmonic Process (ω = π/12) x t 3 2 1 0 1 2 3 0 20 40 60 80 100 t VIII 32
Aside Harmonic Processes: II since X t = 2 cos (ω)x t 1 X t 2, can perfectly predict X t given X t 1 & X t 2 note: {X t } is example of a deterministic stationary process regarding {X t } as an AR(2) process with {Z t } WN(0, 0), have φ(z) = 1 φ 1 z φ 2 z 2 = 1 2 cos (ω)z + z 2, which has roots e ±iω since φ(e iω ) = 1 2 cos (ω)e iω + e i2ω = 1 (e iω + e iω )e iω + e i2ω = 0, where we have made use of 2 cos (ω) = e iω + e iω since e ±iω 2 = cos 2 (ω) + sin 2 (ω) = 1, roots are on unit circle reconsider example ω = π/12, which has period 2π ω = 24 VIII 33
Root Plot for Harmonic Process y 2 1 0 1 2 * * 2 1 0 1 2 x VIII 34