Leture 2: ARMA Models Bus 41910, Autumn Quarter 2008, Mr Ruey S Tsay Autoregressive Moving-Average (ARMA) models form a lass of linear time series models whih are widely appliable and parsimonious in parameterization By allowing the order of an ARMA model to inrease, one an approximate any linear time series model with desirable auray [This is similar to using rational polynomials to approximate a general polynomial; see the impluse response funtion] In what follows, we assume that {a t } is a sequene of independent and identially distributed random variables with mean zero and finite variane σ 2 That is, {a t } is a white noise series Sometimes, we use σ 2 a to denote the variane of a t A Autoregressive (AR) Proesses 1 AR(p) Model: Z t φ 1 Z t 1 φ p Z t p = + a t or φ(b)z t = + a t, where is a onstant If Z t is stationary with mean E(Z t ) = µ, then the model an be written as φ(b)(z t µ) = a t The latter is the parameterization implemented in the R pakage 2 Stationarity: All zeros of the polynomial φ(b) lie outside the unit irle (Why?) 3 Moments: (Assume stationarity) Mean: Taking expetation of both sides of the model equation, we have E(Z t ) φ 1 E(Z t 1 ) φ p E(Z t p ) = + E(a t ) µ φ 1 µ φ p µ = (1 φ 1 φ p )µ = µ = 1 φ 1 φ p This relation between and µ is important for stationary time series Autoovariane funtion: For simpliity, assume = µ = 0 Multiplying both sides of the AR model by Z t l and taking expetations, we have E[(Z t φ 1 Z t 1 φ p Z t p )Z t l ] = E(Z t l a t ) For l = 0, For l > 0, γ 0 φ 1 γ 1 φ p γ p = σ 2 γ l φ 1 γ l 1 φ p γ l p = 0 1
Autoorrelation funtion (ACF): ρ k = γ k γ 0 From autoovariane funtion, we have { σ 2 ρ l φ 1 ρ l 1 φ p ρ l p = γ 0 for l = 0 0 for l > 0 Sine the ACFs satisfy the p-th order differene equation, namely φ(b)ρ l = 0 for l > 0, they deay exponentially to zero as l 4 Yule-Walker equation: Consider the above equations jointly for l = 1,, p, we have ρ 1 ρ 2 ρ p = 1 ρ 1 ρ 2 ρ 2 p ρ 1 p ρ 1 1 ρ 1 ρ 3 p ρ 2 p ρ p 1 ρ p 2 ρ p 3 ρ 1 1 whih is the p-order Yule-Walker equation For given ρ l s, the equation an be used to solve for φ i Of ourse, we an obtain ρ l s for given φ i s and σ 2 via the moment generating funtion or the ψ-weight representation to be disussed shortly 5 MA representation: Z t µ = 1 φ(b) a t = ψ i a t i i=0 where the ψ i s are referred to as the ψ-weights of the model These ψ-weights are also alled the impulse response funtion and an be obtained from the φ i s by equating the oeffiients of B i 1 φ(b) = 1 + ψ 1B + ψ 2 B 2 + Thus, we have φ 1 φ 2 φ p 1 = (1 φ 1 B φ p B p )(1 + ψ 1 B + ψ 2 B 2 + ) ψ 1 = φ 1 ψ 2 = φ 1 ψ 1 + φ 2 ψ 3 = φ 1 ψ 2 + φ 2 ψ 1 + φ 3 ψ p = φ 1 ψ p 1 + φ 2 ψ p 2 + + φ p 1 ψ 1 + φ p p ψ l = φ i ψ l i for l > p i=1 Again, the ψ-weights satisfy the differene equation φ(b)ψ l = 0 for l > p so that they also deay exponentially to zero as l goes to infinite 2
6 The moment generating funtion: 7 A simple example: the AR(1) ase Γ(z) = Stationarity ondition: φ 1 < 1 Mean: µ = ACF: ρ l = φ l 1 1 φ 1 σ 2 φ(z)φ(z 1 ) Z t φ 1 Z t 1 = + a t Yule-Walker equation: ρ 1 = φ 1 ρ 0 = φ 1 MA representation: Z t = so that the ψ-weights are ψ l = φ l 1 The variane of Z t : Var(Z t ) = 1 φ 1 + a t + φ 1 a t 1 + φ 2 1a t 2 + φ 3 1a t 3 + σ2 1 φ 2 1 8 An AR(2) model: (1 φ 1 B φ 2 B 2 )Z t = + a t Stationarity ondition: Zeros of φ(b) are outside the unit irle Mean: µ = 1 φ 1 φ 2 ACF: ρ 0 = 1, ρ 1 = φ 1 /(1 φ 2 ), and ρ j = φ 1 ρ j 1 + φ 2 ρ j 2, for j > 1 Why? Yule-Walker equation: ρ 1 = φ 1 + φ 2 ρ 1 ρ 2 = φ 1 ρ 1 + φ 2, or equivalently, [ 1 ρ1 ρ 1 1 ] [ φ1 φ 2 ] [ ρ1 = ρ 2 ] MA representation: Z t = 1 φ 1 φ 2 + a t + ψ 1 a t 1 + ψ 2 1a t 2 + ψ 3 1a t 3 +, where 1 + ψ 1 B + ψ 2 B 2 + = 1 1 φ 1 B φ 2 B 2 3
The variane of Z t an be obtained from the moment equations as Var(Z t ) = 1 φ 2 1 + φ 2 σ 2 [(1 φ 2 ) 2 φ 2 1] B Moving-Average (MA) Proesses 1 MA(q) Model: Z t = + a t θ 1 a t 1 θ q a t q or Z t = + θ(b)a t 2 Stationarity: Finite order MA models are stationary 3 Mean: By taking expetation on both sides, we obtain E(Z t ) = + E(a t ) θ 1 E(a t 1 θ q E(a t q ) µ = Thus, for MA models, the onstant term is the mean of the proess 4 Invertibility: Can we write an MA model as an AR model? Consider Let π(b) = 1 π 1 B π 2 B 2 = 1 Z t 1 θ(b) (Z t µ) = a t θ(b) We all the π s the π weights of the proess 1 Similar to, for the above AR representation to be meaningful we require that φ(b) πi 2 < i=1 The neessary and suffiient ondition for suh a onvergent π-weight sequene is that all the zeros of θ(b) lie outside the unit irle 5 Autoovariane funtion: Again for simpliity, assume µ = 0 Multiple both sides by Z t l and take expetation We have (1 + θ1 2 + + θq)σ 2 2 for l = 0 γ l = σ 2 q l i=0 θ i θ l+i for l = 1,, q 0 for l > q where θ 0 = 1 This shows that for an MA(q) proess, the autoovariane funtion γ l is zero for l > q This is a speial feature of MA proesses and it provides a onvenient way to identify an MA model in pratie 4
6 Autoorrelation funtion: ρ l = γ l γ 0 Again, from the autoovariane funtion we have ρ l = { 0 for l = q 0 for l > q In other words, the ACF has only a finite number of non-zero lags Thus, for an MA(q) model, Z t and Z t l are unorrelated provided that l > q For this reason, MA models are referred to as short memory time series 7 The moment generating funtion: 8 A simple example: the MA(1) ase Γ(z) = σ 2 θ(z)θ(z 1 ) Invertibility ondition: θ < 1 Mean: E(Z t ) = µ = ACF: Z t = + a t θa t 1 1 for l = 0 θ ρ l = for l = 1 1+θ 2 0 for l > 1 The above result shows that ρ 1 05 for an MA(1) model Variane: γ 0 = (1 + θ 2 )σ 2 Moment generating funtion: Γ(z) = [(1 + θ 2 ) θ(z + z 1 )]σ 2 From the MGF, we have γ 0 = (1 + θ 2 )σ 2, γ 1 = θσ 2, and γ j = 0 for j 2 5
C Mixed ARMA Proesses 1 ARMA(p, q) Model: Z t φ 1 Z t 1 φ p Z t p = + a t θ 1 a t 1 θ q a t q or φ(b)z t = + θ(b)a t Again, for a stationary proess, we an rewrite the model as φ(b)(z t µ) = θ(b)a t 2 Stationarity: All zeros of φ(b) lie outside the unit irle 3 Invertibility: All zeros of θ(b) lie outside the unit irle 4 AR representation: π(b)z t = θ(1) + a t, where π(b) = φ(b) θ(b) The π-weight π i an be obtained by equating the oeffiients of B i in π(b)θ(b) = φ(b) 5 MA representation: Z t = φ(1) + ψ(b)a t, where ψ(b) = θ(b) φ(b) Again, the ψ-weights an be obtained by equating oeffiients Note that ψ(b)π(b) = 1 for all B for an ARMA model This identity has many appliations 6 Moments: (Assume stationarity) Mean: µ = 1 φ 1 φ p Autoovariane funtion: (Assume µ = 0) Using the result σ 2 for l = 0 E(Z t a t l ) = ψ l σ 2 for l > 0 0 for l < 0, and the same tehnique as before, we have (1 θ 1 ψ 1 θ q ψ q )σ 2 for l = 0 γ l φ 1 γ l 1 φ p γ l p = (θ l + θ l+1 ψ 1 + + θ q ψ q l )σ 2 for l = 1,, q 0 for l q + 1 where ψ 0 = 1 and θ j = 0 for j > q 6
Autoorrelation funtion: ρ l satisfies ρ l φ 1 ρ l 1 φ p ρ l p = 0 for l > q You may think that the ACF satisfy the differene equation φ(b)ρ l = 0 for l q + 1 with ρ 1,, ρ q as initial onditions 7 Generalized Yule-Walker Equation: Consider the above equations of ACF for l = q + 1,, q + p, we have ρ q+1 ρ q ρ q 1 ρ q+2 p ρ q+1 p φ 1 ρ q+2 ρ q+1 ρ q ρ q+3 p ρ q+2 p φ 2 = ρ q+p ρ q+p 1 ρ q+p 2 ρ q+1 ρ q whih is referred to as a p-th order generalized Yule-Walker equation for the ARMA(p, q) proess It an be used to solve for φ i s given the ACF ρ i s 8 Moment generating funtion: 9 A simple example: The ARMA(1,1) ase Stationarity ondition: φ < 1 Invertibility ondition: θ < 1 Mean: µ = 1 φ Variane: γ 0 = σ2 (1+θ 2 2φθ) 1 φ 2 ACF: 10 AR representation: where π i = θ i 1 (φ θ) 11 MA representation: where ψ i = φ i 1 (φ θ) Z t = Z t = Γ(z) = θ(z)θ(z 1 ) φ(z)φ(z 1 ) σ2 Z t φz t 1 = + a t θa t 1 (1 φθ)(φ θ) ρ 1 = 1 + θ 2 2φθ, ρ l = φρ l 1 for l > 1 1 θ + π 1Z t 1 + π 2 Z t 2 + + a t 1 φ + a t + ψ 1 a t 1 + ψ 2 a t 2 + φ p 7
1 Illurative Examples Some examples of ARMA models are given below with demonstration US quarterly growth rate of GDP Daily returns of the US-EU exhange rate Chemial onentration readings, Series A, of Box and Jenkins (1976) 8