Estimation for ARMA Processes with Stable Noise Matt Calder & Richard A. Davis Colorado State University rdavis@stat.colostate.edu 1
ARMA processes with stable noise Review of M-estimation Examples of M-estimation LS LAD MLE Bootstrapping Simulations 2
ARMA(p,q) model with heavy tailed noise....... X t φ 1 X t-1 φ p X t-p = Z t + θ 1 Z t-1 + + θ q Z t-q, a. {Z t } ~ IID(α) with Pareto tails (0 < α < 2)...... b. φ(z) = 1 φ 1 z φ p z p and θ(z) = 1 + θ 1 z + + θ q z q have no common zeroes and no zeroes on or inside the unit circle. Shorthand notation. φ(b)x t = θ(b)z t, {Z t } ~ IID(α) (Β=backward shift) β= (φ 1,..., φ p, θ 1,..., θ q ) T (parameter vector) 3
Examples of AR(2) models. 0 50 100 150 200 250-400 0 200 400 0 200 400 600 800 0 50 100 150 200 250 4
0 50 100 150 200 250-500 0 500-1000 0 500 1000 0 50 100 150 200 250 5
Review of M-Estimation for ARMA Models Data. X 1,..., X n Model. (ARMA(p,q)) φ(b)x t = θ(b)z t, {Z t } ~ IID(α). Loss function. ρ( ) Criterion. Minimize where n Σ t=1 T n (β) = ρ(z t (β)) with respect to β, Z t (β ) = 0 and X t = 0, for t < 1,... Z t (β ) = φ(b)x t θ 1 Z t-1 (β ) θ q Z t-q (β ), t >0. 6
Result (Davis, Knight & Liu `92, Davis `95). If {Z t } ~SαS (symmetric α stable) and ψ(x)= ρ'(x) satisfies 1. Lipschitz of order τ > max(α 1,0). 2. Ε ψ(z 1 ) < if α < 1, 8 3. Ε ψ(z 1 ) = 0 and Var(ψ(Z 1 ) ) <, if α > 1, then n 1/α (β M β) d η where β M is the M-estimate of β. The limit random vector η is the minimizer of a stochastic process. 8 7
Εxample (MA(1)). Data. X 1,..., X n Model. X t = Z t + θ Z t-1, LAD estimation: Minimize θ 0 < 1, {Z t } ~ IID(α) n Σ t=1 n Σ t=1 T n (θ) = ( ρ(z t (θ)) - ρ(z t (θ 0 )) ) = (ρ(x t θx t-1 +θ 2 X t-2 - (-θ) t-1 X 1 ) - ρ (Z t (θ 0 ))) Set u=n 1/α (θ θ 0 ),... 8
S n (u) = T n (θ 0 + un -1/α ) n Σ t=1 = (ρ(z t (θ 0 + un -1/α )) - ρ(z t (θ 0 )) ) (Not a convex function of u even if ρ is convex!) Linearize Z t (θ 0 + un -1/α ) to get n Σ t=1 S n (u) (ρ(z t (θ 0 ) + un -1/α Z t ' (θ 0 )) ρ(z t (θ 0 )) ) where -Z t ' (θ 0 ) is the AR(1) process Y t = θ 0 Y t-1 +Z t. Result : u n := argmin(s n (u)) = n 1/ α (θ M θ 0 ) d η:= argmin(s (u)) 9
M-Estimation Examples 1. LS (least squares). ρ(x) = x 2 does not satisfy assumptions 1 3 of previous slide. However, ( n / ln n ) 1/α (β LS β) η LS d Remark: Estimation procedures which are inherently second order based will have scaling factor (n / ln n) 1/α. Examples are moment estimation (Davis & Resnick `85, `86) Yule-Walker estimation for AR's (Davis & Resnick `85, `86) Whittle estimate (and max Gaussian likelihood?) (Mikosch, Gadrich, Kluppelberg and Adler `95) 10
Moreover, β M β β LS β p 0 2. LAD (least absolute deviations). ρ(x) = x does not satisfy assumptions 1 3 of previous slide either. However, n 1/α (β LAD β) d η LAD 11
3. MLE (maximum likelihood). Suppose Z t has pdf f and ρ(x) = ln f(x). Then β MLE, which minimizes T n (β) = ln f(z t (β)), is an approximate MLE estimator. If one chooses f to be the symmetric λ stable density f λ, then ρ(x) = ln f λ (x) satisfies the assumptions of the result mentioned previously so that n Σ t=1 n 1/α d (β MLE,λ β) η λ Call β MLE,λ the maximum (λ-stable) likelihood estimate. 12
Remark. Can minimize n Σ t=1 T n (β) = ln f λ (Z t (β)) with respect to both λ and β to obtain pseudo-mle's of both parameters. 13
Bootstrapping the M-Estimate (Davis and Wu `95). Data. X 1,..., X n Model. X t = φ 1 X t-1 + + φ p X t-p +Z t, {Z t } ~ IID(α) M-estimate. φ Estimated residuals. Z t =X t φ 1 X t-1 +... φ p X t-p Bootstrap sample. X * t = φ 1 X * t-1 +... + φ p X * * t-p +Z t for t = 1,..., m n, where {Z * t }~IID(F n ), F n = empirical df of Z p+1,..., Z n. BS M-estimate. φ... 14
Result. If m n / n 0, then P(m 1/α n (φ φ) p X n ) P( η ). Removing the dependence on normalizing constants. Let M n = max{ X 1,..., X n } M * m = max{ X * 1,..., X * m }. Then M n (φ φ) d w and P(M * m (φ φ) p X n ) P( w ). 15
Simulation Comparison. Principal objectives: compare performance of β MLE,α, β LAD, and β LS. compare performance of β MLE,α, β MLE, 1, β MLE,α, and β LAD. investigate performance of the MLE estimator of α. 3 Models ({Z t } ~ SαS) AR(1) : X t =.4 X t-1 + Z t MA(1): X t = Z t +.8 Z t-1 ARMA(1,1): X t =.4 X t-1 + Z t +.8 Z t-1 Sample size = 200, replications = 10,000 16
α = 1.75 Model True Values β β β LAD LS MLE, α M.1 φ =.4.397 (.0465).397(.0474).398(.0394) M.2 θ =.8.802 (.0317).803(.0330).803(.0270) M.3 φ =.4.389 (.0456).397(.0525).398(.0440) θ =.8.807 (.0353).804(.0363).803(.0296) α = 1.0 Model True Values β β β LAD LS MLE, α M.1 φ =.4.3995 (.0073).3971(.0263).3999(.0061) M.2 θ =.8.8000 (.0043).8005(.0207).7997(.0039) M.3 φ =.4.3997 (.0083).3979(.0320).3999(.0071) θ =.8.8005 (.0048).8012(.0232).7996(.0048) 17
α =.50 Model True Values β β β LAD LS MLE, α M.1 φ =.4.3997 (.00058).3988(.01022).4000(.00007) M.2 θ =.8.7978 (.00226).8004(.01052).7576(.04242) M.3 φ =.4.3988 (.00142).3986(.01396).3998(.00071) θ =.8.8000 (.00012).8014(.01179).7529(.04715) 18
M.1, φ α=1.75 M.2, θ α=1.75 0.25 0.40 0.55 LAD LS MLE 0.65 0.80 0.95 LAD LS MLE M.1, φ α=1.0 M.2, θ α=1.0 0.35 0.45 0.75 0.85 LAD LS MLE LAD LS MLE 19
M.1, φ, α=0.5 M.2, θ α=0.5 0.36 0.40 0.44 LAD LS MLE 0.76 0.80 0.84 LAD LS MLE 20
M.1, φ α=1.75 M.2, θ α =1.75 0.25 0.40 0.55 0.65 0.80 0.95 LAD MLE Cchy Est. LAD MLE Cchy Est. M.1, φ α=0.8 M.2, θ α=0.8 0.390 0.405 LAD MLE Cchy Est. 0.790 0.805 LAD MLE Cchy Est. 21
Estimation of α 0 500 1500 2500 M.1, φ α = 0.80 500 1000 1500 2000 0 M1, φ α = 1.75 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 M.2, θ α = 0.80 M.2, θ α = 1.75 0 500 1500 2500 0 500 1000 1500 2000 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 22
Robustness of the estimates. Z 1 has a Pareto tail with exponent α = 4.0. M.1, φ α =4.0 M.2, θ α =4.0 0.30 0.40 0.50 0.70 0.80 0.90 LAD LS Cauchy LAD LS Cauchy 23
Conclusions Parameters of an ARMA model with heavy tailed noise can be estimated quite well. LAD estimates almost as good as MLE when noise is stable. Drawbacks of MLE: computationally more difficult requires an estimate of α MLE estimation of α performs well but is not very robust. 24