GCG-type Methods for Nonsymmetric Linear Systems

GCG-type Methods for Nonsymmetric Linear Systems Tsung-Ming Huang Department of Mathematics National Taiwan Normal University December 6, 2011 T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 1 / 45

Outline 1 GCG method(generalized Conjugate Gradient) 2 BCG method (A: unsymmetric) 3 The polynomial equivalent method of the CG method 4 Squaring the CG algorithm 5 Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 2 / 45

Recall: A is s.p.d. Consider the quadratic functional F (x) = 1 2 xt Ax x T b Consider Ax = b min x R n F (x) = F (x ) ϕ(x) = 1 2 (b Ax)T A 1 (b Ax) = F (x) + 1 2 bt A 1 b, (1) where 1 2 bt A 1 b is a constant. Then Ax = b ϕ(x ) = min ϕ(x) = [ min F (x)] + 1 x Rn x R n 2 bt A 1 b T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 3 / 45

Algorithm: Conjugate Gradient method (CG-method) Input: Given s.p.d. A, b R n and x 0 R n and r 0 = b Ax 0 = p 0. 1: Set k = 0. 2: repeat 3: Compute α k = pt k r k p T k Ap ; k 4: Compute x k+1 = x k + α k p k ; 5: Compute r k+1 = r k α k Ap k = b Ax k+1 ; 6: Compute β k = rt k+1 Ap k p T k Ap ; k 7: Compute p k+1 = r k+1 + β k p k ; 8: Set k = k + 1; 9: until r k = 0 Numerator: r T k+1 ((r k r k+1 )/α k ) = ( r T k+1 r k+1)/α k Denominator: p T k Ap k = (r T k + β k 1p T k 1 )((r k r k+1 )/α k ) = (r T k r k)/α k. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 4 / 45

Remark 1 CG method does not need to compute any parameters. It only needs matrix vector and inner product of vectors. Hence it can not destroy the sparse structure of the matrix A. The vectors r k and p k generated by CG-method satisfy: p T i r k = (p i, r k ) = 0, r T i r j = (r i, r j ) = 0, p T i Ap j = (p i, Ap j ) = 0, i < k i j i j x k+1 = x 0 + k i=0 α ip i minimizes F (x) over x = x 0 + < p 0,, p k >. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 5 / 45

GCG method(generalized Conjugate Gradient) GCG method is developed to minimize the residual of the linear equation under some special functional. In conjugate gradient method we take ϕ(x) = 1 2 (b Ax)T A 1 (b Ax) = 1 2 rt A 1 r = 1 2 r 2 A 1, where x A 1 = x T A 1 x. Let A be a unsymmetric matrix. Consider the functional f(x) = 1 2 (b Ax)T P (b Ax), where P is s.p.d. Thus f(x) > 0, unless x = A 1 b f(x ) = 0, so x minimizes the functional f(x). T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 6 / 45

Different choices of P: (i) P = A 1 (A is s.p.d.) CG method (classical) (ii) P = I GCR method (Generalized Conjugate Residual). Here {r i } forms A-conjugate. f(x) = 1 2 (b Ax)T (b Ax) = 1 2 r 2 2 (iii) Consider M 1 Ax = M 1 b. Take P = M T M > 0 GCGLS method (Generalized Conjugate Gradient Least Square). (iv) Similar to (iii), take P = (A + A T )/2 (note: P is not positive definite) and M = (A + A T )/2 we get GCG method (by Concus, Golub and Widlund). In general, P is not necessary to be taken positive definite, but it must be symmetric (P T = P ). Therefore, the minimality property does not hold. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 7 / 45

Let (x, y) o = x T P y = (x, y) o = (y, x) o. Algorithm: GCG method Input: Given A, b R n and x 0 R n and r 0 = b Ax 0 = p 0. 1: Set k = 0. 2: repeat 3: Compute α k = (r k, Ap k ) o /(Ap k, Ap k ) o ; 4: Compute x k+1 = x k + α k p k ; 5: Compute r k+1 = r k α k Ap k = b Ax k+1 ; 6: Compute β k = (Ar k+1, Ap i ) o /(Ap i, Ap i ) o, for i = 0, 1,..., k; 7: Compute p k+1 = r k+1 + k i=0 β(k) i p i ; 8: Set k = k + 1; 9: until r k = 0 T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 8 / 45

In GCG method, the choice of {β (k) i } k i=1 satisfy: (r k+1, Ap i ) o = 0, i k (2a) (r k+1, Ar i ) o = 0, i k (2b) (Ap i, Ap j ) o = 0, i j (2c) Theorem 1 x k+1 = x 0 + k i=0 α kp i minimizes f(x) = 1 2 (b Ax)T P (b Ax) over x = x 0 + < p 0,, p k >, where P is s.p.d. (The proof is the same as that of classical CG method). If P is indefinite, which is allowed in GCG method, then the minimality property does not hold. x k+1 is the critical point of f(x) over x = x 0 + < p 0,, p k >. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 9 / 45

Question Can the GCG method break down? i.e., Can α k in GCG method be zero? Consider the numerator of α k : (r k, Ap k ) o = (r k, Ar k ) o [by Line 7 in GCG Algorithm and (2a) ] = r T k P Ar k = r T k AT P r k [Take transpose] = r T k (P A + A T P ) r k. (3) 2 From (3), if (P A + A T P ) is positive definite, then α k 0 unless r k = 0. Hence if the matrix A satisfies (P A + A T P ) positive definite, then GCG method can not break down. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 10 / 45

From Lines 5 and 7 in GCG Algorithm, r k and p k can be rewritten by r k = ψ k (A)r 0, p k = ϕ k (A)r 0, (4a) (4b) where ψ k and ϕ k are polynomials of degree k with ψ k (0) = 1. From (4a) and (2b) follows that (r k+1, A i+1 r 0 ) o = 0, i = 0, 1,..., k. (5) From (4b) and Line 6 in GCG Algorithm, the numerator of β (k) i expressed by can be (Ar k+1, Ap i ) o = r T k+1 AT P Ap i = r T k+1 AT P Aϕ i (A)r 0. (6) T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 11 / 45

If A T P can be expressed by A T P = P θ s (A), (7) where θ s is some polynomial of degree s. Then (6) can be written by (Ar k+1, Ap i ) o = rk+1 T AT P Aϕ i (A)r 0 = rk+1 T P θ s(a)aϕ i (A)r 0 = (r k+1, Aθ s (A)ϕ i (A)r 0 ) o. (8) From (5) we know that if s + i k, then (8) is zero, i.e.,(ar k+1, Ap i ) o = 0. Hence β (k) i = 0, i = 0, 1,..., k s. But only in the special case s will be small. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 12 / 45

For instance : (i) In classical CG method, A is s.p.d, P is taking by A 1. Then A T P = AA 1 = I = A 1 A = A 1 θ 1 (A), where θ 1 (x) = x, s = 1. So, β (k) i = 0, for all i + 1 k, it is only β (k) k 0. (ii) Concus, Golub and Widlund proposed GCG method, it solves M 1 Ax = M 1 b. (A: unsymmetric), where M = (A + A T )/2 and P = (A + A T )/2 (P may be indefinite). Check condition (7): (M 1 A) T P = A T M 1 M = A T = M(2I M 1 A) = P (2I M 1 A). Then θ s (M 1 A) = 2I M 1 A, where θ 1 (x) = 2 x, s = 1. Thus β (k) i = 0, i = 0, 1,..., k 1. Therefore we only use r k+1 and p k to construct p k+1. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 13 / 45

Check condition A T P + P A: (M 1 A) T M + MM 1 A = A T + A indefinite The method can possibly break down. (iii) The other case s = 1 is BCG (BiCG) (See next paragraph). Remark 2 Except the above three cases, the degree s is usually very large. That is, we need to save all directions p i (i = 0, 1,..., k) in order to construct p k+1 satisfying the conjugate orthogonalization condition (2c). In GCG method, each iteration step needs to save 2k + 5 vectors (x k+1, r k+1, p k+1, {Ap i } k i=0, {p i} k i=0 ), k + 3 inner products (Here k is the iteration number). Hence, if k is large, then the space of storage and the computation cost can become very large and can not be acceptable. So, GCG method, in general, has some practical difficulty. Such as GCR, GMRES (by SAAD) methods, they preserve the optimality (P > 0), but it is too expensive (s is very large). T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 14 / 45

Modification: (i) Restarted: If GCG method does not converge after m + 1 iterations, then we take x k+1 as x 0 and restart GCG method. There are at most 2m + 5 saving vectors. (ii) Truncated: The most expensive step of GCG method is to compute β (k) i, i = 0, 1,..., k so that p k+1 satisfies (2c). We now release the condition (2c) to require that p k+1 and the nearest m direction {p i } k i=k m+1 satisfy the conjugate orthogonalization condition. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 15 / 45

BCG method (A: unsymmetric) BCG is similar to the CG, it does not need to save the search direction. But the norm of the residual by BCG does not preserve the minimal property. Solve Ax = b by considering A T y = c (phantom). Let à = [ A 0 0 A T ], x = [ x y ], b = [ b c ]. Consider [ Take P = 0 A T A 1 0 ] à x = b. (P = P T ). This implies à T P = P Ã. From (7) we know that s = 1 for à x = b. Hence it only needs to save one direction p k as in the classical CG method. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 16 / 45

Apply GCG method to à x = b [ ] x0 1: Given x 0 =, compute p ˆx 0 = r 0 = b à x 0 = 0 2: Set k = 0. 3: repeat 4: Compute α k = ( r k, à p k) o /(à p k, à p k) o ; 5: Compute x k+1 = x k + α k p k ; 6: Compute r k+1 = r k α k à p k ; 7: Compute β k = (à r k+1, à p k) o /(à p k, à p k) o ; 8: Compute p k+1 = r k+1 + β k p k ; 9: Set k = k + 1; 10: until r k = 0 [ r0 ˆr 0 ]. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 17 / 45

Simplification (BCG method) 1: Given x 0, compute p 0 = r 0 = b Ax 0. 2: Choose ˆr 0, ˆp 0 = ˆr 0. 3: Set k = 0. 4: repeat 5: Compute α k = (ˆr k, r k )/(ˆp k, Ap k ); 6: Compute x k+1 = x k + α k p k ; 7: Compute r k+1 = r k α k Ap k, ˆr k+1 = ˆr k α k A T ˆp k ; 8: Compute β k = (ˆr k+1, r k+1 )/(ˆr k, r k ); 9: Compute p k+1 = r k+1 + β k p k, ˆp k+1 = ˆr k+1 + β k ˆp k ; 10: Set k = k + 1; 11: until r k = 0 From above we have (Ã p k, Ã p k) o = [ p T k AT, ˆp T k A] [ 0 A T A 1 0 ] [ Apk A T ˆp k ] = 2(ˆp k, Ap k ). T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 18 / 45

BCG method satisfies the following relations: rk T ˆp i = ˆr k T p i = 0, i < k (9a) p T k AT ˆp i = ˆp T k Ap i = 0, i < k (9b) rk T ˆr i = ˆr k T r i = 0, i < k (9c) Definition 2 (9c) and (9b) are called biorthogonality and biconjugacy condition, respectively. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 19 / 45

Property: (i) In BCG method, the residual of the linear equation does not satisfy the minimal property, because P is taken by ( ) 0 A T P = A 1 0 and P is symmetric, but not positive definite. The minimal value of the functional f(x) may not exist. (ii) BCG method can break down, because Z = (ÃT P + P Ã)/2 is not positive definite. From above discussion, α k can be zero. But this case occurs very few. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 20 / 45

GCG GCR, GCR(k) BCG Orthomin(k) CGS Orthodir BiCGSTAB Orthores QMR GMRES(m) TFQMR FOM Axelsson LS T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 21 / 45

The polynomial equivalent method of the CG method Consider first A is s.p.d. CG-method 1: r 0 = b Ax 0 = p 0. 2: Set k = 0. 3: repeat 4: α k = (r k,p k ) (p k,ap k ) = (r k,r k ) (p k,ap k ) ; 5: x k+1 = x k + α k p k ; 6: r k+1 = r k α k Ap k ; 7: β k = (r k+1,ap k ) (p k,ap k ) = (r k+1,r k+1 ) (r k,r k ) ; 8: p k+1 = r k+1 + β k p k ; 9: k = k + 1; 10: until r k = 0 Equivalent CG-method 1: r 0 = b Ax 0 = p 0, p 1 = 1, ρ 1 = 1. 2: Set k = 0. 3: repeat 4: ρ k = rk T r k, β k = ρ k ρ k 1 ; 5: p k = r k + β k p k 1 ; 6: σ k = p T k Ap k, α k = ρ k σ k ; 7: x k+1 = x k + α k p k ; 8: r k+1 = r k α k Ap k ; 9: k = k + 1; 10: until r k = 0 T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 22 / 45

Remark 3 1. E k = r T k A 1 r k = min x x0 +K k b Ax 2 A 1 2. r T i r j = 0, p T i Ap j = 0, i j. From the structure of the new form of the CG method, we write r k = ϕ k (A)r 0, p k = ψ k (A)r 0 where ϕ k and ψ k are polynomial of degree k. Define ϕ 0 (τ) 1 and ϕ 1 (τ) 0. Then we find with and with p k = ϕ k (A)r 0 + β k ψ k 1 (A)r 0 ψ k (A)r 0 ψ k (τ) ϕ k (τ) + β k ψ k 1 (τ), r k+1 = ϕ k (A)r 0 α k Aψ k (A)r 0 ϕ k+1 (A)r 0 ϕ k+1 (τ) ϕ k (τ) α k τψ k (τ). (10a) (10b) (11a) (11b) T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 23 / 45

The polynomial equivalent method of the CG method : 1: ϕ 0 1, ϕ 1 0, ρ 1 = 1. 2: for k = 0, 1, 2,... do 3: ρ k = (ϕ k, ϕ k ), β k = ρ k ρ k 1 ; 4: ψ k = ϕ k + β k ψ k 1 ; 5: σ k = (ψ k, θψ k ), α k = ρ k σ k ; 6: ϕ k+1 = ϕ k α k θψ k ; 7: end for where θ(τ) = τ. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 24 / 45

The minimization property reads We also have E k = (ϕ k, θ 1 ϕ k ) = min ϕ P N (ϕ, θ 1 ϕ) ϕ(0) 2. (ϕ i, ϕ j ) = 0, i j from (r i, r j ) = 0, i j. (ψ i, θψ j ) = 0, i j from (p i, Ap j ) = 0, i j. Theorem 3 Let [, ] be any symmetric bilinear form satisfying [ϕχ, ψ] = [ϕ, χψ] ϕ, ψ, χ P N. Let the sequence of ϕ i and ψ i be constructed according to PE algorithm, but using [, ] instead (, ). Then as long as the algorithm does not break down by zero division, then ϕ i and ψ i satisfy with θ(τ) τ. [ϕ i, ϕ j ] = ρ i δ ij, [ψ i, θψ j ] = σ i δ ij T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 25 / 45

Bi-Conjugate Gradient algorithm 1: Given r 0 = b Ax 0, p 1 = ˆp 1 and ˆr 0 arbitrary. 2: for k = 0, 1, 2,... do 3: ρ k = ˆr k T r k, β k = ρ k /ρ k 1 ; 4: p k = r k + β k p k 1, ˆp k = ˆr k + β k ˆp k 1 ; 5: σ k = ˆp T k Ap k, α k = ρ k /σ k ; 6: r k+1 = r k α k Ap k, ˆr k+1 = ˆr k α k A T ˆp k ; 7: x k+1 = x k + α k p k ; 8: end for Property: r k = b Ax k, r T i ˆr j = 0, i j and p T i AT ˆp j = 0, i j. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 26 / 45

Consider Ax = b, A : nonsymmetric. Given x 0, r 0 = b Ax 0, let ˆr 0 be a suitably chosen vector. Define [, ] by [ϕ, ψ] = ˆr T 0 ϕ(a)ψ(a)r 0 = (ϕ(a T )ˆr 0 ) T ψ(a)r 0 and define p 1 = ˆp 1 = 0. (If A symmetric : (ϕ, ψ) = r T 0 ϕ(a)ψ(a)r 0). Then we have r k = ϕ k (A)r 0, ˆr k = ϕ k (A T )ˆr 0, p k = ψ k (A)r 0, ˆp k = ψ k (A T )ˆr 0 with ϕ k and ψ k according to (10b) and (11b). Indeed, these vectors can be produced by the Bi-Conjugate Gradient algorithm: T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 27 / 45

Squaring the CG algorithm: CGS Algorithm (SISC, 1989, Sonneveld) Assume that Bi-CG is converging well. Then r k 0 as k. Because r k = ϕ k (A)r 0, ϕ k (A) behaves like contracting operators. Expect: ϕ k (A T ) behaves like contracting operators (i.e., ˆr k 0). But quasi-residuals ˆr k is not exploited, they need to be computed for the ρ k and σ k. Disadvantage: Work of Bi-CG is twice the work of CG and in general A T v is not easy to compute. Especially if A is stored with a general data structure. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 28 / 45

Improvement: Using Polynomial equivalent algorithm to CG. Since ρ k = [ϕ k, ϕ k ] and σ k = [ψ k, θψ k ], [, ] has the property [ϕχ, ψ] = [ϕ, χψ]. Let ϕ 0 = 1. Then ρ k = [ϕ 0, ϕ 2 k ], σ k = [ϕ 0, θψ 2 k ]. Remark 4 { ϕk+1 = ϕ k α k θψ k, ψ k = ϕ k + β k ψ k 1. ρ k = ˆr T k r k = (ϕ k (A T )ˆr 0 ) T (ϕ k (A)r 0 ) = ˆr T 0 ϕ 2 k (A)r 0, σ k = ˆp T k Ap k = (ψ k (A T )ˆr 0 ) T A(ψ k (A)r 0 ) = ˆr T 0 Aψ 2 k (A)r 0. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 29 / 45

Purpose: 1 Find an algorithm that generates the polynomial ϕ 2 k and ψ2 k rather than ϕ k and ψ k. 2 Compute approximated solution x k with r k = ϕ 2 k (A)r 0 as residuals (try to interpret). Because ρ k = ˆr T 0 r k with r k = ϕ 2 k (A)r 0, ˆr k and ˆp k need not to be computed. How to compute ϕ 2 k and ψ2 k? ψ 2 k = [ϕ k + β k ψ k 1 ] 2 = ϕ 2 k + 2β kϕ k ψ k 1 + β 2 k ψ2 k 1, ϕ 2 k+1 = [ϕ k α k θψ k ] 2 = ϕ 2 k 2α kθϕ k ψ k + α 2 k θ2 ψ 2 k. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 30 / 45

Since ϕ k ψ k = ϕ k [ϕ k + β k ψ k 1 ] = ϕ 2 k + β k ϕ k ψ k 1, we only need to compute ϕ k ψ k 1, ϕ 2 k and ψ2 k. Now define for k 0 : From Φ k = ϕ 2 k, Θ k = ϕ k ψ k 1, Ψ k 1 = ψ 2 k 1. ψ 2 k = ϕ 2 k + 2β k ϕ k ψ k 1 + β 2 kψ 2 k 1, ϕ k+1 ψ k = (ϕ k α k θψ k )ψ k = ϕ k ψ k α k θψ 2 k, ϕ 2 k+1 = ϕ 2 k 2α k θϕ k ψ k + α 2 kθ 2 ψ 2 k = ϕ 2 k α k θ(ϕ k ψ k α k θψ 2 k) α k θϕ k ψ k = ϕ 2 k α k θ(ϕ k+1 ψ k + ϕ k ψ k ), we have Y k = ϕ k ψ k = Φ k + β k Θ k, Ψ k = Φ k + 2β k Θ k + βkψ 2 k 1 = Y k + β k (Θ k + β k Ψ k 1 ), Θ k+1 = Y k α k θψ k, Φ k+1 = Φ k α k θ(y k + Θ k+1 ). T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 31 / 45

Bi-Conjugate Gradient 1: Given r 0 = b Ax 0, p 1 = ˆp 1 and arbitrary ˆr 0. 2: for k = 0, 1, 2,... do 3: ρ k = ˆr T k r k, β k = ρ k /ρ k 1 ; 4: p k = r k + β k p k 1, ˆp k = ˆr k + β k ˆp k 1 ; 5: σ k = ˆp T k Ap k, α k = ρ k /σ k ; 6: r k+1 = r k α k Ap k, ˆr k+1 = ˆr k α k A T ˆp k ; 7: x k+1 = x k + α k p k ; 8: end for CGS 1: Φ 0 1, Θ 0 Ψ 1 0, ρ 1 = 1. 2: for k = 0, 1, 2,... do 3: ρ k = [1, Φ k ], β k = ρ k /ρ k 1 ; 4: Y k = Φ k + β k Θ k ; 5: Ψ k = Y k + β k (Θ k + β k Ψ k 1 ); 6: σ k = [1, θψ k ], α k = ρ k /σ k ; 7: Θ k+1 = Y k α k θψ k ; 8: Φ k+1 = Φ k α k θ(y k + Θ k+1 ); 9: end for T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 32 / 45

CGS 1: Φ 0 1, Θ 0 Ψ 1 0, ρ 1 = 1. 2: for k = 0, 1, 2,... do 3: ρ k = [1, Φ k ], β k = ρ k /ρ k 1 ; 4: Y k = Φ k + β k Θ k ; 5: Ψ k = Y k + β k (Θ k + β k Ψ k 1 ); 6: σ k = [1, θψ k ], α k = ρ k /σ k ; 7: Θ k+1 = Y k α k θψ k ; 8: Φ k+1 = Φ k α k θ(y k + Θ k+1 ); 9: end for CGS Variant 1: Given r 0 = b Ax 0, q 0 = p 1 = 0, ρ 1 = 1. 2: for k = 0, 1, 2,... do 3: ρ k = ˆr 0 T r k, β k = ρ k /ρ k 1 ; 4: u k = r k + β k q k ; 5: p k = u k + β k (q k + β k p k 1 ); 6: v k = Ap k ; 7: σ k = ˆr 0 T v k, α k = ρ k /σ k ; 8: q k+1 = u k α k v k ; 9: r k+1 = r k α k A(u k +q k+1 ); 10: x k+1 = x k + α k (u k + q k+1 ); 11: end for Define r k = Φ k (A)r 0, q k = Θ k (A)r 0, p k = Ψ k (A)r 0 and u k = Y k (A)r 0. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 33 / 45

Since r 0 = b Ax 0, r k+1 r k = A(x k x k+1 ), we have that r k = b Ax k. So this algorithm produces x k of which the residual satisfy r k = ϕ 2 k (A)r 0. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 34 / 45

Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems (SISC, 1992, Van der Vorst) Bi-CG method 1: Given r 0 = b Ax 0, (ˆr 0, r 0 ) 0, ρ 0 = 1, ˆp 0 = p 0 = 0. 2: for k = 1, 2,... do 3: ρ k = (ˆr k 1, r k 1 ); 4: β k = ρ k /ρ k 1 ; 5: p k = r k 1 + β k p k 1, ˆp k = ˆr k 1 + β k ˆp k 1 ; 6: v k = Ap k ; 7: α k = ρ k /(ˆp k, v k ); 8: x k = x k 1 + α k p k ; 9: Stop here, if x k is accurate enough. 10: r k = r k 1 α k v k = r k 1 α k Ap k ; 11: ˆr k = ˆr k 1 α k A T ˆp k ; 12: end for T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 35 / 45

Property: (i) r k ˆr 0,..., ˆr k 1 and ˆr k r 0,..., r k 1. (ii) three-term recurrence relations between {r k } and {ˆr k }. (iii) It terminates within n steps, but no minimal property. Since it implies that r Bi CG k = ϕ k (A)r 0, ˆr Bi CG k = ϕ k (A T )ˆr 0, (r k, ˆr i ) = ( ϕ k (A)r 0, ϕ i (A T )ˆr 0 ) = (ϕi (A)ϕ k (A)r 0, ˆr 0 ) = 0, i < k. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 36 / 45

CGS Method 1: Given x 0, r 0 = b Ax 0, (r 0, ˆr 0 ) 0, ˆr 0 = r 0, ρ 0 = 1, p 0 = q 0 = 0. 2: for k = 1, 2,... do 3: ρ k = (ˆr 0, r k 1 ), β k = ρ k /ρ k 1 ; 4: u k = r k 1 + β k q k 1 ; 5: p k = u k + β k (q k 1 + β k p k 1 ); 6: v k = Ap k ; 7: α k = ρ k /(ˆr 0, v k ); 8: q k = u k α k v k ; 9: w k = u k + q k ; 10: x k = x k 1 + α k w k ; 11: Stop here, if x k is accurate enough. 12: r k = r k 1 α k Aw k ; 13: end for We have r CGS k = ϕ k (A) 2 r 0. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 37 / 45

From Bi-CG method we have r Bi CG k = ϕ k (A)r 0 and p k+1 = ψ k (A)r 0. Thus we get ψ k (A)r 0 = (ϕ k (A) + β k ψ k 1 (A)) r 0, and ϕ k (A)r 0 = (ϕ k 1 (A) α k Aψ k 1 (A)) r 0, where ψ k = ϕ k + β k ψ k 1 and ϕ k = ϕ k 1 α k θψ k 1. Since ( ϕk (A)r 0, ϕ j (A T )ˆr 0 ) = 0, j < k, it holds that if and only if ϕ k (A)r 0 ˆr 0, A T ˆr 0,..., (A T ) k 1ˆr 0 ( ϕ j (A)ϕ k (A)r 0, ˆr 0, ) = 0 for some polynomial ϕ j of degree j < k for j = 0, 1,..., k 1. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 38 / 45

In Bi-CG method, we take ϕ j = ϕ j, ˆr k = ϕ k (A T )ˆr 0 and exploit it in CGS to get rk CGS = ϕ 2 k (A)r 0. Now r k = ϕ k (A)ϕ k (A)r 0. How to choose ϕ k polynomial of degree k so that r k satisfies the minimum. Like polynomial, we can determine the optimal parameters of ϕ k so that r k satisfies the minimum. But the optimal parameters for the Chebychev polynomial are in general not easily obtainable. Now we take where ϕ k η k (x), η k (x) = (1 ω 1 x)(1 ω 2 x) (1 ω k x). Here ω j are suitable constants to be selected. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 39 / 45

Define r k = η k (A)ϕ k (A)r 0. Then r k = η k (A)ϕ k (A)r 0 = (1 ω k A)η k 1 (A) (ϕ k 1 (A) α k Aψ k 1 (A)) r 0 = {(η k 1 (A)ϕ k 1 (A) α k Aη k 1 (A)ψ k 1 (A))} r 0 ω k A {(η k 1 (A)ϕ k 1 (A) α k Aη k 1 (A)ψ k 1 (A))} r 0 = r k 1 α k Ap k ω k A(r k 1 α k Ap k ) T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 40 / 45

and p k+1 = η k (A)ψ k (A)r 0 = η k (A) (ϕ k (A) + β k ψ k 1 (A)) r 0 = η k (A)ϕ k (A)r 0 + β k (1 ω k A)η k 1 (A)ψ k 1 (A)r 0 = η k (A)ϕ k (A)r 0 + β k η k 1 (A)ψ k 1 (A)r 0 β k ω k Aη k 1 (A)ψ k 1 (A)r 0 = r k + β k (p k ω k Ap k ). Recover the constants ρ k, β k, and α k in Bi-CG method. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 41 / 45

We now compute β k : Let ˆρ k+1 = (ˆr 0, η k (A)ϕ k (A)r 0 ) = ( η k (A T )ˆr 0, ϕ k (A)r 0 ). From Bi-CG we have ϕ k (A)r 0 all vectors µ k 1 (A T )ˆr 0, where µ k 1 is an arbitrary polynomial of degree k 1. Consider the highest order term of η k (A T ) (when computing ˆρ k+1 ) is ( 1) k ω 1 ω 2 ω k (A T ) k. From Bi-CG method, we also have ρ k+1 = ( ϕ k (A T )ˆr 0, ϕ k (A)r 0 ). The highest order term of ϕ k (A T ) is ( 1) k α 1 α k (A T ) k. Thus β k = (ˆρ k /ˆρ k 1 ) (α k 1 /ω k 1 ), T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 42 / 45

because β k = ρ ( k α1 α k 1 (A T ) k 1ˆr ) 0, ϕ k 1 (A)r 0 = ρ k 1 (α 1 α k 2 (A T ) k 2ˆr 0, ϕ k 2 (A)r 0 ) ( α1 α k 1 ω 1 ω ω k 1 1 ω k 1 (A T ) k 1ˆr ) 0, ϕ k 1 (A)r 0 = ( α1 α k 2 ω 1 ω ω k 2 1 ω k 2 (A T ) k 2ˆr ) 0, ϕ k 2 (A)r 0 = (ˆρ k /ˆρ k 1 ) (α k 1 /ω k 1 ). Similarly, we can compute ρ k and α k. Let r k = r k 1 γay, x k = x k 1 + γy (side product). Compute ω k so that r k = η k (A)ϕ k (A)r 0 is minimized in 2-norm as a function of ω k. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 43 / 45

Bi-CGSTAB Method 1: Given x 0, r 0 = b Ax 0, ˆr 0 arbitrary, such that (r 0, ˆr 0 ) 0, e.g. ˆr 0 = r 0, ρ 0 = α = ω 0 = 1, v 0 = p 0 = 0. 2: for k = 1, 2,... do 3: ρ k = (ˆr 0, r k 1 ), β = (ρ k /ρ k 1 )(α/ω k 1 ); 4: p k = r k 1 + β(p k 1 ω k 1 v k 1 ); 5: v k = Ap k ; 6: α = ρ k /(ˆr 0, v k ); 7: s = r k 1 αv k ; 8: t = As; 9: ω k = (t, s)/(t, t); 10: x k = x k 1 + αp k + ω k s (= x k 1 + αp k + ω k (r k 1 αap k )); 11: Stop here, if x k is accurate enough. 12: r k = s ω k t (= r k 1 αap k ω k A(r k 1 αap k ) = r k 1 A(αp k + ω k (r k 1 αap k )); 13: end for T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 44 / 45

Preconditioned Bi-CGSTAB-P: Rewrite Ax = b as à x = b with à = K 1 1 AK 1 2, where x = K2 1 x and b = K1 1 b. Then p k := K1 1 p k, ṽ k := K1 1 v k, r k := K1 1 r k, s := K1 1 s, t := K1 1 t, x k := K 2 x k, r 0 := K T 1 ˆr 0. T.M. Huang (NTNU) GCG-type Methods for Nonsymmetric LS December 6, 2011 45 / 45