Matrices and vectors. Matrix and vector. a 11 a 12 a 1n a 21 a 22 a 2n A = b 1 b 2. b m. R m n, b = = ( a ij. a m1 a m2 a mn. def

Transcript

1 Matrices and vectors Matrix and vector a 11 a 12 a 1n a 21 a 22 a 2n A = a m1 a m2 a mn def = ( a ij ) R m n, b = b 1 b 2 b m Rm

2 Matrix and vectors in linear equations: example E 1 : x 1 + x 2 + 3x 4 = 4, E 2 : 2x 1 + x 2 x 3 + x 4 = 1, E 3 : 3x 1 x 2 x 3 + 2x 4 = 3, E 4 : x 1 + 2x 2 + 3x 3 x 4 = 4 coefficient matrix unknown vector A def = x def = x 1 x 2 x 3 x 4 right hand side vector (RHS) , b def = ,

3 General linear equations E 1 : a 11 x 1 + a 12 x 2 + a 1n x n = b 1, E 2 : a 21 x 1 + a 22 x 2 + a 2n x n = b 2, E n : a n1 x 1 + a n2 x 2 + a nn x n = b n,

4 General linear equations A def = E 1 : a 11 x 1 + a 12 x 2 + a 1n x n = b 1, E 2 : a 21 x 1 + a 22 x 2 + a 2n x n = b 2, E n : a n1 x 1 + a n2 x 2 + a nn x n = b n, a 11 a 12 a 1n b 1 a 21 a 22 a 2n, b def b 2 =, x def = a n1 a n2 a nn b n x 1 x 2 x n

5 General linear equations A def = E 1 : a 11 x 1 + a 12 x 2 + a 1n x n = b 1, E 2 : a 21 x 1 + a 22 x 2 + a 2n x n = b 2, E n : a n1 x 1 + a n2 x 2 + a nn x n = b n, a 11 a 12 a 1n b 1 a 21 a 22 a 2n, b def b 2 =, x def = a n1 a n2 a nn b n equation E j maps to row j of A: ( ) a j1, a j2,, a jn, 1 j n a 1k a 2k variable x k relates to column k of A:, 1 k n a nk x 1 x 2 x n

6 Gaussian Elimination with partial pivoting (I) a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n, x = equation E j maps to row j of A, x 1 relates to pivoting: choose largest entry in absolute value: a 11 a 21 a n1 x 1 x 2 x n piv def = argmax 1 j n a j1, ( a piv,1 = max 1 j n a j1 ) exchange equations E 1 and E piv (E 1 E piv )

7 Gaussian Elimination with partial pivoting (II) A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn, b = b 1 b 2 b n, x = x 1 x 2 x n elimination of x 1 from E 2 through E n : ( E j a ) j1 E 1 (E j ), a j1 a 11 1, 2 j n new row j of A ( aj1 a j2 a jn ) a j1 a 11 ( a11 a 12 a 1n ) a 11 = ( 0 a j2 a j1 a 11 a 12 a jn a j1 a 11 a 1n ), 2 j n new j th component of b b j a j1 a 11 b 1

8 Gaussian Elimination with partial pivoting (III) new A = a 11 a 12 a 1n 0 a 22 a 2n 0 a n2 a nn overwrite ==== A, where a jk = a jk a j1 a 11 a 1k, 2 j n, 2 k n new b = b 1 b 2 b n overwrite ==== b where b j = b j a j1 a 11 b 1, 2 j n, 2 k n (bold faced entries are computed from elimination process)

9 Gaussian Elimination with partial pivoting (IV) a 11 a 12 a 1n 0 a 22 a 2n A =, b = 0 a n2 a nn b 1 b 2 b n

10 Gaussian Elimination with partial pivoting (IV) a 11 a 12 a 1n 0 a 22 a 2n A =, b = 0 a n2 a nn b 1 b 2 b n repeat same process on E s for s = 2,, n 1: pivoting: choose largest entry in absolute value: piv def = argmax s j n a js, ( a piv,s = max 1 j n a js ) exchange equations E s and E piv (E s E piv )

11 Gaussian Elimination with partial pivoting (IV) a 11 a 12 a 1n 0 a 22 a 2n A =, b = 0 a n2 a nn b 1 b 2 b n repeat same process on E s for s = 2,, n 1: pivoting: choose largest entry in absolute value: piv def = argmax s j n a js, ( a piv,s = max 1 j n a js ) exchange equations E s and E piv (E s E piv ) eliminating x s from E s+1 through E n : ( E j a ) js E s (E j ), s + 1 j n a ss a jk = a jk a js a ss a sk overwrite ==== a jk, s + 1 j n, s + 1 k n, b j = b j a js a ss b s overwrite ==== b j, s + 1 j n

12 Gaussian Elimination with partial pivoting: summary a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n

13 Gaussian Elimination with partial pivoting: summary a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv def = argmax s j n a js, E s E piv

14 Gaussian Elimination with partial pivoting: summary a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: piv def = argmax s j n a js, E s E piv eliminating xs from E s+1 through E n : a jk = a jk a js a ss a sk overwrite ==== a jk, s + 1 j n, s + 1 k n, b j = b j a js a ss b s overwrite ==== b j, s + 1 j n

15 Equations after Gaussian Elimination, backward substitution A = a 11 a 12 a 1n 0 a 22 a 2n 0 0 a nn, b = b 1 b 2 b n

16 Equations after Gaussian Elimination, backward substitution A = a 11 a 12 a 1n 0 a 22 a 2n 0 0 a nn, b = b 1 b 2 b n for s = n, n 1,, 1: x s = b s n k=s+1 a skx k a ss matlab command x = A\b

17 Gaussian Elimination: cost analysis (I) for s = 1, 2,, n 1: pivoting: (n s + 1 comparisons) piv = argmax s j n a js, E s E piv

18 Gaussian Elimination: cost analysis (I) for s = 1, 2,, n 1: pivoting: (n s + 1 comparisons) piv = argmax s j n a js, E s E piv eliminating x s from E s+1 through E n : a jk = a jk a js a ss a sk, s + 1 j n, s + 1 k n, b j = b j a js a ss b s, s + 1 j n Counting operations compute piv for each s, and perform swaps E s E piv for each s, compute ratios a js a ss for each j s + 1 for each s, compute a jk for each pair of j, k s + 1 for each s, compute b j for each j s + 1 Do not count integer operations; do not count memory costs

19 Gaussian Elimination: cost analysis (II) for s = 1, 2,, n 1: pivoting: (n s + 1 comparisons) piv = argmax s j n a js, E s E piv (total comparisons: n 1 n(n+1) s=1 (n s + 1) = 2 1 )

20 Gaussian Elimination: cost analysis (II) for s = 1, 2,, n 1: pivoting: (n s + 1 comparisons) piv = argmax s j n a js, E s E piv (total comparisons: n 1 n(n+1) s=1 (n s + 1) = 2 1 ) eliminating xs from E s+1 through E n : a jk = a jk a js a ss a sk, s + 1 j n, s + 1 k n, b j = b j a js a ss b s, s + 1 j n (total cost for computing { a js a ss }: n 1 n(n 1) s=1 (n s) = 2 ) (total cost for computing {a jk }: n 1 s=1 2(n s)2 = n(n 1)(2n 1) (total cost for computing {b j }: n 1 s=1 2(n s) = n(n 1) ) 3 )

21 Gaussian Elimination: cost analysis (II) for s = 1, 2,, n 1: pivoting: (n s + 1 comparisons) piv = argmax s j n a js, E s E piv

22 Gaussian Elimination: cost analysis (II) for s = 1, 2,, n 1: pivoting: (n s + 1 comparisons) piv = argmax s j n a js, E s E piv eliminating xs from E s+1 through E n : a jk = a jk a js a ss a sk, s + 1 j n, s + 1 k n, b j = b j a js a ss b s, s + 1 j n about 2(n s) 2 additions and multiplications for each s grand total, up to n 3 term: 2 3 n3 additions and multiplications

23 Gaussian Elimination: performance on mac desktop, in octave, GEPPruntime n 3

24 NO scaled partial pivoting

25 Gaussian Elimination with partial pivoting blue: A R n n, b R n random

26 Gaussian Elimination with partial pivoting blue: A R n n, b R n random red: A R n n is Wilkinson matrix A = , b R n random

27 Gaussian Elimination with complete pivoting, A R 2 2 ( a11 a A = 12 a 21 a 22 ) ( b1, b = b 2 ) ( x1, x = x 2 ) pivoting: choose largest entry in absolute value: (ip, jp) def = argmax 1 i,j 2 a ij, ( a ip,jp = max 1 j n a ij ) exchange equations E 1 and E ip (E 1 E ip ) exchange columns/variables 1 and jp perform Gaussian Elimination without pivoting on the resulting 2 2 system

28 Gaussian Elimination with complete pivoting a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n

29 Gaussian Elimination with complete pivoting a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: (ip, jp) def = argmax s i,j n a ij, E s E ip swap matrix columns/ variables s and jp

30 Gaussian Elimination with complete pivoting a 11 a 12 a 1n a 21 a 22 a 2n A =, b = a n1 a n2 a nn b 1 b 2 b n for s = 1, 2,, n 1: pivoting: choose largest entry in absolute value: (ip, jp) def = argmax s i,j n a ij, E s E ip swap matrix columns/ variables s and jp eliminating xs from E s+1 through E n : a jk = a jk a js a ss a sk overwrite ==== a jk, s + 1 j n, s + 1 k n, b j = b j a js a ss b s overwrite ==== b j, s + 1 j n more numerically stable, but too expensive in practice

31 Matrix Algebra Definition: Let a 1,1 a 1,2 a 1,m a 2,1 a 2,2 a 2,m A = a n,1 a n,2 a n,m b 1,1 b 1,2 b 1,m b 2,1 b 2,2 b 2,m B = b n,1 b n,2 b n,m C def = def = (a ij ) R n m, def = (b ij ) R n m, then α a 1,1 + β b 1,1 α a 1,2 + β b 1,2 α a 1,m + β b 1,m α a 2,1 + β b 2,1 α a 2,2 + β b 2,2 α a 2,m + β b 2,m α a n,1 + β b n,1 α a n,2 + β b n,2 α a n,m + β b n,m = α A + β B R n m

32 Matrix Algebra: example Let A = ( ) ( 4 2 8, B = ) R 2 3, then C ( def = A 2B = ) R 2 3

33 Matrix Algebra Let A, B, C R n m, and let λ, µ R

34 Matrix-vector product vs linear equations a 11 x 1 + a 12 x 2 + a 1n x n = b 1, a 21 x 1 + a 22 x 2 + a 2n x n = b 2, a n1 x 1 + a n2 x 2 + a nn x n = b n,

35 Matrix-vector product vs linear equations A = a 11 x 1 + a 12 x 2 + a 1n x n = b 1, a 21 x 1 + a 22 x 2 + a 2n x n = b 2, a n1 x 1 + a n2 x 2 + a nn x n = b n, a 11 a 12 a 1n b 1 a 21 a 22 a 2n, b = b 2, x = a n1 a n2 a nn b n x 1 x 2 x n

36 Matrix-vector product vs linear equations a 11 x 1 + a 12 x 2 + a 1n x n = b 1, a 21 x 1 + a 22 x 2 + a 2n x n = b 2, a n1 x 1 + a n2 x 2 + a nn x n = b n, a 11 a 12 a 1n b 1 x 1 a 21 a 22 a 2n A =, b = b 2, x = x 2 a n1 a n2 a nn b n x n a 11 x 1 + a 12 x 2 + a 1n x n A x def a 21 x 1 + a 22 x 2 + a 2n x n =, for any A Rn m, x R m a n1 x 1 + a n2 x 2 + a nn x n equations become: A x = b

37 Matrix-vector product is dot product A x def = a 11 x 1 + a 12 x 2 + a 1n x n a 21 x 1 + a 22 x 2 + a 2n x n a n1 x 1 + a n2 x 2 + a nn x n (A x) j = ( ) ( ) a j1 x 1 + a j2 x 2 + a jn x n = aj1 a j2 a jn x 1 x 2 x n

38 Matrix-vector product, example A = A x = , x = ( 3 1 ( 3 1 ) = )

39 Matrix-matrix product Let a 11 a 12 a 1m b 11 b 12 b 1p a 21 a 22 a 2m A= b 21 b 22 b 2p Rn m, B= Rm p a n1 a n2 a nm b m1 b m2 b mp

40 Matrix-matrix product Let a 11 a 12 a 1m b 11 b 12 b 1p a 21 a 22 a 2m A= b 21 b 22 b 2p Rn m, B= Rm p a n1 a n2 a nm b m1 b m2 b mp Column partition: b 1j B = ( ) def b 2j b 1 b 2 b p, bj = Rm, j = 1,, p b mj

41 Matrix-matrix product Let a 11 a 12 a 1m b 11 b 12 b 1p a 21 a 22 a 2m A= b 21 b 22 b 2p Rn m, B= Rm p a n1 a n2 a nm b m1 b m2 b mp Column partition: b 1j B = ( ) def b 2j b 1 b 2 b p, bj = Rm, j = 1,, p Definition: A B = A ( b 1 b 2 b p ) def = ( A b 1 A b 2 A b p ) R n p b mj

42 Matrix-matrix product Let a 11 a 12 a 1m b 11 b 12 b 1p a 21 a 22 a 2m A= b 21 b 22 b 2p Rn m, B= Rm p a n1 a n2 a nm b m1 b m2 b mp Column partition: b 1j B = ( ) def b 2j b 1 b 2 b p, bj = Rm, j = 1,, p Definition: A B = A ( b 1 b 2 b p ) def = ( A b 1 A b 2 A b p ) R n p b mj entry-wise formula: (A B) jk = (A b k ) j = ( a j1 a j2 a jm ) bk = m a ji b ik i=1

43 Matrix-matrix product, example Let A = R 3 2, B = ( ) R 2 3 C = A B = R 3 3

44 Theorem: Let A R n m, B R m p, C R p k, Then A (B C) = (A B) C Proof: (A (B C)) st = = = = m a si (B C) it i=1 m i=1 a si p b ij c jt j=1 ( p m ) a si b ij j=1 i=1 p (A B) sj c jt j=1 = ((A B) C) st c jt

45 Special matrices square matrix A R n n d 1 d diagonal matrix D = 2 dn identity matrix I = upper-triangular matrix U = lower-triangular matrix L = l 11 l 21 l 22 u 11 u 12 u 1n u 22 u 2n l n1 l n2 l nn u nn