On the linear convergence of the general alternating proximal gradient method for convex minimization

Transcript

1 2014c9 $ Ê Æ Æ 118ò 13Ï Sept., 2014 Operations Research Transactions Vol.18 No.3 2ÂOCqFÝŽ{ 5Âñ Û 1 M ^1, Á éü Œ à¼êú3 5åe4z K, 3O {µee, J Ñ2ÂOCqFÝŽ{. 3 ½^ e, TŽ{äkÛ9 5Âñ5. êš L ²TŽ{kÐêŠLy. ' c O { 2ÂOCqFÝŽ{ ÛÂñ Q- 5Âñ ã aò O êæ aò 90C25 On the linear convergence of the general alternating proximal gradient method for convex minimization WAN Rui 1 XU Zi 1, Abstract In this paper, we propose a general alternating proximal gradient method for linear constrained convex optimization problems with the objective containing two separable functions. Our method is based on the framework of alternating direction method of multipliers. The global and linear convergence of the proposed method is established under certain conditions. Numerical experiments show that the algorithm has good numerical performance. Keywords alternating direction method of multipliers, general alternating proximal gradient method; global convergence, Q-linear convergence Chinese Library Classification O Mathematics Subject Classification 90C25 0 Ú ó Ä3 5å^ e, ü Œ à¼êú4z K, ÙêÆ. min fx + gy x R n,y Rp s. t. Ax + By = b, 0.1 Ù A R m n, B R m p, b R m. f : R n R, g : R p RÑ à¼ê. ÂvFÏµ2013c1014F * Ä7 8µI[g, ÆÄ7]Ï 8 No þ ŒÆnÆêÆX, þ , Department of Mathematics, College of Sciences, Shanghai University, Shanghai , China ÏÕŠö Corresponding author, xuzi@shu.edu.cn

2 2, M ^ 18ò Cc5, Ï à`z DÕ`zÚ$ `z K A^, ²Úåé õïä<úó óšö '5. ùa Kéõ5uØ a [1]!$ ÝW [2,3]! ã?n [4,5] ÚÚO [6]. '5 KäkDÕ½ö$. éu. 0.1, «k { O { Alternating Direction Method of Multipliers, { ADMMŽ{. S ª x k+1 =argmin x L µ x, y k, λ k, y k+1 =argmin y L µ x k+1, y, λ k, λ k+1 = λ k 1 µ Axk+1 + By k+1 b, 0.2 Ù L µ x, y, λ = fx + gy λ T Ax + By b + 1 Ax + By b 2 2 O2. KF¼ ê. ADMMŽ{éuã?n!Ø a!åìæs!œ½5yúúo JÑ zà`z KÑé ^. `³3ur Œ5 K=z $ ê f K, ù3 S JpOŽÇ. ADMMŽ{ÛÂñ5 ²Øy. Cc5, k'admmž{9ùc/ž{âñ ÇJ kéõ. GoldfarbÚ Ma [7] y²8i¼ê1w, ÙFÝLipschitzëYž, ^ADMMJacobid/, 8 I¼êeü Ç O 1 1 k, Œ±\ O k. fx Úgy k ¼ê1w, Ù 2 FÝLipschitzëY, ^Gauss-Seideld/, Goldfarb [8] y²âñ Ç O 1 k. HeÚ Yuan [9] ^C Øªn, 3ü f K k Œ cje, y² Douglas-RachfordO Ž{Âñ Ç O 1 k. Goldstein[10] 38I¼êrà, Ù ¼ê g¼ê, ü f KÑŒ ^ e, y²admmc/ªé ó8i¼êšâñ Ç O 1 k. 3fÚg k rà, ÙFÝ LipschitzëY 2 ^ e, DengÚYin [11] ^2ÂADMMŽ{, y² 5Âñ Ç O 1, Ù c < 1. c k éu. 0.1, Ma [12] JÑOCqFÝŽ{ Alternating Proximal Gradient Method, { APGMŽ{, ÙS ª x k+1 =argmin x fx + 1 x x k τ 1 A T Ax k + By k b µλ k 2 τ 2, 1 y k+1 =argmin y gy + 1 y y k τ 2 B T Ax k+1 + By k b µλ k 2 τ 2, λ k+1 = λ k 1 µ Axk+1 + By k+1 b, y²τ 1 < 1/λ max A T A, τ 2 < 1/λ max B T Bž, dapgms{x k, y k, λ k } Âñ. `:3uš / f K. Äud, JÑ APGMŽ{, =2ÂAPGMŽ{, ÙS ª x k+1 =argmin x fx+ 1 x x k τ 1 A T Ax k +By k b µλ k 2 τ x xk T P x x k, y k+1 =argmin y gy+ 1 y y k τ 2 B T Ax k+1 +By k b µλ k 2 τ y yk T Qy y k, λ k+1 = λ k 1 µ Axk+1 + By k+1 b, Ù P, Q é Ý, P, QŒ±Šâ Sœ¹ÀJ. 0.4

3 3Ï 2ÂOCqFÝŽ{ 5Âñ Û 3 e5, y²2âapgmž{ûâñ5 5Âñ Ý. d, ŠXe b. b KKT:, P u = x, y, λ T, Kx, y, λ vkkt^ : b0.2 ¼êf ÚgÑ à¼ê. dfúgà5œ, 3v f, v g 0, A T λ fx, 0.5 B T λ gy, 0.6 Ax + By b = s 1 s 2, x 1 x 2 v f x 1 x 2 2, x 1, x 2, s 1 fx 1, s 2 fx 2, 0.8 t 1 t 2, y 1 y 2 v g y 1 y 2 2, y 1, y 2, t 1 gy 1, t 2 gy XJfÚg rà, Kv f, v g > 0, džv f Úv g O fúgrà~ê. b0.3 τ 1 < 1/λ max A T A τ 2 < 1/λ max B T B. 1 ÛÂñ! y²2âapgmž{ûâñ5. z[12] Ún3.1Ú½n3.2y ²aq, Œ±y²XeÚn1.1Ú½n1.1. Pu k = x k, y k, λ k T cs :, u = x, y, λ T b0.1 KKT:, H = P Q + 1 I, Ù P = P + 1 µτ 1 I 1 µ AT A, ÀJÜ P, Q, H½. µi Ún1.1 3b á^ e, 3η > 0, d2âapgmž{ 0.4 S{x k, y k, λ k } v u k u 2 H u k+1 u 2 H η u k u k+1 2 H + 2v f x k+1 x 2 + 2v g y k+1 y dún1.1, Œ±2ÂAPGMŽ{ 0.4 ÛÂñ5. ½n1.1 3b á^ e, 2ÂAPGMŽ{0.4S{x k, y k, λ k } Âñ. 0.1 `. y² d 1.1 Œ, u k u 2 H 4~ke., ± uk u 2 H Âñ, {uk }3 ; p, 3Âñf{u kj }, Ù u kj = x kj, y kj, λ kj T, Plim j u kj = û, Ù û = x, ŷ, λ T. e y²û KKT:. dún1.1œ, k ž, u k u k+1 0, =x k x k+1 0, y k y k+1 0, λ k λ k+1 0. Ï λ k+1 = λ k 1 µ Axk+1 + By k+1 b, Œ± Ax k + By k b 0.

4 4, M ^ 18ò ±j, k A x + Bŷ b = Ï { x k+1 =argmin x fx+ 1 x x k τ 1 A T Ax k +By k b µλ k } τ 1 2 x xk T P x x k, Œ `5^ P + 1 I 1 µτ 1 µ AT A x k x k+1 1 µ AT By k y k+1 + A T λ k+1 fx k Ón, { y k+1 = argmin y gy+ 1 y y k τ 2 B T Ax k+1 +By k b µλ k 2 τ 2+ 1 } 2 2 y yk T Qy y k, Œ `5^ Q + 1 I 1 µ BT B y k y k+1 + B T λ k+1 gy k Ï 1.3 éuz kñ á, ±k P + 1 I 1 µτ 1 µ AT A x kj x kj+1 1 µ AT By kj y kj+1 + A T λ kj+1 fx kj j ž, kx k j xk j+1 0, yk j yk j+1 0, ± A T λ f x. 1.6 Ón, Œ B T λ gŷ. 1.7 d 1.2, 1.6, 1.7, Œû = x, ŷ, λ T v. 0.1 KKT^, qï 0.1 à`z K, ±. 0.1 `. û 1 = x 1, ŷ 1, λ 1 T, û 2 = x 2, ŷ 2, λ 2 T {x k, y k, λ k T }ü 4 :, Œû 1, û 2 Ñ. 0.1 `. d 1.1 Œ u k+1 û i 2 H u k û i 2 H, i = 1, P ^ ü>4 lim k uk û i H = η i < +, i = 1, 2. u k û 1 2 H u k û 2 2 H = 2 u k, û 1 û 2 H + û 1 2 H û 2 2 H, 1.9, Œ η 2 1 η 2 2 = 2 û 1, û 1 û 2 H + û 1 2 H û 2 2 H = û 1 û 2 2 H, 1.10

5 3Ï 2ÂOCqFÝŽ{ 5Âñ Û 5 Óž, η 2 1 η 2 2 = 2 û 2, û 1 û 2 H + û 1 2 H û 2 2 H = û 1 û 2 2 H, 1.11 Ü 1.10 Ú 1.11 Œ û 1 û 2 2 H = 0. l{x k, y k, λ k T }4 :. ± u k Âñ u. 2 5Âñ!, y²2âapgmž{ 5Âñ5. =y²3δ > 0, u k u 2 H u k+1 u 2 H δ u k+1 u 2 H 2.1 á, Ù u b0.1 KKT:, ={u k }Q- 5Âñ. e, ü«œ/o λ k+1 λ 2. kšxeb. b2.1 ¼êgFÝ g LipschitzëY, = y 1, y 2 R p, k gy 1 gy 2 L g y 1 y 2 á, Ù L g glipschitz~ê. Ún2.1 3b Úb2.1 á^ e, bb1, Kéu? 0 < ρ 1 < 1, k á, Ù λ k+1 λ 2 c 1 y k+1 y 2 + c 2 y k y k c 1 = L 2 g1 ρ 1 1 λ 1 min BBT > 0, c 2 = ρ 1 1 Q + 1 I 1 µ BT B 2 λ 1 min BBT > 0. y² d `5^ 0.6 Ú 1.4 ±9 g LipschitzëY, Œ± B T λ k+1 λ + Q + 1 I 1 µ BT B y k y k+1 2 = gy k+1 gy a 2.2.b L 2 g y k+1 y ^Øª [11] u + v ρ 1 v ρ 1 u 2, 0 < ρ 1 < >Œ±=z B T λ k+1 λ + Q + 1 I 1 µ BT B y k y k Q 1 + I 1 ρ 1 µ BT B 2 y k y k ρ 1 λ min BB T λ k+1 λ

6 6, M ^ 18ò éá 2.3 Ú 2.5, k 1 1 ρ 1 λ min BB T λ k+1 λ 2 L 2 g y k+1 y + 2 Q+ 1 1 I 1 ρ 1 µ BT B 2 y k y k+1 2, = 2.6 λ k+1 λ 2 L 2 g1 ρ 1 1 λ 1 min BBT y k+1 y 2 +ρ 1 1 Q+ 1 I 1 µ BT B 2 λ 1 min BBT y k y k ùp, 0 < ρ 1 < 1, dbb1, Œc 1 > 0, c 2 > 0. ùpc 1, c 2 O X 2.2.a b ½Â. Ún á. ½n2.1 3Ún2.1b^ e, egrà, A, K3δ > 0, 2.1 á. y² Ï u k u 2 H u k+1 u 2 H x k x k+1 2 P+ y k y k+1 2 Q+ 1 I 1 ρ λmaxbt BI +µ ρ λk λ k v f x k+1 x 2 + 2v g y k+1 y 2, Ù ρ = µ 1+τ2λmaxBT B 2. λ k+1 = λ k 1 µ Axk+1 + By k+1 b, K λ k λ k+1 2 = 1 µ 2 Axk+1 x + By k+1 y 2. ^ 2.4 Œ, µ ρ λ k λ k+1 2 = µ ρ µ 2 Ax k+1 x + By k+1 y 2 Ù 0 < ρ 2 < 1. éá 2.8, 1.1 Œ±=z µ ρ [ µ 2 1 ρ 2 λ min A T A x k+1 x λ max B T B y k+1 y 2] ρ 2 = 1 τ 2λ max B T B [ 1 ρ 2 λ min A T A x k+1 x λ max B T B y k+1 y 2]. 2.8 ρ 2 u k u 2 H u k+1 u 2 H x k x k+1 2ˆP + y k y k+1 2 Q+ 1 µτ I 1 2 ρ λmaxbt BI [ + 2v f + 1 τ 2λ max B T B ] 1 ρ 2 λ min A T A x k+1 x 2 [ + 2v g + 1 τ 2λ max B T B 1 1 ] λ max B T B y k+1 y 2. ρ 2

7 3Ï 2ÂOCqFÝŽ{ 5Âñ Û 7 ^Ún2.1, Œ± u k u 2 H u k+1 u 2 H c 3 x k+1 x 2 + c 4 y k+1 y 2 + c 5 λ k+1 λ 2, 2.9 Ù c 3 = 2v f + 1 τ 2λ max B T B 1 ρ 2 λ min A T A, λ max B T B ρ 2 c 4 = 2v g + 1 τ 2λ max B T B c 1 c 2 λ min Q + 1 I 1 ρ λ maxb T BI c 5 = λ min Q + 1 I 1 ρ λ maxb T BI, 2.11 /c c 1, c 2 OX 2.2.a b ª ½Â. Ï 0 < ρ 2 < 1, dba, Œc 3 > 0. qï v g > 0, Àρ 1, ρ 2 ve ü ^ : 0 < ρ 1 < N < 1, 0 < M < ρ 2 < 1 ž, c 4 > 0, Ù N = L 2 g M = Q + 1 I 1 µ BT B 2 4µv g λ max B T B1 τ 2 λ max B T B > 0, λ min Q + 1 I 1 ρ λ maxb T BI 2v g + 1 τ 2λ max B T B 1 1 ρ 2 λ max B T B c 2 > 0 y, w,kc 5 > 0. { δ = min c 3 λ 1 max P, c 4 λ 1 max Q + 1 I, c 5 µ 1} > 0, K 2.1 y. b2.2 ¼êfFÝ f LipschitzëY, = x 1, x 2 R n, k fx 1 fx 2 L f x 1 x 2 á, Ù L f flipschitz~ê. Ún2.2 3b Úb2.2 á^ e, ba1, Kéu? 0 < t < 1Ú? t 1 > 0, k > 0, λ k+1 λ 2 c 6 x k+1 x 2 + c 7 x k x k c 8 y k y k á, Ù c 6 = L 2 f 1 t 1 λ 1 min AAT > 0, c 7 = t t 1 λ 1 min AAT λ max P T P > 0, c 8 = t 1 µ t 1 λ max B T AA T Bλ 1 min AAT a 2.13.b 2.13.c

8 8, M ^ 18ò y² d `5^ 0.5 Ú 1.3 ±9 f LipschitzëY, Œ± A T λ k+1 λ + P x k x k+1 1 µ AT By k y k+1 2 = fx k+1 fx 2 L 2 f x k+1 x 2. ^ 2.4, 0 < t < 1k A T λ k+1 λ + P x k x k+1 1 µ AT By k y k+1 1 t A T λ k+1 λ P x k x k+1 1 t µ AT By k y k tλ min AA T λ k+1 λ P x k x k+1 1 t µ AT By k y k ^e Øª [11] u + v t 1 u t 1 v 2, t 1 > K P x k x k+1 1 µ AT By k y k t P x k x k t µ AT By k y k t 1 λ max P T P x k x k t 1 1 µ λ maxb T AA T B y k y k ± A T λ k+1 λ + P x k x k+1 1 µ AT By k y k+1 1 tλ min AA T λ k+1 λ t t 1 λ max P T P x k x k t 1 1 t µ λ maxb T AA T B y k y k éá 2.14 Ú 2.18 Œ± 1 1 tλ min AA T λ k+1 λ 2 L 2 f x k+1 x t 1 1 λ max t P T P x k x k t t 1 1 µ λ maxb T AA T B y k y k ± λ k+1 λ 2 c 6 x k+1 x 2 + c 7 x k x k c 8 y k y k+1 2. À0 < t < 1Út 1 > 0, dba1, ±kc 6 > 0, c 7 > 0, c 8 0, Ù c 6, c 7, c 8 Od 2.13.a, 2.13.b ± c ½Â. Ún á.

9 3Ï 2ÂOCqFÝŽ{ 5Âñ Û 9 ½n2.2 3Ún2.2b^ e, efrà, B, K3δ > 0, 2.1 á. y² éu? 0 < t 2 < 1, k µ ρ λ k λ k+1 2 = µ ρ µ 2 Ax k+1 x + By k+1 y 2 µ ρ [ µ 2 1 t 2 λ min B T B y k+1 y λ max A T A x k+1 x 2] t 2 = 1 τ 2λ max B T B éá 1.1 Ú 2.20, Œ± [ 1 t 2 λ min B T B y k+1 y λ max A T A x k+1 x 2]. t u k u 2 H u k+1 u 2 H x k x k+1 2ˆP + y k y k+1 2 Q+ 1 I 1 ρ λmaxbt BI ÏB [ + [ + 2v f + 1 τ 2λ max B T B 2v g + 1 τ 2λ max B T B 1 1 ] λ max A T A x k+1 x 2 t 2 ] 1 t 2 λ min B T B y k+1 y 2., ±c 8 > 0, Ù c 8 d 2.13.c ½Â. ^Ún2.2, Œ± u k u 2 H u k+1 u 2 H c 9 x k+1 x 2 + c 10 y k+1 y 2 + c 11 λ k+1 λ 2, 2.21 Ù c 9 = 2v f + 1 τ 2λ max B T B 1 1 λ max A T A t 2 c 6 c 8 λ min Q + 1 I 1 ρ λ maxb T BI, 2.22 c 10 = 2v g + 1 τ 2λ max B T B 1 t 2 λ min B T B, 2.23 c 11 = λ min Q + 1 I 1 ρ λ maxb T BI /c 8, 2.24 c 6, c 8 OX 2.13.a c ª ½Â. Ï 0 < t 2 < 1, dbb, ±w,kc 10 > 0, c 11 > 0. qï v f > 0, Àt, t 1, t 2 ve n ^ : 1 0 < t < < 1, 1 + N 1 λ max t 1 > µ P T λ P min Q + 1 I 1 λ max B T AA T B ρ λ maxb T BI λ min P > 0, 1 < t 2 < M 1 ž, c 9 > 0, Ù M 1 = 4µv f λ max A T A1 τ 2 λ max B T B > 0,

10 10, M ^ 18ò N 1 = t 1 L 2 f 1 + t 1 λ max P T P λ min P 2v f + 1 τ 2λ max B T B 1 1 t 2 λ max A T A > 0. δ = min{c 9 λ 1 max P, c 10 λ 1 max Q + 1 I, c 11 µ 1 } > 0, K 2.1 y. íø f ÚgÑ rà¼êž, KA1, f LipschitzëY½B1, g LipschitzëYž, 3δ > 0, 2.1 á. Œ±w, P = 0, Q = 0ž, z[12] APGMŽ{ JÑŽ{A~, ±, Œ±APGMŽ{ 0.3 3ƒÓ^ e, Q- 5Âñ. 3 êšá Ä5 K [13], ÙêÆ. min x x R n 1 + α x β Ax b 2 2, 3.1 Ù A R m n, α > 0, β > 0 ëê, x 1 = n i=1 x i. 3.1 Œ±d= min y 1 + α x 2 x R n,y R n β Ax b 2 2 én wñ8i¼ê'ux, y Œ. s. t. x y = ÀXe4 ØÓP, QŠ, 5ÿÁ2ÂOCqFÝŽ{ÛÂñ5 9ÛÜÂñ Ý. œ/1: P = 0, Q = 0; œ/2: P = 0, Q = 1 µ 1 I; œ/3: P = 1 µ 1 µτ 1 I, Q = 0; œ/4: P = 1 µ 1 µτ 1 I, Q = 1 µ 1 I. À½m = 250, n = æ^xe { Å ÝA. k z ƒññlio ÙN0, 1 ÅÝ,, éa1 þ?1ioz. 2 ¹k25 š" þdõ þx 0 R n, z þþñlio þb = Ax 0 + ε, Ù ε N0, 10 3 I. À ½.ëêα = 0.1, β = 0.01, Ž{ëêµ = 0.01, τ 1 = 0.95, τ 2 = ÅØÓ ÝA, Ù ëêñøc, õgáuys :u k `u ØkaqêŠLy, l? À4 ã, ã1, Ù, î L«S gê, p L«Ø. lã1, Œ±w u k u 2 H Åì~ ªCu0, =d2âapgmž{ 0.4 S S{u k }Âñ `u. lã2 w u k+1 u 2 H / uk u 2 H Š301ƒm, =3δ > 0, uk u 2 H uk+1 u 2 H δ uk+1 u 2 H, l`²{uk } Q- 5Âñ.

11 3Ï 2ÂOCqFÝŽ{ 5Âñ Û P=0,Q=0 P=1/µ 1/µ*τ1I,Q=0 P=0,Q=1/µ 1/µ*τ2I P=1/µ 1/µ*τ1I,Q=1/µ 1/µ*τ2I P=0,Q=0 P=1/µ 1/µ*τ1I,Q=0 P=0,Q=1/µ 1/µ*τ2I P=1/µ 1/µ*τ1I,Q=1/µ 1/µ*τ2I u k u 2 H u k u 2 H P=0,Q=0 P=1/µ 1/µ*τ1I,Q=0 P=0,Q=1/µ 1/µ*τ2I P=1/µ 1/µ*τ1I,Q=1/µ 1/µ*τ2I P=0,Q=0 P=1/µ 1/µ*τ1I,Q=0 P=0,Q=1/µ 1/µ*τ2I P=1/µ 1/µ*τ1I,Q=1/µ 1/µ*τ2I u k u 2 H 10 2 u k u 2 H ã1 2ÂOCqFÝŽ{ÛÂñ u k+1 u 2 H / uk u 2 H P=0,Q=0 P=1/µ 1/µ*τ1I,Q=0 P=0,Q=1/µ 1/µ*τ2I P=1/µ 1/µ*τ1I,Q=1/µ 1/µ*τ2I ã2 2ÂOCqFÝŽ{ 5Âñ ë z [1] Yang J, Zhang Y. Alternating direction algorithms for l 1 problems in compressive sensing [J]. SIAM Journal on Scientific Computing, 2011, 33:

12 12, M ^ 18ò [2] Candès E J, Recht B. Exact matrix completion via convex optimization [J]. Foundations of Computational Mathematics, 2009, 9: [3] Candès E J, Tao T. The power of convex relaxation: near-optimal matrix completion [J]. IEEE Translations on Information Theory, 2009, 56: [4] Qin Z W, Goldfarb D, Ma S Q. An alternating direction method for total variation denoising [EB/OL]. [ ]. [5] Wang Y, Yang J, Yin W T, et al. A new alternating minimization algorithm for total variation image reconstruction [J]. SIAM Journal on Imaging Sciences, 2008, 1: [6] Yuan X M. Alternating direction methods for sparse covariance selection selection [EB/OL]. [ ]. FILE/2009/09/2390.pdf [7] Goldfarb D, Ma S Q. Fast multiple splitting algorithms for convex optimization [J]. SIAM Journal on Optimization, 2012, 222: [8] Goldfarb D, Ma S Q, Scheinberg K. Fast alternating linearization methods for minimizing the sum of two convex functions [J]. Mathematical Programming Series A, 2013, : [9] He B S, Yuan X M. On non-ergodic convergence rate of douglas-rachford alternating direction method of multipliers multipliers [EB/OL]. [ ]. xmyuan/paper/adm-hy-jan16.pdf. [10] Goldstein T, Donoghue B O, Setzer S, et al. Fast alternating direction optimization methods [R]. California: University of California, Los Angeles, 2012, [11] Deng W, Yin W T. On the global and linear convergence of the generalized alternating direction method of multipliers [R]. State of Texas: Rice University, 2012, [12] Ma S Q. Alternating proximal gradient method for convex minimization [EB/OL]. [ ]. FILE/2012/09/3608.pdf. [13] Zou H, Hastie T. Regularization and variable selection via the elastic net [J]. Journal of the Royal Statistical Society: Series B Statistical Methodology, 2005, 672: