q norm regularizing least-square-support-vector-machine linear classifier algorithm via iterative reweighted conjugate gradient

Transcript

1 DOI: /CTA Control Theory & Applcatons Vol. 31 No. 3 Mar q,,,, (, ) : L 2 (square support vector machne algorthm, LS SVM),, q = 2. q LS SVM, 0 < q <, q. cq, c q,,,.,, L 2LS SVM. : q ; (LS SVM); : TP181 : A q norm regularzng least-square-support-vector-machne lnear classfer algorthm va teratve reweghted conjugate gradent LIU Jan-we, LI Ha-en, LIU Yuan, FU Je, LUO Xong-ln (Research Insttute of Automaton, Chna Unversty of Petroleum, Bejng , Chna) Abstract: The L 2 norm penalty least-square-support-vector-machne algorthm (LS SVM) has been extensvely studed and s probably the most wdely used machne learnng algorthm. The regularzaton parameter n LS SVM s predetered wth default value q = 2. Based on the teratve reweghted conjugate gradent algorthm, the q norm regularzng LS-SVM s proposed wth 0 < q < 1, a ratonal number. We desgn a grd method to change the value of two adjustable parameters, the regularzaton parameter c and the order q of regularzaton, by performance ndcators of predcton error rate. On the selected values of c and q, usng the teratve reweghted conjugate gradent algorthm for solvng classfcaton object functon and fndng the mum predcton error, we can mprove the feature selecton and predct the error rate. The expermental results on real datasets n dfferent felds ndcate that the predcton performance s more accurate than L 2 norm LS-SVM, and can carry out feature selecton. Key words: q-norm regularzaton; LS SVM; Iteratve reweghted conjugate gradent method 1 (Introducton) Vapnk (support vector machne, SVM), L 2 (L 2 norm support vector machne, L 2 SVM [1] ). (x, y ),,, m, x R n, y { 1, +1}, SVM J(w) = pen(w) + C m L(y, wx ), w f(x)=sgn( w, x +b), J(w) pen(w), L(y, wx). L 2 SVM w J(w) = 1 2 wt w + C m max(1 y, w T x, 0). (1) L 2 (L 2 -norm least square support vector machne, L 2 LS SVM) SVM, SVM C m ξ wξ 2 2 w 2 2, (2) s.t. y [ x, w + b] = 1 ξ, = 1,, m, [2 3] : ; : E-mal: lujw@cup.edu.cn; Tel.: : 973 (2012CB720500); ( ); ( ) (JCXK ).

2 3 : q 335 [ ] [ ] [ ] 0 y T b 0 w 1 SVM, =, (3) y Ω + I/C a 1 v [16], Lu Yufeng L 0 L 1 pen(w) : a = [a 1 a n ], Ω = z T = λ w z, 0 + (1 λ) w 1 SVM [17], z T, L 0 SVM = [x 1 y 1 x m y m ], y = [y 1 y m ], 1 v = L 1 SVM, [1 1]. Lu Yufeng L 1, L 2 L 2 LS SVM pen(w) = w q (q {1, 2}) SVM [18]., L 2 SVM. L 1, Hu,,., Zou : pen(w) = λ w 1 + (1 λ) w 2,, [19]. ; 2 L 1, L 2.,. Su-In Lee L 1., Lopez 1 Norm SVM [4] [20], Zhenqu Lu, L q [21],,, LopezL 0, (recever-operatng characterstc, ROC) (2) wa [5] C m ξ (area under the curve, AUC) n λ j a 2 w,ξ,a 2 2 j, q. [22 24], s.t. y [( m a k x k )x + b] = 1 ξ, = 1,, m. L q, 0 < q < 1. k=1 q SVM, [6]q SVM, (1) L 2 L q w q = ( m w j q ) 1/q, 0 < q < 1, (1) w v, v < w < v,, [7]. [8 12], [13] L q LS SVM, LS SVM s = m a y x x j + b, ξ 2 s s k k=1,,m µ = max s k s, k k=1,,m k=1,,m (2) C m ξ 2 2 C m µ ξ 2 2 (3), Ω (Cµ ) 1, β q, β [ ] [ ] 0 y T 0 s.t. β =, y Ω + I/C 1 v β = [b a] T, 1 q 2,, Enrque BrtoIRMS [14]. Kloft L q SVM [15], q > 1., L 1, L 2 SVM,. Zhu J L 1 pen(w) =,, LS SVM,,, L q q, 0 < q <. LS SVM, q,, : q,,, q q LS SVM, q. q LS SVM(L q LS SVM) : w,ξ 1 2 m ξ 2 + c q w q q, (4) s.t. y (x T w + b) = 1 ξ, = 1, 2,, m, 0 < q <. (4), q = 2, L 2 LS SVM, q =1, L 1 LS SVM. 0<q <1, q =1, q = 2,? [25 27], L q LS SVM., Tany Zhou[28] Nesterov (Nesterov s method)svm, SVM, SVM Perf [29], Pegasos [30], SVM Lght [31] LIBSVM [32],

3 336 31,.,, q c, q , 14 ; c , 18. q c (4),,,, q. k, w (k) + λ (k) d (k) q 2 w (k 1) + λ (k 1) d (k 1) q 2, k, w (k) +λ (k) d (k) q 2 k 1 w (k 1) + λ (k 1) d (k 1) q 2, n n w (k) w (k 1) + λ (k) d (k) q + λ (k 1) d (k 1) q 2 w (k) + λ (k) d (k) 2, q,,.,, :,,, ;,, w,, [33 34].,, [35 36],,.,,,.,,,,,,,. 5: 1) q LS SVM, 0 < q,, q,. 2) L q LS SVM, q c, q cl q LS SVM. 3),,, q,. 4) w, w, w,,. 5) L q LS SVM Tr11 Arcene Sylva PIE Yaleb 32 32,cq, LS SVM. 2 L q LS SVM(L q LS SVM classfer) 2.1 (Objectve functon) X m n, m n. m, n, X m n x R m, y {±1}, w R m, L q LS SVM w,b f (w, b) = 1 N m L(y, x T w + b) + η w q q, (5) : w q = ( n w j q ) 1/q, 0 < q <, w j w j, L( ), η. L q LS SVM f 1 m 1(w, ξ) = ξ 2 + c w,ξ w,ξ 2 q w q q, (6) s.t. y (x T w + b) = 1 ξ, = 1, 2,, m, c (5) η [37]., w T :=[w 1 w n b] T, x T :=[x 1 x n 1], : f 1 m 2(w, ξ) = ξ 2 + c w,ξ w,ξ 2 q w q q, (7) s.t. y (x T w) = 1 ξ, = 1, 2,, m., ξ f 2(w) = 1 m (1 y (x T w)) 2 + c w 2 q w q q. (8) 2.2 λ (k) (Soluton for conjugate gradent step-sze) h(w) = w q = ( n w j q ) 1/q, k(w) = h q (w), sgn w j = w j w j,

4 3 : q < q <, 1 m (y 2 (x T ) 2 (λ (k) d (k) ) 2 + k(w) 2 = q w q 2 j w j. (9) w j f(w) w j, f = m y x,j (y (x T ) 1) + w j c w j q 2 w j, j = 1, 2,, n. (10) g (k) := f g k, w w (k) k, λ (k) k. : w (k+1) = w (k) λ (k) g (k), g (k) = m y () x (,j) (y (x T w (k) 1) + c w (k) j q 2 w (k) j., w w 0, d (1), d (1) = g (1). k, (8) w (k+1) = w (k) + λ (k) d (k), (11) f 2 (w) = 1 m (1 y (x T (w (k) + λ (k) d (k) ))) c q w(k) + λ (k) d (k) q q. (12) w (k) + λ (k) d (k) q 2 w (k 1) + λ (k 1) d (k 1) q 2, k w (k) + λ (k) d (k) q 2 k 1 w (k 1) + λ (k 1) d (k 1) q 2, f 2 (w) 1 m (1 y (x T (w (k) + λ (k) d (k) ))) c n w (k 1) +λ (k 1) d (k 1) q 2 w (k) +λ (k) d (k) 2. q (13) k w (k) d (k), (13) λ (k) : f 2 (λ) λ (k) 1 m (y 2 (x T ) 2 (w (k) + λ (k) d (k) ) 2 2 2y (x T (w (k) + λ (k) d (k) ))) + c q w(k 1) + λ (k 1) d (k 1) q 2 w (k) + λ (k) d (k) 2 = 2y 2 (x T ) 2 w (k) λ (k) d (k) 2y (x T λ (k) d (k) ))+ c q w(k 1) + λ (k 1) d (k 1) q 2 (2w (k) λ (k) d (k) + (λ (k) d (k) ) 2 ). (14), λ (k),. c 1 = c q w(k 1) + λ (k 1) d (k 1) q 2, (14) f 2 (λ (k) ) λ (k) m (y (x T ) 2 λ (k) (d (k) ) 2 + 2y 2 (x T ) 2 w (k) d (k) 2y (x T d (k) )) + c 1 (2w (k) d (k) + 2λ (k) (d (k) ) 2 ). (15) f 2(λ (k) ) = 0, λ (k) λ (k) m y (x T d (k) ) y 2 (x T ) 2 w (k) d (k) (k) c 1 w (k) d y 2 (x T ) 2 (d. (k) ) 2 + c 1 (d (k) ) 2 (16) 2.3 (Soluton for teratve reweghted conjugate gradent model) (8) c 1 = c q w(k 1) + λ (k 1) d (k 1) q 2, w f 2(w)= 1 2 (1 Y Xw)T (1 Y Xw)+c 1 w T w, (17) : Y =dag{y 1, y 2,, y m }, X =(x T 1, x T 2,, x T m) T, w =(w (1), w (2),, w (m) ) T. (17): w f 2(w) = 1 2 wt Gw + b T w + a, (18) : G = (Y X) T (Y X) + c 1 I, b = 2Y X, a = 1. [38]β k : β k = gt k+1(gk+1 T gk T ) d T k (gt k+1 gt k ) = gt k+1g k+1 gk Tg. (19) k d (1) = g (1), d (k+1) = g (k+1) + β k+1 d (k), β k+1 = g(k+1), g (k+1). g (k), g (k) w (k+1) = w (k) + λd (k), (20) : d (k) k,,

5 338 31, λ., g (k), d (k) 0, g (k) d (k),. f (k) k, τ 1 : f (k) f (k 1) > τ 2 ; τ 2, (g (k), d (k) ) < τ 2, d (k),, (g (k), d (k) ) τ 2,, g (k)., τ 1 = 1e 5, τ 2 = 4 9 π. 2.4 (Iteratve algorthm of reweghted conjugate gradent) L q LS SVM: 1), (x 1, y 1 ),, (x n, y n ),, c q. 2) L q LS SVM. w = [1 1], t = 1, τ 1 = 1e 5, τ 2 = 4 9 π, τ 3 = 1e 5; : rw: w 0, rw()=0; w 0, rw() = c w q 2, tmp1 = y. X = [ m y x,1 m y x,2 tmp2 = tmp1 w 1. m y x,n ] T, 3) c, q,, 500, t = 1g (1), d (1) = g (1). t > 1 k, f (k) f (k 1) < τ 1 τ 1, g (k) = tmp1 T tmp2 + rw T w (k), d (k) d (k) = g (k) + β k d (k 1), β k = g(k), g (k) g (k 1), g (k 1)., τ 2, (g (k), d (k) ) < τ 2, d (k), (g (k), d (k) τ 2, d (k) = g (k) ; 4) w (k+1) = w (k) + λd (k), w (k). w w j w w, 0. 5), L q LS SVM Procedure Iter-Reweg-ConjG Input: pars of samples(x 1, y 1 ),, (x n, y n ) Output: w Intalzaton: setc = (ones(1, 18) 2). ( 4:13), q = 0.2 : 0.2 : 4 w = [1,, 1], t = 1, τ 1 = 1e 5, τ 2 = 4 9 π, τ 3 = 1e 5 for to length(c) do for to length(q) do for t=1 to 500 do f t=1 then g (k) = tmp1 T tmp2 + rw T w (k) d (1) = g (1) else f f (t) f (t 1) > τ 1 then g (t) (t) = tmp1t tmp2+rwt w d (t) = g (t) + β t d (t 1) β t = g(t), g (t) g (t 1), g (t 1) computaton: (g (t), d (t) ) f (g (t), d (t) ) τ 2 then d (t) = g (t) end f end f end f update: w t+1 := w t + λ (t) d (t) select features: f w (t) < τ 3 w (t) then w (t) = 0 end f end for end for end for return w end Iter-Reweg-ConjG 3 (Complexty analyss) O(m 2 ), m.,. c, q,, L q LS SVM O(m 2 s t), s t c, q,,,,

6 3 : q L p SVM, c, q, s t, L 2 LS SVM s L p SVM Table 1 The runnng tme results of L p SVM on 10 datasets c q L q LS SVM /s PIE Yaleb Yaleb Tr Arcene Gsette (Experments) 4.1 (Pretreatment), L q LS SVM 3. 3 (bladder) (colon) (lymphoma). znovyev/prncmanf2006/, data/rawdata/, telecom-parstech.fr/ gfort/glm/programs.html. (bladder) 2215; (colon) 2000; (lymphoma) 4026.,. (bladder) 57, 42, 15; (colon) 62, 46, 16 ; (lymphoma) 96, 72, 24. L q LS SVM L 2 LS SVM,, : Tr11 (http: // glaros. dtc. umn. edu/ gkhome/cluto/cluto)arcene Sylva(http: // PIE Yaleb ( dengca/ Data/data.html)., [ 1,1], x, {1, 2,, m}, : 1, x < 1, f(x) = x, 1 < x < 1, 1, x 1.. : g(x) = f(x) ,. 4.2 L 2 LS SVM(Concluson and comparson to L 2 LS SVM) q 0<q < (bladder) (colon) (lymphoma), q. q > 4, 0 < q < 4, q q. cq,, q, qc. q, c., (c, q). 2, c, q. 2, L 2, L 1, L Table 2 Predcton error rates of the three cancel datasets c q /% (bladder) (colon) (lymphoma) ,. q = 1, q (0, 2], q c, (c, q), 10, q, c,,.,. (c, q), 10,. :

7 cq,. 3(a),,, 3(b),. 3(a) Table 3(a) Select features numbers on best classfcaton error /% (bladder) (colon) (lymphoma) (b) Table 3(b) Classfcaton error on best feature selecton /% (bladder) (colon) (lymphoma) L 2 LS SVM, LS SVMlab v1.7 (http: //www. esat. kuleuven.be/ssta/ lssvmlab/) L q LS SVM. LS SVM, 10, C, 3 10 sg C 4(a). 4(b) L q LS SVMqL 2 LS SVM sgc, 3. 4L 2 LS SVM, q, q L 2 LS SVM. 4(a) 3 C sg Table 4(a) The values of regularzaton parameter C and kernel wdth sg on the three real datasets C sg (bladder) (colon) (lymphoma) (b) 3 Table 4(b) Comparson of test error rates on the three cancel datasets LS SVM/% q /% (bladder) (colon) (lymphoma) , L q LS SVM L 2 LS SVM. 10 : 3 Tr11 Arcene Sylva PIE Yaleb Yaleb 32 32,, , 59 64, , 128, 1 64, , 2 Yaleb 0102 ; 26 27, 128, Yaleb 2627 ; PIE L q LS SVM, C, q. L 2 LS SVM 10, Table 5 Predcton error rates of the ten datasets and compare wth L 2 LS SVM C q L q LS SVM / % L 2 LS SVM / % PIE Yaleb Yaleb Tr Arcene Gsette

8 3 : q , L q LS SVM LS SVM, PIE 0102 Yaleb 2627 L q LS SVM L 2 LS SVM. L q LS SVM. 5 (Concluson and prospect) L q LS SVM, L 2, L 1, L 0, q, q, 0 < q <, c, q,.,.,, Tr11 Arcene Sylva PIE Yaleb 32 32,,, L 2 LS SVM.,, LS SVM,. (References): [1],,. p [J]., 2012, 38(1): (LIU Janwe, LI Shuangcheng, LUO Xongln. Classfcaton algorthm of support vector machne va p-norm regularzaton [J]. Acta Automatca Snca, 2012, 38(1): ) [2] SUYKENS J A K, VAN GESTEL T, DE BRABANTER K, et al. Least Squares Support Vector Machnes [M]. Sngapore: World Scentfc, [3] SUYKENS J A K, VANDEWALLE J. Least squares support vector machne classfers [J]. Neural Processng Letters, 1999, 9(3): [4] LOPEZ J, DORRONSORO J R. Least 1 norm SVMs: a new SVM varant between standard and LS SVMs [C] //Proceedngs of the 18th European Symposum on Artfcal Neural Networks. Bruges, Belgum, 2010, [5] LOPEZ J, DE BRABANTER J, SUYKENS J A K, et al. Sparse L s- SVMs wth L 0-norm mzaton [C] //Proceedngs of the 19th European Symposum on Artfcal Neural Networks. Bruges, Belgum: Elsever, 2011, [6] TAN J Y, ZHANG C H, DENG N Y. Cancer related gene dentfcaton va p-norm support vector machne [C] //Proceedngs of the Fourth Internatonal Conference on Computatonal Systems Bology. Suzhou, Chna: World Publshng Corporaton, 2010: [7] BRADLEY P S, MANGASARIAN O L, STREET W N. Feature selecton va mathematcal programg [J]. INFORMS Journal on Computng, 1998, 10(2): [8] SUYKENS J A K, DE BRABANTER J, LUKAS L, et al. Weghted least squares support vector machnes: robustness and sparse approxmaton [J]. Neurocomputng, 2002, 48(1): [9] VALYON J, HORVATH G. A weghted generalzed LS SVM [J]. Electrcal Engneerng and Computer Scence, 2003, 47(3/4): [10] ZHOU L, LAI K K, YEN J. Credt scorng models wth AUC maxmzaton based on weghted SVM [J]. Internatonal Journal of Informaton Technology & Decson Makng, 2009, 8(4): [11] LESKI J M. Iteratvely reweghted least squares classfer and ts L 2-and L 1-regularzed Kernel versons [J]. Bulletn of The Polsh Academy of Scences: Techncal Scences, 2010, 58(1): [12] SUN D, LI J, WEI L. Credt rsk evaluaton: Support vector machnes wth adaptve L q penalty [J]. Journal of Southeast Unversty (Englsh Edton), 2008, 24(S): 3 6. [13] LIU J, LI J, XU W, et al. Weghted L q adaptve least squares support vector machne classfers-robust and sparse approxmaton [J]. Expert Systems wth Applcatons, 2011, 38(3): [14] BRITO A E. Iteratve adaptve extrapolaton appled to SAR mage formaton and snusodal recovery [D]. EI Paso: The Unversty of Texas, [15] KLOFT M, BREFELD U, SONNENBURG S, et al. Effcent and accurate L p-norm multple kernel learnng [J]. Advances n Neural Informaton Processng Systems, 2009, 22(22): [16] ZHU J, ROSSET S, HASTIE T, et al. 1-norm support vector machnes [J]. Advances n Neural Informaton Processng Systems, 2004, 16(1): [17] LIU Y, WU Y. Varable selecton va a combnaton of the L 0 and L 1 penaltes [J]. Journal of Computatonal and Graphcal Statstcs, 2007, 16(4): [18] LIU Y, HELEN ZHANG H, PARK C, et al. Support vector machnes wth adaptve L q penaltes [J]. Computatonal Statstcs & Data Analyss, 2007, 51(12): [19] MARTIN S. Tranng support vector machnes usng Glbert s algorthm [C] //Proceedngs of the Ffth IEEE Internatonal Conference on Data Mnng. Houston, USA: IEEE, 2005: [20] LEE S I, LEE H, ABBEEL P, et al. Effcent L 1 regularzed logstc regresson [C] //Proceedngs of the 21st Natonal Conference on Artfcal Intellgence. Menlo Park, CA: AAAI Press, 2006: [21] LIU Z, JIANG F, TIAN G, et al. Sparse logstc regresson wth Lp penalty for bomarker dentfcaton [J]. Statstcal Applcatons n Genetcs and molecular Bology, 2007, 6(1): [22] BRADLEY P S, MANGASARIAN O L. Feature selecton va concave mzaton and support vector machnes [C] //Proceedngs of the 15th Internatonal Conference on Machne Learnng. Madson, USA: Morgan Kaufmann, 1998, [23] FOUCART S, LAI M J. Sparsest solutons of underdetered lnear system va L q-mzaton for 0 < q < 1 [J]. Appled and Computatonal Harmonc Analyss, 2009, 26(3): [24] FAN J, LI R. Varable selecton va nonconcave penalzed lkelhood and ts oracle propertes [J]. Journal of the Amercan Statstcal Assocaton, 2001, 96(456): [25] PYTLAK R. Conjugate Gradent Algorthms n Nonconvex Optmzaton [M]. New York: Sprnger, 2009.

9 [26] SCHIROTZEK W. Nonsmooth Analyss [M]. New York: Sprnger, [27] MISHRA S K, GIORGI G. Invexty and Optmzaton [M]. New York: Sprnger, [28] ZHOU T, TAO D, WU X. NESVM: a fast gradent method for support vector machnes [C] //Proceedngs of the Tenth IEEE Internatonal Conference on Data Mnng. Sydney, Australa: IEEE, 2010: [29] JOACHIMS T. Tranng lnear svms n lnear tme [C] //Proceedngs of the 12th ACM SIGKDD Internatonal Conference on KnowledgeDscovery and Data Mnng. Phladelpha, USA: ACM, 2006: [30] SHALEV-SHWARTZ S, SINGER Y, SREBRO N, et al. Pegasos: prmal estmated sub-gradent solver for svm [J]. Mathematcal Programg, 2011, 127(1): [31] JOACHIMS T. Makng large-scale SVM learnng practcal [C] //Proceedngs of Advances n Kernel Methods-Support Vector Machnes. Cambrdge, USA: MIT Press, 1999: [32] CHANG C C, LIN C J. LIBSVM: a Lbrary for support vector machnes [J]. ACM Transactons on Intellgent Systems and Technology (TIST), 2011, 2(3): 27. [33] SINDHWANI V, KEERTHI S S. Large scale sem supervsed lnear SVMs [C] //Proceedngs of the 29th annual nternatonal ACM SIGIR conference on Research and development n nformaton retreval. New York: ACM, 2006: [34] SONNENBURG S, FRANC V. COFFIN: a computatonal framework for lnear SVMs [C] //Proceedngs of the 27nd Internatonal Conference on Machne Learnng. Hafa, Israel: Omnpress, 2010: [35] LIU J, LI S C, LUO X. Iteratve reweghted nonnteger norm regularzng SVM for gene expresson data classfcaton [J/DB]. Computatonal and Mathematcal Methods n Medcne, ID [36] CHEN P C, HUANG S Y, CHEN W J, et al. A new regularzed least squares support vector regresson for gene selecton [J]. BMC Bonformatcs, 2009, 10(1): [37] BERTSEKAS D P. Nonlnear Programg (Second edton) [M]. Belmont, England: Athena Scentfc, [38],. [M]. :, (YUAN Yaxang, SUN Wenyu. Optmzaton Theory and Method [M]. Bejng: Scence Press, : (1966 ),,,,, E-mal: lujw@ cup.edu.cn; (1988 ),,,, E-mal: lhaen1988@163.com; (1989 ),,,, E-mal: ysht 25@sna.com; (1988 ),,,, E-mal: luylujw@sna.com; (1963 ),,,,, E-mal: luoxl@cup.edu.cn.