Kernel Methods and their Application for Image Understanding

Vol 1 No SIG 12(CVIM 1) Jan 1960 Kernel Methods and their Application for Image Understanding Kenji Nishida and Takio Kurita Support vector machine (SVM) has been extended to build up nonlinear classifier using the kernel trick It is recognized as one of the best models for two class classification among the many methods currently known because it is devised to obtain high performance for unlearned data This paper reviews kernel methods centering on the SVM and introduces some examples of applications for image understanding 1 (Support Vector Machine, SVM) 1) 3) 21 4) 6) 7) 2 8) 10) Neuroscience Research Institute, National Institute of Advanced Industrial Science and Technology 1 9) 2 1960 Vapnik Optimal Separating Hyperplane 1990

2 Jan 1960 shrinkage 2 y = sign(w T x h) (1) 2 (±1) w h sign(u) u > 0 1 u 0 1 1 1 2 2 C 1C 2 1 1 N x 1,, x N t 1,, t N Lagrange α i ( 0)i = 1,, N t i(w T x i h) 1, i = 1,, N (2) H1: w T x h = 1 H2: w T x h = 1 2 2 1 w w h H2 H1 1 ( 1-1 ) t i(w T x i h) 1, (i = 1,, N) (3) L(w) = 1 2 w 2 (4) L( w, h, ) = 1 2 w 2 α i{t i(w T x i h) 1} (5) w h w = α it ix i (6) 0 = α it i (7) α i t i = 0 (8) 0 α i, i = 1,, N (9)

Vol 1 No SIG 12(CVIM 1) 3 L D( ) = α i 1 2 α iα jt it jx T i x j (10) i,j=1 Lagrange α i ( 0)i = 1,, N α i 0 α i > 0 x i w T x h = 1 w T x h = 1 α i 0 x i α i (i 0) w w = αi t i x i (11) S h w T x h = 1 w T x h = 1 1 w 211 x s, s S h = w T x s t s (12) α i (i 0) y = sign(w T x h ) = sign( α i t ix T i x h ) (13) α i = 0 α i > 0 γ 2 211 2 H2 H1 ( 1-1 ) H1 H2 ξ i ( 0) ξ i w ξ i w (14) ξ i 0, t i (w T x i h) 1 ξ i, L(w, ) = 1 2 w 2 + γ ( i = 1,, N) (15) ξ i (16) Lagrange

4 Jan 1960 α i ν i L(w, h,, ) = 1 2 w 2 + γ ξ i α i {t i(w T x i h) (1 ξ i)} ν i ξ i (17) whν i 0 w = α it ix i (18) 0 = α i t i (19) α i = γ ν i (20) α i t i = 0 (21) 0 α i γ, i = 1,, N (22) L D ( ) = α i 1 2 α i α j t i t j x T i x j (23) i,j=1 α i H1 H2 ( ) H1 H2 α i α i = 0 H1 H2 0 < αi < γ H1 H2 α i = γ ξ i 0 H1 H2 22 x φ(x) φ L D φ(x 1) φ(x 2) φ(x 1 ) T φ(x 2 ) = K(x 1, x 2 ) (24) x 1 x 2 φ(x 1 ) φ(x 2 ) K(x 1, x 2 ) K K K(x 1, x 2) = (1 + x T 1 x 2) p (25) Gauss K(x 1, x 2 ) = exp ( x1 x 2 2 2σ 2 ) (26) K(x 1, x 2) = tanh ( ax T 1 x 2 b ) (27)

Vol 1 No SIG 12(CVIM 1) 5 Kernel 2 x w y 15 1 05 0 05 3 1 221 (10) (23) L D L D ( ) = α i 1 2 = α i 1 2 α i α j t i t j φ(x i ) T φ(x j ) i,j=1 α i α j t i t j K(x i, x j ) (28) i,j=1 (13) y = sign(w T φ(x) h ) = sign( α i t i φ(x i ) T φ(x) h ) = sign( α i t i K(x i, x) h ) (29) y = A T x (30) Gauss Radial Basis Function (RBF) A = [a ij ] (31) A Σ T Σ B 15 4 2 2 15 1 05 0 05 1 15 2 ( ) 4 23 8) 10) x = (x 1,, x M ) T K J = tr(ˆσ 1 T ˆΣ B ) (31) ˆΣ T ˆΣ B y Σ BA = Σ T AΛ (A T Σ T A = I) (32) Λ x

6 Jan 1960 Σ T = Σ B = (x i x T )(x i x T ) T K ω k ( x k x T )( x k x T ) T (33) k=1 ω k = N k /N x k x T C k x C k (30) Φ(x) 1 y = a T Φ(x) (34) a a = α iφ(x i) (35) y = α iφ(x i) T Φ(x) = α i K(x i, x) = T k i (x) (36) k i (x) = (K(x 1, x),, K(x N, x)) T T Σ (K) B J = T Σ (K) (37) Σ (K) B = Σ(K) λ (38) Σ (K) B Σ(K) Σ (K) = Σ(K) + αi (39) 0 3 (25)(26) (27) (Empirical Kernel Map) 9),12) (x 1,, x l ) Φ l : x K(, x) (x1,,x l ) = (K(x 1, x),, K(x l, x)) (40) (x 1,, x l ) (40) x (x 1,, x l ) k i (x) l l K K K ij = K(x i, x j ) (41) (13) y = sign( αi t ix T i x h ) = sign( αi t i k T i (x i )k i (x) h ) (42) i s (42) k T i (x i )k i (x) = KK (43) i s i s (42)

Vol 1 No SIG 12(CVIM 1) 7 y = sign( α i t ikk h ) (44) (29) Φ l : x K 1/2 k i (x) (45) K 1/2 K λ i λ i K (5(B)DT(distance y = sign( α i t i x T i x h ) = sign( α i t ik T i (x i)k 1 2 K 1 2 ki(x) h ) i s = sign( α i t ikk 1 K h ) i s = sign( α i t ik(x i, x) h ) (46) (29) K(x, y) l (x 1,, x l ) l (45) 4 41 Chamfer Distance Gavrila Chamfer Distance 11) Gavrila 924 Chamfer Distance Chamfer Distance Gauss 300 900 Chamfer distance (empirical kernel map) 9),12) Chamfer distance I, J(5(A)) I e, J e transformation) I d, J d (5(C)) DT I J chamfer distance D chamfer (I, J) = 1 J d (i) (47) I e i I e I e I I e J d (i) I J chamfer distance I e DT J d D chamfer (I, J) D chamfer (J, I) I J [ K = [K ij] = exp( D ] chamfer(i, j) ) (48) 2σ 2 45 k emp (x) = K 1 2 k(x) (49) MIT CBCL 2700 100 300 Chamfer Distance 905 False Positive 108

8 Jan 1960 I e Je (A) (B) (C) D chamfer D chamfer 5 (A) (B) (C)DT (J,I) (i,j) False Positive 13) (False Positive ) 42 2 14) 100 200 C = {k(x i) i = 1,, n f } C = {k(x k ) k = 1,, n f }, (50) n f n f k f = 1 n f Σ f = 1 n f n f n f k(x i), (k(x i) k f )(k(x i) k f ) T (51) k T = ω f k + n f N ω f n f k(x k ) (52) k=1 = n f N N = n f + n f Σ (f) Σ (f) B + 1 N = ω f Σ f = ω f ( k f k T )( k f k T ) T n f (k(x k ) k T )(k(x k ) k T ) T (53) k=1 J = tr(ˆσ (f) 1 ˆΣ (f) B ) (54) y = A T k(x) A Σ (f) B A = Σ(f) AΛ (AT Σ (f) A = I) (55) y min(n f, N) = n f 2 1 Σ (f) = Σ(f) + αi (56) MIT CMU eb 325 1000 1

Vol1 NoSIG12(CVIM1) 9 1 9917 % 9815 % 9950 % SVM 9834 % 9969 % 9790 % MATLAB R GUI http://wwwcsientuedutw/~cjlin/libsvm/ SVM light : Cornell University T Joachims C Solaris, indows, Cygwin, Linux 4 http://svmlightjoachimsorg/ 6 6 5 1998 41 15) 1) VNVapnik, Statistical Learning Theory, John iley & Sons (1998) 2),,,, No444, pp52-58 (2000) 3), -,, Vol42, No7, pp676-683 (2001) 4) EOsuna, RFreund and FGirosi, Training Support Vector Machines: an Application to Face Detection, Proc of IEEE Computer Society Conf on Computer Vision and Pattern Recognition, pp130-136, 1997 5) CPPapageorgiou and MOren and TPoggio, A General Framework for Object Detection, Proc Fifth Int l Conf on Computer Vision, 6) AMohan, CPPapageorgiou and TPoggio, Example-Based Object Detection in Images by Components, IEEE Trans on Pattern Analysis and Machine Intelligence, Vo23, No4, pp349-361, 2001 7) THastie, RTibshirani, JFriedman, The Elements of Statistical Learning - Data Mining, Inference, and Prediction, Springer-Verlag, 2001 8) KRMuller, SMika, GRatsch, KTsuda, BScholkopf, An introduction to kernel-based learning algorithms, IEEE Trans On Neural Networks, Vol12, No2, pp181-201, 2001 9) BScholkpf and AJSmola, Learning with Kernels, The MIT Press, 2002 10),, 2003 11) DMGavlira, Pedestrian Detection from a Moving Vehicle, Proc of Europian Conference URL on Computer Vision, pp37-49, 2000 12) BSchölkopf, SMika, C J C Burges, LIBSVM: National Taiwan University Chih- P Knirsch, KR Müller, GRätsch, and Jen Lin AJSmola, Input Space Versus Feature Space in Kernel-Based Methods, Trans on Neural C

10 Jan 1960 Networks, val10, No5, pp1000-1017, 1999 13),,, VolJ84-D-II, No8, pp1781-1789, 2001 14) TKurita, TTaguchi, A modification of kernel-based Fisher discriminant analysis for face detection, Proc of the Fifth Inter Conf on Automatic face and Gesture Recognition, 20-21 May 2002, ashington, DC, pp300-305, 2002 15),, 2005-CVIM-149, pp 333-342, 2005 (200533 ) (2005825 ) 33 56 60 ICOT 33 56 2 3 (NRC) IEEE CS