Gaussian Processes Classification Combined with Semi-supervised Kernels

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download "Gaussian Processes Classification Combined with Semi-supervised Kernels"

Βηθζαθά Αλεξανδρίδης
5 χρόνια πριν
Προβολές:

1 35 7 Vol. 35, No ACTA AUTOMATICA SINICA July, : 1) ; 2) ; 3),. :,.,.,,, TP391 Gaussian Processes Classification Combined with Semi-supervised Kernels LI Hong-Wei 1 LIU Yang 1 LU Han-Qing 1 FANG Yi-Kai 2 Abstract In this paper, we present a semi-supervised algorithm to learn Gaussian process classifiers, which is combined with nonparametric semi-supervised kernels in the presence of unlabeled data. This algorithm mainly includes the following aspects: 1) The spectral decomposition of graph Laplacians is used to obtain kernel matrices incorporating labeled and unlabeled data; 2) The convex optimization method is employed to learn the optimal weights of kernel matrix eigenvectors, which construct the nonparametric semi-supervised kernels; 3) The proposed semi-supervised learning algorithm is obtained by incorporating semi-supervised kernels into Gaussian process model. The main characteristic of the proposed algorithm is that we employ the nonparametric semi-supervised kernels based on the entire dataset into the Gaussian process model, which has an explicit probabilistic interpretation, and can model the uncertainty among the data and solve the complex non-linear inference problems. The effectiveness of the proposed algorithm is demonstrated by the experimental results in comparison with other related works in the literature. Key words Gaussian processes, semi-supervised learning, kernel method, convex optimization, [1]. :,.,,,.,., Received June 16, 2008; in revised form November 16, 2008 (863 ) (2006AA01Z315), ( ), ( ) Supported by National High Technology Research and Development Program of China (863 Program) (2006AA01Z315), Key Project of National Natural Science Foundation of China ( ), and National Natural Science Foundation of China ( ) National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing Nokia Research Center, China, Beijing DOI: /SP.J ,,.,,,,,., [2 9],,,.,,,,.,,,.,.

2 7 : 889,., : 1) (Support vactor machine, SVM) Margin,. [1], (Null-category noise model, NCNM),, NCNM Margin, (Transductive support vector machine, TSVM) [1,10],. Rogers [4] (Multinomial probit) NCNM, Lawrence ; 2),,,. [11 14], ( ),,, (Spectral clustering) [12], (Diffusion kernels) [13], (Gaussian random field) [14],, ( ). Sindhwani [15], (Reproducing kernel Hilbert spaces, RKHS),,,,, [16].,,,,,.,, (Kernel alignment), RKHS,,, [2],, µ Σ f = (f 1,, f n ) N(µ, Σ) (1), f i. f(x) m(x) ( ) K(x, x ) f(x) GP (m(x), K(x, x )) (2), x i, : m(x) = E[f(x)], : K(x, x ) = E[(f(x) m(x))(f(x ) m(x ))].,, f(x),,.,, X = {X l, X u }, X l = {x 1,, x l }, Y l = {y 1,, y l }, X u = {x l+1,, x l+u }. f(x) : f(x, θ) GP (0, K),, K R (), θ K. f, X Y. p(y = +1 f(x)) = Φ(f(x)) (3), Φ S (Sigmoid), (Logistic) (Cumulative Gaussian). f,, l+u l+u p(y f) = p(y i f i ) = Φ(y i f i ) (4) p(f) p(y f),

3 p(f y, X, θ) = p(y f)p(f X, θ) p(y X, θ) (5) x t, f(x t ) p(f t y, X, θ, x t ) = p(f t f, X, θ, x t )p(f y, X, θ)df (6) x t p(y t y, X, θ, x t ) = p(y t f t )p(f t y, X, θ, x t )df t (7), : f(x), ( Sigmoid ),. 1.3, 1.,,. Fig. 1 1 The graphical representation of Gaussian processes in the discriminative framework 1, x i X l, f i, x i K y i, y i, x i K,,. x j X u, y j, x j K, x j f,.,, RKHS X = {X l, X u },, { } 1 l f = arg min C(f, x i, y i ) + Ω( f H ) (8) l, C, f ; f H f, f. (Representer theorem) [17], l f (x) = α i K(x, x i ) (9), f {x i }.,,., {x i, y i } l {x i, y i } l+u i=l+1, f = arg min f H { 1 l + u } C(f, x i, y i ) + Ω( f H) (10), H,. Sindhwani [15] H ( ), H. V, S, H H : f, g H = f, g H + Sf, Sg V. H, f H H, H, H K(x i, y j ) [15] K(x i, y j )=K(x i, y j ) K T x (I +MK) 1 MK x (11), K x = [K(x 1, x),, K(x l+u, x)] T, M, K(x i, x j ), RKHS,,

4 7 : 891,,.,,,. RKHS. 2.2 RKHS :, (),. X = {X l, X u } G = {V, E}, V, E, W = {w ij }, x i x j, w ij = 0. D ii = j W ij W, : L = D W, L L = λ i φ i φ T i (12), λ i L, λ 1 λ l+u, φ i L. λ i : λ i = r(λ i ), K = r(λ i )φ i φ T i, r(λ i ) 0 (13) K : f 2 K = f T K 1 f., f,,.,,,,. r(λ), r(λ) λ, φ i ( λ i ) K. r(λ),. r(λ),, RKHS, K = α i φ i φ T i, α i 0 (14), α i : α i α i+1, α i, f, α i. K, RKHS K(x i, y j )=K (x i, y j ) K T x (I +LK) 1 LK x (15) 2.3 [18 19],. :,,. ( ),. A(K, T ) = K, T F K, K F T, T F (16) K K, ; T,, T ij = y i y j,, y i = y j, T ij = 1, T ij = 1.,,, : 1),, ; 2),, ; 3) :,. 2.4 (16),. : α i α i+1,,,., s. t. max K [A(K, T )] (17) K = α i K i (18) tr(k ) = 1 (19)

5 α i 0 (20) α i α i+1, i = 1,, l + u (21), K i = φ i φ T i, φ i. (19), (20) K. (17), A(K, T ) K, T F, T T ij +1 1, (18) α i,,. K, (15) RKHS K, ,,, 2., X, Y, f, K RKHS. p(y t y, X, K, x t ) = p(y t f t )p(f t y, X, K, x t )df t (23) (4) p(y f) Sigmoid, p(f 0, K), (22) p(f X, K) (23) p(y t X, K, x t ). : p(f X, K) q(f X, K) = N(f m, Σ),. (6), f x t q(f y, X, K, x t ) = N(f t µ t, σt 2 ), µ t = k T t K 1 m (24) σ 2 t = k(x t, x t ) k T t ( K 1 K 1 Σ K 1 )k t (25), k t x t X. Probit, x t q(y t =+1 y, X, K, x t )= Φ(f t )N(f t µ t, σ 2 t )df t = ( ) µ t Φ (26) 1 + σ 2 t Fig. 2 2 The graphical structure of Gaussian processes combined with semi-supervised kernels : 1), L; 2) L, φ i ; 3) K, (15) RKHS K; 4) K (7), f GP (0, K) K,. (4), (5) (7) p(f X, K) N(f 0, K) l+u = p(y X, K) Φ(y i f i ) (22), q(f X, K) = N(f m, Σ) m Σ (Expectation propagation, EP) [20]. EP f, (Kullback-Leibler divergence) [21]., EP. 3.2 : 1) : K, x t, O(N 2 ), N ; 2) :, N N, O(N 3 ), EP ( θ), K (Cholesky), N 3 /6;,

6 7 : 893 α i, α i 0,,, O(N 2 ), O(N 3 ),., [22], O(N log N),., ( ),,, ( ), ; ,. [11], 1. One vs Two Odd vs Even Cedar Buffalo,. One vs Two 1 2, Odd vs Even 0, 2, 4, 6, 8 1, 3, 5, 7, 9, Pc vs Mac Baseball vs Hockey 20,. Pc vs Mac 1 943, Baseball vs Hockey , One vs Two Odd vs Even, (Euclidean 10-nearest-neighbor, 10NN) ; Pc vs Mac Baseball vs Hockey, (Cosine similarity 10-nearest-neighbor, 10NN). 1 Table 1 The information of database on Experiment 1 One vs Two Euclidean 10 NN Odd vs Even Euclidean 10 NN Pc vs Mac Cosine similarity 10 NN Baseball vs Hockey Cosine similarity 10 NN [15], 2. G50c, ; Coil , ; Uspst USPS,. 2 Table 2 The information of database on Experiment 2 G50c Coil Uspst , 5,., 20,,.,, (), RBF (Radial basis function),, RBF ( K(x, x ) = θ 1 exp (x ) x ) T (x x ) (27) 2θ2 2, θ 1 θ 2 RBF, q(y X, θ) q(y X, θ) = p(y f)q(f X, θ)df (28) q(y X, θ), RBF θ i, θ i (6) (7) RBF. 3 6 ( ), Zhu [11] ( ), ( ) ( )., SVM.,,. ( ).,.

7 One vs Two (%) Table 3 The average accuracies on the One vs Two datasets (%) (RBF) [11] [11] Odd vs Even (%) Table 4 The average accuracies on the Odd vs Even datasets (%) (RBF) [11] [11] Pc vs Mac (%) Table 5 The average accuracies on the Pc vs Mac datasets (%) (RBF) [11] [11] Baseball vs Hockey (%) Table 6 The average accuracies on the Baseball vs Hockey datasets (%) (RBF) [11] [11] Table 7 7 (%) The average accuracies of the results on Experiment 2 (%) SVM [15] LapSVM [15] G50c Coil Uspst ,, RKHS, SVM (Laplacian SVM, LapSVM), SVM LapSVM 4, 4,, ;, RKHS,,., ; RKHS, G50c [15], [15], RKHS. 5.,,,,.,,

7 : 895 ;,,,,. References 1 Chapelle O, Scholkopf B, Zien A. Semi-Supervised Learning. Cambridge: MIT Press, 2006 2 Rasmussen C E, Christopher K I W. Gaussian Processes for Machine Learning.

1 55 4 Rogers S, Girolami M. Multi-class semi-supervised learning with the E-truncated multinomial probit Gaussian process. In: Proceedings of the Gaussian Processes in Practice Workshop.

The Journal of Machine Learning Research, 2005, 6: 1783 1816 6 Lawrence N D, Candela J Q. Local distance preservation in the GP-LVM through back constraints.

Learning for larger datasets with the Gaussian process latent variable model. In: Proceedings of the 11th International Workshop on Artificial Intelligence and Statistics.

8 7 : 895 ;,,,,. References 1 Chapelle O, Scholkopf B, Zien A. Semi-Supervised Learning. Cambridge: MIT Press, Rasmussen C E, Christopher K I W. Gaussian Processes for Machine Learning. Cambridge: MIT Press, Rasmussen C E. Advances in Gaussian processes. In: Proceedings of the Neural Information Processing Systems Conference. Cambridge, USA: MIT Press, Rogers S, Girolami M. Multi-class semi-supervised learning with the E-truncated multinomial probit Gaussian process. In: Proceedings of the Gaussian Processes in Practice Workshop. Berlin, German: Springer, Lawrence N D. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. The Journal of Machine Learning Research, 2005, 6: Lawrence N D, Candela J Q. Local distance preservation in the GP-LVM through back constraints. In: Proceedings of the 23rd International Conference in Machine Learning. Pittsburgh, Pennsylvania: ACM, Lawrence N D. Learning for larger datasets with the Gaussian process latent variable model. In: Proceedings of the 11th International Workshop on Artificial Intelligence and Statistics. San Juan, Puerto Rico: Morgan Kaufmann Publishers, Lawrence N D, Moore A J. Hierarchical Gaussian process latent variable models. In: Proceedings of the 24th International Conference in Machine Learning. Corvalis, Oregon: ACM, Urtasun R, Darrell T. Discriminative Gaussian process latent variable model for classification. In: Proceedings of the 24th International Conference in Machine Learning. Corvalis, Oregon: ACM, Joachims T. Transductive inference for text classification using support vector machines. In: Proceedings of the 16th International Conference on Machine Learning. San Francisco, USA: Morgan Kaufmann Publishers, Zhu X J, Kandola J, Ghahramani Z, Laffert J. Nonparametric transforms of graph kernels for semi-supervised learning. In: Proceedings of the Conference on Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, Chapelle O, Weston J, Scholkopf B. Cluster kernels for semisupervised learning. In: Proceedings of the Conference on Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, Kondor R I, Lafferty J D. Diffusion kernels on graphs and other discrete input spaces. In: Proceedings of the 19th International Conference on Machine Learning. San Francisco, USA: Morgan Kaufmann Publishers, Zhu X J, Ghahramani Z, Lafferty J D. Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the 20th International Conference on Machine Learning. Washington D. C., USA: MIT Press, Sindhwani V, Niyogi P, Belkin M. Beyond the point cloud: from transductive to semi-supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning. Bonn, Germany: MIT Press, Sindhwani V, Chu W, Keerthi S S. Semi-supervised Gaussian processes classifiers. In: Proceedings of the International Joint Conference on Artificial Intelligence. San Francisco, USA: Morgan Kaufmann Publishers, Kimeldorf G S, Wahba G. Some results on Tchebycheffian spline functions. Journal of Mathematical Analysis and Applications, 1971, 33(1): Cristianini N, Shawe-Taylor J, Elisseeff A, Kandola J. On kernel-target alignment. In: Proceedings of the Conference on Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, Lanckriet G R G, Cristianini N, Bartlett P, Ghaoui L E, Jordan M I. Learning the kernel matrix with semidefinite programming. The Journal of Machine Learning Research, 2004, 5: Minka T P. A Family of Algorithms for Approximate Bayesian Inference [Ph. D. dissertation], Massachusetts Institute of Technology, USA, Kullback S. Information Theory and Statistics. New York: Dover, Smith B, Bjrstad P, Gropp W. Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge: Cambridge University Press, 1996.,.. hwli@nlpr.ia.ac.cn (LI Hong-Wei Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. His research interest covers image and video processing, and machine learning. Corresponding author of this paper.).,. liuyang@nlpr.ia.ac.cn (LIU Yang Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. His research interest covers image and video processing, and machine learning.).,. luhq@nlpr.ia.ac.cn (LU Han-Qing Professor at the Institute of Automation, Chinese Academy of Sciences. His research interest covers image and video processing, and multimedia technique.) ykfang@gmail.com (FANG Yi-Kai Researcher at Nokia Research Center, China. He received his Ph. D. degree at the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science in His research interests covers image processing and pattern recognition.)

Σχετικά έγγραφα

ER-Tree (Extended R*-Tree)

ER-Tree (Extended R*-Tree) 1-9825/22/13(4)768-6 22 Journal of Software Vol13, No4 1, 1, 2, 1 1, 1 (, 2327) 2 (, 3127) E-mail xhzhou@ustceducn,,,,,,, 1, TP311 A,,,, Elias s Rivest,Cleary Arya Mount [1] O(2 d ) Arya Mount [1] Friedman,Bentley