Random Forests Leo. Hitoshi Habe 1

Transcript

1 1 tree forestleo Breiman 2001 Random Forests Hitoshi Habe 1 Abstract: Random Forests is a machine learning framework that consists of many decision trees. It can be categorized as an ensemble classifier in which each decision tree performs as a weak classifier. Since it was originally developed by Leo Breiman in 2001, it has been applied to various application scenarios including computer vision. Especially, recent successful results have been attracting researcher s attention. This report introduces a generalized definition of Random Forests and discusses important factors when one would like to apply it to their purpose. Keywords: Decision Tree, Ensamble Learning, Bootstrap 1. Support Vector Machine[7] Adaboost[10] Random Forests[4] Random Forests Leo Breiman 2001 Lepetit [16] Shotton [24] 1 habe@info.kindai.ac.jp Department of Informatics, School of Science and Engineering, Kinki University 3-4-1, Kowakae, Higashi-Osaka-shi, Osaka , Japan Random Forests Random Forests Random Forests [2], [6], [8], [20] [8] c 2012 Information Processing Society of Japan 1

2 *1 2. Random Forests Random Forests treeforest decision tree ensemble learning weak classifier T. K. Ho [14][15] Breiman Random Forests[4] bagging [3] Breiman 2.1 decision tree 1 node split functionleaf d v = (x 1, x 2,, x d ) R d Random Forests Random Forests binary decision tree * j h(v, θ j ) {0, 1}, (1) v j θ θ = (ϕ, ψ, τ) ϕ d v ψ τ h(v, θ j ) = [τ 1 > ϕ(v) ψ > τ 2 ], (2) [ ] 1 0 τ 1 = τ 2 = ϕ 2 2 [11], [16] 2 2 SIFT 2 1 *1 Random Forests Random Forest Randomized Decision Trees Randomized Decision Forests Leo Breiman Random Forests Random Forests *2 [8] c 2012 Information Processing Society of Japan 2

3 CT c p(c v) Random Forests 2.2 Random Forests Random Forests T S = {v} S 0 ( S) C Random Forests (1) (2) 2 t S t 0 S S 1 θ j T T T j T T j / T T ρ = T j {1,..., T } ρ = T ρ = 1 ρ S t 0 S i S i = {v} S L i SR i j S j = S L j SR j, SL j SR j =, SL j = S 2j+1, S R j = S 2j+2 h 1 h θ j θ j θ j = arg max θ j T j I j (3) I j I j = I(S j, S L j, S R j, θ j ), (4) I j S c C split class class : Large : Small class 3 4 c 2012 Information Processing Society of Japan 3

4 H(S) = c C p(c) log(p(c)), (5) S H(S) H(S) I = H(S) i 1, 2 S i S H(Si ) (6) I I 3 S R j θ j S j S L j (1) D (2) (3) 2.3 Random Forests Random Forests 5 v T h 5 c t( {1,, T }) p t (c v) p(c v) = 1 T T p t (c v), (7) t=1 p(c v) = 1 Z T p t (c v), (8) t=1 Z p(c v) * Random Forests (1) T (2) D (3) ρ (4) (5) (6) Random Forest [8] T D Breiman [4] Random Forests 4. Random Forests Random Forests v c {c k } Random Forests (1) SVM (2) (3) *3 c 2012 Information Processing Society of Japan 4

5 (4) (1) 2 Random Forests Random Forests 3 T 6 c = 0, 1 c={0, 1} c p(c x) 6(a) 6(b) T (a) D D v (a) (b) Shotton [23] Microsoft Kinect for Xbox 360 CVPR 2011 Kinect p R 2 c {lefthand, righthand, head, } *4 [23] p 5. Random Forests 5.1 v y Y R n p(y v) Random Forests [1] Random Forests Random Forests 7 p(y v) y(x) = n i=0 w ixi *4 Shotton 31 c 2012 Information Processing Society of Japan 5

6 h θ 3 I j I j = log( Λ y ) log( Λ y ).(9) v S j 1 {L, R} v Sj i Λ y [8] 7 p(y v) = 1 T 5.2 T p t (y v) (10) t T D D 5.3 Random Forests [11], [17] 3 CT [9] CT 8 (a) Random Forests (b) Forests Random Forests c,x 1 1 c 2,x2 c 3 c 4,x 4 c 5 (a) c, x (b) Random Forests 7 8 Random Forests c 2012 Information Processing Society of Japan 6

7 6. Random Forests [8] 6.1 {v} {v} Random Forests v p t (v) p(v) = 1/T T t=1 p t(v) Parzen Random Forests [8] 6.2 Random Forests affinity matrix Random Forests 6.3 Random Forests Random Forests I s j Iu j I j = I u j + αis j 6.4 Random Forests Extremely Randomized Trees[13] 2 ρ = 1 Random ferns[19] Random Forests Random Forests [22] 7. Random Forests Open CV[18] OpenCV CvRTrees Waffles[12] Waffles Random Forests Linux Windows OSX randomforest-matlab[21] Leo Breiman [5] MATLAB FORTRAN77 Random Forests 8. Random Forests c 2012 Information Processing Society of Japan 7

8 [2], [6], [8], [20] [1] Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J.: Classification and Regression Trees, Statistics/Probability Series, Vol. 19, Wadsworth (1984). [2] Breiman, L.: [3] Breiman, L.: Bagging Predictors, Machine Learning, Vol. 24, No. 2, pp (1996). [4] Breiman, L.: Random Forests, Machine Learning, Vol. 45, No. 1, pp (2001). [5] Breiman, L. and Cutler, A.: breiman/ RandomForests/cc software.htm. [6] Computer Vision Tutorial: Random Forests [7] Cortes, C. and Vapnik, V.: Support-vector networks, Machine Learning, Vol. 20, No. 3, pp (1995). [8] Criminisi, A., Shotton, J. and Konukoglu, E.: Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning., Technical Report MSR-TR , Microsoft Research (2011). [9] Criminisi, A., Shotton, J., Robertson, D. and Konukoglu, E.: Regression Forests for Efficient Anatomy Detection and Localization in CT Studies, Medical Computer Vision Recognition Techniques and Applications in Medical Imaging, Vol. 6533, pp (2011). [10] Freund, Y.: A Decision-Theoretic Generalization of On- Line Learning and an Application to Boosting, Journal of Computer and System Sciences, Vol. 55, No. 1, pp (1997). [11] Gall, J., Yao, A., Razavi, N. and Van Gool, L.: Hough Forests for Object Detection, Tracking, and Action Recognition, Pattern Analysis and Machine Intelligence, IEEE Transactions on, Vol. 33, No. 11, pp (2011). [12] Gashler, M. S.: Waffles: A Machine Learning Toolkit, Journal of Machine Learning Research, Vol. MLOSS 12, pp (2011). [13] Geurts, P., Ernst, D. and Wehenkel, L.: Extremely randomized trees, Machine Learning, Vol. 63, No. 1, pp (2006). [14] Ho, T. K.: Random decision forests, Document Analysis and Recognition 1995 Proceedings of the Third International Conference on (Kavavaugh, M. and Storms, P., eds.), Proceedings of the 3rd International Conference on Document Analysis and Recognition, Vol. 1, IEEE, pp (1995). [15] Ho, T. K.: The random subspace method for constructing decision forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 8, pp (1998). [16] Lepetit, V., Lagger, P. and Fua, P.: Randomized Trees for Real-Time Keypoint Recognition, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR05, Vol. 2, pp (2005). [17] Okada, R.: Discriminative generalized hough transform for object dectection, Computer Vision, 2009 IEEE 12th International Conference on, pp (2009). [18] OpenCV: [19] Ozuysal, M., Calonder, M., Lepetit, V. and Fua, P.: Fast keypoint recognition using random ferns., IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 3, pp (2010). [20] Random Forests Wikipedia: forest. [21] Randomforest-matlab: [22] Saffari, A., Leistner, C., Santner, J., Godec, M. and Bischof, H.: On-line Random Forests, 2009 IEEE 12th International Conference on Computer Vision Workshops ICCV Workshops, Vol. 32, No. 2, pp (2009). [23] Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A. and Blake, A.: Real- Time Human Pose Recognition in Parts from Single Depth Images, CVPR 2011, Vol. 2, IEEE, pp (2011). [24] Shotton, J., Johnson, M. and Cipolla, R.: Semantic texton forests for image categorization and segmentation, 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp. 1 8 (2008). c 2012 Information Processing Society of Japan 8