Identifying Scenes with the Same Person in Video Content on the Basis of Scene Continuity and Face Similarity Measurement

Transcript

1 Identifying Scenes with the Same Person in Video Content on the Basis of Scene Continuity and Face Similarity Measurement Tatsunori Hirai, Tomoyasu Nakano, Masataka Goto and Shigeo Morishima Abstract We present a method that can automatically annotate when and who is appearing in a video stream that is shot in an unstaged condition. Previous face recognition methods were not robust against different shooting conditions, such as those with variable lighting, face directions, and other factors, in a video stream and had difficulties identifying a person and the scenes the person appears in. To overcome such difficulties, our method groups consecutive video frames (scenes) into clusters that each have the same person s face, which we call a facial-temporal continuum, and identifies a person by using many video frames in each cluster. In our experiments, accuracy with our method was approximately two or three times higher than a previous method that recognizes a face in each frame TEL TEL Faculty of Science and Engineering, Waseda University (3-4-1, Ohkubo, Shinjuku-ku, Tokyo , Japan) National Institute of Advanced Industrial Science and Technology (AIST) (1-1-1, Umezono, Tsukuba-shi, Ibaraki , Japan) 1 JST CREST

2 ) TRECVID 2) TREC Video Retrieval Evaluation Bag-of-Features MKL-SVN TRECVID2010 TRECVID2010 3) TRECVID 4)5)

3 1 The image of facial-temporal continuum ➀ 2 ➁ ➂ 2 ➃ ➄ 2 ➅ 2 Outline of the method N i(i =1 N) I H i (I) 1 1 D(H i,h i+1 ) D(H i,h i+1 )= I H i+1 (I) H i (I) H i+1 (I)+H i (I) (1) D(H i,h i+1 )

4 4 Example of face detection and detected facial feature points (green plots). 3 An example of transition of shot detection feature in a video stream Active Structure Appearance Model ASAM 6) 7) ASAM ASAM Active Appearance Model AAM Active Shape Model ASM ASAM ASAM ASAM 3. 3 I 1 I 2 Sum of Squared Difference(SSD R SSD = width i (I 1 (i, j) I 2 (i, j)) 2 (2) height j R SSD 2 SSD 2

5 5 3 Adjusting face direction by reconstructing 3D face form ASAM Blanz 8) Blanz Blanz Histogram of Oriented Gradient HOG 9)10) HOG HOG Scale-Invariant Feature Transform Invariant SIFT 11) HOG 5 5 HOG HOG 9

6 HOG HOG feature around the facial feature points and reactions to facial expression by changing cell size HOG n n =5 2 Faces in the wild ) 500

7 Performance of face detection and effect of tracking. [% ] [% ] Let it be / The Beatles Hey Jude / The Beatles Get Back / The Beatles Two of us / The Beatles The Beatles Can You Keep A Secret?/ Wait & See / For You / Final Distance / PV Promotion Video Promotion Video The Beatles 4 Promotion Video4 8 The Beatles 4 Face in the wild 495 The Beatles ASAM = (3) The Beatles 2 Promotion Video 3 3 The Beatles 3 HOG

8 2 ( ) Comparison of face recognition ratios (comparison of the number of frames). [% ] [% ] The Beatles Face recognition ratio with facial-temporal continuum. [%] The Beatles Average of error rate in each facial-temporal continuum. [%] The Beatles ASAM 5 Average of error rate between facial-temporal continua. [%] The Beatles

9 IPA IT IT pp (2011) 2 TRECVID 3 CVIM pp.1-8 (2011) 4 FacePassenger TM 4 3 pp (2011) 5 3 PRMU pp (2006) 6 J94-D 4 pp (2011) 7 A. Irie M. Takagiwa K. Moriyama T. Yamashita Improvements to Facial Contour Detection by Hierarchical Fitting and Regression Asian Conference on Pattern Recognition (ACPR) pp (2011) 8 V. Blanz A. Mehl T. Vetter H. SeidelA Statistical Method for Robust 3D Surface Reconstruction from Sparse DataSymp. on 3D Data Processing Visualization and Transmission pp (2004) 9 N. Dalal B. Triggs Histograms of Oriented gradients for human detection Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp (2005) 10 HOG AdaBoost FPGA CPSY pp (2011) 11 D. G. LoweObject Recognition from Local Scale-Invariant Features Proc. IEEE International Conference on Computer vision (ICCV) pp (1999) 12 T. L. Berg A. C. Berg J. Edwards D. A. ForsythWho s in the Picture Proc. Neural Information Processing Systems (NIPS) pp (2004)