38 8 Vol. 38, No ACTA AUTOMATICA SINICA August, , (Graphic processing unit, GPU). : 1) GPU Based Fast 3D-Object Modeling with Kinect

Transcript

1 38 8 Vol. 38, No ACTA AUTOMATICA SINICA August, 2012 GPU Kinect Kinect, (Graphic processing unit, GPU). : 1) ; 2)., ;,.,. (Loop-closure),.,,.,.,. DOI,, Kinect,,,,. GPU Kinect., 2012, 38(8): /SP.J GPU Based Fast 3D-Object Modeling with Kinect LIU Xin 1 XU Hua-Rong 2 HU Zhan-Yi 1 Abstract The cost-effective 3D-object modeling is a topic in computer vision field that is full of significance. In this paper, we report a method for object modeling with Kinect, as well as its implementation on graphic processing unit (GPU). Our modeling solution is divided into two steps: one is the system calibration, and the other is the fast 3D-object modeling. We present a easy-to-use calibration method for the first step, and a fully automatic modeling method for the other. Besides, our method is robust to registration of point clouds. For the well-known loop closure problem, we present a global registration method. Our method is validated by the modeling of several difficult real-world objects. In addition, the modeling of the objects with occlusion is also investigated and satisfactory results are obtained. Key words 3D object modeling, graphic processing unit (GPU), Kinect, occlusion, loop-closure Citation Liu Xin, Xu Hua-Rong, Hu Zhan-Yi. GPU based fast 3D-object modeling with Kinect. Acta Automatica Sinica, 2012, 38(8): , 3D,.,,.,,,.,,. Kinect, Manuscript received January 10, 2012; accepted May 22, 2012 ( ), (2011H0032) Supported by National Natural Science Foundation of China ( ) and the Key Project of Fujian Province (2011H0032) Recommended by Associate Editor JIA Yun-De National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing Department of Computer Science and Technology, Xiamen University of Technology, Xiamen Kinect Xbox360,, [1 6]. Engelhard [1] Kinect RGB-D SLAM,. SURF [7],, ICP (Iterative closest point) [8]. Henry [2 3] Kinect, ICP.,. Izadi [4 5] GPU,. ICP,, (Loop-closure). Tong [6] Kinect ( ).

2 8 : GPU Kinect 1289.,..,,,.,., Kinect : Kinect ; Kinect, [9 10] ; Kinect CMOS,, Kinect ;, Kinect,. GPU Kinect GPU., Kinect, GPU. : 1). [1 5], Kinect. 2),. [1 3, 6],.., [4 5]. 3),. 4) Loop-closure. [6],.,,,. 1 ICP EM-ICP ICP EM-ICP. ICP. X = {x 1, x 2,, x n } Y = {y 1, y 2,, y m }, ICP R t, X ( ) Y (Y Y R t ). ICP [7, 11 12],. EM-ICP ICP, : ICP, X x j (j = 1, 2,, n), Y ;, EM-ICP, Y x j,., EM-ICP R t, X Y. n m E = α ij d 2 ij, d ij = x j (Ry i + t) (1) j=1 i=1, α ij x j y i, : α ij = 1 ( d 2 ) ij exp (2) C i σ 2 p ( ) d 2 n+1 ( ) C i = exp 0 d 2 + exp ik σ 2 p k=1 σ 2 p, σ p d 0. (1) [11] λ i = E = (3) m λ 2 i x i (Ry i + t) (4) i=1 n αij, x i = 1 λ i j=1 EM-ICP EM-ICP n αij x j (5) j=1 1. R 0 I, t 0 0, k 0, σ p sigma p, σ recon sigma inf, d 2 0 d while σ p > σ recon do 3. i, j (i = 1, 2,, n; j = 1, 2,, m), (2) α ij. 4. (4) R t [11 12]. 5. R k R, t k t, k k σ p σ p sigma factor. 7. end while. ICP, EM-ICP, [13 14]. EM-ICP [13 16] : Kinect GPU, 1..,. ω : ω = v sω 0 tm (6) 360

3 , v s (step/s), ω 0, tm. ( R 01 t 01 ) = ( R 12 t 12 ) = = ( R n 2,n 1 t n 2,n 1 ) (7), Kinect Kinect,, : (a) (a) Turntable 1 Fig (b) (b) System configuration System platform,,., Kinect tm 0., RGB,., (X 0, X 1,, X n 1 ),, X 0.,,., Kinect :. P, Kinect d, P.,, P. Kinect Kinect Kinect, Kinect.,.,,. : 1) tm 0. (6), ω 01 = ω 12 = = ω n 2,n 1 (ω i,i+1 ). ( R i,i+1 t i,i+1 ) (i = 0, 1,, n 1), : ( R 01 t 01 ) ( R 12 t 12 ) ( R n 2,n 1 t n 2,n 1 ) (8) 2) (8),.,, (R 0 t 0 ). : 1),, ; 2),,.,. (R 0 t 0 ),,.,., Kinect, ,.,. 2. Fig. 2 2 Flow diagram of the system : 4 Kinect Kinect

4 8 : GPU Kinect 1291 (R 0 t 0 ) ; 5 ; 6 GPU. 3 Kinect Kinect 3.1 Kinect Kinect [10, 17] : 1), α, β, b K IR. K IR = f uir 0 u 0IR 0 f vir v 0IR (9), (f uir f vir ), (u 0IR v 0IR )., P d ( ), P (x IR y IR z IR ) T. x IR = b(u IR u 0IR ) d y IR = bf uir(v IR v 0IR ) f vir d z IR = 1 α(d β) (10) (11) (12) 2) f urgb 0 u 0RGB K RGB = 0 f vrgb v 0RGB (13) 0 0 1, (f urgb f vrgb ), (u 0RGB v 0RGB ). 3) T RGB IR = (R RGB IR t RGB IR ). Kinect [17]. 3.2 Kinect EM-ICP : (R 0 t 0 ).,,, tm 0. n (X 0, X 1,, X n 1 )., (I 0), EM-ICP (R i,i+1 t i,i+1 ) (i = 0, 1,, n 2). (R i,i+1 t i,i+1 ), X 0 X n 1 (R i 0,n 1 t i 0,n 1): ( R i 0,n 1 t i 0,n ) = ( R i,i+1 t i,i ) n 1 (14) X 0 X n 1 (X n 1 X n 1 R 0,n 1 t 0,n 1 ) ε i. (R0,n 1 i t i 0,n 1) (R 0 t 0 ). : (R 0 t 0 ) = arg min ε ( ) R i 0,n 1 t i 0,n 1 (15) i, ,, c. P, Kinect d, (10) (12) P., (16) (17), P (x RGB y RGB z RGB ) T (u RGB v RGB ) T. 1 z RGB x RGB y RGB z RGB u RGB v RGB 1 = R RGB IR x IR y IR z IR + t RGB IR (16) = K RGB x RGB y RGB z RGB (17) (R 0 t 0 ), ICP., (R 0 t 0 ), while do 2. Kinect. 3. RGB X i. 4. for X i c X j (j = i c, i c + 1,, i 1; j 0) do 5. ˆR ji ˆt ji: ( ) ( ) i j ˆRji ˆt ji R0 t 0 = ˆR ji ˆt ji, ICP X j X i R ji t ji. 7. ( ˆR ji tˆ ji) (R ji t ji) ε.

5 if ε < ε inf do 9. R ji R ji, t ji t ji 10. else 11. R ji ˆR ji, t ji ˆt ji 12. end if. 13. end for. 14. i i end while. 4.2, (X 0, X 1,, X n 1 ).,, c X i (i = 0, 1,, c 1) c X j (j = n c, n c+1,, n 1),., 2 5,, (I 0), EM-ICP X 0 X n 1 (R 0,n 1 t 0,n 1 )., X j (j = n c, n c + 1,, n 1) X i (i = 0, 1,, c 1) ( ˆR ji ˆt ji ): ( ) ( ˆRji ˆt ji R0 T R0 T t 0 = ) j ( ) ( ) n 1 i R0,n 1 T R0,n 1t T 0,n 1 R0 T R0 T t (18), ( ˆR ji ˆt ji ), ICP. 4.3,, X i X 0 R 0i t 0i (i = 1, 2,, n 1)., SVD [18] R 0i. X i X j ( R ij t ij ) (i, j = 0, 1,, n 1; i j). Ā R. R 01 I R 02 0 I 0 0 Ā = R 0,n I Rn 2,n 1 I (19) R = I R 01 R 02. R 0,n 1 (20), Ā rel n, rel, n. R n 3. : Ā R = 0 (21) Ā A. R 01 I R 02 0 I 0 0 A = R 0,n I R n 2,n 1 I (22) AR ( ) R R, : A SVD, : R = arg min R AR (23) A = UΣV T (24), V R : R = R 00 R 01. R 0,n 1 R 0i (i = 1, 2,, n 1) : (25) R 0i = R 0iR T 00 (26), R 0i t 0i (i = 1, 2,, n 1). B āa. R 01 I R 02 0 I 0 0 B = R 0,n I R n 2,n 1 I (27)

6 8 : GPU Kinect 1293 āa = 0 t 01 t 02. t 0,n 1 (28) Ā, B rel n. X 0 x 0, X 1, X 2,, X n 1 x 1, x 2,, x n 1. b. x 0 x b = 1 (29). : x n 1 B b āa = 0 (30) B B. R0i R ij : R ij = R 0j RT 0i (31) R ij, (31) R ij, B. R 01 I R 02 0 I 0 0 B = R 0,n I R n 2,n 1 I (32) āa a 0 t 01 a = t 02 (33). t 0,n 1 [19] b b b = ( B T B ) 1 B T a = x 0 x 1 x 2. x n 1 (34) X 0 X i (i = 1, 2,, n 1) t 0i = x i R 0i x 0 (35) 4.4 R 0i t 0i (i = 1, 2,, n 1), X i (i = 1, 2,, n 1) X 0,., Kinect Kinect,,.. RGB,.. 5 GPU : Kinect ; ; ; ;. GPU. CUDA [20] GPU. 5.1 Kinect Kinect GPU : 1) EM-ICP ; 2) X 0 X n 1. EM-ICP CUDA 5 : 1) d ij ; 2) C = (α ij ) n m ; 3) x i; 4) ; 5) R t. 5 Kernel 4 CUBLAS [21] : Sgemv, Saxpy, Sgemm Sscal. [14]. X 0 X n Kernel. X n 1, CUDA-thread. 5, CUDA-thread GPU- KD, x i. Kernel. 3. X 0 X n 1 1. X 0 GPU-KD,. 2. X 0 X n for R i,i+1 t i,i+1 do 4. X n 1 (X n 1 R i,i+1 t i,i+1 ). 5. GPU-KD, X n 1.

7 ε. 7. if ε < ε min do 8. R 0 R i,i+1, t 0 t i,i+1, ε min ε 9. end if. 10. end for. 5.2 Kinect i D i, GPU RGB. CUDA Kernel. CUDA-thread CUDA Kernel CUBLAS : Sgemm. Kernel 5.1. S Sgemm, Kernel Xi., CPU, R k t k. X i, GPU-KD. 4. c 1. X i GPU-KD,. 2. for X j (i c j < i) do 3. X j. 4. ˆR ji ˆt ji ( ) ( ) i j ˆRji ˆt ji R0 t 0 = R 0 ˆR ji, t 0 ˆt ji. 6. for k = 1,, maxiterations do 7. GPU-KD, X j. 8. Xi = {x i1, x i2,, x im }. 9. X j Xi R t. 10. R k R, t k t. 11. end for. 12. end for.. CUDA-thread, RGB,. 6, 7, : ( ) ( ) ( ). 3 ( ), 1.,,.,, 1, 3, tm 0 = 1 s; c = 3., c c EM-ICP ICP,,. 5.4 A SVD CULA [22] : Sgesvd., CUBLAS. 5.5 Kernel Fig. 3 3 Object images and reconstruction results 4 :,, ;, ; ;

8 8 : GPU Kinect ,,.,,,. 4., ;,.,,.,,. ; 3) ; 4) ; 5) ; 6) ; 7) ; 8) ; 9)., 2., 2 %.,.,, X 0 X n 1 (X n 1 R 0,n 1 t 0,n 1 ), ,,. 1 Table 1 Some meta-information and reconstruction results (mm) Fig. 5 Measurements Table 2 2 Results of measurements (mm) (mm) (mm) (%) Fig. 4 Fusion models,,. 5, : 1) ; 2),,. 2, Kinect,., 4. ( 46 ),

9 Table 3 Residual-errors (mm) X ,,,. Kinect,. References 1 Engelhard N, Endres F, Hess J, Sturm J, Burgard W. Realtime 3D visual SLAM with a hand-held RGB-D camera. In: Proceedings of the 2011 RGB-D Workshop on 3D Perception in Robotics at the European Robotics Forum. Västeras, Sweden: Robotdalen, 2011 Table 4 4 (s) Time consuming of objects reconstruction (s) 2 Henry P, Krainin M, Herbst E, Ren X, Fox D. RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments. In: Proceedings of the 12th International Symposium on Experimental Robotics. Delhi, India: IEEE, 2010 Kinect Kinect.,. GPU, 50 s.,.,.,., : 1).,. 2) Kinect. Kinect 0.5 m,,., 3 Du H, Henry P, Ren X F, Cheng M, Goldman D B, Seitz S M, Fox D. Interactive 3D modeling of indoor environments with a consumer depth camera. In: Proceedings of the 13th International Conference on Ubiquitous Computing. Beijing, China: IEEE, Izadi S, Newcombe R A, Kim D, Hilliges O, Molyneaux D, Hodges S, Kohli P, Davison A, Fitzgibbon A. KinectFusion: real-time dynamic 3D surface reconstruction and interaction. In: Proceedings of the 2011 International Conference on Computer Graphics and Interactive Techniques. Vancouver, Canada: ACM, Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 2011 Annual ACM Symposium on User Interface Software and Technology. Santa Barbara, CA: ACM, Tong J, Zhou J, Liu L G, Pan Z G, Yan H. Scanning 3D full human bodies using Kinects. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(4): Bay H, Ess A, Tuytelaars T, van Gool L. Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110(3): Besl P J, McKay H D. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): Konolige K, Mihelich P. Technical description of Kinect calibration [Online], available: kinect calibration/technical, November 3, Gu Zhao-Peng. A Study on Monocular Simultaneous Localization and Mapping [Ph. D. dissertation]. Institute of Automation, Chinese Academy of Sciences, China, 2011 (. [ ],,, 2011)

10 8 : GPU Kinect Liu Y H. Automatic registration of overlapping 3D point clouds using closest points. Image and Vision Computing, 2006, 24(7): Horn B K P. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, 1987, 4(4): Granger S, Pennec X. Multi-scale EM-ICP: a fast and robust approach for surface registration. In: Proceedings of the 7th European Conference on Computer Vision. Copenhagen, Denmark: Springer-Verlag, Tamaki T, Abe M, Raytchev B, Kaneda K. Softassign and EM-ICP on GPU. In: Proceedings of the st International Conference on Networking and Computing. Washington DC, USA: IEEE, Dewaele G, Devernay F, Horaud R. Hand motion from 3D point trajectories and a smooth surface model. In: Proceedings of European Conference on Computer Vision. Prague, Czech: Springer, Tamaki T. Pose estimation and rotation matrices. IEICE Technical Report SIS , 2009, 109(203): Herrera C D, Kannala J, HeikkilAa J. Accurate and practical calibration of a depth and color camera pair. In: Proceedings of the 14th international conference on Computer analysis of images and patterns. Seville, Spain: Springer, Martinec D, Pajdla T. Robust rotation and translation estimation in Multiview reconstruction. In: Proceedings of 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Minneapolis, Minnesota, USA: IEEE, Hartley R, Zisserman A. Multiple View Geometry in Computer Vision (Second edition). London: Cambridge University Press, CUDA 3.0. CUDA Programming Guide 3.0 [Online], available: cuda/, December 5, CUBLAS library 2.0. CUBLAS Library User s Guide [Online], available: compute/cuda/, December 5, CULA library. CULA Library User s Guide [Online], available: December 5, GPU.. liux skylark@tom.com (LIU Xin Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. He received his bachelor degree from Beijing Normal University in His research interest covers 3D reconstruction and GPU computing. Corresponding author of this paper.).. huarong xu@sina.com (XU Hua-Rong Associate professor in the Department of Computer Science and Technology, Xiamen University of Technology. His research interest covers computer vision and image processing.).,,. huzy@nlpr.ia.ac.cn (HU Zhan-Yi Professor at the Institute of Automation, Chinese Academy of Sciences. His research interest covers robot vision, which include camera calibration and 3D reconstruction, vision guided robot navigation, image based modeling and rendering.)