38 8 Vol. 38, No. 8 2012 8 ACTA AUTOMATICA SINICA August, 2012 GPU Kinect 1 2 1. Kinect, (Graphic processing unit, GPU). : 1) ; 2)., ;,.,. (Loop-closure),.,,.,.,. DOI,, Kinect,,,,. GPU Kinect., 2012, 38(8): 1288 1297 10.3724/SP.J.1004.2012.01288 GPU Based Fast 3D-Object Modeling with Kinect LIU Xin 1 XU Hua-Rong 2 HU Zhan-Yi 1 Abstract The cost-effective 3D-object modeling is a topic in computer vision field that is full of significance. In this paper, we report a method for object modeling with Kinect, as well as its implementation on graphic processing unit (GPU). Our modeling solution is divided into two steps: one is the system calibration, and the other is the fast 3D-object modeling. We present a easy-to-use calibration method for the first step, and a fully automatic modeling method for the other. Besides, our method is robust to registration of point clouds. For the well-known loop closure problem, we present a global registration method. Our method is validated by the modeling of several difficult real-world objects. In addition, the modeling of the objects with occlusion is also investigated and satisfactory results are obtained. Key words 3D object modeling, graphic processing unit (GPU), Kinect, occlusion, loop-closure Citation Liu Xin, Xu Hua-Rong, Hu Zhan-Yi. GPU based fast 3D-object modeling with Kinect. Acta Automatica Sinica, 2012, 38(8): 1288 1297, 3D,.,,.,,,.,,. Kinect, 2012-01-10 2012-05-22 Manuscript received January 10, 2012; accepted May 22, 2012 (60905020), (2011H0032) Supported by National Natural Science Foundation of China (60905020) and the Key Project of Fujian Province (2011H0032) Recommended by Associate Editor JIA Yun-De 1. 100190 2. 361024 1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190 2. Department of Computer Science and Technology, Xiamen University of Technology, Xiamen 361024. Kinect Xbox360,, [1 6]. Engelhard [1] Kinect RGB-D SLAM,. SURF [7],, ICP (Iterative closest point) [8]. Henry [2 3] Kinect, ICP.,. Izadi [4 5] GPU,. ICP,, (Loop-closure). Tong [6] Kinect ( ).
8 : GPU Kinect 1289.,..,,,.,., Kinect : Kinect ; Kinect, [9 10] ; Kinect CMOS,, Kinect ;, Kinect,. GPU Kinect GPU., Kinect, GPU. : 1). [1 5], Kinect. 2),. [1 3, 6],.., [4 5]. 3),. 4) Loop-closure. [6],.,,,. 1 ICP EM-ICP ICP EM-ICP. ICP. X = {x 1, x 2,, x n } Y = {y 1, y 2,, y m }, ICP R t, X ( ) Y (Y Y R t ). ICP [7, 11 12],. EM-ICP ICP, : ICP, X x j (j = 1, 2,, n), Y ;, EM-ICP, Y x j,., EM-ICP R t, X Y. n m E = α ij d 2 ij, d ij = x j (Ry i + t) (1) j=1 i=1, α ij x j y i, : α ij = 1 ( d 2 ) ij exp (2) C i σ 2 p ( ) d 2 n+1 ( ) C i = exp 0 d 2 + exp ik σ 2 p k=1 σ 2 p, σ p d 0. (1) [11] λ i = E = (3) m λ 2 i x i (Ry i + t) (4) i=1 n αij, x i = 1 λ i j=1 EM-ICP 1. 1. EM-ICP n αij x j (5) j=1 1. R 0 I, t 0 0, k 0, σ p sigma p, σ recon sigma inf, d 2 0 d 02. 2. while σ p > σ recon do 3. i, j (i = 1, 2,, n; j = 1, 2,, m), (2) α ij. 4. (4) R t [11 12]. 5. R k R, t k t, k k + 1. 6. σ p σ p sigma factor. 7. end while. ICP, EM-ICP, [13 14]. EM-ICP [13 16]. 2 2.1 : Kinect GPU, 1..,. ω : ω = v sω 0 tm (6) 360
1290 38, v s (step/s), ω 0, tm. ( R 01 t 01 ) = ( R 12 t 12 ) = = ( R n 2,n 1 t n 2,n 1 ) (7), Kinect Kinect,, : (a) (a) Turntable 1 Fig. 1 2.2 (b) (b) System configuration System platform,,., Kinect tm 0., RGB,., (X 0, X 1,, X n 1 ),, X 0.,,.,. 2.2.1 Kinect :. P, Kinect d, P.,, P. Kinect. 2.2.2 Kinect Kinect, Kinect.,.,,. : 1) tm 0. (6), ω 01 = ω 12 = = ω n 2,n 1 (ω i,i+1 ). ( R i,i+1 t i,i+1 ) (i = 0, 1,, n 1), : ( R 01 t 01 ) ( R 12 t 12 ) ( R n 2,n 1 t n 2,n 1 ) (8) 2) (8),.,, (R 0 t 0 ). : 1),, ; 2),,.,. (R 0 t 0 ),,.,., Kinect,. 2.2.3,.,. 2. Fig. 2 2 Flow diagram of the system : 4 Kinect Kinect
8 : GPU Kinect 1291 (R 0 t 0 ) ; 5 ; 6 GPU. 3 Kinect Kinect 3.1 Kinect Kinect [10, 17] : 1), α, β, b K IR. K IR = f uir 0 u 0IR 0 f vir v 0IR 0 0 1 (9), (f uir f vir ), (u 0IR v 0IR )., P d ( 0 2 047), P (x IR y IR z IR ) T. x IR = b(u IR u 0IR ) d y IR = bf uir(v IR v 0IR ) f vir d z IR = 1 α(d β) (10) (11) (12) 2) f urgb 0 u 0RGB K RGB = 0 f vrgb v 0RGB (13) 0 0 1, (f urgb f vrgb ), (u 0RGB v 0RGB ). 3) T RGB IR = (R RGB IR t RGB IR ). Kinect [17]. 3.2 Kinect EM-ICP : (R 0 t 0 ).,,, tm 0. n (X 0, X 1,, X n 1 )., (I 0), EM-ICP (R i,i+1 t i,i+1 ) (i = 0, 1,, n 2). (R i,i+1 t i,i+1 ), X 0 X n 1 (R i 0,n 1 t i 0,n 1): ( R i 0,n 1 t i 0,n 1 0 1 ) = ( R i,i+1 t i,i+1 0 1 ) n 1 (14) X 0 X n 1 (X n 1 X n 1 R 0,n 1 t 0,n 1 ) ε i. (R0,n 1 i t i 0,n 1) (R 0 t 0 ). : (R 0 t 0 ) = arg min ε ( ) R i 0,n 1 t i 0,n 1 (15) i,. 4 4.1,, c. P, Kinect d, (10) (12) P., (16) (17), P (x RGB y RGB z RGB ) T (u RGB v RGB ) T. 1 z RGB x RGB y RGB z RGB u RGB v RGB 1 = R RGB IR x IR y IR z IR + t RGB IR (16) = K RGB x RGB y RGB z RGB (17) (R 0 t 0 ), ICP., (R 0 t 0 ),. 2. 2. 1. while do 2. Kinect. 3. RGB X i. 4. for X i c X j (j = i c, i c + 1,, i 1; j 0) do 5. ˆR ji ˆt ji: ( ) ( ) i j ˆRji ˆt ji R0 t 0 = 0 1 0 1 6. ˆR ji ˆt ji, ICP X j X i R ji t ji. 7. ( ˆR ji tˆ ji) (R ji t ji) ε.
1292 38 8. if ε < ε inf do 9. R ji R ji, t ji t ji 10. else 11. R ji ˆR ji, t ji ˆt ji 12. end if. 13. end for. 14. i i + 1. 15. end while. 4.2, (X 0, X 1,, X n 1 ).,, c X i (i = 0, 1,, c 1) c X j (j = n c, n c+1,, n 1),., 2 5,, (I 0), EM-ICP X 0 X n 1 (R 0,n 1 t 0,n 1 )., X j (j = n c, n c + 1,, n 1) X i (i = 0, 1,, c 1) ( ˆR ji ˆt ji ): ( ) ( ˆRji ˆt ji R0 T R0 T t 0 = 0 1 0 1 ) j ( ) ( ) n 1 i R0,n 1 T R0,n 1t T 0,n 1 R0 T R0 T t 0 0 1 0 1 (18), ( ˆR ji ˆt ji ), ICP. 4.3,, X i X 0 R 0i t 0i (i = 1, 2,, n 1)., SVD [18] R 0i. X i X j ( R ij t ij ) (i, j = 0, 1,, n 1; i j). Ā R. R 01 I 0 0 0 R 02 0 I 0 0 Ā = R 0,n 1 0 0 0 I........ 0 0 0 Rn 2,n 1 I (19) R = I R 01 R 02. R 0,n 1 (20), Ā rel n, rel, n. R n 3. : Ā R = 0 (21) Ā A. R 01 I 0 0 0 R 02 0 I 0 0 A = R 0,n 1 0 0 0 I........ 0 0 0 R n 2,n 1 I (22) AR ( ) R R, : A SVD, : R = arg min R AR (23) A = UΣV T (24), V R : R = R 00 R 01. R 0,n 1 R 0i (i = 1, 2,, n 1) : (25) R 0i = R 0iR T 00 (26), R 0i t 0i (i = 1, 2,, n 1). B āa. R 01 I 0 0 0 R 02 0 I 0 0 B = R 0,n 1 0 0 0 I........ 0 0 0 R n 2,n 1 I (27)
8 : GPU Kinect 1293 āa = 0 t 01 t 02. t 0,n 1 (28) Ā, B rel n. X 0 x 0, X 1, X 2,, X n 1 x 1, x 2,, x n 1. b. x 0 x b = 1 (29). : x n 1 B b āa = 0 (30) B B. R0i R ij : R ij = R 0j RT 0i (31) R ij, (31) R ij, B. R 01 I 0 0 0 R 02 0 I 0 0 B = R 0,n 1 0 0 0 I........ 0 0 0 R n 2,n 1 I (32) āa a 0 t 01 a = t 02 (33). t 0,n 1 [19] b b b = ( B T B ) 1 B T a = x 0 x 1 x 2. x n 1 (34) X 0 X i (i = 1, 2,, n 1) t 0i = x i R 0i x 0 (35) 4.4 R 0i t 0i (i = 1, 2,, n 1), X i (i = 1, 2,, n 1) X 0,., Kinect Kinect,,.. RGB,.. 5 GPU : Kinect ; ; ; ;. GPU. CUDA [20] GPU. 5.1 Kinect Kinect GPU : 1) EM-ICP ; 2) X 0 X n 1. EM-ICP CUDA 5 : 1) d ij ; 2) C = (α ij ) n m ; 3) x i; 4) ; 5) R t. 5 Kernel 4 CUBLAS [21] : Sgemv, Saxpy, Sgemm Sscal. [14]. X 0 X n 1 3. 3 4 5 6 Kernel. X n 1, CUDA-thread. 5, CUDA-thread GPU- KD, x i. Kernel. 3. X 0 X n 1 1. X 0 GPU-KD,. 2. X 0 X n 1. 3. for R i,i+1 t i,i+1 do 4. X n 1 (X n 1 R i,i+1 t i,i+1 ). 5. GPU-KD, X n 1.
1294 38 6. ε. 7. if ε < ε min do 8. R 0 R i,i+1, t 0 t i,i+1, ε min ε 9. end if. 10. end for. 5.2 Kinect i D i, GPU RGB. CUDA Kernel. CUDA-thread. 5.3 4. CUDA Kernel CUBLAS : Sgemm. Kernel 5.1. S Sgemm, Kernel Xi., CPU, R k t k. X i, GPU-KD. 4. c 1. X i GPU-KD,. 2. for X j (i c j < i) do 3. X j. 4. ˆR ji ˆt ji ( ) ( ) i j ˆRji ˆt ji R0 t 0 = 0 1 0 1 5. R 0 ˆR ji, t 0 ˆt ji. 6. for k = 1,, maxiterations do 7. GPU-KD, X j. 8. Xi = {x i1, x i2,, x im }. 9. X j Xi R t. 10. R k R, t k t. 11. end for. 12. end for.. CUDA-thread, RGB,. 6, 7, : ( ) ( ) ( ). 3 ( ), 1.,,.,, 1, 3, tm 0 = 1 s; c = 3., c c EM-ICP ICP,,. 5.4 A SVD CULA [22] : Sgesvd., CUBLAS. 5.5 Kernel Fig. 3 3 Object images and reconstruction results 4 :,, ;, ; ;
8 : GPU Kinect 1295. 3.,,.,,,. 4., ;,.,,.,,. ; 3) ; 4) ; 5) ; 6) ; 7) ; 8) ; 9)., 2., 2 %.,.,, X 0 X n 1 (X n 1 R 0,n 1 t 0,n 1 ), 3. 3 3,,. 1 Table 1 Some meta-information and reconstruction results (mm) 195 164 332 1 169 981 82 121 213 367 036 81 201 220 305 162 310 1 114 669 163 280 707 175 98 187 319 312 90 120 170 49 308 5 Fig. 5 Measurements Table 2 2 Results of measurements (mm) (mm) (mm) (%) 1 87.3 86.1 1.2 1.37 2 33.2 32.8 0.4 1.20 3 174.1 171.9 2.2 1.26 4 78.7 77.9 0.8 1.01 5 33.1 32.5 0.6 1.81 6 24.7 24.3 0.4 1.62 7 84.2 82.9 1.3 1.54 8 50.1 49.4 0.7 1.49 9 163.1 161.6 1.5 0.92 4 Fig. 4 Fusion models,,. 5, : 1) ; 2),,. 2, Kinect,., 4. ( 46 ),
1296 38 50. 3 Table 3 Residual-errors (mm) X 0 0.217 20 166 0.187 6 288 0.494 4 252 0.378 21 064 0.345 11 662 0.221 7 145 0.198 578.,,,. Kinect,. References 1 Engelhard N, Endres F, Hess J, Sturm J, Burgard W. Realtime 3D visual SLAM with a hand-held RGB-D camera. In: Proceedings of the 2011 RGB-D Workshop on 3D Perception in Robotics at the European Robotics Forum. Västeras, Sweden: Robotdalen, 2011 Table 4 4 (s) Time consuming of objects reconstruction (s) 2 Henry P, Krainin M, Herbst E, Ren X, Fox D. RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments. In: Proceedings of the 12th International Symposium on Experimental Robotics. Delhi, India: IEEE, 2010 Kinect 1.5 46.0 3.3 50.8 1.5 46.0 2.5 50.0 1.5 46.0 2.2 49.7 1.5 46.0 1.9 49.4 1.5 46.0 3.1 50.6 1.5 46.0 2.3 49.8 1.5 46.0 1.5 49.0 7 Kinect.,. GPU, 50 s.,.,.,., : 1).,. 2) Kinect. Kinect 0.5 m,,., 3 Du H, Henry P, Ren X F, Cheng M, Goldman D B, Seitz S M, Fox D. Interactive 3D modeling of indoor environments with a consumer depth camera. In: Proceedings of the 13th International Conference on Ubiquitous Computing. Beijing, China: IEEE, 2011. 75 84 4 Izadi S, Newcombe R A, Kim D, Hilliges O, Molyneaux D, Hodges S, Kohli P, Davison A, Fitzgibbon A. KinectFusion: real-time dynamic 3D surface reconstruction and interaction. In: Proceedings of the 2011 International Conference on Computer Graphics and Interactive Techniques. Vancouver, Canada: ACM, 2011 5 Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 2011 Annual ACM Symposium on User Interface Software and Technology. Santa Barbara, CA: ACM, 2011. 559 568 6 Tong J, Zhou J, Liu L G, Pan Z G, Yan H. Scanning 3D full human bodies using Kinects. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(4): 643 650 7 Bay H, Ess A, Tuytelaars T, van Gool L. Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110(3): 346 359 8 Besl P J, McKay H D. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): 239 256 9 Konolige K, Mihelich P. Technical description of Kinect calibration [Online], available: http://www.ros.org/wiki/ kinect calibration/technical, November 3, 2011 10 Gu Zhao-Peng. A Study on Monocular Simultaneous Localization and Mapping [Ph. D. dissertation]. Institute of Automation, Chinese Academy of Sciences, China, 2011 (. [ ],,, 2011)
8 : GPU Kinect 1297 11 Liu Y H. Automatic registration of overlapping 3D point clouds using closest points. Image and Vision Computing, 2006, 24(7): 762 781 12 Horn B K P. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, 1987, 4(4): 629 642 13 Granger S, Pennec X. Multi-scale EM-ICP: a fast and robust approach for surface registration. In: Proceedings of the 7th European Conference on Computer Vision. Copenhagen, Denmark: Springer-Verlag, 2002. 418 432 14 Tamaki T, Abe M, Raytchev B, Kaneda K. Softassign and EM-ICP on GPU. In: Proceedings of the 2010 1st International Conference on Networking and Computing. Washington DC, USA: IEEE, 2010. 179 183 15 Dewaele G, Devernay F, Horaud R. Hand motion from 3D point trajectories and a smooth surface model. In: Proceedings of European Conference on Computer Vision. Prague, Czech: Springer, 2004. 495 507 16 Tamaki T. Pose estimation and rotation matrices. IEICE Technical Report SIS2009-23, 2009, 109(203): 59 64 17 Herrera C D, Kannala J, HeikkilAa J. Accurate and practical calibration of a depth and color camera pair. In: Proceedings of the 14th international conference on Computer analysis of images and patterns. Seville, Spain: Springer, 2011. 437 445 18 Martinec D, Pajdla T. Robust rotation and translation estimation in Multiview reconstruction. In: Proceedings of 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Minneapolis, Minnesota, USA: IEEE, 2007. 1 8 19 Hartley R, Zisserman A. Multiple View Geometry in Computer Vision (Second edition). London: Cambridge University Press, 2004 20 CUDA 3.0. CUDA Programming Guide 3.0 [Online], available: http://developer.download.nvidia.com/compute/ cuda/, December 5, 2011 21 CUBLAS library 2.0. CUBLAS Library User s Guide [Online], available: http://developer.download.nvidia.com/ compute/cuda/, December 5, 2011 22 CULA library. CULA Library User s Guide [Online], available: http://www.culatools.com/, December 5, 2011. 2004. GPU.. E-mail: liux skylark@tom.com (LIU Xin Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. He received his bachelor degree from Beijing Normal University in 2004. His research interest covers 3D reconstruction and GPU computing. Corresponding author of this paper.).. E-mail: huarong xu@sina.com (XU Hua-Rong Associate professor in the Department of Computer Science and Technology, Xiamen University of Technology. His research interest covers computer vision and image processing.).,,. E-mail: huzy@nlpr.ia.ac.cn (HU Zhan-Yi Professor at the Institute of Automation, Chinese Academy of Sciences. His research interest covers robot vision, which include camera calibration and 3D reconstruction, vision guided robot navigation, image based modeling and rendering.)