Automatic Video Segmentation Employing Object/Camera Modeling Techniques PROEFSCHRIFT

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download "Automatic Video Segmentation Employing Object/Camera Modeling Techniques PROEFSCHRIFT"

Αμφιτρίτη Παπαδόπουλος
7 χρόνια πριν
Προβολές:

1 Automatic Video Segmentation Employing Object/Camera Modeling Techniques PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de Rector Magnificus, prof.dr.ir. C.J. van Duijn, voor een commissie aangewezen door het College voor Promoties in het openbaar te verdedigen op donderdag 15 december 2005 om uur door Dirk Sven Farin geboren te Tübingen, Duitsland

2 Dit proefschrift is goedgekeurd door de promotoren: prof.dr.ir. P.H.N. de With en Prof. Dr.-Ing. W.W.J. Effelsberg CIP-DATA LIBRARY TECHNISCHE UNIVERSITEIT EINDHOVEN Farin, Dirk S. Automatic video segmentation employing object/camera modeling / by Dirk Sven Farin. - Eindhoven : Technische Universiteit Eindhoven, volumes. Proefschrift. - ISBN X NUR 959 Trefw.: videotechniek / digitale beeldverwerking / beeldcodering / beeldherkenning. Subject headings: image segmentation / motion estimation / computer vision / object detection. Schlagwörter: Videosegmentierung / Bewegungsschätzung / Objekterkennung. c Copyright 2005 Dirk Farin All rights are reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission from the copyright owner.

3 Contents 1 Introduction Motivation Video editing and scene composition Object-oriented video coding Automatic video analysis D analysis and reconstruction The video-object segmentation problem Object-oriented video coding in MPEG Automatic video segmentation system (Thesis Part I) Design goals Segmentation-algorithm overview Framework of the segmentation algorithm Extensions to the segmentation system Segmentation using object models (Thesis Part II) From camera motion to 3-D models (Thesis Part III) Contributions of the author I An Automatic Video Segmentation System 31 2 Projective Geometry Introduction Projective spaces Homogeneous coordinates Lines in the projective plane Geometric transformations in 2-D Projective transformation Affine motion Projective motion i

4 ii Contents 2.4 Geometric transformations in 3-D Affine motion in 3-D Rotation in 3-D Perspective projection Image acquisition Intrinsic camera parameters Extrinsic camera parameters Camera motion in a static environment Inter-image transformation Summary and notational conventions Feature-based Motion I: Point-Correspondences Introduction Basics of feature-based motion estimation From feature-points to motion parameters Interest-point detectors Moravec interest-point detector Shi-Tomasi detector Harris corner detector SUSAN corner detector Evaluation Computing feature-correspondences Fast greedy algorithm Evaluation Summary Feature-Based Motion II: Parameter Estimation Introduction Computing motion model parameters One-dimensional affine motion Two-dimensional affine motion One-dimensional projective motion Two-dimensional projective motion Non-linear least-squares estimation Robust estimation algorithms Breakdown of least-squares fit on data with outliers Robust estimation using RANSAC Robustness of the RANSAC algorithm Summary

5 Contents iii 5 Background Reconstruction Introduction Frame alignment Motion models for sprite generation Geometry of background image generation Long-term motion estimation Background estimation Introduction and previous work The SimMat background-estimation algorithm Results Summary of the background reconstruction module Multi-Sprite Backgrounds Introduction Limitations of the single-sprite approach Detecting degenerated transforms Examples of single-sprite inefficiencies Example case: camera zoom-out Example case: horizontal camera pan Example case: camera zoom-in Sprite cost definitions Bitstream length Coded sprite area Sprite buffer size Adding a resolution preservation constraint and limiting sprite buffer requirements Multi-sprite partitioning algorithm Cost matrix calculation and reference frame placement Optimal sequence partitioning Experiments and results Integration into the segmentation system Online calculation of constrained sprites Coding multi-sprites in MPEG-4 streams Conclusions Background Subtraction Introduction Pixel-based classification Distance metrics Influence of the color-space Classes of errors Evaluation method

6 iv Contents Results Multi-pixel based significance tests Classification using a χ 2 test Extension to color images Fast implementation Evaluation Classification using Markov random fields MRF model for segmentation masks Obtaining a MAP estimate Extension to color images Optimization algorithm Evaluation Sources of errors and robustness improvements Map of misregistration risk Map of interpolation errors Integrating risk maps into the segmentation process Postprocessing the object mask Filling holes in the object Heuristics for removing clutter in the mask Overview of the segmentation process Results and Applications Algorithm modules Variants of the segmentation system Surveillance with a static camera Surveillance with a moving camera Offline video analysis Online video segmentation and transmission Implementation Segmentation results Applications of the segmentation system MPEG-4 video coding Video editing Pseudo 3-D video generation Video-object recognition Extensions MPEG-4 coding with sprite-mode detection Camera auto-calibration Absolute coordinate transfer Object models

7 Contents v II Segmentation Using Object Models Object Detection based on Graph-Models I: Cartoons Introduction Principle of region-based graph matching Model editor Automatic color segmentation Feature extraction and matching criteria Color Size Distance Shape Orientation Node and edge costs Generalization of costs for 1 : N-matching Matching algorithm Candidate-region selection Matching algorithm : N-matching Results Conclusions Object Detection based on Graph Models II: Natural Introduction Segmentation system architecture Step 1: motion detection Step 2: model matching Model editor Model detection Step 3: spatial segmentation Spatial segmentation algorithm Merging criterion Experiments and results Conclusions Appendix: notes on ellipse processing Manual Segmentation and Signature Tracking Introduction From Intelligent to Corridor Scissors Intelligent Scissors algorithm Problems of the Intelligent Scissors tool The Corridor Scissors tool

8 vi Contents Experiments and results with Corridor Scissors Shortest circular paths Definition of circular paths Computation of shortest circular paths Computational complexity Signature tracking A first tracking algorithm Signature tracking algorithm Circular-path search with object signatures Tracking results Summary and conclusions Discussion on the signature tracking technique Appendix: step-by-step examples III From Camera Motion to 3-D Models Estimation of Physical Camera Parameters Introduction Geometry of background image generation Global motion estimation Previous work Estimation of focal length Linear camera calibration Calibration using the image of the absolute conic Integration of multi-sprite motion estimation Non-linear camera calibration Parameterization Generalizing to multi-sprites Optimization algorithm Recovering rotation angles Experimental results Conclusions Camera Calibration for the Analysis of Sport Videos Introduction and previous work Calibration-algorithm principle Overview of the calibration system Court-line pixel detection Filter 1: luminance threshold Filter 2: non-flat regions Filter 3: linear structure

9 Contents vii 13.5 Line-parameter estimation Line detection with the Hough transform Line detection with RANSAC Line-segment boundary detection Court-model fitting Fast fitting method Robust fitting method Fast calibration-parameter rejection test Model tracking Experiments Conclusions Panoramic Video and Floor Plan Reconstruction Introduction From background sprites to panoramic images Visualization of panoramic images Floor plan reconstruction Chapter outline Capturing panoramic images and video Panoramic image generation Cameras for recording panoramic videos Visualization of panoramic videos Reconstruction of rectangular rooms The circular arc of possible camera locations Searching for the camera position Creating a virtual room visualization Reconstruction of floor plans Previous work Reconstruction algorithm concept Modeling the floor plan geometry Estimating the floor plan parameters Improving the convergence behaviour Initialization of the floor plan layout Obtaining wall textures from the panoramic images Experimental Results Conclusions Conclusions Discussion on the individual chapters Chapter 3 and 4: camera-motion estimation Chapter 5: background estimation Chapter 6: multi-sprites

10 viii Contents Chapter 7: background subtraction Chapter 9 and 10: graph-based object models Chapter 11: Corridor Scissors and circular paths Chapter 12: physical camera-parameter extraction Chapter 13: camera calibration for sport videos Chapter 14: floor plans from panoramic images Explicit vs. implicit models Future of segmentation IV Appendices 465 A Video-Summarization with Scene Preknowledge 467 A.1 Introduction A.2 Summarization algorithm A.2.1 Feature extraction A.2.2 Determining segment boundaries A.2.3 Clustering A.2.4 Integration of domain-knowledge A.3 Evaluation A.4 Conclusions B Efficient Computation of Homographies From Four Correspondences 479 C Robust Motion Estimation with LTS and LMedS 483 D Additional Test Sequences 487 E Color Segmentation Using Region Merging 489 E.1 Introduction E.1.1 The region-merging algorithm E.2 Merging criteria E.2.1 Mean luminance difference E.2.2 Ward s criterion E.2.3 Mean/Ward mixture E.2.4 Linear-luminance model E.2.5 Border criterion E.3 Criteria properties E.3.1 General behaviour E.3.2 Comparison E.4 Multi-stage merging

11 Contents ix E.4.1 Applying a watershed presegmentation E.5 Results and conclusions F Shape-Based Analysis of Object Behaviour 501 F.1 Classification of object shapes F.2 Simple model for object behaviour F.3 Behaviour analysis References 507 Summary 527 Samenvatting 533 Zusammenfassung 539 Acknowledgments 545 Biography 547

12 x Contents

Σχετικά έγγραφα

Spectroscopy on Metal-Halide Lamps under Varying Gravity Conditions PROEFSCHRIFT

Spectroscopy on Metal-Halide Lamps under Varying Gravity Conditions PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de Rector Magnificus, prof.dr.ir.