Virtual Visual Servoing: A framework for augmented reality

Note about this page

This page has been created in 2002 and deals with the work presented in the Eurographics'02 conference. It does not present any results dealing with the paper to be published at ISMAR'03, Tokyo, October 2003. An new demonstration dealing with these particular results will be proposed in a short time.

Overview

In this demonstration we illustrate a formulation of pose computation involving a full scale non-linear optimization: Virtual Visual Servoing (VVS). We consider the pose computation problem as similar to 2D visual servoing [Sundareswaran98]. Visual servoing or image-based camera control [Hutchinson96][Espiau92] allows to control a camera wrt. to its environment. More precisely it consists in specifying a task (mainly positioning or target tracking tasks) as the regulation in the image of a set of visual features. A set of constraints are defined in the image space. A control law that minimize the error between the current and desired position of these visual features can then be automatically built. This approach has proved to be an efficient solution to camera positioning task within the robotics context and more recently in computer graphics. Considering pose as an image-based visual servoing problem takes advantage of all the background knowledge and the results in this research area. It allows us to propose a very simple and versatile formulation of this important problem. One of the main advantages of this approach is that it allows to consider within the same process different geometrical features. We show how this framework scale easily when the camera parameters are unknown or modified.

Principle : Pose computation by virtual visual servoing.

The principle of our algorithm is to iteratively modify using a visual servoing control law the position of a virtual camera in order to register the desired features extracted from the image (in green) and the current one obtained by back-projection of the object model (in blue) for a given pose. Image (a) corresponds to the initialization while in image (d) registration has been achieved and the pose is computed. This figure illustrates the registration convergence for one image. It also illustrates the minor influence of the initialization, indeed the initial position/orientation of the camera is very different from the final computed one. In this example, straight lines are considered to compute pose.



see mpeg movie

Results

The architecture of our system is similar to those proposed in [Billinghurst01]. Since our approach is based on the visual servoing framework we rely on a library dedicated to such system and called ViSP (Visual servoing platform). This library is written in C++ and proposed both the features tracking algorithms and the pose computation and calibration algorithms. A new software component based on Open Inventor has been written to allow the insertion of virtual objects in the images. All the experiments presented in the next paragraph have been carried out on a simple PC with an Nvidia 3D board and an Imaging technology IC-Comp framegrabber.

Augmented reality using precise patterns.

In a first time we report augmented reality experiments that use precise patterns to estimate the camera viewpoint. Such experiments show that this approach may be efficiently used in interactive application such as (but not restricted to) collaborative immersive workplace, cultural heritage or architecture interactive visualization interactive video game (by moving and orienting a simple pattern the player may modify on line its perception of the game).

• in this experiment we show that a limited number of features are required to compute the pose. Indeed, the pose related to ``ghost'' is computed using two circles (the first image depicts the result of the image processing where the related circles appear in green). An other object, a ```virtual graveyard'', is also inserted. Therefore, an other pose is computed for this object using four points (the center of each black dot). The two poses and the rendering of the scene is handled at video rate.

see mpeg movie

Augmented reality in non controlled situations.

We then consider ``real'' images acquired using a commercial recorder. In such experiments, the image processing may be very complex. Indeed extracting and tracking reliable points in real environment is a real issue. Therefore it is important to consider other image feature than simple points. We demonstrate the use of circles, lines, and cylinders. In various experiments, the features are tracked using the Moving edges algorithm at video rate.

• We first consider a context that can be encountered in an industrial environment. A camera is mounted on the end-effector of a 6 d.o.f robot. The primitives considered in the virtual servoing process are two cylinders and a straight line. Pose computation is handled in less than 3 ms (except in the very first image). Note that some features disappear over time. The size of the interaction matrix and of the error vector is modified consequently.
  
  
  
  
• In the second experiments, we consider an outdoor scene. Here again, we consider a cylinder and two lines to compute the pose. Despite very noisy images (wind in the trees, etc.) tracking is achieved along a 700 and 1400 image sequences. The first image display the tracked lines and cylinder limbs as well as 3D informations inserted after the pose computation (the reference frame and the projection of the cylinder). The other images are extracted from the sequence after the insertion of virtual objects.
  
  
  
  
  
• The third experiments features a similar experiments but features are partially not visible by the camera (left and right image). The algorithm still generate (up to a certain limit) acceptable results. On the right image note that explicit occlusions is not handled (this was not the scope of this demonstration).
  
  
  
  see second mpeg movie

Publications

E. Marchand, F. Chaumette. Virtual Visual Servoing: a framework for real-time augmented reality. in EUROGRAPHICS 2002 Conference Proceeding, G. Drettakis, H.-P. Seidel (eds.), Computer Graphics Forum, Volume 21(3), Saarebrücken, Germany, September 2002.

E. Marchand, F. Chaumette. A new formulation for non-linear camera calibration using virtual visual servoing. Rapport de Recherche IRISA, No 1366, Janvier 2001.

Other Related Papers

V. Sundareswaran, R. Behringer. Visual servoing-based augmented reality. In IEEE Int. Workshop on Augmented Reality, San Francisco, November 1998.

M. Billinghurst, H. Kato, I. Poupyrev. The magicbook: Moving seamlessly between reality and virtuality. IEEE Computer Graphics and Applications, 21(3):6-8, May 2001.

B. Espiau, F. Chaumette, P. Rives. A new approach to visual servoing in robotics. IEEE Trans. on Robotics and Automation, 8(3):313-326, June 1992.

S. Hutchinson, G. Hager, P. Corke. A tutorial on visual servo control. IEEE Trans. on Robotics and Automation, 12(5):651-670, October 1996.

E. Marchand, N. Courty. Image-based virtual camera motion strategies. In S. Fels and P. Poulin, editors, Graphics Interface Conference, GI2000, pages 69-76, Montreal, Quebec, May 2000. Morgan Kaufmann.