Mutual information based tracking

Contact : Eric Marchand
Date : 2011

Overview

we present a direct image registration approach that uses Mutual Information (MI) as a metric for alignment. The proposed approach is robust, real-time and gives an accurate estimation of a set of 2D motion parameters. MI is a measure of the quantity of information shared by signals. Although it has the ability to perform robust alignment with illumination changes, multi-modality and partial occlusions, few works propose MI-based applications related to spatio-temporal image registration or object tracking in image sequences due to some optimization problems.

In this work, we propose a new optimization method that is adapted to the MI cost function and gives a practical solution for real time tracking. We show that by refining the computation of the Hessian matrix and using a specific optimization approach, the registration results are far more robust and accurate than the existing solutions while the computation is cheaper. A new approach is also proposed to speed up the computation of the derivatives and keep an equivalent optimization efficiency.

Related publications

1. A. Dame, E. Marchand. Second order optimization of mutual information for real-time image registration. IEEE Trans. on Image Processing, 21(9):4190-4203, September 2012.
2. A. Dame, E. Marchand. Accurate real-time tracking using mutual information. In IEEE Int. Symp. on Mixed and Augmented Reality, ISMAR'10, Pages 47-56, Seoul, Korea, October 2010.

Validation

Metaio Benchmark

The validity of the approach has been validated through several experiments. The tracker has been evaluated on the metaio benchmark and gives very good results.

AR application

Many sequences have also been treated and shows the accuracy of the tracker as it is show in the video opposite. Since the estimation of the displacement is very accurate, the use of the proposed approach gives a new solution to augmented reality applications.

Multi-modal registration

The next experiment illustrates the capabilities of the presented mutual information-based image registration process in alignment applications between map and aerial images. The reference image is a map template provided by IGN (Institut Géographique National) that can easily be linked to Geographic Information System (GIS) and the sequence has been acquired using a moving USB camera focusing on a poster representing the satellite image corresponding to the map.

A non-linear relationship exists between the intensities of the map and aerial image and this link can be evaluated by the MI function. Mutual information can therefore allow for aligning the satellite image using the map image. this video shows, the selected initial position can be rather far from the correct position.

Potential AR applications of multimodal registration

Registering a map and an aerial image sequence is an extreme case, but registration between aerial and satellite (or any combination of such modalities), acquired at different time (and thus different) can be considered. Potential applications include visual odometry, aircraft or drone localization, pilot assistance, etc. Infrared cameras (although still expensive) are widely used by civilians and, obviously, military aircraft. Such a registration process with a simple satellite image may prove to be very helpful for the pilots especially when landing (night or day) on a small and ILS free airport. Considering that aircraft position is fully known, additional information about runway, other aircraft positions or military targets may thus be easily displayed in the pilot helmet. Although, we mentioned here applications in the aeronautic area, it is clear that other domains may be targeted such as energy monitoring, robotics, urbanism, architecture, defense, ...