Enhancing the Point Feature Tracker by Adaptive Modelling of the Feature Support

Introduction

• A technique is proposed for enhancing the differential point feature tracker under following assumptions:
  • extended real sequences acquired from a moving car
  • mainly translational forward motion of the observer
  • a usual perspective camera (FOV ~ 30 degrees)
  • the principal objective: long feature lifetimes (the envisioned application area is visual servoing)
• special considerations for features only partly projected from a distinctive quasi-planar 3D surface:
  • usual feature size in tracking with warp correction: 15 x 15 pixels
  • at 50 m, this projects to a planar patch of over 1 x 1 m:
  • the odds for including a 3D surface discontinuity are high:
    ⇒ the non-rectangular features become increasingly important
• the desired functionality is illustrated in the figures right and below:

The feature support

• The ``strongest'' part of a feature, which is consistently tracked throughout the sequence is termed the feature support
• idea: to determine the support at runtime, by testing the temporal variability of individual pixels in the warped feature
• for each feature pixel x the current estimate of a distinct normal distribution is updated as follows:
  μ_x,t = (1-α)·μ_x,t-1 + α·x_t
  σ_x,t² = (1-α)·σ_x,t-1² + α·(x_t-μ_x,t)²
  
• α ∈ (0,1) represents the learning rate, how many previous frames are taken into account in the estimate
• best results obtained by:
  • detecting spikes in σ_x,t by thresholding
  • excluding persistently the corresponding pixels
  • adapting the modulating factor α to the interframe change of geometric warp parameters
• the effectiveness of the technique is illustrated in the figures on the right

Experimental results

• The experiments were performed on 8 sequences taken from cars performing a substantial movement in the forward direction:

  rennes:	Cycab approaching IRISA from the north
  rennes2:	Cycab in the inner court of IFSIC
  compiegne:	towards a square in the sunlight
  compiegne2:	traversing a large square into a street
  compiegne3:	a very narrow street
  compiegne4:	a street bordered by buildings and trees
  antibes:	some trees on the left and far away (15Hz)
  antibes2:	a narrow downhill winding street

• the videos of the results obtained for each of the above sequences can be accessed from this page (please see the paper for more details on obtained results)

  rennes: a, b	rennes2	antibes	antibes2
  compiegne	compiegne2	compiegne3	compiegne4

• the videos were prepared with mencoder, using the msmpeg4v2 codec, which should be playable on all platforms

Conclusions

• A technique for increasing the feature lifetimes in sequences acquired from a moving car has been presented
• the starting point was an adapted version of the differential tracker with warp correction
  • the warp models translation (2D), isotropic scaling (1D) and a linear feature-wide photometric change (2D)
  • the optimization of the warp parameters is performed at an adequate level of the 1:2:4 image pyramid
  • besides the usual RMS residual, the tracking is also evaluated by the ``blank wall'' criterion and the smoothness of the magnitude change
• the ill-behaved parts of the feature are detected by a statistical test which has been used in background subtraction
• the technique addresses ``almost good'' features, in which the planarity assumption is only partially satisfied due to:
  • occlusions and disocclusions
  • complex photometric variations
  • small structural developments
• further improvements might be obtained by using the recovered motion information to regulate the modulation speed α
• an eventual faster convergence would allow the monitoring procedure to rely more heavily on the size and the shape of the feature support
• the recovered suport could be used to move the tracked part of the feature towards a better behaved part of the scene

References

• J. Shi, C.Tomasi. Good features to track. In: Proc. of CVPR. (1994) 593-600
• C. Stauffer, W. Grimson. Adaptive background mixture models for real-time tracking. In: Proc. of CVPR, IEEE (1999) II: 246-252
• H. Jin, P. Favaro, S. Soatto. Real-time feature tracking and outlier rejection with changes in illumination. In: Proc. of ICCV. Volume 1. (2001) 684-689
• S. Baker, I. Matthews. Lucas-Kanade 20 years on: A unifying framework. Int. J. Comput. Vis. 56 (2004) 221-255
• S. Segvic, A. Remazeilles, F. Chaumette. Enhancing the Point Feature Tracker by Adaptive Modelling of the Feature Support. In European Conf. on Computer Vision, ECCV'2006, LNCS, Graz, Austria, Mai 2006.