Links

Home

Publications

Research

Gallery

Jean-Michel Morel           Guoshen Yu

morel[AT]cmla.ens-cachan.fr    yu[AT]cmap.polytechnique.fr

News: The ASIFT source code and online demo are now published in the journal IPOL! 2011.02.24

News: frequently asked questions on ASIFT. --- 2009.11.22

Summary:

A fully affine invariant image comparison method, Affine-SIFT (ASIFT) is introduced. While SIFT is fully invariant with respect to only four parameters namely zoom, rotation and translation, the new method treats the two left over parameters : the angles defining the camera axis orientation. Against any prognosis, simulating all views depending on these two parameters is feasible. The method permits to reliably identify features that have undergone very large affine distortions measured by a new parameter, the transition tilt. State-of-the-art methods hardly exceed transition tilts of 2 (SIFT), 2.5 (Harris-Affine and Hessian-Affine) and 10 (MSER). ASIFT can handle transition tilts up 36 and higher.

References:

• J.M. Morel and G.Yu, ASIFT: A New Framework for Fully Affine Invariant Image Comparison, SIAM Journal on Imaging Sciences, vol. 2, issue 2, 2009.

• G.Yu and J.M. Morel, ASIFT: An Algorithm for Fully Affine Invariant Comparison,  Image Processing On Line, 2011. DOI:10.5201/ipol.2011.my-asift.

• G. Yu and J.M. Morel, A Fully Affine Invariant Image Comparison Method, Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, 2009.

• J.M. Morel and G.Yu, Is SIFT Scale Invariant?, Inverse Problems and Imaging, vol 5, no. 1, Feb., 2011

Additional Materials:

• Slides [one-hour version] [20-minute version]

• Poster

Online demo:  

An online demo (version beta) that allows you to try ASIFT with your own images is available here. If you are interested in ASIFT and want to try it with your own images, we suggest that you start with the user friendly online demo.

Source code : NEW!!!

• Standalone program in C++

• Ready to be complied

• Reproduce exactly the same results as in the online demo

The ASIFT source code and online demo are now published in the journal IPOL.

Frequently asked questions:

We have received many correspondences from the ASIFT readers and users. Some typical questions are posted here.

Dataset:

An image dataset for systematic evaluation of robustness to absolute and transition tilt of the image matching algorithms is available here.

When does it work?

The SIFT method works to compare 2D objects or 3D objects with flat enough details, taken from similar view angles but at arbitrary distances.

The typical failure cases are:

• The illumination conditions are different (for instance daylight/nightlight).

• The object has a reflecting surface (typically cars are mirrors ; they change completely aspect under different view angles).

• The object has a strong 3D structure: in that case a change of view angle alters drastically its aspect.

• The object has a self similar or periodic structure: then "true" mismatches occur.

• The view angle is too different.

See our failure case study.

ASIFT corrects the last problem : if the object is under view has similar illumination conditions, has rather flat surface, and is not a mirror, then ASIFT retrieves the object even under extreme changes of angle. In technical terms, ASIFT is more affine invariant than SIFT.

Other popular methods addressing this problem exist: MSER and Hessian-Affine and Harris-Affine. All these methods will be compared below.

Examples:

ASIFT is compared with the four state-of-the-art algorithms the SIFT, Harris-Affine, Hessian-Affine and MSER detectors, all coded with the SIFT descriptor. Various types of images (size 600×450) were used for the experiments. The SIFT software is from D. Lowe. The Harris-Affine, Hessian-Affine and MSER programs are from the web site of K. Mikolajczyk.

• Planar objects (paintings, journals, walls, etc)

• Monuments and constructions.

• 3D objects (statues, cars, etc)

• Complex scenes.

• Object deformation.

• Video tracking.

In the examples below, the powerful Moisan-Stival procedure is applied to eliminate matches incoherent with epipolar geometry. (download .pdf) Matches are connected by white segments.

Video Tracking

　

1. Planar objects
Adam taken from short distance (zoom ×1) at frontal view and at 75 degree angle. Absolute tilt t = 4 (middle), <4 (left), >4 (right).  Not shown Harris-Affine: 3 matches. Hessian-Affine: 1 match
ASIFT: 202 matches	SIFT: 15 matches	MSER: 5 matches
Adam taken from short distance (zoom ×10) at frontal view and at 65 degree angle. Absolute tilt t = 2.4.  Not shown Harris-Affine: 3 matches. Hessian-Affine: 0 match.
ASIFT: 341 matches	SIFT: 5 matches	MSER: 4 matches
Adam taken from short distance (zoom ×10) at frontal view and at 80 degree angle. Absolute tilt t = 5.8.  Not shown Harris-Affine: 1 match. Hessian-Affine: 0 match.
ASIFT: 75 matches	SIFT: 1 match	MSER: 2 matches
Magazine taken from middle distance (zoom ×4) at frontal view and at 80 degree angle. Absolute tilt t = 5.8.  Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
ASIFT: 349 matches	SIFT: 0 match	MSER: 17 matches
Magazine taken with absolute tilt t1= t2= 2, with longitude angles Φ1= 0 deg, Φ2= 50 deg.  Transition tilt  τ = 3.  Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
ASIFT: 881 matches	SIFT: 3 matches	MSER: 87 matches
Magazine taken with absolute tilt t1= t2= 4, with longitude angles Φ1= 0 deg, Φ2= 90 deg.  Transition tilt  τ = 16.  Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
ASIFT: 88 matches	SIFT: 1 match	MSER: 9 matches
Facade at frontal view and at 75 degree angle. Absolute tilt  t = 3.8. Not shown SIFT: 0 match. Harris-Affine: 1 match.
ASIFT: 68 matches
Hessian-Affine: 1 match
MSER: 2 matches
Graffiti No.1 vs No. 6 (images from K. Mikolajczyk).  Transition tilt  τ ~ 3.2. Not shown SIFT: 0 match. Hessian-Affine: 1 match.
ASIFT: 721 matches
Harris-Affine: 3 matches
MSER: 70 matches
Direction Transition tilt τ ~ 2.6. Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
ASIFT: 50 matches
SIFT: 0 match
MSER: 1 match
Parkings Transition tilt τ ~ 15. Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
ASIFT: 78 matches
SIFT: 0 match
MSER: 0 match
Stump Transition tilt τ ~ 2.6.  Not shown Harris-Affine: 2 matches. Hessian-Affine: 1 match.
ASIFT: 168 matches	SIFT: 1 match	MSER: 6 matches
Back to top
2. Monuments and constructions
Notre Dame Transition tilt τ ~ 1.4. Not shown Hessian-Affine: 1 match. MSER: 0 match.
ASIFT: 35 matches
SIFT: 10 matches
Harris-Affine: 1 match
Pentagon Not shown SIFT: 6 matches.  Harris-Affine: 2 matches.
ASIFT: 28 matches	Hessian-Affine: 8 matches	MSER: 17 matches
Round building Transition tilt τ ~ [1.8, ∞).  Not shown Harris-Affine: 5 matches. Hessian-Affine: 7 matches.
ASIFT: 139 matches	SIFT: 19 matches	MSER: 13 matches
Palace of Versailles Transition tilt τ ~ 1.8.  Not shown Harris-Affine: 2 matches. Hessian-Affine: 1 match.
ASIFT: 67 matches	SIFT: 26 matches	MSER: 4 matches
Ecole Polytechnique taken at frontal view and at 65 degree angle. Absolute tilt t = 2.4.  Not shown Harris-Affine: 2 matches. Hessian-Affine: 2 matches.
ASIFT: 101 matches	SIFT: 13 matches	MSER: 4 matches
Back to top
3. 3D objects
Statue of Liberty  τ ~ [1.3, ∞). Not shown Hessian-Affine: 1 match. MSER: 1 match.
ASIFT: 32 matches
SIFT: 9 matches
Harris-Affine: 1 match
Statue Transition tilt τ ~ [1.6, ∞). Not shown Harris-Affine: 7 matches. Hessian-Affine: 2 matches.
ASIFT: 67 matches	SIFT: 26 matches	MSER: 4 matches
Car Not shown Harris-Affine: 0 match.  MSER: 1 match.
ASIFT: 28 matches	SIFT: 1 match	Harris-Affine: 1 match
Can Transition tilt τ ~ [2.3, ∞). Not shown SIFT: 0 match. Hessian-Affine: 3 matches.
ASIFT: 287 matches
Harris-Affine: 6 matches
MSER: 22 matches
Back to top
4. Complex scenes
Office Transition tilt  τ ~ 3.  Not shown SIFT: 0 match. Harris-Affine: 0 match.
ASIFT: 88 matches
Hessian-Affine: 1 match
MSER: 3 matches
Bottles (proposed by the authors of MSER)  Transition tilt τ ~ [1.6, 3.0]. Not shown SIFT: 10 matches. Hessian-Affine: 11 matches.
ASIFT: 254 matches
Harris-Affine: 23 matches
MSER: 22 matches
Coffee room Transition tilt τ ~ [1.5, 3.3]. Not shown Harris-Affine: 0 match. Hessian-Affine: 3 matches.
ASIFT: 125 matches
SIFT: 13 matches
MSER: 5 matches
Back to top
5. Object deformation (images from Ling and Jacobs)
SpongeBob Not shown Harris-Affine: 6 matches.  MSER: 4 matches.
ASIFT: 370 matches	SIFT: 75 matches	Harris-Affine: 8 matches
CVPR Not shown Harris-Affine: 25 matches.  MSER: 17 matches.
ASIFT: 528 matches	SIFT: 160 matches	Hessian-Affine: 55 matches
Flag Not shown Hessian-Affine: 10 matches.  MSER: 2 matches.
ASIFT: 141 matches	SIFT: 31 matches	Harris-Affine: 15 matches
Girl Not shown Harris-Affine: 45 matches (1 on the cloth).  Hessian-Affine: 22 matches (1 on the cloth).
ASIFT: 659 matches	SIFT: 304 matches	MSER: 35 matches
Toy Not shown Harris-Affine: 0 match. MSER: 0 match.
ASIFT: 33 matches
SIFT: 4 matches
Hessian-Affine: 1 match
Back to top
Failure: day-and-night illumination change --- all methods fail!
Six images of Notre-Dame under different illumination conditions are compared. The number of matches of ASIFT and SIFT are shown. (Harris-Affine, Hessian-Affine and MSER find less matches than SIFT.) Little view angle change is presented. The red arrows imply recognition failure. 　 In general, matching succeeds between day images and between night images. However, under day-and-night illumination change, all methods fail. 　 Description: -  All methods fail under day-and-night illumination change (pairs 1-2, 1-6, 2-3, 2-4, 2-5, 3-6, 4-6, 5-6). - Matching succeeds between night images, with a substantial scale change (pair 2-6). - Matching succeeds between day images under same weather condition, with or without a substantial scale change (pairs 1-4, 3-5). - Matching succeeds between day images under different weather conditions, without a substantial scale change (pairs 1-3, 3-4). - Matching fails between day images under different weather conditions, with a substantial scale change (pairs 1-5, 4-5).
Failure: strong relief effect --- all methods fail! ASIFT, SIFT, Harris-Affine, Hessian-Affine, MSER: 0 match. The image on the right in close-up shows strong relief effect.
Failure: Repetitive shapes --- "good" false matches Matches can be arbitrary among repetitive shapes.
ASIFT: 169 matches, many are "good" false matches (for example the matches between the left-most window in the image above and the second window in the image below). 　 Due to the viewpoint change, SIFT, Harris-Affine, Hessian-Affine and MSER find much less matches (respectively 30, 4, 9 and 4), among which many are "good" false matches as well.
Back to top