Photos by Russell Hewett
In this project, you will create a program to automatically stitch two or more photographs into a single image. In doing so, you will explore correspondence using interest points, robust matching with RANSAC, and homography. You can also investigate cylindrical and spherical projection and other extensions of photo stitching and homography as bells and whistles.
To stitch two images, you must first establish point correspondences. In the first part of this assignment, you will do so manually. In the second part, you will use automatically computed interest points and SIFT descriptors and RANSAC to establish the correspondences. With four or more correspondences, you can compute the homography matrix
You may not use some built-in matlab functions, such as
For this first part, you will manually select the corresponding points and write code to compute the homography (warping) and compositing. Each step is described in detail below. Create a function
that stitches two images after asking the user to provide at least four correspondences.
You're strongly encouraged to use your own photographs. You are also welcome to try using samples by Russ Hewett (small, large). The photographs must be of the same scene, and the translation of the camera must be small compared to the depth of the scene. The pictures should overlap quite a bit but not entirely (e.g., 30%-70%). Examples of scene include panoramas, waterfalls, and interesting indoor environments -- anything with interesting texture that is not well-contained within a single photograph.
Manual Point Correspondence
For a pair of photographs, click on four or more (I suggest more) pairs of points that correspond to the same place on the same object in the scene. You can do this using
Computing the Homography
Write a function
The last step is to create a single image that combines the different parts of the scene captured by each photograph. Write a function
that projects both image onto a planar surface and blends. The easiest way to do this is to create two larger images in the same coordinate space as the first image -- one for each projected image. Copy pixels from the first image, and warp pixels from the second image using the recovered homography (hopefully, this still feels familiar from the morphing project). Adjust the intensity of the 2nd image so that its average intensity matches the average intensity of the 1st image in the overlapping region. Then, create a mask (e.g., based on whether a pixel is closer to the center of image 1 or image 2) and blend. For extra points, you can create two images, one of the projection from each image and then to blend using Laplacian pyramid blending.
Before working on Part II, make sure that you have Part I working. Now, it's time to fully automate the process. First, you should automatically compute interest points and descriptors, using provided code. Then, you should find initial matches to those descriptors and, finally, use the RANSAC algorithm to find a good set of matching points with the homography that corresponds them. Create a function
Computing Interest Points
To compute interest points, use the SIFT code provided by Andrea Vedaldi:
I strongly recommend using the latest of the pre-compiled binaries if possible. Read the included documentation for how to use the
Finding Good Correspondences with RANSAC
Unfortunately, not all of the initial corresponding points can be trusted. Fortunately, we can deal with some outliers using RANSAC. Write a function
that robustly computes the homography from a set of corresponding points that include outliers. The basic idea is to randomly select four points, compute the homography from these four points, see how many other points agree (if a point from
For compositing, you can use your
Laplacian pyramid blending (up to 20 pts)
To get a seamless blend, you should adjust for the overall gain and then use a Laplacian pyramid blend. For 15 points, do the 2-band blending; 20 points for multi-band blending. If you've already gotten extra points for re-implementing Laplacian pyramid blending for a previous project, you can reuse it here, but do a nice before/after comparison for at least two or three panoramas.
Fun with homography (up to 10 pts)
Try other uses of homography. Put your face on a billboard photographed from the street, or blend an image into a football field. Manually select the four corresponding points and use your
Stitching multiple images (10 pts)
Show the result for at least one multi-image panoramic stitching. In this case, the multiple images can be done sequentially. This is a good one to combine with the stitching on non-planar surfaces task.
Stitching onto non-planar surfaces (up to 30 pts)
There's a limit to how much rotation you can deal with when projecting onto a plane. It's much more common to use cylindrical projection for large panoramas. It's also possible to project onto a sphere or a cube. You can get up to 20 points for doing one of these, and 5 points each for doing the others. Show at least one example for each time, preferrably with multiple images. Hint: the easiest way to do this is to figure out the (non-linear) projection from im1 onto the non-planar surface; then to map im2, project from im2 to im1 using
Stitching multiple images with bundle adjustment (25 pts)
Not for the faint of heart, implement and demonstrate the bundle adjustment described by Brown and Lowe (2007) for 25 pts.
Other ideas (up to 30 pts)
If you want to try something else and you want points for it, let me know ahead of time, and I'll tell you how much it's worth.
To turn in your assignment, place your index.html file and any supporting media in your project directory: http://web.engr.illinois.edu/~netid/cs498dwh/proj5/, where "netid" is your netid (e.g., dhoiem). Also, e-mail me the zipped code in "netid_proj5.zip" and include a link to the project page in the text. See project instructions for details. In the e-mail, tell me how many points you think you should get for the core project and any bells and whistles.
Use both words and images to show us what you've done. Please:
The core assignment is worth 100 points, as follows:
You can also earn up to 125 extra points for the bells & whistles mentioned above.