# Programming Project #4: Face Morphing CS498: Computational Photography

## Face Labeling Due Date: 11:59pm on Wed, Oct. 12, 2011

See under "Computing the Mean Face" and instructions sent to the Google group: e-mail me a .txt with your face points.

## Project Due Date: 11:59pm on Monday, Oct. 24, 2011

### Overview

The current class faces and annotations are here.

In this assignment you will produce a "morph" animation of your face into another student's face. If all goes well, I'll stitch together the results into a morph through the faces of the whole class.

A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors. The cross-dissolve is the easy part; controlling and doing the warp is the hard part. The warp is controlled by defining a correspondence between the two pictures. The correspondence should map eyes to eyes, mouth to mouth, chin to chin, ears to ears, etc., to get the smoothest transformations possible.

We will take pictures of everyone in the class. Each student will then get a starting image ("picture A") and an ending image ("picture B") for the animation sequence. Picture A will be the photo of your face. Picture B will be the student whose image name is after yours in alphabetical order. Instructions for downloading your image and annotation were sent via Google groups. On Oct 13, I'll make all face images and annotations available.

You'll morph still picture A into still picture B and produce 61 frames of animation numbered 0-60, where frame 0 must be identical to picture A and frame 60 must be identical to picture B. In the video, each frame will be displayed for 1/30 of a second. Name your frames morph_name1_name2_??.jpg where ?? is 00 to 60 (do use two digits, even for 01, 02, etc.) and name1 and name2 are the basenames of the two images (e.g., "derekh").

Here is a workflow summary for the creating a morph:

1. Mark up corresponding points
2. Triangulate using Delaunay triangulation
For each morphed frame:
3. Interpolate geometry by weighted average of point locations
4. Warp each image to the interpolated geometry by finding affine warps between triangles and doing affine projections on points
5. Linearly blend warped images as a weighted average

Now, we describe these steps in more detail.

### Defining Correspondences

First, you will need to define pairs of corresponding points on the two images by hand. The more points, the better the morph, generally. I suggest using the standard correspondence format described under "Computing the Mean Face", as you will need to provide this for your own face anyway. The simplest way is to use the cpselect tool or to write your own little tool using ginput and plot commands (with hold on and hold off). I found the latter easier and provided a tool in the Google groups instructions.

Now, you need to provide a triangulation of these points that will be used for morphing. You can compute a triangulation any way you like, or even define it by hand. A Delaunay triangulation (see dalaunay and related functions) is a good choice since it does not produce overly skinny triangles. You can compute the Delaunay triangulation on either of the point sets (but not both -- the triangulation has to be the same throughout the morph!). But the best approach would probably be to compute the triangulation at midway shape (i.e. mean of the two point sets) to lessen the potential triangle deformations.

### The Morph

You need to write a function:

```morphed_im = morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac);
```

that produces a warp between im1 and im2 using point correspondences defined in im1_pts and im2_pts (which are both n-by-2 matrices of (x,y) locations) and the triangulation structure tri. Note that tri can be computed using delaunay from one set of points, as mentioned earlier. The parameters warp_frac and dissolve_frac control shape warping and cross-dissolve, respectively. In particular, images im1 and im2 are first warped into an intermediate shape configuration controlled by warp_frac, and then cross-dissolved according to dissolve_frac. For interpolation, both parameters lie in the range [0,1]. They are the only parameters that will vary from frame to frame in the animation. For your starting frame, they will both equal 0, and for your ending frame, they will both equal 1.

Given a new intermediate shape, the main task is implementing an affine warp for each triangle in the triangulation from the original images into this new shape. This will involve computing an affine transformation matrix A between two triangles:

```A = computeAffine(tri1_pts,tri2_pts)
```

A set of these transformation matrices will then need to be used to implement an inverse warp (as discussed in class) of all pixels. Functions tsearch and interp2 can come very handy here. But note that you are not allowed to use Matlab's build-in offerings for computing transformations, (e.g. imtransform, cp2tform, maketform, etc). Note, however, that tsearch assumes that your triangulation is always Delaunay. In our case, this might not always be true -- you may start with a Delaunay triangulation, but through the course of the morph it might produce skinny triangles and stop being Delaunay. David Martin from Boston College has kindly given access to his versions of tsearch that work on any triangulation: mytsearch.m. Note that this will return a "NaN" value for pixels that are not in any triangle. Two ways around this: (1) Set all such pixels to white (1 in each channel); (2) add points at the corners of the image to the original set of corresponding points (so that all pixels are in some triangle). I prefer the first option. Don't worry too much about using fancier interpolation methods. In my experience, just mapping to the nearest pixel seems to work ok.

### Computing the "Mean Face"

Several fun things are possible with our new morpher. For example, we can compute the mean face of CP students. This would involve: 1) computing the average shape, 2) warping all faces into that shape, and 3) averaging the colors together. However, this also requires a consistent labeling of all the faces. So, what we will do is ask everyone to label their own face, but do it in a consistent manner as shown in the following two images: points, point_labels. For each face image, you should put on your website a text file with coordinates of x,y positions, one per line (43 lines total). Please name the file with the same name as your picture, but with a ".txt" extension. You can read/write such files with the save -ascii and load -ascii commands. E.g.,
pts = [x(:) y(:)];
save -ascii ./derek.txt pts;

If everyone does this in a timely manner, then everyone can use all of the data to compute their mean image. Show the mean image that you got, as well as 1) your face warped into the average geometry, and 2) the average face warped into your geometry. To get you started, here are the faces and points from the last class.

### Bells and Whistles

• (10 points) Automatically align two faces by translation, rotation, and uniform scaling, using the sets of corresponding points. For example, you could scale so that the points have the same standard deviation, rotate so that the principal axis have the same orientation, and translate so that they have the same mean. Alternatively, you could solve a system of linear equations.
• (up to 20 points) Try morphing something other than one-to-one face morphs (e.g., dog to cat, a crappy car to a nice car, etc.). Getting the full 20 points requires multiple cool results. Aging morphs are also fun.
• (up to 20 points) Use the mean face to produce caricatures, masculitizing and feminizing transforms and any other fun things you can think of. For example, extrapolate from the mean face past your geometry to create an extreme version of you.
• (up to 20 points) Use the class data to compute a PCA basis for the face space. Try performing caricatures and other transformations in the new basis. Compare your results to doing this in normal basis. Are the results better?
• (up to 20 points) Implement a non-triangle based morphing algorithm. There are a lot out there, meaning this Bell/Whistle is somewhat underdefined. Take it on carefully.

### Deliverables

To turn in your assignment, place your index.html file and any supporting media in your project directory: netid.projects.cs.illinois.edu/cs498dh/proj4/index.html, where "netid" is your netid (e.g., dhoiem). Also, e-mail me the zipped code in "netid_proj4.zip" and include a link to the project page in the text. See project instructions for details. In the e-mail, tell me how many points you think you should get for the core project and any bells and whistles.

Use both words and images to show us what you've done. Please:

• Write out each frame of the morphing into a file named "morph_name1_name2_??.jpg" (XX = 00 to 60) in a single folder. Zip and e-mail me your folder. Display an even sampling of five to ten frames on your page. Also, show each image with a plot of corresponding points. Send me your morph.m code. Explain any difficulties or artifacts.
• Show the video morph of face A into face B. You may be able to create the video using Matlab's movie or avifile or using your written jpg files with PhotoLapse or a similar tool. The movie file could be linked as a file or embedded, e.g., using YouTube.
• Next, show the mean class face on your page and include relevant code in the e-mail. Also, include the morph with mean geometry/your texture and vice versa (as described earlier).
• Describe and bells and whistles under a separate heading and include relevant code.

### Scoring

The core assignment is worth 100 points, as follows:

• 70 points for correct implementation of the morph.m function and the resulting .jpg frames.
• 10 points for the face morphing video.
• 20 points for the mean face result.

You can also earn up to 90 extra points for the bells & whistles mentioned above.

### Tips and Tricks

• I'm not supplying starter code this time. You should write a script that reads in the two source images and, using a loop, creates and saves a morph for each frame of the video. You'll want to write functions for computing affine transformations and per-frame morphing.
• Check to see if "mytsearch" returned a nan value using "isnan". E.g., if the triangle indices of each pixel are stored in "ind_t", then "ind = isnan(ind_t); newim(repmat(ind, [1 1 3])) = 1;" would set pixels without triangle assignments to white.

### Acknowledgments

Thanks to Alexei Efros for creating this cool assignment and giving permission to use it.