## Lab Assignment

The raw frames of the video sequences you will work with in this lab are available in the following directory:

video_database

Download the files into a directory.## A. Display a Video Sequence

Download the Matlab skeleton code for part a: lab6a.m and read it carefully. A significant part of the work has been already done for you. Here we will use MATLAB to generate a video sequence from two images then display the video. The video is displayed at a frame rate which you can specify.

1. We will read in the files "tank_01.raw" and "tank_10.raw" using the MATLAB commands described on the "Coding" webpage of this lab.(Note that both of these images are of size 486x486). Let X and Y respectively denote the Matlab arrays holding "tank_01.raw" and "tank_10.raw".

2. We shall use linear intensity interpolation in time to generate 19 intermediate frames where X is the first frame and Y is the last frame. The 19 intermediate frames will be used to ``fade'' from X to Y. In all there will be 21 frames including the first frame (X) and the last frame (Y). After creating the 21 images, we shall assemble them in an "image deck" for display using the MATLAB commands "getframe" and "movie". Typehelp getframeandhelp moviefor help. Try different frame rates when displaying your video sequence.

Write all your code for this part of the lab to the filelab6a.m.## B. Decimation and Interpolation of Video Frames

You have seen ways to decimate and interpolate images in the spatial domain; this extends very naturally to decimation and interpolation in time (between image frames).

We will apply interpolation and decimation in time for the following video sequence:video_database/fn0ijwhere ij = 00,01,02,...,19,20. This is the famous "football" video sequence which is used a lot in digital video processing. The skeleton code for this lab is available here: lab6b.m. Many of the following tasks have already been done for you in the skeleton code:

First store all of these frames in the suitable format in MATLAB to visualize the video sequence.

Subsample this video sequence by 2, then predict each missing frame by using linear interpolation in the time domain. In other words, you are going to create a new sequence, let's call it "football2_0ij" where ij = 00,01,...,19,20. This sequence will be created as follows:football2_000 = fn000Find the average MSE per pixel per frame between this synthesized sequence and the original football sequence. In order to do this simply find the MSE per pixel for each frame, then average these values for all the frames. Recall that for each frame, MSE per pixel is given by

football2_001 = (fn000 + fn002)/2

football2_002 = fn002

football2_003 = (fn002 + fn004)/2

.

.

football2_019 = (fn018 + fn020)/2

football2_020 = fn020MSE = (1/(m*n))\sum{m} \sum{n} [ synthesized_frame(m,n) - original_frame(m,n) ]^2where m = 352 and n = 240 for this case, and where \sum{m} and \sum{n} means summing in the m and n direction respectively. Visualize this new sequence in MATLAB and compare it with the original sequence.

Subsample the original fn0ij sequence by 4, then predict missing frames by using linear interpolation in time domain. This time, you expect to see more discrepancy between the synthesized sequence and the original sequence. You can call this new synthesized sequence "football4_0ij" where ij = 00,01,...,19,20 . Again find the average MSE per pixel per frame between this synthesized sequence and the original football sequence. Visualize it in MATLAB. Compare it with the original sequence and the previously synthesized sequence, which was formed by subsampling by 2 and then performing linear interpolation.

Write all the MATLAB code for part B of this lab to the filelab6b.m.## C. Pyramid Decomposition in Time

Pyramid decomposition is useful for video compression. We'll take a look at how it can be used for lossless compression. We will do this part of the lab for the image sequencevideo_database/tank_ij.rawwhere ij = 01,02,..,09. So you will access 9 frames for this part. These frames are 486x486 in size. Download the skeleton code for pyramid decomposition: lab6c.m. Pyramid decomposition in time works as follows:

1. Discard every other frame (downsample by 2), then reinterpolate the missing frames using linear interpolation. Calculate the difference between each discarded frame and its corresponding reinterpolated frame. Keep these four difference frames.

2. Take the downsampled set (five original frames) and downsample by a factor of two to get three images. Again, reinterpolate using linear interpolation to recover the two discarded frames, and calculate the difference frames.

Here we have three original frames and six difference frames. This is the compressed video data. Now we reconstruct the original video:

3. Start with the three original frames, interpolate (linear interpolation again) and add the difference frames to the reinterpolated frames.

5. Interpolate (linear interpolation) once more and add the four corresponding difference frames to complete the reconstruction. Note that this is a perfect (zero error) reconstruction.

We are transmitting 3 original images and 6 difference images. In general we can operate at a lower bit rate since difference frames typically have low intensities, and hence contain less information.

Write all the MATLAB code for part C of this lab to the filelab6c.m.