Tutorial: Render Synthetic Objects using Blender 2.8

This page explains and illustrates how to render synthetic objects into photographs for project 4 using Blender 2.8. You can install from here. This page explains how to build your scene from scratch. You can load the provided sample ibl.blend file into Blender as a reference, but please note that the equirectangular image is likely in a different rotation than the one you generated, so there are some differences in camera parameters.

Key concepts

Your 3D scene has the following components:

Camera: specifies the viewing position and direction for rendering. In my scene, I set the position as (X=6m, Y=0m, Z=0m) and the orientation as (X=90 deg, Y=0 deg, Z=90 deg). The easiest way to tell if your camera orientation is correct is to insert a reflective sphere into the scene with no other objects, render it with your lighting environment, and check that it looks like your original picture of the mirrored sphere.
Background image: This is for reference to help you set up your scene in a way that will make sense with your background image. You can drag an image into the viewport, or follow the method shown in the step-by-step below.
Objects: These are what will be rendered. Usually, the minimum set of objects is a plane for a supporting surface (needed so that objects can cast shadows and reflections onto something) and some objects on top, e.g. sphere, cube, bunny, or other objects of your choosing. Objects have a position, orientation, and scale, which can be set using the move/rotate tools on the left, or the object properties transform settings on the right. If you want the objects to be on your plane, it's usually helpful to raise and lower them so that they pass through the plane and then adjust so they are just at the threshold of passing through. Objects also have a material, minimally the reflectivity and color, which you can set using the material properties, which in my interface looks like a reddish beach ball, 2nd to last button in the properties panel on the lower right. To make a mirror, use "Glossy BSDF" as the surface and set Roughness to 0. Diffuse and Glass are other common ones to try.
Environment Map: This is what is used to light your scene. There should be no other light sources in your scene collection. Set it using "World Properties" (globe icon in lower-right panel) and selecting Environment Texture as Color and loading your equirectangular HDR image. Strength can be adjusted to produce a pleasing result.

To generate the image, you need to render. First, you set the Render Engine to Cycles using the Render Properties (icon looks kind of like a clipboard or tv). Set the Sampling Render number to 1000 (lower for speed, higher for quality). For your final scene composite, you'll render once with all the objects, once with only the supporting plane, and one more time to generate the object masks. Make sure to save your complete scene before modifying your scene for these different renderings.

Step by step instructions

First clear out initial scene including sample mesh and lighting, and setup camera position

Next, go into perspective camera mode, and setup viewport dimension. You want the same dimension as your background image for this.

We then want to load background image onto this viewport. Click on 'g' to move background image around, and 's' to resize it.

Then, we want to add local scene. That is, add simple geometry (usually planes suffice) to recreate the geometry in the scene near where you'd like to insert objects. For best results, this should be close to where you placed the spherical mirror. Feel free to use the sample scene provided and move the vertices of the plane to match the surface you'd like to recreate (ignore the inserted bunny/teapot/etc for now). Once you're happy with the placement, add materials to the local scene: select a piece of local scene geometry, go to Properties->Materials, add a Diffuse BSDF material, and change the "Color" to roughly match the color from the photograph.

Finally, insert synthetic objects into the scene. There are basic models in Blender, and you can find others online: Turbosquid, Google 3D Warehouse, DModelz, etc. Add interesting materials to your inserted objects as well. Once finished, your scene should now look something like the right image below.

Then, add your equirectangular HDR image that you generated in your Jupyter Notebook: write_hdr_image(eq_image, 'equirectangular.hdr').
In the World tab on property panel, make sure Surface="Background" and Color="Environment Texture". Locate your saved HDR image in the filename field below "Environment Texture".

Once finished, your scene should now look something like the right image below.

Rendering

We can now render the scene to see a preview of what the inserted objects will look like. Make sure "Cycles" is the Render Engine in the Render Properties panel, and Render the scene (F12). Under Sampling in the Render Properties, you can set the Render value higher to improve quality or lower for a faster result. Recall that the solution of the HDR image is up to an known scale factor, so you may need to adjust the brightness strength. You can adjust the overall brightness by setting World Properties --> Surface --> Strength. You also may need to adjust your scene, e.g. to pick different materials or make sure that shadows cast by your objects don't go beyond the supporting plane. Once satisfied with the result, save the rendered image to disk.

This is a demo of how the Cycles render will work:

My rendered scene is down below; as you can tell, this is not quite the final result. To seamlessly insert the objects, we need to follow the compositing procedure outlined by Debevec (Section 6 of the paper). This requires rendering the scene twice (both with and without the inserted objects), and creating an inserted object mask.

Next, we'll render the "empty" scene (without inserted objects). Create a copy of your Blender scene and name it something like ibl-empty.blend. Open up the copy, and delete all of the inserted objects (but keep the local scene geometry such as support plane). Render this scene and save the result.

Finally, we need to create an object mask. The mask should be 0 where no inserted objects exist, and greater than 0 otherwise. First, create another duplicate of your scene and open it up (e.g. ibl-mask.blend). We can create the mask quickly using Blender by manipulating object materials and rendering properties:

In the top panel, Choose 'Eevee Render'
Select an object
Go to the materials panel
Remove the Material
Click "New" to add a new material
In the new material properties, under "Emissive"
Repeat for all inserted objects
In World panel, remove equirectangular lighting that was added previously
Render the scene and save your mask as a PNG (or some lossless format)
The PNG will contain a masked image where pixel intensities over 0.5 indicates True for object mask

After these steps, you should have the something akin to the following three rendered images:

Compositing

To finish the insertion, we will use the above rendered images to perform "differential render" compositing. This can be done using a simple pixel-wise equation. Let R be the rendered image with objects, E be the rendered image without objects, M be the object mask, and I be the background image. The final composite is computed with:

composite = M*R + (1-M)*I + (1-M)*(R-E)*c

The first two terms effectively pastes the inserted objects into the background image, and the third term adds the lighting effects of the inserted objects (shadows, caustics, interreflected light, etc), modulated by c. Set c=1 initially, but try different values to get darker or lighter shadows/interreflections. The final compositing result I achieved using my image-based light is at the top of the page.

Some other tips on using Blender

Save your Blender file regularly, and always before closing (on some operating systems, Blender will close without prompting to save).
To move more than one object at once, select multiple objects using shift. Pressing 'a' deselects all objects/vertices.
You can edit vertices directly in "Edit mode" (tab toggles between Object and Edit modes).