Partial Light Field Tomographic Reconstruction From a Fixed-Camera Focal Stack
 

Melon
"Partial light field tomographic reconstruction from a fixed-camera focal stack", IEEE trans. on Image Processing, submitted, 2015.(pdf)
contact: A. Mousnier, E. Vural, C. Guillemot

The code is available for download here.

Abstract

This paper describes a novel approach to partially reconstruct high-resolution 4D light fields from a stack of differently focused photographs taken with a fixed camera. First, a focus map is calculated from this stack using a simple approach combining gradient detection and region expansion with graphcut. Then, this focus map is converted into a depth map thanks to the calibration of the camera. We proceed after this with the tomographic reconstruction of the epipolar images by back-projecting the focused regions of the scene only. We call it masked back-projection. The angles of back-projection are calculated from the depth map. Thanks to the high angular resolution we achieve, we are able to render puzzling perspective shifts although the original photographs were taken from a single fixed camera at a fixed position. To the best of our knowledge, our method is the first one to reconstruct a light field by using a focalstack captured with an ordinary camera at a fixed viewpoint.

Depth map

We designed a two-steps algorithm for focus map estimation: first, we detect strong gradients on every image of the focus stack, then we use a graph-cut algorithm to expand the zones where strong gradients have been detected. Thanks to the prior calibration of the camera, we can then transform this focus map into a depth map.

process

Masked back-projection

A conventional photograph is a projection of the 4D light field. Thus, a focal stack can be viewed as a collection of projections of the 4D light field along different directions. We propose to retrieve the epipolar images from their projections with an algorithm that we name masked back-projection. The conventional backprojection method is based on the Radon transform. The original image intensity is recovered from its projections by first filtering the projections, and then back-projecting them onto the plane of the image to be reconstructed.


In order to achieve an accurate reconstruction with the traditional back-projection algorithm, a high number of projections is needed, whereas the number of projections is limited by the number of images in the focal stack in our problem. Nevertheless, the epipolar image to be reconstructed has a special structure, which can be approximated as a set of overlapping lines with different slopes.

We thus propose a modified version of the back-projection algorithm adapted for our problem, which exploits this prior information on the epipolar image.  We first back-project entirely the background. Then, we backproject only the in-focus parts of the projections, from the second farthest to the foreground, in the order of decreasing depth, as depicted in the figure below.


bp

An important difference between our back-projection procedure and the traditional one is that we overwrite the reconstructed epipolar image throughout the back-projections, while the classical method sums them up. The linear patterns in the epipolar image correspond to points on objects in a real 3D scene; therefore, processing the objects in the order of decreasing depth and overwriting the previous pixels throughout the back-projection is coherent with the physicalfact that distant objects are occluded by the closer ones.

Results

Depth maps

Perspective shifts with ground truth depth maps

perspective shift cards
To play the video click on the image

Perspective shifts with computed depth map

perspective shift rifle
To play the video click on the image
perspective shift s10
To play the video click on the image
perspective shift s20
To play the video click on the image
perspective shift rifle
To play the video click on the image

Refocusing

refocusing rifle
To play the video click on the image
perspective shift rifle
To play the video click on the image
perspective shift rifle
To play the video click on the image
perspective shift rifle
To play the video click on the image

Extended focus

extendedFocus_flower