Super-rays for Efficient Light Field Processing

Matthieu Hog , Neus Sabater and Christine Guillemot

Abstract

Light field acquisition devices allow capturing scenes with unmatched post-processing possibilities. However, the huge amount of high dimensional data poses challenging problems to light field processing in interactive time. In order to enable light field processing with a tractable complexity, in this paper, we address the problem of light field over-segmentation. We introduce the concept of super-ray, which is a grouping of rays within and across views, as a key component of a light field processing pipeline. The proposed approach is simple, fast, accurate, easily parallelisable, and does not need a dense depth estimation. We demonstrate experimentally the efficiency of the proposed approach on real and synthetic datasets, for sparsely and densely sampled light fields. As super-rays capture a coarse scene geometry information, we also present how they can be used for real time light field segmentation and correcting refocusing angular aliasing

Over-Segmentation Results

We show the original light field a long with colour coded super-rays assignments plus visualisation were each ray has the single segment average color and the estimated coarse depth.

Dense Synthetic Dataset

Input Light (top) Super-rays (top)
Average Color (btm) Propagated Depth (btm)

Budha [1]

Stilllife 2 [1]

Sparse Synthetic Dataset

Input Light (top) Super-rays (top)
Average Color (btm) Propagated Depth (btm)

Scene 4 [2]

Tricycle

Dense Illum Dataset

Input Light (top) Super-rays (top)
Average Color (btm) Propagated Depth (btm)
Flowers IMG_1306_eslf [3]

Sparse Real Dataset

Input Light (top) Super-rays (top)
Average Color (btm) Propagated Depth (btm)
Tsukuba [4] Biergarten [5]

Comparison with matched superpixels

Here super pixels are computed separately on each view and merged later (using the ground truth depth).

Input Light field Super-rays Superpixels
Budha [1] Scene 4 [2] Tsukuba [4]

Applications Results

Real Time Segmentation

Applying graph cut using each super-ray as graph node greatly increases the speed of the optimisation (here we go from 3s to 1ms).
Input Light field (top) Super-rays(top)
Our Segmentation (btm) Segmentation[2] (btm)

Correcting Angular Aliasing

We create intermediate views using the coarse depth information of each super-ray (green pixels are missing values).

Interpolated view Real view View Position
The generated views are not convincing enough to be used as is but can be used to correct angular aliasing during refocusing of sparse light fields.
Refocus Raw Refocus Corrected

Download

Citation

@ARTICLE{
hog17super,
author={Matthieu Hog, Neus Sabater, Christine Guillemot},
journal={IEEE Journal of Selected Topics in Signal Processing},
title={Super-rays for Efficient Light Field Processing},
year={2017},
volume={PP},
number={99},
pages={1-1},
doi={10.1109/JSTSP.2017.2738619},
ISSN={1932-4553}
}

References

[1] Globally consistent multi-label assignment on the ray space of 4d light fields, Sven Wanner and Bastian Goldluecke (CVPR 2013)
[2] Light Field Segmentation Using a Ray-Based Graph Structure , Matthieu Hog, Neus Sabater, and Christine Guillemot (ECCV 2016)
[3] Learning-based view synthesis for light field cameras , Kalantari, Nima Khademi, Ting-Chun Wang, and Ravi Ramamoorthi. (TOG 2016)
[4] A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, Daniel Scharstein and Richard Szeliski (IJCV 2002)
[5] Efficient Multi-image Correspondences for On-line Light Field Video Processing, Łukasz Dąbała, Matthias Ziegler, Piotr Didyk, Frederik Zilly, Joachim Keinert, Karol Myszkowski, Hans-Peter Seidel, Przemysław Rokita and Tobias Ritschel (Computer Graphics Forum 2016)