Structure and Material Optimization Using a ToF Depth Camera – Time of Flight cameras – Dataset creation

View Github Repository Open presentation in a new window

johannesmik

See all presentation from johannesmik

Structure and Material Optimization Using a ToF Depth Camera – Time of Flight cameras – Dataset creation

0 0

thesis-presentation

On Github johannesmik / thesis-presentation

Master's thesis presentation

Structure and Material Optimization Using a ToF Depth Camera

Johannes Mikulasch

Advisor: Christian Kerl

August 21, 2015

Slides: github.com/johannesmik/thesis-presentation

Computer Vision Group

Department of Informatics, Technische Universität München

Introduction

Can we improve depth and material information?

Outline

Time of Flight cameras

Dataset creation

Material estimation

Optimization

Outlook, Conclusion

Time of Flight cameras

Kinect One

512 × 424 px depth image (real-time).

512 × 424 px infrared image.

1920 × 1080 px color image.

Image: ifixit.com teardown

Time of Flight cameras

Working principle 1: pulse modulation

Round trip time t of a light burst gives distance d=c⋅t2

But light is fast, thus precision is challenging

Δd=1 mm⟹Δt=6.ˉ6 ps.

… and no intensity image.

Time of Flight cameras

Working principle 2: continuous-wave modulation

Measure phase shift φ of an intensity modulated, continuous light signal

t=φ2fmodπ⟹d=cφ4fmodπ.

Time of Flight cameras

Infrared images

material color image ⇎ material infrared image

Time of Flight cameras

Artifacts on specular objects

Time of Flight cameras

Development

precision, cost,

devices become smaller

PMD Tec press release (June 2015): Project Tango

Dataset creation

Synthetic datasets of color, depth, infrared images

Dataset creation

Implementation

OpenGL3 and Python

Fragment shaders:diffuse, specular, normal, depth, data

Load extern .obj with textures

Camera free animatable

Material estimation

Material properties

k_d: diffuse reflectivity

k_s: specular reflectivity

n: hardness

Phong model

\begin{align} I =& I_{\text{in}} k_d \cos \theta + I_{\text{in}} k_s \left(\cos \theta'\right)^n \end{align}

Dataset creation

Scenes 1

A specular sphere.

Monkey Suzanne (blender): diffuse face, specular eyes.

infrared light, pointlight, coincident Kinect One

Dataset creation

Scenes 2

Three spheres, one diffuse, two specular.

Three Boxes, same materials as the spheres.

Desired challenge: can you see specular highlights

Dataset creation

Advantages

Ground truth known

Easy to create new scenes

Disadvantage

Noise and artifacts are not realistic

Material estimation

Cluster color and depth using k-means

Relate surface normals to infrared intensity

Fit the Phong model through it

\Longrightarrow Estimate material properties k_d, k_s, n for each cluster.

Material estimation

Clustering

k-means to find cluster centers \vec{\mu} of 6-dimensional data \vec{x}:

\begin{align} \vec{x} = (\lambda_p, \lambda_p, \lambda_d, \lambda_c)^T \begin{pmatrix}x \\ y \\ \mathcal{D} \\ \mathcal{C} \end{pmatrix} \end{align}

Material estimation

Clustering -- how to choose k?

\begin{equation} I_k = \sum_{i}^{k} \sum_{j, c(j) = i} \parallel \vec{x}_j - \mu_i \parallel_2^2. \end{equation}\begin{equation} \hat{k} = \underset{k}{\mathrm{argmin~}} k \qquad\text{subject to} \qquad \frac{I_{k+1}}{I_{k}} > \tau \end{equation}

Material estimation

Extracting

Normals calculated from depth image

Light direction / viewing direction is known

Obtain data pairs (\Theta_h, \mathcal{I}) for each pixel in each cluster

Material estimation

Fitting

A constrained optimization fits the Phong model through data

\Longrightarrow gives k_d, k_s and n value for each cluster

Material estimation

Results

Material estimation

Results 2

Optimization

Böhme et al. (2008) proposed this for a diffuse relighting model.

Extend this to specular relighting model.

Böhme et al. (2008), Shading constraint improves accuracy of time-of-flight measurements.

Optimization

Energy function

\begin{align*} \hat{\mathcal{D}}, \hat{\mathcal{M}} =& \underset{\mathcal{D}, \mathcal{M}}{\mathrm{argmax~}} p(\mathcal{D}, \mathcal{M} | \mathcal{D}^O, \mathcal{I}^O) \\ =& \underset{\mathcal{D}, \mathcal{M}}{\mathrm{argmax~}} p(\mathcal{D}^O | \mathcal{D}) p(\mathcal{I}^O | \mathcal{D}, \mathcal{M}) p(\mathcal{D}) p(\mathcal{M}). \end{align*}

boils down to optimizing this energy:

\begin{align*} \hat{\mathcal{D}}, \hat{\mathcal{M}} = \underset{\mathcal{D}, \mathcal{M}}{\mathrm{argmin}} & \overbrace{\sum_{j} \frac{(\mathcal{D}_i - \mathcal{D}_i^O)^2}{2 \sigma_{d}^2}}^{\text{depth data term}} + \overbrace{\sum_{j} \frac{(\mathcal{I}_i^O - I_{spec}(\mathcal{D}, \mathcal{M}))^2}{2 \sigma_{i}^2}}^{\text{shading constraint term}} \\ +& \underbrace{w_d \sum_{i}\sum_{j \in n(i)} \lVert \vec{n}_j - \vec{n}_i \rVert_2^2}_{\text{smooth depth term}} + \underbrace{w_m \sum_{i}\sum_{j \in n(i)} \lvert \mathcal{M}_j - \mathcal{M}_i \rvert}_{\text{smooth material term}}. \end{align*}

A non-linear conjugate gradient minimization was used.

Optimization

Implementation

Python and PyCUDA

Performance critical code in CUDAHuge speedup over initial implementation on the CPU/OpenGL

Energy derivative: central difference

Optimization

Results A

The first experiment

compares diffuse and specular optimizations.

Optimization

Results A -- Diffuse

Optimization

Results A -- Specular

Optimization

Results A

Comparing RMSE improvement

RMSE (ground, initial) \leftrightarrow RMSE (ground, optimized)

Improvement \mathcal{D} - 1.6 % I(\mathcal{D}, \mathcal{M}) 13.8% k_d - 31.4 %
Improvement \mathcal{D} - 5.08 % I(\mathcal{D}, \mathcal{M}) 18.1 k_d - 44.2 % k_s - 2.1 %

 

\Longrightarrow tune free parameters \sigma_d^2, \sigma_i^2, w_d, and w_m.

Optimization

Results B

Evaluate on real data.

Optimization

Results B

Optimization

Results B

Outlook

Segmentation

  • Use another segmentation method
  • Include infrared image for segmentation
  • Results probabilities

Optimization

  • Investigate in different setups and initializations
  • Split the smooth material parameter w_m into three different ones
  • Realistic model for light attenuation
  • Better examination possibilities for the optimization
  • Extension to more general lighting models (BRDF)

Conclusion

  • Proposed and implemented a method to estimate material properties from a single RGB-D-IR image
  • Implemented an optimization method to improve depth and material information
  • More future ivestigation necessary

 

 

Thanks for your attention!

1
Master's thesis presentation Structure and Material Optimization Using a ToF Depth Camera Johannes Mikulasch Advisor: Christian Kerl August 21, 2015 Slides: github.com/johannesmik/thesis-presentation Computer Vision Group Department of Informatics, Technische Universität München