DeepLight: Light Source Estimation for Augmented Reality using Deep Learning

Information

• Publication Type: Journal Paper with Conference Talk
• Workgroup(s)/Project(s):
  • Virtual Reality Group
• Date: June 2019
• Journal: The Visual Computer
• Volume: 35
• Open Access: yes
• Number: 6
• Lecturer: Hannes Kaufmann
• Event: Computer Graphics International 2019
• DOI: 10.1007/s00371-019-01666-x
• Call for Papers: Call for Paper
• Conference date: 2019
• Pages: 873 – 883
• Keywords: Light source estimation, Augmented reality, Photometric registration, Deep learning

Abstract

This paper presents a novel method for illumination estimation from RGB-D images. The main focus of the proposed method is to enhance visual coherence in augmented reality applications by providing accurate and temporally coherent estimates of real illumination. For this purpose, we designed and trained a deep neural network which calculates a dominant light direction from a single RGB-D image. Additionally, we propose a novel method for real-time outlier detection to achieve temporally coherent estimates. Our method for light source estimation in augmented reality was evaluated on the set of real scenes. Our results demonstrate that the neural network can successfully estimate light sources even in scenes which were not seen by the network during training. Moreover, we compared our results with illumination estimates calculated by the state-of-the-art method for illumination estimation. Finally, we demonstrate the applicability of our method on numerous augmented reality scenes.

Additional Files and Images

Weblinks

• https://link.springer.com/article/10.1007/s00371-019-01666-x
• Entry in the publication database of TU-Wien
• DOI: 10.1007/s00371-019-01666-x

BibTeX

@article{kan-2019-dli,
  title =      "DeepLight: Light Source Estimation for Augmented Reality
               using Deep Learning",
  author =     "Peter K\'{a}n and Hannes Kaufmann",
  year =       "2019",
  abstract =   "This paper presents a novel method for illumination
               estimation from RGB-D images. The main focus of the proposed
               method is to enhance visual coherence in augmented reality
               applications by providing accurate and temporally coherent
               estimates of real illumination. For this purpose, we
               designed and trained a deep neural network which calculates
               a dominant light direction from a single RGB-D image.
               Additionally, we propose a novel method for real-time
               outlier detection to achieve temporally coherent estimates.
               Our method for light source estimation in augmented reality
               was evaluated on the set of real scenes. Our results
               demonstrate that the neural network can successfully
               estimate light sources even in scenes which were not seen by
               the network during training. Moreover, we compared our
               results with illumination estimates calculated by the
               state-of-the-art method for illumination estimation.
               Finally, we demonstrate the applicability of our method on
               numerous augmented reality scenes.",
  month =      jun,
  journal =    "The Visual Computer",
  volume =     "35",
  number =     "6",
  doi =        "10.1007/s00371-019-01666-x",
  pages =      "873--883",
  keywords =   "Light source estimation, Augmented reality, Photometric
               registration, Deep learning",
  URL =        "https://www.cg.tuwien.ac.at/research/publications/2019/kan-2019-dli/",
}