Maximilian Staats
Alpine Terrain Relighting
[
thesis]
Information
- Publication Type: Master Thesis
- Workgroup(s)/Project(s): not specified
- Date: 2024
- Second Supervisor: Pedro Hermosilla-Casajus
- Open Access: yes
- First Supervisor: Manuela Waldner
- Pages: 70
- Keywords: Shadow-Removal, Shadow-Detection, Orthophotos, Digital Elevation Models, Deep-Learning, Generative Adversarial Models, Computer Vision
Abstract
Aerial orthophotos together with digital elevation models (DEMs) allow the rendering of 3D representations of the earth, including alpine terrain. These virtual landscapes provide the opportunity to simulate light conditions at different times of the day, aiding in trip planning. However, orthophotos used as texture often contain large shadows stemming from cliffs and rocks, which significantly impact the visual quality of relighted textures. The necessary single-image shadow-removal process presents a crucial problem for the computer vision domain, which also functions as a prerequisite for many other tasks like segmentation and classification. Many promising approaches have already been developed, but unlike previous methods, this study tries to capitalize on the availability of DEMs to enhance the shadow removal process. Shadows in orthophotos are inherently linked to the underlying geospatial topology, and DEMs provide a valuable source of information for mitigating their impact. Therefore, this thesis explores the integration of DEMs into a state-of-the-art deep learning pipeline. DEMs are examined for their role in generating training sets and as supplementary input for a multi-modal network. Notably, 3D geometry derived from DEMs complemented by ray-tracing is used to generate artificial shadows with realistic shapes. Subsequently, an experiment is conducted with the created dataset to empirically test if additional elevation data is beneficial for the performance of the models. Additionally, the model’s ability to generalize from artificial to real shadows was probed. The experiment on virtual shadows showed that providing additional elevation data to the shadow-removal network does yield significantly better results with a medium to large effect size. Initially, all trained models failed to generalize to real shadow data. Downsizing the dataset to a lower level of detail mitigated this problem. Together with an analysis of the output of each network layer, it was concluded that the reason for the subpar real data performance are remaining small-scale shadows in the train set. A visual analysis of the improved models showed noticeable improvements with the generated realistic shadow shapes compared to random ones. Moreover, the utility of additional elevation data as input for the models was demonstrated.Additional Files and Images
Weblinks
BibTeX
@mastersthesis{staats-2024-atr,
title = "Alpine Terrain Relighting",
author = "Maximilian Staats",
year = "2024",
abstract = "Aerial orthophotos together with digital elevation models
(DEMs) allow the rendering of 3D representations of the
earth, including alpine terrain. These virtual landscapes
provide the opportunity to simulate light conditions at
different times of the day, aiding in trip planning.
However, orthophotos used as texture often contain large
shadows stemming from cliffs and rocks, which significantly
impact the visual quality of relighted textures. The
necessary single-image shadow-removal process presents a
crucial problem for the computer vision domain, which also
functions as a prerequisite for many other tasks like
segmentation and classification. Many promising approaches
have already been developed, but unlike previous methods,
this study tries to capitalize on the availability of DEMs
to enhance the shadow removal process. Shadows in
orthophotos are inherently linked to the underlying
geospatial topology, and DEMs provide a valuable source of
information for mitigating their impact. Therefore, this
thesis explores the integration of DEMs into a
state-of-the-art deep learning pipeline. DEMs are examined
for their role in generating training sets and as
supplementary input for a multi-modal network. Notably, 3D
geometry derived from DEMs complemented by ray-tracing is
used to generate artificial shadows with realistic shapes.
Subsequently, an experiment is conducted with the created
dataset to empirically test if additional elevation data is
beneficial for the performance of the models. Additionally,
the model’s ability to generalize from artificial to real
shadows was probed. The experiment on virtual shadows showed
that providing additional elevation data to the
shadow-removal network does yield significantly better
results with a medium to large effect size. Initially, all
trained models failed to generalize to real shadow data.
Downsizing the dataset to a lower level of detail mitigated
this problem. Together with an analysis of the output of
each network layer, it was concluded that the reason for the
subpar real data performance are remaining small-scale
shadows in the train set. A visual analysis of the improved
models showed noticeable improvements with the generated
realistic shadow shapes compared to random ones. Moreover,
the utility of additional elevation data as input for the
models was demonstrated.",
pages = "70",
address = "Favoritenstrasse 9-11/E193-02, A-1040 Vienna, Austria",
school = "Research Unit of Computer Graphics, Institute of Visual
Computing and Human-Centered Technology, Faculty of
Informatics, TU Wien",
keywords = "Shadow-Removal, Shadow-Detection, Orthophotos, Digital
Elevation Models, Deep-Learning, Generative Adversarial
Models, Computer Vision",
URL = "https://www.cg.tuwien.ac.at/research/publications/2024/staats-2024-atr/",
}