Sketch to 3d-Model using Deep Learning and Differentiable Rendering

Information

• Publication Type: Master Thesis
• Workgroup(s)/Project(s):
  • Rendering and Modeling
• Date: March 2023
• Date (Start): November 2022
• Date (End): 3. March 2023
• Second Supervisor: Clemens Havas (Department Creative Technologies, University of Applied Sciences Salzburg, Salzburg, Austria)
• Diploma Examination: 3. May 2023
• Note: Thesis created in cooperation with Department Creative Technologies, University of Applied Sciences Salzburg, Salzburg, Austria
• Open Access: yes
• First Supervisor: Philipp Erler
• Keywords: deep learning, single-view model generation, topology, differentiable rendering, 2.5d intermediate representation

Abstract

Deriving 3D models from sketches has been a relevant research topic for years now, mainly due to its potential to simplify the realization of ideas and drafts for certain professions like artists, engineers and designers. With the raise of neural networks, new methods start to form utilizing the learning capability of those networks. One such method is the application of differentiable rendering, where a base mesh is deformed until the difference between the differentiable renderings and the target image is reasonably small. However, those target images, e.g., normal or depth maps, are not inherently available to the system, since the only input is a sketch. Therefore, intermediate representations predicted from the sketch need to be generated, that provide the required information.

The aim of this thesis is to provide a an end-to-end framework that reconstructs 3d models from 2d sketches using intermediate depth and normal representations and differentiable rendering. In addition to that, the topology of the sketch is determined and a respective base mesh is used for the mesh deformation. By comparing this setup to an established network as well as conducting an ablation study to reveal the effects the improvements have, the system and its limitations are evaluated. This will determine if the proposed optimisations facilitate input generalisability beyond class/object restrictions and improve mesh quality.

Additional Files and Images

Weblinks

• Source Code
  Repository on Github

BibTeX

@mastersthesis{hofer_kerstin-2023-sketch2model,
  title =      "Sketch to 3d-Model using Deep Learning and Differentiable
               Rendering",
  author =     "Kerstin Hofer (Department Creative Technologies, University
               of Applied Sciences Salzburg, Salzburg, Austria)",
  year =       "2023",
  abstract =   "Deriving 3D models from sketches has been a relevant
               research topic for years now, mainly due to its potential to
               simplify the realization of ideas and drafts for certain
               professions like artists, engineers and designers. With the
               raise of neural networks, new methods start to form
               utilizing the learning capability of those networks. One
               such method is the application of differentiable rendering,
               where a base mesh is deformed until the difference between
               the differentiable renderings and the target image is
               reasonably small. However, those target images, e.g., normal
               or depth maps, are not inherently available to the system,
               since the only input is a sketch. Therefore, intermediate
               representations predicted from the sketch need to be
               generated, that provide the required information.  The aim
               of this thesis is to provide a an end-to-end framework that
               reconstructs 3d models from 2d sketches using intermediate
               depth and normal representations and differentiable
               rendering. In addition to that, the topology of the sketch
               is determined and a respective base mesh is used for the
               mesh deformation. By comparing this setup to an established
               network as well as conducting an ablation study to reveal
               the effects the improvements have, the system and its
               limitations are evaluated. This will determine if the
               proposed optimisations facilitate input generalisability
               beyond class/object restrictions and improve mesh quality.",
  month =      mar,
  note =       "Thesis created in cooperation with Department Creative
               Technologies, University of Applied Sciences Salzburg,
               Salzburg, Austria",
  address =    "Favoritenstrasse 9-11/E193-02, A-1040 Vienna, Austria",
  school =     "Research Unit of Computer Graphics, Institute of Visual
               Computing and Human-Centered Technology, Faculty of
               Informatics, TU Wien",
  keywords =   "deep learning, single-view model generation, topology,
               differentiable rendering, 2.5d intermediate representation",
  URL =        "https://www.cg.tuwien.ac.at/research/publications/2023/hofer_kerstin-2023-sketch2model/",
}