Description
Creating textures for meshes is costly and tedious, and automatizing this process can be very helpful. In recent years, neural network-based solutions have gained significant traction. They usually repurpose a 2D image synthesis method and apply it on the surface of a mesh, either by utilizing a differentiable renderer or by slicing a 3D volume. However, it is also possible to do this directly on a curved surface by modifying a 2D network, so instead of using 2D convolutional layers, it can use surface convolutional layers with the same weights [1]. This approach exploits the fact, that curved surfaces tend to be locally flat, however, depending on the mesh, NN's architecture, etc., this assumption may be broken or the NN can lose track of the local context, which may prevent applying this approach to more complex NN-based approaches for synthesizing images.
The objective is to use the surface-based approach and evaluate how suitable it is for other methods that work in 2D.
[1] Kovács, Áron Samuel, Pedro Hermosilla, and Renata G. Raidou. "Surface‐aware Mesh Texture Synthesis with Pre‐trained 2D CNNs." Computer Graphics Forum. 2024.
Tasks
In this project, you will be tasked with researching the state of the art of NN-based image/texture synthesis and evaluating how well they behave when applied directly on a curved surface, possibly making adjustments to account for the differences between flat and curved images. You will also need to conduct a study to show the validity of this approach.
Requirements
- Knowledge of the English language (source code comments and final report should be in English)
- Knowledge of Python (+ PyTorch or TensorFlow), Rust, and Cuda
Environment
The project should be implemented as a standalone application for Windows and Linux.