Description
With the advent of Large Language Models (LLMs) such as GPT-4, new opportunities arise to bridge the gap between natural language descriptions and visual outputs. This project aims to explore how LLMs can be leveraged to modify the sentiment conveyed by volume renderings. The objective is to create a system where users can input a desired sentiment (e.g., serious, hopeful, ominous) through natural language, and the system will automatically adjust the rendering parameters to align with that sentiment.
Tasks
The tasks of this thesis are the following:
- Sentiment Mapping: Develop a methodology for mapping natural language sentiment descriptions to visual attributes of volume renderings.
- LLM Integration: Integrate LLMs with volume rendering software, enabling automatic modification of renderings based on the sentiment derived from user inputs.
- Evaluation: Evaluate the effectiveness of the system by comparing sentiment-modified renderings against user expectations and traditional, manually-tuned renderings.
Requirements
Interest and knowledge in visualization. Familiarity and understanding of volume rendering concepts in particular.
Good programming skills (Python for VTK or C# for Unity) and API integration (GPT-4 via OpenAI's API).
Familiarity with Machine Learning and NLP.
Creativity and enthusiasm.
Environment
Potentially, GPT-4 and VTK/Unity 3D, but to be determined depending on the background of the student.
References
[1] https://ieeexplore.ieee.org/abstract/document/10132377
[3] https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/iet-ipr.2019.1270
[4] https://doi.org/10.1109/CWIT.2017.7994829
[5] https://arxiv.org/pdf/2312.04494