Instructions

 

You can use following buttons to interact with the MusicVis:

  • Left Mouse Button: Rotate the object
  • Right Mouse Button: Translate the object
  • Middle Mouse Button: Zoom in/out
  • Gain Highy Frequency Slider: Changes the gain factor of the mapping function, such that the higher frequencies are gained a little bit.
  • Play Button: Starts the music playback
  • Stop Button: Stops the music playback
  • Drop *.mp3 Files into the window and they will be loaded in background. If they are ready the music playback stops and you have to press play.

Short Descripton

 

The goal of our Visualization exercise was to build a Music Visualization application that deforms a 3D-Mesh according to the music. This Web-Application relies primarily on WebGL and other new technologies and frameworks. It was developed with Google Chrome, which supports all the libraries and is not tested on other browsers. (see Technical Details).

What’s the benefit of this app?

The whole application is more or less just for fun. It generates interesting images according to a piece of music, similar to music visualization in popular music software.

Why are we doing this?

This application is written during an exercise for the Visualization class at the Vienna University of Technology. More specific the course is held by the CG-Institute and you can find the course page here.

Technical Details

 

Used Frameworks

The rendering application is written in Javascript and HTML5. And uses the following APIs and Frameworks:

  • WebGL - Used to Render the stuff
  • Web Audio API - Play and process the Music
  • Three.js - Lightweight 3D Library (Makes the life a lot easier when using WebGL)
  • jQuery - Javascirpt Framework to simplify the HTML document traversing and much more.
The Web Audio API as we use it can only play songs within the Webbrowser Chrome, because other Browser implement the API differently or does not support this API at all. As well as there are many limitations when WebGL is used to render the stuff. By now many nice Features of new OpenGL versions are missing in WebGL and special "hacks" were needed to get the application running.

Under the hood

To deform a 3D-mesh like MusicVis does it, some math is needed. The whole idea is based on Spectral Geometry processing. It was not our idea how to do all the stuff and the original paper describing the algorithms as we implemented them is "Stereo music visualization trhough manifold harmonics" by T. Lewiner et al. Another good reading is "Spectral Geometry processing with Manifold Harmonics" by B. Vallet and B. Lévy on which the first paper is based more or less.

Now we are getting to the interesting part. How the whole thing works. This can be described by following steps:

  • Preprocessing: Calculate the so called Manifold Harmonics Basis (MHB) of the mesh (we used MATLAB to calculate this part)
  • Rendering-Preprocessing: Calculate the Manifold harmonics Transform (MHT) on the original mesh to get the transformed vertices (this transforms the vertices from the geometric space to the frequency space). This is also a preprocessing step, but now done in the real application using JavaScript.
  • Audio-Processing: Get the frequencies of the played music. These frequencies are than mapped to the frequencies of the mesh (we use a linear mapping by now).
  • Rendering: Calculate the inverse MHT to get back into the geometric space, but before this is done the Audio-Frequencies are multiplied to the transformed vertices. This multiplication results in the transformation of the vertices if they are transformed back to the geometric space. This whole processing step is written as Vertex-Shader to get a realtime performance of the application.
I think this short description is not enough to really understand how the whole system works, but please feel free to read the papers and/or our sourcecode.

References

 

  • Project Page of "Spectral Geometry Processing through Manifold Harmonic" by Bruno Vallet and Bruno Lévy with Demo and Paper
  • Homepage with the paper "Stereo music visualization through manifold harmonics" by T. Lewiner et al. with some additional information.
  • Discrete Differential Geometry Forum page with nice tutorials and papers (helped to understand the Discrete Exterior Calculus (DEC)).