Improvements for the visualizations in Sonic Analyst

Description

Sonic Analyst is a tool for building combinations of audio feature extractors, so called Sonic Characters. Its primary use is to playfully observe feature extractors in real time. By doing so, the user should be able to tweak the feature extractor definition to better suit his needs.

The application is based on a high-performance audio framework (Sonic.framework) that establishes audio tracks and interfaces with the audio hardware on a system level (using Audio Units). Sonic Analyst is querying this particular framework to fetch feature data that is going to be visualized by the application graphical framework.

The concept for the application and parts of the audio framework have been written within the context of a bachelor thesis. This lab course should further improve this and also build the foundation for interesting visualizations.

This page was written for the InfoVis lab course 2010. For up-to-date information about Sonic Analyst and a complete description of the concept, please visit http://www.sonicanalyst.com.

Implementation report

The following features have been implemented during this course:

• The Sonic framework has been improved and a file-playing feature has been added (before that only live input streaming was possible).
• Stability of adding/removing elements in the Sonic Analyst has been improved.
• The visualization of generic feature data (dim x time) has been abstracted in its own interfaces which build the foundation for a SonicCharacter visualization plugin system.
• The threading model has been greatly improved. The visualization system is driven by its own high-priority display linked thread. Input handlers and other main-thread based tasked are serialized using Grand Central Dispatch queues and synchronized with the render-thread by using fast dispatch-semaphores.
• A 3D waterfall visualization was implemented and chosen as the default (as most generic) visualization type. This implementation uses an efficient ring-buffer mechanism backed by OpenGL Pixel Buffer Object which are used for feature data streaming.

For more information look at the doxygen documentation or the sources directly.

Instructions to install

To run the application you have to copy the binaries to the following locations:

• Unzip Sonic.framework.zip and copy Sonic.framework to /Library/Frameworks
• Unzip the SncZeroCrossingRate.component.zip and SncRMSAmplitude.component.zip and copy the components to /Library/Audio/Plug-Ins/Components.
• Finally, launch SonicAnalyst.app.

Future releases will have all shipped components properly bundled in the SonicAnalyst.app folder structure.

Instructions to test

Please be aware that this application is not in its complete state yet. It might crash badly if you push the wrong buttons. A more stable release will be available here. The following instructions for this build have been tested under Mac OS 10.6.3 with a late 2009 MacBook Pro and a NVIDIA 9600GT:

1. Under system preferences, select any aggregate audio device as your default audio device. If you are using the MacBook's onboard Audio Input/Output you might have to manually create them using the Audio Midi Editor.
2. Launch the application
3. Create a track in the top-bar under Create->Track
4. Create a sonic character in the top-bar under Create->SonicCharacter
5. Create a SNC Feature unit using the + button in the previously created track (e.g. SncRMSAmplitude).
6. Drag and drop the red SNC feature unit from the track down to the root element of the sonic character.
7. The 3D waterfall visualization should play in the render window. Use alt+mouse drag to tumble view, right mouse drag to zoom.

Sources + Documentation (Graphics package)

Follow this link to get the source files for the Graphics package and this one for the doxygen documentation. Please note that these sources are only a subproject of the much bigger complete SonicAnalyst-application. You will not be able to compile without the overall package. A complete source package of the audio framework, the Sonic Analyst app and its Graphic framework might be published in the future at http://www.sonicanalyst.com.