Synesthesia project


Exhibited at Siggraph2016, Tokyo Game Show 2016, Asia Haptics 2016, Dubai Design Week 2016, UIST2016, Dubai International Film Festival 2016.



imagine a connected sensory world...

Synesthesia Project is about exploring the kinds of new sensory experiences that we can have using new media technologies such as virtual reality.  The project builds on previous music visualisation work, and adds a whole new dimension of full-body high-fidelity haptic vibration provided by colleagues at Embodied Media, Keio Media Design.

Synesthesia being demoed at the VR Village at Siggraph2016.

Synesthesia being demoed at the VR Village at Siggraph2016.

The virtual environment consists of a succession of interactive environments made up using procedural meshes that respond to music.  A natural mapping between sound and colour (as described in other sections) enhances immersion and presence by presenting consistent stimuli in visual and audial sensory channels, creating a world where our senses are hyper-connected, inspired by synesthesia.


While the sound creates the geometries and colours in the kaleidoscopic environment, the geometries in turn reach out and touch the user's body causing vibrational stimuli to add another sensory channel by which the reality of the world becomes objective in the mind of the user.  

The result is a level of sensory integration not possible even in reality.

I am interested in whether we can use mediated environments such as these to create new physiological phenomena in the brain and associated new types of experience.  I have been conducting experiments to determine whether the types of cross-modal sensory interactions made possible through VR, such as audio visualisation, can have an impact on the feeling of presence, and we are also conducting studies to measure whether multi-modal immersive mediated environments produce physical correlates and alter activity in the brain.  How far can we push human experience using media technology, and what doors will these new experiences open?