“Seeing Sound” is an interdisciplinary collaborative research project to explore how to effectively communicate characteristics of sound with creative visualisation. The concept and interest have been initiated with a recent interactive multimedia system developed at ICSRiM (University of Leeds Interdisciplinary Centre for Scientific Research in Music) for a real-time interactive performance at the AIC2013 (12th International Colour Science Congress) with the Royal Northern Sinfonia at Sage Gateshead. The project lead, Kia Ng, is inspired to extend the research further to better understand different stages of the trans-domain mapping system; with better insights into human cross-modal perception to enhance the effectiveness of the communication, appropriate and meaningful mapping strategies and optimal rendering approaches for different application contexts.

With the Leeds Creative Labs partnership with Jane Wood (Creative Producer) and Shay Moradi (Partner, RITH), the team is exploring how to capture and analyse sound-visual perception, designing artistic and meaningful mapping models and effective rendering techniques.

Music-colour association has a rich history within both the sciences and arts. An early scientific association of the two domains is detailed by Newton (1704). Historically, visual and auditory artists have mutually served as each other’s inspiration. Musical timbre is frequently described as the “tone colour”. Our initial step in sound-colour mapping is centred around recorded sound-colour mapping from synaesthetic composers. Synaesthesia is a neurological phenomenon where by stimulation of one sensory modality results in an extra sensory perceptual response in another. Common manifestations of this sensation include the perception of colour for music, phonemes, numerals and letters, and ‘tactile shapes’ for taste.

Research into the music-colour synaesthetes perception of stimulatory audio has produced varied responses, reflecting the subjectivity of the phenomenon. Colour synaesthesia is generally individual. However, there are several features that exhibit more comment trends. These include: (i) pitch and brightness; (ii) loudness to size; (iii) colour and frequency. Many composers and artists, including Messiaen, Scriabin and Steen report synaesthetic responses that influence their work.

With better understanding of our cross-modal association and more general mapping models, it is anticipated that interactive/create feedback can be carried out with more intuitive and effective to convey creative and meaningful communication. The mapping models and rendering techniques shall be utilised for an interactive installation at the Centre for Life later this year and it is hoped that the research outcomes can be applied in a verity of application contexts including technology-enhanced learning, gaming, live performances and contribute towards accessibilities.