Syn(es)thetic Reality: simulating synesthesia for the non synesthetic

Vinay Khare, Luis Edgardo Fraguada, and Elizabeth Esther Bigger. 2019. Syn(es)thetic reality: simulating synesthesia for the non synesthetic. In Proceedings of the 23rd International Symposium on Wearable Computers (ISWC ‘19). Association for Computing Machinery, New York, NY, USA, 290โ€“295. https://doi.org/10.1145/3341163.3346940

Presented at the 23nd (2019) International Symposium on Wearable Computers in London, United Kingdom.

abstract

Syn(es)thetic Reality explores a new way of sensing the world by understanding sounds through colors. It looks to simulate projective chromesthesia, an experience of seeing colors involuntarily as a result of sound input. The project achieves this through an augmented reality web application to be run on mobile phones via a wearable device. By displaying audio based color content composited over the camera feed, the user’s visual perception is altered to reflect their auditory perception. The basic principle is to correlate sound frequencies to hue and amplitude to saturation and visualise these colors creatively.