Algorithmic Chromo-synesthesia

Throughout history, composers (and many others) have described a phenomenon in which musical notes manifest as colors in the mind’s eye. Consider this chart by Fred Callopy showing three centuries’ of recorded synesthesia:

While I don’t personally experience synesthesia, I am a very visual thinker, and in self-teaching guitar I wondered at the potential of this system for improving learning. By studying the commonalities in the above chart, I created this system for color association:

I then painted the fretboard of my guitar according to these colors, which produced a beautiful artifact:

Then, I started mapping the colors to sheet music. Here are some of the results:

“Frère Jacques”, where red = C, yellow = E, blue = G, etc. and length of rectangle indicates length of note, such that a square is a quarter note:

Below, “Ode to Joy” introduced problems of identifying adjacent octaves; the low G in row 3 is differentiated from the higher G in other rows by a decrease in brightness, such that hue still corresponds to note but brightness corresponds to octave:

Unlike “Frère Jacques” and “Ode to Joy”, which are very bright rainbows, the color palette generated by sheet music for “Fur Elise” is decidedly unsettling, much like the tune itself:

Note the very light yellow note in the 5th line; this is the very high E, as higher octaves are identified by increased brightness in the same way that lower octaves (such as the mustard-colored low Es preceding it) are darker.

Here are some experiments I’ve done trying to perform pitch detection; first using FFT as part of Processing’s minim library, then using the Google Teachable Machines audio software (still in beta testing):

WARNING: This video starts with a very high-pitched sound

 

Future plans for this project include building a program that can read sheet music and automatically generate graphics like the ones above (which were manually generated), the ability to translate guitar tabs, experiments into chords and harmonic ratios (possibly building a harmonograph), and building an interactive installation that allows instantaneous visual transcription of instruments, using Processing and a microphone to generate synesthetic paintings in real time!

Check out my list of computational music resources, and be sure to email jane@universalities.com if you have an idea for an addition to the list.

Image dump below; formatting and explanations to be added in the future:

Leave a Reply

Your email address will not be published. Required fields are marked *