Algorithmic Chromo-synesthesia


My latest side project: color-encoding music scores with palettes assigned according to the circle of fifths, and building networks of songs from their midi files using Mido & Plotly in Python.

Check out the notebook here and the GitHub here!

Throughout history, composers (and many others) have described a phenomenon in which musical notes manifest as colors in the mind’s eye. Consider this chart by Fred Callopy showing three centuries’ of recorded synesthesia:

While I don’t personally experience synesthesia, I am a very visual thinker, and in self-teaching guitar I wondered at the potential of this system for improving learning. By studying the commonalities in the above chart, I created this system for color association:

I then painted the fretboard of my guitar according to these colors, which produced a beautiful artifact:

Then, I started mapping the colors to sheet music. Here are some of the results:

“Frère Jacques”, where red = C, yellow = E, blue = G, etc. and length of rectangle indicates length of note, such that a square is a quarter note:

Below, “Ode to Joy” introduced problems of identifying adjacent octaves; the low G in row 3 is differentiated from the higher G in other rows by a decrease in brightness, such that hue still corresponds to note but brightness corresponds to octave:

Unlike “Frère Jacques” and “Ode to Joy”, which are very bright rainbows, the color palette generated by sheet music for “Fur Elise” is decidedly unsettling, much like the tune itself:

Note the very light yellow note in the 5th line; this is the very high E, as higher octaves are identified by increased brightness in the same way that lower octaves (such as the mustard-colored low Es preceding it) are darker.

Here are some experiments I’ve done trying to perform pitch detection; first using FFT as part of Processing’s minim library, then using the Google Teachable Machines audio software (still in beta testing):

WARNING: This video starts with a very high-pitched sound


Future plans for this project include building a program that can read sheet music and automatically generate graphics like the ones above (which were manually generated), the ability to translate guitar tabs, experiments into chords and harmonic ratios (possibly building a harmonograph), and building an interactive installation that allows instantaneous visual transcription of instruments, using Processing and a microphone to generate synesthetic paintings in real time!

Check out my list of computational music resources, and be sure to email if you have an idea for an addition to the list.

Image dump below; formatting and explanations to be added in the future:

One thought on “Algorithmic Chromo-synesthesia

  1. Kia Ora from NZ. Thank you for posting this. It makes great reading. I came across it doing a search on the terms: colours of chords synesthesia – Have a great day or night wherever you are.

Leave a Reply

Your email address will not be published. Required fields are marked *