Algorithmic Chromo-synesthesia


My latest side project: color-encoding music scores with palettes assigned according to the circle of fifths, and building networks of songs from their midi files using Mido & Plotly in Python.

Check out the notebook here and the GitHub here!

Throughout history, composers (and many others) have described a phenomenon in which musical notes manifest as colors in the mind’s eye. Consider this chart by Fred Callopy showing three centuries’ of recorded synesthesia:

While I don’t personally experience synesthesia, I am a very visual thinker, and in self-teaching guitar I wondered at the potential of this system for improving learning. By studying the commonalities in the above chart, I created this system for color association:

I then painted the fretboard of my guitar according to these colors, which produced a beautiful artifact:

Then, I started mapping the colors to sheet music. Here are some of the results:

“Frère Jacques”, where red = C, yellow = E, blue = G, etc. and length of rectangle indicates length of note, such that a square is a quarter note:

Below, “Ode to Joy” introduced problems of identifying adjacent octaves; the low G in row 3 is differentiated from the higher G in other rows by a decrease in brightness, such that hue still corresponds to note but brightness corresponds to octave:

Unlike “Frère Jacques” and “Ode to Joy”, which are very bright rainbows, the color palette generated by sheet music for “Fur Elise” is decidedly unsettling, much like the tune itself:

Note the very light yellow note in the 5th line; this is the very high E, as higher octaves are identified by increased brightness in the same way that lower octaves (such as the mustard-colored low Es preceding it) are darker.

Here are some experiments I’ve done trying to perform pitch detection; first using FFT as part of Processing’s minim library, then using the Google Teachable Machines audio software (still in beta testing):

WARNING: This video starts with a very high-pitched sound


Future plans for this project include building a program that can read sheet music and automatically generate graphics like the ones above (which were manually generated), the ability to translate guitar tabs, experiments into chords and harmonic ratios (possibly building a harmonograph), and building an interactive installation that allows instantaneous visual transcription of instruments, using Processing and a microphone to generate synesthetic paintings in real time!

Check out my list of computational music resources, and be sure to email if you have an idea for an addition to the list.

Image dump below; formatting and explanations to be added in the future:

4 thoughts on “Algorithmic Chromo-synesthesia

  1. Kia Ora from NZ. Thank you for posting this. It makes great reading. I came across it doing a search on the terms: colours of chords synesthesia – Have a great day or night wherever you are.

    1. Oh super cool, thank you for sharing! It was actually on my list to make a set of stickers sometime, so glad they already exist 🙂

  2. Thanks for posting. I was making my own circle of color after reading Musicophilia by Oliver Sacks. Your work is what I’m looking to get to, but you added the frequency transformation which is nice. Apparently humans likely get born with synesthesia that diminishes as the senses specialize during the first few months of age. The point is chromatic scale matches how the brain processes sound internally. The difference I had was to map each note to the color while keeping the ratios between the frequencies the same as the ratios with colors with the goal of having visually pleasing colors based on the frequency difference between them. I’m happy to share my spreadsheets if you’re curious. Red and purple took too many notes, so I’m not sure how helpful it would be.
    The idea of increasing brightness with frequency is brilliant (no pun intended).

Leave a Reply

Your email address will not be published. Required fields are marked *