Synesthesia. Gift or Curse?

‘Music is liquid Architecture; Architecture is frozen Music’ – Goethe

Do you want to hear the sound of time? Or maybe taste the music? Or see its colors? Or the dance of sculptures? And no, we have not just taken some…. mushrooms, before writing this article. But we really discovered this world of new perceptions and we want to share it with you. More exactly, we discovered the ones that can share it with you.
You might think, we will give you a text written by artists. Not exactly. Although, it is true that artists along the history have used association and metaphors to describe different dimensions, emotions, actions, so on. Or they might really experience it like this, who knows? For example, avant-garde experiments were made to discover new artistic languages and expressions in order to build the “art of the future”. Kandinsky boldly stated that “the color may have a smell, and the taste may have a sound”. Based on his reflections, a new theory of color and sound appeared, according to which a certain sound gives a certain color. Another important concept developed during the avant-garde period is the synesthetic concept based on the transfer of sensations associated with the space of the MACROcosmos and the MICROcosmos. This type of synesthetic concepts created new, abstract imagery, abstract artistic languages (abstract painting, serial music, phonetic poetry).

“There was a smell of Time in the air tonight”, writes Ray Bradbury in The Martian Chronicles.

How did Time smell like? Like dust and clocks and people. And if you wondered what Time sounded like, it sounded like water running in a dark cave and voices crying and dirt dropping down upon hollow box lids, and rain. Time looked like snow dropping silently into a black room or it looked like a silent film in an ancient theatre one hundred billion faces falling like those New Year balloons down and down into nothing. That was How Time smelled and looked and sounded.

These are only a few examples of how artists imagined shapes, sounds, colors, by combining them. And then (or before?) scientists realized that some people are really able to have a special form of perception. This is synesthesia.
A neurological condition when the brain is wired differently: the stimulation of one sensory or cognitive pathway leads to experiences in a second sensory or cognitive pathway.
And we promised that we can make you feel this world.
Well, we have found a team determined to set up a VR experience, which will make the public perceive the world through the eyes and the perception of the Synesthetic. The project is called Synesthesia, and we sat down with the director Ola Pankratova to discuss all the details.

TechvangArt(TVA): We have e-met at NewImages Festival, how did you start your journey that led you to the festival?
Ola Pankratova: I came up with the idea a year ago, and we just started filming Synesthesia as a plain documentary, so we started applying for different pitching, grants, so on. That is how we came across Stereopsia, a very big XR forum in Belgium, and they suggested me to consider making it into an XR project. And that is how it transformed into what we have right now. I came to pitch the film to Brussels, a year ago, and there I met my producer Boris Baum. And what we have planned is a documentary for 1’30 min., and a VR version for 15 minutes.
And that’s how we’re pitching it with the different formats at the different festivals. And it’s exactly how we got picked up for the NewImages Festival. For us, it was very important to be present to be pitched among the best XR projects

TechvangArt(TVA): And what was your gain at the festival? Even if you were not between the winners, you had a lot of meetings and networking opportunities.
Ola Pankratova: I am always thinking that is not about the winning, it’s about being present, it’s about networking, and we were very pleased with the results for us we were contacted by the biggest XR-studios from Canada and France and now we are at the stage of discussions with whom we will be moving forward

TVA: To be honest, it is a fascinating topic, because it is the neurological condition that everybody wants to have it! 😀
Ola Pankratova: Exactly. But let me ask you, did you know about this condition before you saw the pitch?
TVA: Yes, because my first degree is psychology…
Ola Pankratova: So, you got an advantage, but people really don’t know about this condition. For example, when I was speaking in Brussels, there was a woman that has a son with autism, and she came up to me after my pitch and said: “ I never heard about this condition, but now I know that my four year old son has it. Because he was telling me that he’s seeing music in colour and I thought that he was making things up. But now I know that this is for real”.
We are supported by the Synesthesia International Association, and they are helping me with a lot of literature. And so I learned that it hasn’t been studied lately, I think in the 20th century researchers haven’t touched upon it at all. And people just didn’t know about it at all.
I learned that 4% of the human population has it. It may seem like a very small amount of people but at the same time, when you talk to others and they say, “oh yeah and I think I also have synesthesia because sometimes I envision this or sometimes, I can see in colors”. And people just don’t know how to apply this in their real life.
And I understood the importance of the social aspect: not to just show it as a superpower, but to educate people and to tell society this is normality for us. And we have not only fun stories about synesthesia, but also very tense stories when people really do suffer from this condition.

TVA: How do you choose your subject, because we have 60-80 different forms of synesthesia, so you choose according to frequencies or types, or do you just ‘pick’ any subjects…?
Ola Pankratova: It is not about just ‘picking’ anybody. The IASAS has provided me with a wide database of people who have synesthesia. I was consulting with them before selecting. For me the main criteria were not only the fact of having synesthesia, but also the personal story. Because the crucial aspect is storytelling, it’s not just about the VR and having the helmet on. In addition, we wanted to showcase different types of synesthesia and not only just the most common ones. That’s why we selected very rare forms, such as mirror touch synesthesia (when you can feel, what another person feels). And that’s how we’re going to apply it for a XR installation. We’re going to have special jackets so whenever a character in the story is supposed to feel something – the viewer will feel the same sensations with the help of this jacket, he will have all the vibrations during the installation.
And I also wanted to have diverse stories from all over the world, because we’re doing this film with an international team. I’m based in Russia, my producer is based in Brussels, and a lot of my protagonists are based in the US. We want to showcase that these people are all around the world. It doesn’t matter where you live, and in what society you live in, it touches upon different genders, different ages.

TVA: Can you tell us what characters are you gonna have in the film?
Ola Pankratova: The main protagonist is going to be our conductor from Moscow. He has an orchestra here, and he has just found out he has synesthesia. So, we’re going to actually film him during the process – how it is being tested, how the tests are evolving, how you can diagnose synesthesia. And obviously, we’re going to show how he’s applying synesthesia in his work, in directing the orchestra.
Another character that we have is a medical worker from the USA – she has mirror-touch synesthesia. She’s from a family of medical workers, and her mother wanted her to be a surgeon. But with a mirror-touch synesthesia is pretty much impossible – she was feeling everything what was happening DURING her work at the hospital. Or for example, that she cannot watch the Game of Thrones series, because she literally feels all the injuries and pain and everything that’s going on. So, she will tell the story about how she became a manual therapist, how she found out when she was 45 years old, that she has synesthesia and that all of these years she has been struggling with this.
You can imagine that sometimes that could be a gift for these people and sometimes it could be a curse. And she also educates people on how to manage and how to get and adapt in a society with this kind of condition.
We will have an artist that draws and creates art, with the help of music, and we have a girl who is a sommelier, so she tastes wine with the help of the surrounding sounds.

But I also want to stress that this is not a disease. This is not a disability. We’re making a film about people with superpowers, and we want to portray them as very special people, and how you can actually apply if you discover that you have one after watching our film. And maybe this is exactly a key to reveal your creative genius.
TVA: And you will make the VR, from the first-person point of view?
Ola Pankratova: Yes, the VR, it will be from the first-person point of view. When you put the helmet on, you completely transfer into the brain of this person. So you can choose between four different states, whatever experience you want to try and so you will just immerse for 15 minutes in his story.

TVA: Sounds exciting! At what stage are you right now with the filming?
Ola Pankratova: Because we are still at the development stage, we’re just ready to go from pre-production to production. Also, due to the situation, it is really hard and we just still don’t know yet, when we’re going to be able to start filming. But I hope soon. Another aspect is that we still have to raise money – it’s important for us to make it, as we planned it together with my producer, this is the subject that we don’t want to make any compromises. But during all our pitching sessions, I can see the involvement and interest of different partners that want to get involved with the project. And then the post-production is going to be very precise and long, it is an important part of our process because we will have a lot of graphics, sound design. So, maybe you just ask in a year….

All visual materials were integrated with author consent © Ola Pankratova

more about project : https://www.filmsdelarecre.com/synestesia
Team : Ola Pankratova Director, Amaury La Burthe VR Exec Producer, Beatrice Lartigue VR Art Directors, Arnaud Desjardins VR Art Directors, Fabien Togman VR Art Directors
Producer:Boris Baum
Producers : Les Films de la Récré (Belgium)