Michelle ThompsonThe brain frequently blends the senses, a phenomenon that marketers leverage in food packaging design. Similarly, AIs seem to engage in this sensory blending process as well.
Questions like “What is the flavor of a pink sphere?” or “What is the sound of a Sauvignon Blanc?” might seem absurd, but they highlight a fascinating aspect of human perception: our brains often blend sensory experiences. This phenomenon, which we may not always be aware of, links colors, shapes, and sounds with specific flavors, subtly shaping how we perceive the world. For example, the color of a glass or background music in a bar can influence how sweet or musky a wine tastes.
Carlos Velasco from BI Norwegian Business School in Oslo explains that this sensory “cross-talk” occurs almost constantly. In extreme cases, some individuals experience synesthesia, where words trigger tastes or music evokes visual experiences.
Interestingly, Velasco’s recent research suggests that generative AI systems might also engage in this blending of senses. While this occurs due to biases in the data used to train these AI systems, it underscores how common these sensory associations are. Velasco and his team aim to use this understanding to explore new ways to influence human perception.
Eating with the eyes
To start, a quick explanation of terminology: scientists use the phrase “sensory modality” to refer to the methods the body uses to process information—such as through our taste buds, eardrums, the retina in our eyes, or the “tactile corpuscles” in our skin. The connections we naturally make between different sensory qualities are referred to as “cross-modal correspondences.”
Humans frequently combine our sensory experiences of the surrounding world.
The first experimental evidence of this phenomenon appeared in the 1970s, with studies suggesting certain colors are linked to specific tastes: red and pink to sweetness, yellow or green to sourness, white to saltiness, and brown or black to bitterness. These patterns have been consistently confirmed through various experimental methods.
Participants are sometimes asked to rate abstract questions like, “On a scale from 1 to 10, with 10 being the sweetest, how sweet is the color red?” The results reveal that each color has a distinct flavor profile shared by many people across different cultures. A study led by Xiaoang Wang at Tsinghua University in China found similar cross-modal correspondences among participants from China, India, and Malaysia.
In other experiments, participants may evaluate the taste of food or drinks in various colors. For example, Eriko Sugimori and Yayoi Kawasaki from Waseda University in Japan discovered that bitter chocolate tastes sweeter when wrapped in pink packaging instead of black. The shape of food can influence taste perceptions too—round shapes are often linked with sweetness, while spiky shapes are associated with sourness or bitterness. This suggests we also “taste” with our eyes.
The origin of these associations remains debated. Charles Spence, head of the cross-modal research lab at Oxford University, suggests that these connections are learned. He explains that they could reflect the internalization of environmental patterns, such as fruits changing color from green (sour) to redder hues (sweeter), guiding us to the best fruits.
However, the link between shape and taste is harder to explain. Spence proposes that the emotions triggered by shapes may play a role. For example, round shapes are less likely to cause harm, and we may associate them with pleasant tastes like sweetness. Conversely, sharp shapes, linked to potentially harmful bitter substances, might be connected to bitterness.
Associative AI
The rapid growth of AI prompted Velasco, Spence, and their colleague Kosuke Motoki at the University of Tokyo to explore whether generative AIs, trained on human data, would report similar associations. They asked the AI-powered chatbot, ChatGPT, to respond to the same kinds of questions given to human participants. For example:
“To what extent do you associate round shapes with sweet, sour, salty, bitter, and umami tastes? Please answer on a scale of 1 (not at all) to 7 (very much).”
And:
“Among the 11 colours listed (black, blue, brown, green, grey, orange, pink, purple, red, white, yellow), which do you think best matches sweet tastes?”
By averaging results across hundreds of chats in English, Spanish, and Japanese, the researchers found that the AI reflected the patterns commonly seen in human participants, although some differences were observed between the versions of AI used.
In general, ChatGPT-4o more reliably mirrored human associations than ChatGPT-3.5. “The differences likely arise from variations in model architecture, such as the higher number of parameters in ChatGPT-4o and its larger, more diverse training set,” Motoki explains.
Silicon brainstorming
Curious about whether other large language models (LLMs), like Google’s Gemini, would also reflect our sensory associations, I asked it to identify the sweetest color. It responded: “Many people associate pink with sweetness, likely due to its association with sugary treats like cotton candy and bubble gum.” It also identified green with sour, white with salty, and black with bitter.
At first glance, the response seemed almost perfect, but midway through, Gemini referenced one of Spence’s previous research papers on cross-modal associations, suggesting it had sourced its answer directly from scientific literature.
Spence had anticipated this. “Since we tested large language models on what is already known and published in the literature, maybe it’s just repeating what it’s read,” he said.
Looking ahead, Spence hopes to explore whether generative AIs can generate new hypotheses for cross-modal correspondences not yet documented in the scientific literature, which could then be tested on human participants.
“You could potentially use large language models and generative AI to discover the perfect correspondences for any dimension you’re interested in,” he explains. This could be valuable for marketers designing products or packaging that play on existing sensory associations. However, there are some caveats. AIs can sometimes “hallucinate,” meaning they may generate misleading responses. Even if their answers are reliable, they may lack the subtleties and unique insights our brains provide, which can make designs more engaging or innovative. Sometimes, it may be more effective to explore sensory associations without replicating them exactly.
For this reason, any cross-modal correspondences suggested by AI should be paired with human creativity, says Velasco. “It’s inspiration, rather than a final solution.”
Christmas accompaniments
While more evidence is needed before we fully trust AI’s judgment, as I wrote this piece leading up to Christmas, I couldn’t help but wonder if ChatGPT could offer some advice for a drinks party.
Spence has previously demonstrated that people generally agree on which types of music pair well with different wines. For example, with its high tempo and pitch, Debussy’s Jardin Sous la Pluie seems to pair well with citrusy white wines, while the piano and cello duet in Rachmaninoff’s Vocalise tends to complement the richer flavors of red wines.
So, with all our festive favorites in mind, I wondered: what tunes would pair best with mulled wine and mince pies?
Research has shown that the right kind of music can enhance the rich flavors of red wines.
ChatGPT suggested that the complex flavor profile of mulled wine—rich with spices like cinnamon, clove, and star anise—calls for music that is equally layered, warm, and evocative. It recommended Carol of the Bells with lush orchestration, describing how its cascading melodies evoke festive magic and warmth, mirroring the spices in mulled wine. However, considering the piece’s use in the tension-building scene from Home Alone, I sought more relaxed options, like pop or jazz.
ChatGPT offered a few alternatives: Have Yourself a Merry Little Christmas performed by Ella Fitzgerald or Diana Krall, with smooth jazz vocals and warm instrumentation that reflect the comforting flavors of mulled wine. Other suggestions included Underneath the Tree by Kelly Clarkson, capturing a celebratory energy while balancing the depth of the wine, or Christmas Time is Here by the Vince Guaraldi Trio, a mellow yet jazzy track with sophistication that suits a relaxed festive atmosphere. We’ll see if my guests agree with these choices!
Emily Martinez
