Sony already promised that the PS5 will revolutionize gaming sound with its built-in 3D audio processing unit. Now it appears Sony’s next move is to develop AI-powered game soundtracks that react to how you play, or to your emotions, and change the music accordingly.
The ‘Dynamic Music Creation in Gaming’ patent was first discovered by Game Rant, and while it doesn’t specifically mention the PlayStation 5, it has all the hallmarks of a feature that Sony is considering for the forthcoming console.
In the patent, Sony’s engineers describe how they plan to use machine learning to analyze how different ‘musical components’ like rhythm, melody and harmony map to specific emotions, and then create different emotional variants to musical scores that will trigger specific feelings in gamers as they play.
First, Sony will have its composers create musical ‘motifs’ that map to a specific character, in-game activity, locations or even the player’s ‘personality’. Then, these motifs will be coded to play based on Sony’s data on what the player is feeling, or should be feeling, in a given moment.
For example, on the PS5, Sony could take the overworld music from Assassin’s Creed Valhalla and reduce the tempo and percussion for a calmer ambience if the player has stopped moving for a while, up the tension if they’ve locked onto a target, or add some soulful violin if a beloved character appears.
Specific emotions that this dynamic system would try to evoke include Tension, Power, Joy, Wonder, Tenderness, Transcendence, Peacefulness, Nostalgia, Sadness, Sensuality, and Fear.
The engineers plan to analyze tons of popular music in order to turn emotion-inducing music into a science. Their machine-learning system will convert music into electronic scores, match the scores up with reviews from critics and fans, then timestamp specific stretches of music that correspond with specific emotions.
Sony then plans to experiment with ‘mapping emotions to scenarios’. During a stressful moment, rather than predictably play tense music, they claimed that an undercurrent of sensuality could have a ‘better and more interesting effect’ on your experience.
While we don’t know when the ‘Compositional Engine’ will be considered viable, it’s certain that it could work the late 2020-bound PS5 as an effort to differentiate the console from the Xbox Series X. Many of today’s video games already adjust their music based on players’ actions or events in-game, but not with this level of subtlety and personalization.
Sony’s biometric future for PS5?
Sony wants to adjust the soundtrack based on who you are, listing out categories like “young, old, male, female, introvert, [and] extrovert.” But the patent explores how their musical system could go well beyond such basic descriptors.
Gamers could ‘opt in’ to provide their social media data to Sony for personality and behavior analysis. Gamers could also use ‘biometric devices’ to provide data like “electrodermal activity, pulse and respiration, body temperature, blood pressure, brain wave activity, [and] genetic predispositions”.
To track this information, the engineers listed some potential biometric devices that could track users’ data: a ‘thermal or IR camera’, a ‘wearable activity tracker’ or a ‘smartwatch with bio-sensing’.
From there, the system would take your bodily data and adjust the soundtrack to match. Imagine if you have died multiple times to a tough boss, and the PS5 could sense your elevated heart rate, interpret this as frustration, and adjust the soundtrack to play something more soothing.
Many PS5 owners won’t feel comfortable with Sony knowing so much about their health and personality. But if Sony’s machine learning can create profiles of a subset of gamers, they can likely predict average emotional responses to certain game segments. Then they can extrapolate how any gamer might react to a given scene and adjust the music accordingly.
Regardless of privacy concerns, this patent does hint that Sony may be interested in tracking gamers’ health in other contexts. Nintendo has explored sleep-tracking tech for years and its exercise game Ring Fit Adventure was a major hit.
It’s possible Sony has similar aspirations for the PlayStation 5, either to create its own tracking tech, or to connect gamers’ existing smartwatches and fitness trackers to the PS5 to provide real-time data.