The PS5 could use AI to dynamically adjust game audio based on play status and mood.

Sony's "Dynamic Music Creation in Gaming" patent, discovered by GameRant, describes how machine learning algorithms can analyze different parts of a musical score and provide musical variations in response to game play conditions. The idea is that the music will further stimulate the player's emotions, leading to a more engaging experience.

The patent explains that the AI system analyzes both the score and the game scene and generates metadata for both, effectively mapping out which parts trigger specific emotions. Those emotions include tension, joy, surprise, tenderness, nostalgia, sadness, fear, and transcendence.

The system then maps musical motifs based on the emotions that a particular "game vector" (action, character, location, player character) is expected to evoke. The result is expected to provide game music in a more dynamic way that fits how someone is playing the game, rather than what is happening in the game environment.

In short, if certain parts of the game usually have tense music, but the player is not really responding to it, Sony's system can increase the tempo of the music or make the music emotional as well as tense. Many games do something similar, but it only happens under pre-set circumstances, and Sony's patent envisions a more dynamic approach.

Sony does not cite any actual examples of this, but one can speculate how it might work. For example, if there is a particularly action-packed and ominous situation in Assassin's Creed Valhalla or Cyberpunk 2077, the system would not only increase the tempo of the music, but also by adding ominous, dissonant violin sounds or a deeper bass line, to give the soundtrack more emotional weight.

This all sounds pretty clever, but Sony wants to take it a step further and try to understand the player's emotions and adapt the game's music accordingly. Sony plans to do this by analyzing large amounts of data, such as matching review comments with parts of the game that are expected to have an emotional impact, and then time-stamping those parts in the electronic score so that they can be adjusted to evoke specific emotions.

The patent also mentions other data analysis that could fuel AI systems, such as tweaking the soundtrack based on who the player is. Sony could, if given the green light, use personal data to find out if a player is younger or older, introverted or extroverted, and what personality information might be gleaned from social media information.

Believe it or not, Sony may not stop there. The patent also mentions how the biometric device could be used to read a player's pulse, breathing, blood pressure, and other physical information; the PS5 could use that data to tell the AI system how the player is feeling. For example, it could be used to make the music more tense and dramatic if the player has a low heart rate and seems uninterested in the game, which should be exciting.

Such a system might raise privacy issues. However, it would be interesting not only to improve graphics and resolution, but also to add something new to future games.

Although PS5 is not explicitly mentioned in the patent, Sony supports the inclusion of a 3D audio processing unit in the next generation of consoles to provide a more immersive sound experience. Therefore, it is possible that this AI technology will be included in the PS5. And the inclusion of such audio technology may leave Microsoft's Xbox Series X in the dust.

Currently, the Xbox Series X has the edge in power, but the PS5 has a faster SSD storage system. But faster frame rates and more sophisticated effects, as well as a potentially game-changing audio system, could make the PS5 Sony's ace in the hole for winning the coming next-generation console wars.