The emergence of ChatGPT and AI image generators has opened up new possibilities for creating text, images, and videos. Coupled with the rise of the metaverse during the COVID-19 era, people began crafting avatars, exploring virtual spaces, and interacting with others in entirely new ways. Advances in AI and VR technology have since extended the boundaries of the real world into these virtual realms.
So, how will AI and VR transform the way we experience music? Verses offers an answer with a groundbreaking concept: Interactive Music.
Interactive Music is a new form of music consumption proposed by Verses. It combines traditional audio-based music with visual elements, interactive motions, storytelling, and world-building, all powered by generative AI. The result is a multisensory, immersive musical experience like no other.
‘Connecting sound with other elements’ goes beyond simply listening to music—it means engaging with various sensory and interactive components to create a richer, more immersive musical experience.
Verses’ Interactive Music is built on this concept, seamlessly combining visuals, interactive motions, storytelling, and user experiences into one cohesive whole. Each of these four elements is intricately linked to the music, dynamically interacting and evolving in real-time through generative AI technology.
1. Visuals
Interactive Music expands the experience of music from simply ‘listening’ to ‘seeing,’ offering users a more immersive way to engage with sound. This involves incorporating various forms of visual feedback that interact with the music. Examples include 3D objects that change in sync with the rhythm, metaverse environments that evolve with the flow of the music, and 3D virtual instruments that users can play through their movements.
These visual elements go beyond passive observation, creating opportunities for direct interaction with the music. This enables users to experience music in a more intuitive and engaging way. Dynamic visuals that shift with the rhythm or transform metaverse environments not only add vibrancy but also bring the music to life, offering a groundbreaking experience where the sound unfolds in real time right before the user’s eyes.
2. Interactive Motions
Another key feature of Interactive Music is its ability to change in response to the user’s interactive motions. Actions like shaking a smartphone or touching the screen dynamically alter the flow of the music, generating new beats and allowing users to recreate the music in their own unique way.
For instance, in Verses’ interactive single ‘FIGHTMAN’ by the artist SUMIN, every screen tap or device shake transforms the rhythm and beats of the song in real-time. This creates a creative experience where users can actively intervene in and reshape the flow of the music.
The music's instant response to even the smallest movements or touches fosters a deep sense of connection between the user and the sound. It maximizes the joy of musical interaction by making the experience highly personal and engaging. This real-time interplay between user motions and music not only enhances immersion but also elevates participation, unlocking a whole new dimension of music appreciation.
3. Storytelling
The integration of music with metaverse-based storytelling is another remarkable aspect of Interactive Music. In this system, the user’s actions within the metaverse—performed through their avatar—trigger real-time reactions in the music. For instance, when an avatar interacts with objects or follows specific paths in the virtual world, new musical elements are added or existing ones are transformed.
This approach goes beyond passive music consumption, inviting users to actively engage with both the story and the music. It feels much like a video game, where users become creators of ‘musical events’ within the metaverse. This dynamic interaction makes the experience deeply personalized and more immersive, offering users a unique and ever-evolving musical journey.
4. User Experiences
Interactive Music stands out for its ability to respond in real time to every action a user takes within the app. Not only do elements like visuals, interactive motions, and storytelling interact with the music, but the combined interplay of these elements actively shapes the evolution of the music itself. Through this dynamic interaction, users move beyond passive listening, becoming co-creators who directly influence and develop the music.
At the core of this innovation lies generative AI technology. Generative AI reacts instantly to even the smallest user movements, producing organic and unique music in real time. This allows users to effortlessly and intuitively create and experience music infused with their own personal style and creativity.
Ultimately, this interactive experience takes music enjoyment to a new level by introducing the joy of creation. Users craft a distinctive musical world that reflects their preferences and sensibilities, leading to a deeply immersive and personalized experience. As a result, music consumption evolves into a participatory and creative process—an important step forward in how music is experienced in the modern era.
We’ve explored the four key elements of Interactive Music and how they connect with sound.
What’s particularly fascinating is the interconnectedness of these elements. For instance, interactive motion and visuals might combine to alter the flow of visuals in response to user movements. Similarly, the story and visuals could adapt dynamically, with background environments shifting to reflect the evolving narrative shaped by user actions. This system, resembling an organism that feeds on mutual feedback, enables rich auditory and visual experiences.
The Best Way to Understand: Experience It Yourself
The most effective way to grasp the concept of Interactive Music is through direct experience. It’s far more impactful to engage with it firsthand than to understand it theoretically. To help you dive in, here are some of Verses’ flagship products worth exploring:
1.
aespa world
An interactive music world on the metaverse platform Zepeto, where users can enjoy a unique musical experience with the avatars of aespa members brought to life.
2.
FIGHTMAN: SUMIN’s Interactive Single
An interactive single by artist SUMIN, available as an app on the App Store. Users can enjoy SUMIN’s music as it evolves in real time based on their interactive motions.
3.
DJ Hitchhiker's Metaverse Music
A metaverse music service created in collaboration with global EDM musician and producer Hitchhiker. Set in a metaverse environment, the story follows Hitchhiker as they attempt to escape an alien planet through music, offering users an immersive and narrative-driven musical journey.
Verses' Five Core Services
We’ve explored the concept of Interactive Music, and based on this idea, Verses offers five core services. Each service connects sound with other elements in unique ways. Now, let’s take a closer look at these services and dive deeper into the tangible reality of Interactive Music.
1. aespa world
aespa world is an interactive music world created through a collaboration between SM Entertainment and Verses, built on the metaverse platform ZEPETO. This space brings the K-pop group aespa's universe to life in a virtual world, where users can participate through their avatars. Created alongside the release of aespa's mini album [Drama], this virtual world features interactive sounds based on the title track ‘Drama.’ As users interact with objects in the virtual space, new sounds are added in response to their actions.
Additionally, users can collect various items to remix the music or create content to share. They can also learn aespa's choreography, apply fashion items worn by the artists to their avatars, and engage in various other interactions for a richer fan experience.
2. Beat-based Artificial Intelligence Music Video Maker
Beat-based AI Music Video Generator, which won the Innovation Award in the AI category at CES 2024, is a tool that allows users to easily create short-form music videos in the metaverse. By simply tapping the screen in sync with the beat, the AI automatically generates music and visuals in response. No complex software or technical manipulation is required—just an intuitive and simple touch to create a music video, offering users a moment where play transforms into creation.
3. Meta Music System
Meta Music System, which won the Best Innovation Award at CES 2023, is a generative AI-based metaverse music platform that serves as the core technology behind aespa world. Meta Music System recreates a musician’s universe in a 3D virtual space within the metaverse.
Users can participate as characters from the universe through their avatars, allowing fans to engage in more active and immersive fan activities. By stepping into the world of a musician as a character, fans can influence the storyline, remix the music by interacting with 3D objects, and much more.
4. DJ Hitchhiker's Metaverse Music
Hitchhiker's Metaverse Music Space is a metaverse-based music service created in collaboration with world-renowned EDM musician and producer Hitchhiker. This service offers a unique universe centered around a storyline in which Hitchhiker crash-lands on a distant planet and uses music to escape from the unfamiliar world.
Users can explore the metaverse freely as a character dressed in a silver spacesuit and engage in various interactive experiences, much like a game. One of its key features is the real-time, AI-generated music and visuals that respond to user actions. This allows users to enjoy a dynamic music experience.
When users touch the screen, Hitchhiker’s voice activates, and he begins rapping, providing the foundation for users to create their own music. Through this innovative interactive feature, users can actively engage with the music and immerse themselves in a metaverse space where the music and visuals harmonize.
5. FIGHTMAN: SUMIN’s Interactive Single
FIGHTMAN: SUMIN’s Interactive Single, an interactive single created in collaboration with musician SUMIN, won the CES 2022 Innovation Award. This app goes beyond just listening to music, offering an immersive experience where users can actively participate in the music itself.
Within the world created by ‘FIGHTMAN’ app, users can interact with the music by tapping the screen or moving their phone. Through these interactions, the flow of the beat is altered, and new musical elements are generated and played in real time.
This allows users to create their own remixes, fostering a creative musical experience. FIGHTMAN is a prime example of Interactive Music, offering users an active experience where they can not only listen to music but also create and reinterpret it, breaking away from traditional music consumption.
Interactive Music: A Revolutionary Way to Experience Music
Interactive music offers users an entirely new way to engage with music, breaking free from traditional consumption methods. Let’s explore the unique value it brings to users.
1. A Music Experience Tailored for the Metaverse Era
Interactive music enables users to deeply immerse themselves in the artist’s universe, fostering a new level of interactive communication. Through avatars, users can explore the metaverse, experiencing the artist’s story firsthand. This engagement goes beyond passive listening, allowing fans to interact with artists and express their admiration through a novel medium.
Thanks to generative AI technologies, the experience adapts dynamically based on individual user actions, maximizing personalization and interactivity. This creates a uniquely immersive environment where fans feel truly connected to the music and the artist.
2. Turning Play into Creation
One of the standout features of interactive music is its ability to empower users to easily create and share music content in an enjoyable way.
For instance, Beat-based AI Music Video Generator allows users to produce personalized music videos by simply tapping along to a beat. This makes the creative process feel like play while still yielding meaningful content.
Additionally, platforms like Meta Music System enable users to mint their remixes as NFTs, transforming their creations into tangible digital assets. This accessibility to creativity not only lowers barriers but also provides users with opportunities to contribute as creators within the music ecosystem.
3. A New Level of Enjoyment Through Multisensory Experiences
Interactive music engages multiple senses—sight, hearing, and touch—offering a multisensory experience. By blending sound with immersive 3D visuals and responsive user actions, it deepens the connection between the user and the music.
As the music evolves in response to the user’s movements, it fosters a sense of ownership and fulfillment, allowing users to experience joy and satisfaction from their co-created musical journey. This multidimensional interaction enhances how music is enjoyed, amplifying emotional resonance and user engagement.
Interactive music moves beyond traditional music listening by blending participation, creativity, and immersion. Leveraging advancements in metaverse technology, NFTs, and AI, it redefines how music is experienced, empowering users to become active participants in a dynamic and personalized musical universe.
