top of page
Summary
A dynamic music visualization program that is a VR-compatible companion to the Spotify listening experience
Duet processes inputs from Spotify's API and SDK, and produces immersive visuals that respond to live parameters like tempo and dynamics, as well as static parameters like genre and mood
Type
Personal Project
Duration
9/21 - 12/21
Tools
HTML/CSS/JS
p5.js
A-Frame
Spotify API
Spotify SDK
Team
Sammy Levin (Developer)
Sahana Sripadanna (Developer)
Craig Kapp (Supervisor)
Visit any big concert and you'll see awe-inspiring lights and effects that are perfectly timed to the music, courtesy of a team of engineers working behind the scenes. Go to any great house party and you'll notice there's always something fun to look at, whether it's a colorful LED strip, or music videos playing on the TV.
What if there was a way to take the high-fidelity visual experience of a concert and put it in your own home using the devices you already own?
I started by conducting a competitive analysis of the music visualization product landscape to identify pain points and missing features. From what I found, the scene is ripe for disruption.
In-browser visualizers like Kaleidosync and Wavesync integrate with Spotify, but do not significantly change their visuals from song to song because they are only responsive to tempo.
Native desktop visualizers like VSXU and PotPlayer will accept audio input from local files only and are exclusive to specific operating systems and devices
Tune in,
Rock on,
Zone out,
Turn it up.
Duet seeks to enrich the music listening experience by generating visuals that are just as responsive to a song's mood as you are. Whether you're chilling out alone or having fun with friends, open Duet in your browser to create an immersive backdrop to your favorite artists and albums.
A mode for every mood
Blossom
_gif.gif)
Colorful auroras blossom and drift away on the beat. Acceleration, size, and particle motion are all determined by track features.
Crystalize (VR Ready)
_gif.gif)
Geometric crystals appear and explode, releasing shards in to the air. The number of vertices on each crystal is proportional to the average amplitude, and crystals spin when moused over.
Spectrum (VR Ready)
_gif.gif)
Visualize the full sonic spectrum in an immersive 3D world. Look up in VR to see towers of sound cascading above you as you listen.
Mode switching
Giving users the flexibility to change the way they enjoy their music means starting with a robust, fluid method for changing visualization modes. Each visualizer is built as a p5.js canvas or A-Frame scene and is self-contained in an iframe. These iframes are switched in and out, and data objects are passed between them using the postMessage API.
_gif.gif)
With this functionality implemented, the mode selector UI was added to provide quick access to all three visualizations.
Seamless integration
Most existing music visualization programs rely solely on live audio data to determine the visualization style. By integrating with a user's Spotify account, we are able to integrate in-browser playback controls and extract track features like danceability, energy, and acousticness. After the user selects Duet from the list of devices in the Spotify app, we connect the two programs to provide playback controls and visualization parameters
Responsive visuals
Duet checks a range of inputs in order to produce a visualization that is the best match for whichever track the user is listening to. This involves getting and posting information across several endpoints and processing information in p5.js, a JavaScript graphics library. Duet also listens to the user's microphone, so you can sing along and have your voice become part of the visual experience.
Let's dive into the visualization process for one of my favorite songs, "The Great Gig in the Sky" by Pink Floyd.
The user plays a track, and a query is sent to the Spotify Playback SDK to extract basic information about the song
This information is injected into HTML so the user can see what they're listening to in the Duet interface
Using the track's ID that the SDK provided, a query is sent to the Spotify API to extract richer information about the song, including its "track features".
Track features are subjective song qualities like danceability and acousticness that Spotify automatically quantifies when a new track is uploaded to its platform.
Based on song playback and any sound coming from the user's mic, the p5.js audio library performs a Fast Fourier Transform (FFT) analysis on the waveform. This allows us to quantify sonic attributes like amplitude, bass, treble, and peak detection in real time.
With information extracted from both the API and the FFT object, we tied each audio parameter to the visual parameter that we thought best reflected it.
_gif.gif)
Each visualization mode receives these parameters as a message from the playback UI and processes them to create responsive visuals.
The color theme is unique to each song based on its track features, and carries across all 3 visualizations. Visualization attributes like speed, acceleration, size, and number of vertices are determined by FFT analysis and track features
Lessons learned
Duet has been a rewarding opportunity for me to bolster my software development skills by creating something I could see myself using. I developed a companion service that exists as a part of a larger product ecosystem, and am implementing extended reality in a compelling use-case.
-
I identified pain points in the music visualization product landscape through competitive analysis
-
I executed strategic design decisions based on product objectives
-
I architected a tech stack that strategically integrates various platforms into one unified experience
-
I developed generative graphics that are dynamic and object-oriented
Further documentation
bottom of page