DUET

A dynamic music visualization program that is a VR-compatible companion to the Spotify listening experience

Duet processes inputs from Spotify's API and SDK, and produces immersive visuals that respond to live parameters like tempo and dynamics, as well as static parameters like genre and mood

Type

Personal Project

Duration

9/21 - 12/21

Tools

HTML/CSS/JS
p5.js
A-Frame
Spotify API
Spotify SDK

Team

Sammy Levin (Developer)
Sahana Sripadanna (Developer)
Craig Kapp (Supervisor)

Duet Background.jpg
fm1 copy.png
ezgif.com-gif-maker (4).gif
aae5c38f9874259d6e860239ae3fc637.png
Duet Clip 2.gif
oculusquest2-render_edited.png

Visit any big concert and you'll see awe-inspiring lights and effects that are perfectly timed to the music, courtesy of a team of engineers working behind the scenes. Go to any great house party and you'll notice there's always something fun to look at, whether it's a colorful LED strip, or music videos playing on the TV.

What if there was a way to take the high-fidelity visual experience of a concert and put it in your own home using the devices you already own? 

I started by conducting a competitive analysis of the music visualization product landscape to identify pain points and missing features. From what I found, the scene is ripe for disruption. 

Screen Shot 2021-11-26 at 11.22.18 PM.png
Screen Shot 2021-11-26 at 11.25.11 PM.png
Screen Shot 2021-11-26 at 11.24.00 PM.png
Screen Shot 2021-11-26 at 11.28.00 PM.png

In-browser visualizers like Kaleidosync and Wavesync integrate with Spotify, but do not significantly change their visuals from song to song because they are only responsive to tempo.

Native desktop visualizers like VSXU and PotPlayer will accept audio input from local files only and are exclusive to specific operating systems and devices

Tune in,
   Rock on,
      Zone out,

         Turn it up.

Duet seeks to enrich the music listening experience by generating visuals that are just as responsive to a song's mood as you are. Whether you're chilling out alone or having fun with friends, open Duet in your browser to create an immersive backdrop to your favorite artists and albums. 

A mode for every mood

Blossom

ezgif.com-gif-maker (2).gif

Colorful auroras blossom and drift away on the beat. Acceleration, size, and particle motion are all determined by track features.

Crystalize (VR Ready)

ezgif.com-gif-maker (3).gif

Geometric crystals appear and explode, releasing shards in to the air. The number of vertices on each crystal is proportional to the average amplitude, and crystals spin when moused over.

Spectrum (VR Ready)

ezgif.com-gif-maker (4).gif

Visualize the full sonic spectrum in an immersive 3D world. Look up in VR to see towers of sound cascading above you as you listen.

Mode switching

Screen Shot 2021-12-14 at 1.30.38 AM.png

Giving users the flexibility to change the way they enjoy their music means starting with a robust, fluid method for changing visualization modes. Each visualizer is built as a p5.js canvas or A-Frame scene and is self-contained in an iframe. These iframes are switched in and out, and data objects are passed between them using the postMessage API.

ezgif.com-gif-maker (5).gif

With this functionality implemented, the mode selector UI was added to provide quick access to all three visualizations.

Seamless integration

Most existing music visualization programs rely solely on live audio data to determine the visualization style. By integrating with a user's Spotify account, we are able to integrate in-browser playback controls and extract track features like danceability, energy, and acousticness. After the user selects Duet from the list of devices in the Spotify app, we connect the two programs to provide playback controls and visualization parameters

Data Flow.jpg
aae5c38f9874259d6e860239ae3fc637.png
Duet Clip 2.gif

Responsive visuals

Duet checks a range of inputs in order to produce a visualization that is the best match for whichever track the user is listening to. This involves getting and posting information across several endpoints and processing information in p5.js, a JavaScript graphics library. Duet also listens to the user's microphone, so you can sing along and have your voice become part of the visual experience.

Let's dive into the visualization process for one of my favorite songs, "The Great Gig in the Sky" by Pink Floyd.

SDK Track Info.png

The user plays a track, and a query is sent to the Spotify Playback SDK to extract basic information about the song

Screen Shot 2021-11-26 at 6.58.45 PM.png

This information is injected into HTML so the user can see what they're listening to in the Duet interface

API Track Features.png

Using the track's ID that the SDK provided, a query is sent to the Spotify API to extract richer information about the song, including its "track features". 

Track features are subjective song qualities like danceability and acousticness that Spotify automatically quantifies when a new track is uploaded to its platform.

Screen Shot 2021-11-26 at 7.05.35 PM.png

Based on song playback and any sound coming from the user's mic, the p5.js audio library performs a Fast Fourier Transform (FFT) analysis on the waveform. This allows us to quantify sonic attributes like amplitude, bass, treble, and peak detection in real time.

Track Feature Params.jpg

With information extracted from both the API and the FFT object, we tied each audio parameter to the visual parameter that we thought best reflected it.

ezgif.com-gif-maker (1).gif

Each visualization mode receives these parameters as a message from the playback UI and processes them to create responsive visuals.

The color theme is unique to each song based on its track features, and carries across all 3 visualizations. Visualization attributes like speed, acceleration, size, and number of vertices are determined by FFT analysis and track features

Lessons learned

Duet has been a rewarding opportunity for me to bolster my software development skills by creating something I could see myself using. I developed a companion service that exists as a part of a larger product ecosystem, and am implementing extended reality in a compelling use-case.

  • I identified pain points in the music visualization product landscape through competitive analysis

  • I executed strategic design decisions based on product objectives

  • I architected a tech stack that strategically integrates various platforms into one unified experience

  • I developed generative graphics that are dynamic and object-oriented

Further documentation