HelloEnjoy was asked by Interscope Records to create an interactive music experience using WebGL for the British artist Ellie Goulding.
At HelloEnjoy, we believe in creating interactive 3D experiences that are both intuitive and visually engaging. Our goal is to demand very little from the user to reach broader audiences, and at the same time reward them with high-end graphics and aesthetics.
We love music and have always seen a lot of potential in making it visual and interactive, so when Tool of North America, our agents in the US, approached us with this project we jumped in immediately.
The client gave us free creative rein and was very understanding with the experimental nature of the technology, which allowed us to play with different visualisation techniques. In the same way live music visuals make for a heightened experience, we wanted to achieve this feeling of perceiving music in an enhanced way through light and colour, plus interactivity.
Being technology agnostic allows us to focus on the solution that best meets our project's needs. In this case, we believe the best option was WebGL, a new web technology that brings hardware-accelerated 3D graphics to modern browsers without installing additional software.
As a foundation for the visuals to build up along with the music, we analysed the different stems of the song to capture the structure and energy levels for each part of the experience.
The stems were provided by the record label and gave us a new perspective and a deeper understanding of the music. Some of them were used to sync graphic elements and others were combined to obtain high quality volume and audio spectrum data.
Our creative approach was based around the concept of flying over an infinite landscape which reacts to the sound and evolves along with the music. This allows user interaction but also delivers a great experience if the user chooses not to interact.
We used a variety of tools and techniques to create the different elements of the site, some come from traditional game development, while others are common in graphics programming. We have selected a few of them to share with you.
The terrain is made of identical tiles which are placed in the field of view of the camera. We save considerable processing time by reusing them in different places, instead of updating each one individually. On the other hand, we had to make sure to make them seamless so they are fully tileable.
The terrain itself is generated from a 66x66 pixels height map, a greyscale image showing how high the land is. We started by creating it in Photoshop using the Clouds filter, but the resulting terrain was too steep. We obtained a more rounded landscape by applying Gaussian blur, but we knew there was still room for improvement.
To make it look more interesting, we wanted the ground to display different colours and shapes. To do this, we need to render a dynamic texture in real time and apply it to the landscape. We started doing it in canvas at 512x512 resolution, but the performance suffered when redrawing it on each frame. In the end, we opted for having a separate WebGL scene that's rendered onto the texture that's used for the terrain.
This terrain scene works essentially in 2D with its own orthogonal camera (which doesn't have any perspective). Here we use standard planes to create different types of lines and circles at various parts of the song. At this point, we also render the lights emitted from the other objects in the landscape, and finally we use post processing to make it seamless along the sides.
A range of different light emitting objects populate our environment. They are all generated by code and have various shapes, shaders and behaviours.
The spheres are interactive and synced to the beats. For their shading we use a technique called vertex colour, where colours are assigned to each vertex and the graphics hardware smoothly interpolates between them creating lovely gradients.
To fade the spheres to black and white we use a custom shader, with multiply and additive parameters. In graphics cards, RGB colour values range from 0 to 1, instead of 0 to 255. This way, if we multiply by 1 we get the same colour, but if we do it by 0 we obtain black (0,0,0). Adding works as expected, if we add 0 the colour doesn't change, but by adding 1 we get white (1,1,1) as the result is clamped.
The last part of the song is very epic. Starting with a silence, it unleashes a lot of energy at once. Here's where we introduce the light beams.
To create this effect we build a mesh out of three planes rotated 120 degrees apart. Their texture is a simple gradient which is additive blended, so they are rendered on top of each other. The result is a bright stream of light which gives a nice illusion of volume.
The name and avatar image of the users who tweet about the site can be seen during the experience. The real-time tweets are powered by Echo's powerful StreamServer platform, which allows us to quickly collect, store and serve the live data with little engineering effort.
Ellie Goulding's voice is visualised as a sparkling light with trails that move and change shape according to the volume and spectrum frequency of the combined vox stems.
Our first approach was to create the sparkling light with a 3D particle system, but we needed too many particles to achieve the effect we wanted. With so many effects going on, we had little performance left for this task and we realised we needed a different solution.
In order to obtain a realistic effect with good performance, the particle system was rendered in TimelineFX, a powerful particle effects editor used in games, and exported as a sprite sheet.
On the other hand, the trails are created in 3D using alpha and additive blending modes and animated in real-time using a combination of sound spectrum data and trigonometric functions to smooth it.
And finally, a couple of effects are applied to the final render. Bloom adds a nice glow around the objects in the scene, while vignette reduces the brightness at the borders of the image. The combination of both effects produce a more realistic and warmer image, and reinforces the illusion of light and shadow.
We believe real-time 3D graphics are particularly suited to enhance music, allowing users to enjoy it in new and exciting ways by creating a richer experience with immersive visuals and interaction. Although it's important to remember that graphics are meant to serve the experience and no amount of technical wizardry can replace well defined aesthetics, there is little doubt that the advent of hardware accelerated 3D in the web and mobile brings new levels of expressiveness. We have exciting times ahead.
And if you want to see what Hollywood's up to in 2012, check out the best 3D movies.