“Even if you're on the right track, you'll get run over if you just sit there.”

Will Rogers

3D Audio on the Right Track

It's said that when an early motion picture was first shown to the public, women fainted and men ducked from an approaching train. The director made a bold new decision that would alter the course of filmmaking for the next century. Instead of just placing the camera in front of all the action like an audience watching a stage, the director moved the camera to a new position - within the action - to create perspective. There were more changes on the way. About a hundred years ago, the first color and 3D films were being created. In an Avant Garde era when artists were distorting reality, most filmmakers were trying to recreate reality and immerse the viewers into it.

Realistic sound is no exception. In fact, stereo sound was first demonstrated in 1881, and multiple speaker playback in the early 1930's. These early attempts at locating sounds for the listener were impractical though, and required expensive and ungainly equipment. By the late 1950's, consumers could finally experience stereo with LPs and tapes. But until home theater systems became popular, most had to go to a place like a theater or theme park to experience anything beyond stereo sound.

Since digital audio and video have pretty much taken over the world, we've seen a rapid growth of new technologies. Virtual reality is one of those, like the Oculus Rift, a VR headset for the masses. This is a visual VR device. But what's lacking is auditory VR.

Enter the researchers at Microsoft. They've created a way to fool a listener into thinking sounds are coming from a specific location using ordinary headphones. To make it work, the listener's head and shoulders are scanned into 3D software, which then builds a custom filter. Add motion sensors and a camera to track the listener's position, and the fun begins. Sounds can seem to be coming from objects or areas in a room. Imagine making that stuffed Teddy bear in the corner of your child's room actually talk. That model hot rod on the shelf could rev its engine. The uses seem endless.

It's obvious that this new device will be used as a companion to a visual VR headset. But there could be many more uses. Theme park attractions could be revolutionized by creating more realistic environments while eliminating costly sound systems. The movie industry could enhance the viewer's experience. Music artists could create the ultimate mix, with that lead guitarist standing next to the coffee table.

I think the largest user of this technology will be the advertising industry. Imagine walking down the street with your headphones listening to some music when a voice from your left whispers, "Pssst! Hey buddy, wanna buy a watch?" You turn and see the sparkling Bulova timepiece in the window of the jewelry store.


Train photo courtesy Simon Pielow

Did You Know?

• In Paris in 1881, Clément Ader demonstrated the first two-channel sound utilizing telephone transmitters. As part of an exhibition, the Paris Opera performances were heard at a remote location by listeners wearing headphones. Scientific American immediately picked up on the aural spaciousness compared to a single receiver.
• Adre's process was later marketed in France for over 40 years as the Théâtrophone. These coin-operated devices were popular in hotels, cafés, and clubs. There were also many homes that had the Théâtrophone by subscription.
• Similar services were the Electrophone in England and the Telefon Hírmondó in Budapest. In the U.S., there were only one-time experiments.
• The Théâtrophone service also included short news readings, making it a prototype to the telephone newspaper as well as the first example of electronic broadcasting.
• The popularity of radio and the phonograph killed the Théâtrophone by 1932.

Tech Notes

One of the fundamentals of recording and mixing is deciding what kind of spatial experience the listener will have. Creating a realistic one is difficult. If accuracy is not adhered to strictly, the result is a false space. An example of a near realistic auditory recording and playback is to place two microphones closely together in front of a performance. During playback, the listener is positioned in front of two loudspeakers at about the same angle and distance as the microphone were placed. This is a fairly pleasing experience, but isn't 100% accurate because everyone's playback system and environment is different.

A false spatial method is called "pan-pot" stereo. Mono or stereo tracks are "panned" in the mix away from center. An example might be a cowbell panned slightly to one side ('More cowbell!"). Most contemporary pop and rock songs are constructed this way because the tracks were built one performer at a time.

Recording engineer and producer Bruce Swedien (who recorded and mixed the biggest-selling album of all time, Michael Jackson's Thriller) likes to pre-plan his stereo mix. If Swedien wanted that cowbell on the left side of the mix, he would record it in stereo with the performer standing to the left. In the analog tape days this required careful planning because engineers were limited to a finite number of tracks. Most recorders had 16 or 24 tracks. If all instruments are recorded separately in stereo, then that limited the tracks to 8 or 12.

Most stereo recordings are intended to be heard on speakers. Once the listener uses headphones, the aural space changes dramatically. Instead of left and right sounds coming from forward of the listener (from the speakers), they are now coming from extreme left or right. Anything that was intended for the front middle (like the vocalist or lead guitar), is now coming from inside the head, right in the middle. It's a hyper-real experience, and immediately going back to front speakers elicits a lifeless experience.

There is one recording technique that requires listening only on headphones. This ultimate 2-channel spatial experience is called binaural recording. Two microphones are placed either in a mannequin head with false ears, or over the recordist's ears. Whatever the recordist hears is correctly translated to the listener. A common use for binaural recordings is an audio tour. How about a walk through Chinatown on a busy Monday morning? Or a walk through a submarine as she makes her dive to 300 feet. The listener is instantly transported to the locale.

I did this on a project that required me to be in a crowd at a sporting event. I wore headphones that had small microphones (lavalieres) clipped on the outside. To the unsuspecting person, I was just someone listening to some beats. To the listener, I was their portal to the game.

Some resources to listen to binaural recordings (be sure to use your headphones):

A studio music performance from the videographer's perspective

Binaural mixes on Sound Cloud

Sounds of Bach

Learn more about binaural recording

Neil Kesterson

Tags: 3D, virtual reality, surround sound, Dynamix, Dynamix Productions, microsoft, Oculus Rift