Sound Will Age You

JSConf 2013

@BlaineBublitz

gh: phated

@IcedDev

frozenjs.com

hackPHX.com

Gather Around

In Ancient Times

The Proprietary Ones

Horrible Things

<embed src="epic-theme.mp3"></embed>

<object data="epic-theme.swf" type="application/x-shockwave-flash">
  <param name="movie" value="epic-theme.swf" />
</object>

Warriors

Delivered Unto Us

Multimedia

Most Explored

<audio src="epic-theme.mp3"></audio>

<video src="epic-intro.mp4"></video>

Sunny Fields

Few Ventured

into Unexplored Lands

WebAudio vs HTML5 Audio
Codecs
Mobile

Donned Adventurer's Gear

Before My Quest Began

Before my quest had even begun

Sound Will Age You

Sounds are Important

Sounds are Iconic

Sounds are Hard

My Perspective

Let's Look At

WebAudio vs HTML5 Audio
Codecs
Mobile

Best Foot Forward

First implementation of Sound in Frozen - WebAudio only, otherwise nothing

Something Right

Started with WebAudio

Something Horribly Wrong

Only WebAudio

And hope browser support increases

Browser Support?

Firefox Bug | Android Bug

Has been in Chrome since v10, just now getting to iOS and still not in Chrome for Android or Firefox

Start with WebAudio

What Do We Get?

High Performance
Low Latency
Uses Buffers instead of Resource Loading

But Fall Back

Browser support

Initialization

WebAudio

var audioContext;
if(window.AudioContext){
  audioContext = new window.AudioContext();
}

HTML5 Audio

var audio = new Audio();

Loading

WebAudio

var audioBuffer;
if(audioContext){
  var request = new XMLHttpRequest();
  request.open('GET', 'epic-theme.mp3', true);
  request.responseType = 'arraybuffer'; // Need to get an ArrayBuffer

  function decodeAudioData(e){
    // async decode
    audioContext.decodeAudioData(e.target.response, function(buffer){
      audioBuffer = buffer;
    }, onError);
  }

  request.addEventListener('load', decodeAudioData);
  request.send();
}

Loading

HTML5 Audio

audio.src = 'epic-theme.mp3';

Playing

WebAudio

// Place to attach our buffer
var source = audioContext.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioContext.destination); // destination = output
source.start(0); // source.noteOn(0);

HTML5

audio.play();

Volume

WebAudio

var gainNode = audioContext.createGain();
source.connect(gainNode);
// Connect gainNode to destination instead of source
gainNode.connect(audioContext.destination);

gainNode.gain.value = 0.5; // Change gain (volume)

HTML5

audio.volume = 0.5;

So Complicated

Extras!

Mixing, Processing, Filtering
Smooth Transitions
Sound Creation
Modular - Node System
Designed for Future Features

Turn This

Into This

Codecs

canPlayType

var audio = new Audio();
audio.canPlayType('audio/webm');
// "maybe"

audio.canPlayType('audio/webm; codecs="vorbis"');
// "probably"

audio.canPlayType('audio/obscure-type');
// ""

* Chrome on OSX

Full Browser Support

MP3
- Chrome
- IE
- Safari
- Some FF
WebM (or Ogg)
- Chromium
- FF
- Opera
- Chrome, Safari also support these

Which File?

var audio = new Audio();

var sounds = [
  { type: 'audio/mpeg', filename: 'epic-theme.mp3' },
  { type: 'audio/webm', filename: 'epic-theme.webm' }
];

// Try the first sound that doesn't return ''
sounds.some(function(sound){
  return audio.canPlayType(sound.type) && (audio.src = sound.filename);
});
// Chrome, IE, Safari, Some FF: audio.src === 'epic-theme.mp3'
// Chromium, FF, Opera: audio.src === 'epic-theme.webm'

Fiddle

Don't Silence Me

audio.addEventListener('error', function(){
  // Filter the current sound from list
  sounds = sounds.filter(function(sound){
    return audio.src.indexOf(sound.filename) < 0;
  });

  // Try the next sound that doesn't return ''
  sounds.some(function(sound){
    return audio.canPlayType(sound.type)
      && (audio.src = sound.filename);
  });
});

Fiddle

Maybe Try Probably

var sources = sounds.reduce(function(result, sound){
  var canPlay = audio.canPlayType(sound.type);
  // Unshift the probablies onto beginning of the array
  if(canPlay === 'probably') result.unshift(sound.filename);
  // Push the maybes onto the end of the array
  if(canPlay === 'maybe') result.push(sound.filename);
  return result;
}, []);

audio.addEventListener('error', setSrc);

// Use the first sound on the array - probablies before maybes
function setSrc(){
  sources.length && (audio.src = sources.shift());
}

setSrc();

Fiddle

Function-ize

function load(filename){
  var audio = new Audio();
  // Replace static filenames with the variable passed in
  var sounds = [
    { type: 'audio/mpeg', filename: filename + '.mp3' },
    { type: 'audio/webm', filename: filename + '.webm' }
  ];
  // All That Boilerplate
  return audio;
}

load('epic-theme').play();
load('sad-song').play();

Fiddle

It's My Party And I'll AMD If I Want To

define([
  'frozen/plugins/loadSound!epic-theme'
], function(theme){

  // theme will be a Sound object based on Web Audio,
  // HTML5 Audio, or stubbed out if neither available
  // It will also try to use the best available codec

  theme.play();

});

Fiddle

Frozen uses AMD plugins to make loading and using resources easier Can pass a filename without extension - auto determine which codec to use Also determines if WebAudio is available - defaults to that Then HTML5 Audio Otherwise stubbed out sound A lot of these headaches are already abstracted away from you

What's Expected?

audio.play()

audio.load()

media.load()

Causes the element to reset and start selecting and loading a new media resource from scratch.

Spec

media.play()

Sets the paused attribute to false,

loading the media resource

and beginning playback if necessary. If the playback had ended, will restart it from the start.

Spec

Don't Need No Stinkin' Spec

Excuses, Excuses

"In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it. This means the JavaScript play() and load() methods are also inactive until the user initiates playback, unless the play() or load() method is triggered by user action. In other words, a user-initiated Play button works, but an onLoad="play()" event does not."

Source

On the Bandwagon

"This is intended. Autoplay is not honored on android as it will cost data usage."

Source

Meanwhile

We can download terabytes of images without any restrictions

Workaround

var audio = new Audio();

audio.src = 'epic-theme.mp3';

function preloadSound(){
  audio.load();
  document.removeEventListener('touchstart', preloadSound);
}

document.addEventListener('touchstart', preloadSound);

Fiddle

All Is Good

Nothing As It Seems

We Can Do Better!

var audio = new Audio();

audio.src = 'epic-theme.mp3';

function preloadSound(){
  var vol = audio.volume;
  audio.volume = 0;
  audio.play();
  audio.pause();
  audio.volume = vol;
  document.removeEventListener('touchstart', preloadSound);
}

document.addEventListener('touchstart', preloadSound);

Fiddle

Solved?

A New Hope

Streams

Yay Buzzwords

Audio Streams

Layering of Sounds
Complex Audio Streams

For games, we want to use multiple layers of sounds Background track, clock ticking in the corner, footsteps, gun firing Achieve immersion Need these complex audio streams

sound-will-age-you

iceddev

sound-will-age-you

0 1 (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/platform.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();