Web Audio API

The Modular Synth And Sound Processor at Your Fingertips

Philipp Stucki

@_phst

Web Audio API Overview

• Very high level
• Modular routing
• Runs in its own thread
• Low latency
• Parameter automation
• Output nodes: oscillator, sample player, live audio
• In/out nodes: filter, convolver, wave shaper, compressor
• Audio processing in JavaScript
• Spatialized audio for 3D effects
• Real time audio analysis

AudioContext

• Creates all AudioNodes
• Contains the routing graph
• AudioDestinationNode is the final node
• Usually only a single AudioContext is used

AudioContext Interface

                        interface AudioContext {

    readonly attribute AudioDestinationNode destination;

    // (...)

    OscillatorNode createOscillator();
    GainNode createGain();
    DelayNode createDelay(optional double maxDelayTime = 1.0);
    BiquadFilterNode createBiquadFilter();
    ConvolverNode createConvolver();

    // (...)

};
                    

AudioNode

• Building block of an AudioContext
• Implemented by all input/output and processing nodes
• Output can be connected to multiple nodes (fan-out)
• Multiple nodes can be connected as inputs

AudioNode Interface

                        interface AudioNode {

    void connect( ... );
    void disconnect( ... );

    readonly attribute AudioContext context;

    // (...)

};
                    

Scheduling Audio Events

• Audio events are relative to the context's currentTime
• currentTime starts at zero on context creation and increases in real-time
• currentTime cannot be stopped or paused

Example: Oscillator

Open example

- Explain routing graph
- Show Example
                    

OscillatorNode Interface

                        interface OscillatorNode : AudioNode {

    // (...)

    readonly attribute AudioParam frequency;

    void start(double when);
    void stop(double when);

    // (...)

};
                    

Audio Parameters

• Affect aspects of an AudioNode's processing
• Value changes can be immediate
• Value changes can be automated
• a-rate parameter: value is sampled on a per-sample basis
• k-rate parameter: value is sampled on a per-block basis

AudioParam Interface

                        interface AudioParam {

    attribute float value;

    void setValueAtTime(float value, double startTime);
    void linearRampToValueAtTime(float value, double endTime);

    // (...)

    void cancelScheduledValues(double startTime);

};
                    

- direct value change
- set value at given time, remains constant
- set value using continous function
- cancel schedule values
                    

GainNode Interface

                        interface GainNode : AudioNode {

    readonly attribute AudioParam gain;

};
                    

- interesting property: negative gain
                    

Example: Oscillator With Envelope

Open example

BiquadFilterNode Interface

                        enum BiquadFilterType {
    "lowpass", "highpass", "bandpass", "lowshelf", (...)
};

interface BiquadFilterNode : AudioNode {

    attribute BiquadFilterType type;
    readonly attribute AudioParam frequency;
    readonly attribute AudioParam Q;
    readonly attribute AudioParam gain;

    // (...)

};
                    

- shortly explain filter types
- note how all of the parameters are AudioParams
                    

Example: Oscillator, Envelope, Filter

Open example

Filter: Frequency Response

Example: Frequency Modulation

Open example

Aside

JavaScript Typed Arrays

- Used in many APIs: WebGL, Web Audio API, Canvas, etc.
                    

ArrayBuffer

Generic chunk of data, provides no direct access

ArrayBufferView

Interfaces providing context on a ArrayBuffer:

• type
• offset
• number of elements

Example Types

Typed arrays are implementations of ArrayBufferView

• Float32Array
• Int32Array, Int16Array, Int8Array
• Uint32Array, etc.

Float32Array in Action

                        > a = new Float32Array(4);
  [0, 0, 0, 0]

> a[0] = Math.PI;
> a;
  [3.1415927410125732, 0, 0, 0]

> a.length;
  4

> a.buffer;
  ArrayBuffer {}

> a.buffer.byteLength;
  16

                    

Loading Audio Data

                        interface AudioContext {

    // (...)

    void decodeAudioData(ArrayBuffer audioData,
                         DecodeSuccessCallback successCallback,
                         optional DecodeErrorCallback errorCallback);

    // (...)

}
                    

Loading Audio Data

                        function loadBuffer(url, cb) {
    var r = new XMLHttpRequest();

    r.open('GET', url, true);
    r.responseType = 'arraybuffer';

    r.onload = function() {
        ctx.decodeAudioData(r.response, function(buffer) {
            cb.call(this, buffer);
        });
    };

    r.send();
}
                    

AudioBuffer Interface

                        interface AudioBuffer {

    Float32Array getChannelData(unsigned long channel);

    // (...)

};
                    

- in memory audio asset
- raw and uncompressed
- range is [-1;1]
                    

AudioBufferSourceNode Interface

                        Interface AudioBufferSourceNode : AudioNode {

    attribute AudioBuffer? buffer;

    readonly attribute AudioParam playbackRate;

    attribute boolean loop;
    attribute double loopStart;
    attribute double loopEnd;

    void start(...);
    void stop(...);

};
                    

- stop can only be called once
- throw-away objects
- buffer remains, is only assigned
                    

Example: Sample Player

Open example

Convolution

- mathematical operation on two functions
- produces a third function
- think: blending
                    

Impulse Response

ConvolverNode Interface

                        interface ConvolverNode : AudioNode {

    attribute AudioBuffer? buffer;

    // (...)

};
                    

Example: Convolution

Open example

ScriptProcessorNode

                        interface AudioContext {

    // (...)

    ScriptProcessorNode createScriptProcessor(...);

    // (...)
}
                    

ScriptProcessorNode Interface

                        interface ScriptProcessorNode : AudioNode {

    attribute EventHandler onaudioprocess;

    // (...)

};
                    

AudioProcessingEvent Interface

                        interface AudioProcessingEvent : Event {

    readonly attribute double playbackTime;
    readonly attribute AudioBuffer inputBuffer;
    readonly attribute AudioBuffer outputBuffer;

};
                    

AudioBuffer Interface

                        interface AudioBuffer {

    // (...)

    Float32Array getChannelData(unsigned long channel);

};
                    

Example: White Noise Generator

Open example

Example: Polyphonic Synth With Sequencer...

and Graphics ...

in 1K!

Open example

A Square Wave

Example: Square Oscillator in One Line

Open example

Open OscillatorNode example

Web Audio API Spec on OscillatorNode

Bandlimited Square, Time Domain

Fourier Transform

Bandlimited Square, Frequency Domain

Inverse Fourier Transform

Bandlimited Square, Time Domain

Oscillators in the Web Audio API

• Based on Inverse Fourier Transform
• Bandlimited (have limited harmonics)
• Use multiple precalculated wavetables

OscillatorNode Interface

                        enum OscillatorType {
  "sine",
  "square",
  "sawtooth",
  "triangle",
  "custom"
};

interface OscillatorNode : AudioNode {

    void setPeriodicWave(PeriodicWave periodicWave);

    // (...)

};
                    

PeriodicWave Interface

                       interface PeriodicWave {
};
                    

AudioContext.createPeriodicWave

                        interface AudioContext {

    PeriodicWave createPeriodicWave(Float32Array real, Float32Array imag);

    // (...)
}
                    

DelayNode Interface

interface DelayNode : AudioNode {

    readonly attribute AudioParam delayTime;

};
                        

Example: Delay

Open example

Browser Support

• Chrome: Yes
• Firefox: Yes
• Safari: Yes
• Opera: Yes
• iOS Safari: Yes
• Android Chrome: Yes
• Android Firefox: Yes
• Internet Explorer: Sorry...

What else?

DynamicsProcessor

Waveshaper

ChannelSplitter, ChannelMerger

3D Spatialization

Analyser

Next?

• Check out the examples collection
• Build your own applications
• Live input
• WebRTC
• Web MIDI API
• OfflineAudioContext

Possible improvements

• Audio hardware configuration (interface, sample rate, etc.)
• Audio Parameters for ScriptProcessorNodes

Questions?

Philipp Stucki / @_phst

Arrivederci!

Philipp Stucki / @_phst