Streaming to You Live – The Node 0.10 Streams API – Streams in 0.8

View Github Repository Open presentation in a new window

EvanOxfeld

See all presentation from EvanOxfeld

Streaming to You Live – The Node 0.10 Streams API – Streams in 0.8

2 0

novanode-streams-0.10-presentation

Streaming to You Live: The Node 0.10 Streams API

On Github EvanOxfeld / novanode-streams-0.10-presentation

Streaming to You Live

The Node 0.10 Streams API

Created by Evan Oxfeld / @evanoxfeld

NodeJS

  • Server-side JS platform built on Chrome's V8 engine
  • Easily build fast, scalable network applications
  • Event-driven, non-blocking

I/O bound problems

  • E.g. database, LDAP, REST call, call to ext. service
  • You're reading from / writing to slow interfaces
  • I/O bound if the interfaces, not the CPU, not memory constrains the problem

Overview

  • Explore I/O bound problems in Node
  • Explore Streams API

You'll not only have the knowledge to use Streams, but also write your own

Streams Example - Unzip

fs.createReadStream('path/to/archive.zip')
  .pipe(unzip.Parse())
  .pipe(fstream.Writer('output/path'));
  • Similar to unix pipes
  • Streams are readable, writable, or both

Code!

Behind the curtain

Stream in node 0.8 inherits from EventEmitter

util.pump(readable, writable)

  • Sets up event handlers for 'data', 'end'
  • Ignores other events e.g. 'error', 'close'
  • API limits customization, not chainable

Streams in 0.8

	readable.pipe(duplex).pipe(writable)
  • Looks more like JavaScript - pipes are chainable
  • 0.8 was the stable version of node prior to 3/11/2013

Streams in 0.8

  • Readable
    • Emit data events
    • Optionally implement pause() and resume()
  • Writable
    • Implement write() and end()

Awesome! Let's demo!

Issues - Backpressure

Readable Writable

What happened?

  • Naive implementation didn't handle buffering and backpresure
  • Backpressure - stream signals to its source to stop sending data

Issues - Backpressure

Readable Writable

write() returns false; stream is full

Issues - Backpressure

Readable Writable

Writable emits 'drain' to signal it's ok to resume

Issues - Backpressure

More Issues

  • Buffering and backpressure
  • No on.('pipe') method
  • pause() isn't a guarantee
  • Backpressure is a dance, hyperactivity is bad
  • Data events start immediately (big problem at scale)
    • Data starts before event handlers set up

Streams of Tomorrow, Finally Here

Streams in 0.10

  • Readable streams are now "suck" streams
  • Can use Readable to wrap old-style streams
  • Object mode
  • Same composable pipe API
  • Shared base classes for backpressure and buffering
  • Readable/writable/duplex was one base class in Node 0.8
  • highWaterMark option (16kb default) determines how much data is buffered

Streams 0.10 Base Classes

stream.Readable

  • Implement readable._read(size)
  • Queue data from I/O source
    • push(chunk) or unshift(chunk)
  • Users call read([size]) or pipe(dest)
    • read() returns null if less data is buffered than size
    • If objectMode is true, read(n) returns one object

Streams 0.10 Base Classes

stream.Writable

  • Implement writable._write(chunk, encoding, cb)
  • Users call
    • write(chunk, [encoding], [cb])
      • write() returns false if the buffer is full
    • end([chunk], [encoding], [cb])

Streams 0.10 Base Classes

stream.Duplex

  • Inherits from Readable and Writable
  • Implement _read() and _write()
  • Queue data with push() and unshift()
  • Users call Readable and Writable methods
  • Prototypical inheritance from Readable
  • Parasitical inheritance from Writable

Streams 0.10 Base Classes

stream.Transform

  • Inherits from Duplex
  • Implement transform._transform(chunk, encoding, cb)
  • Users call Readable and Writable methods
  • Transform - output connected somewhat to input. Saves implementing both _read() and _write() * Use this.push and this.unshift() to queue data

Streams 0.10 Base Classes

stream.PassThrough

  • Inherits from Transform
  • No method to implement
  • Users call Readable and Writable methods

Example stream.Transform Code!

Unzip Demo

Other Relevant API Changes

process.nextTick()

process.nextTick happens at the end of the current tick, immediately after the current stack unwinds. If you are currently using recursive nextTick calls, use setImmediate instead.

More 0.8 to 0.10 API Changes

Example 0.8 Compatibility!

readable-stream module

Conclusions

  • Stream API is great for solving I/O bound problems
  • Streams2 developed in the open with a parallel user-land module
  • Now you can use and write streams. * Streams2 API overall easier to reason about * That said, spew streams style is occassionally useful * Unzip - entry.autodrain reads on every 'readable'
@substackStreams make programming in node simple, elegant, and composable.

References

Questions?