Streams

What are they?

• Working with String and Buffer (objectMode available)
• Processing HUGE amount of data on-the-fly
• Pipes&Filters model

Under the hood

Event emitter

• Observer pattern
• .on('name', fn)
• .emit('name'[, arg1, ...])

Event emitter leaks

Event emitter leaks

• Technically not a leak. But it is.
• Clean up after yourself!
• .once(), if possible

Backpressure

• Slow writable vs fast readable

For consumers

Reading

• 'data' event (flowing mode)
• .read(bytes) (non-flowing mode)

(Switch between modes with .pause() and .resume())

Writing

• .write(chunk, [enc], [cb])
• .end([chunk], [enc], [cb])

Pipes!

• $ cat in.txt | compress | encrypt > out.txt

• fsReadStream
    .pipe(compress)
    .pipe(encrypt)
    .pipe(fsWriteStream);

Complex example: Stream Playground

An entire build system built on pipes: Gulp

For implementors

• ._read(bytes) for readable, .push(chunk) to the internal buffer
• ._write(chunk, enc, cb) for writable
• ._transform(chunk, enc, cb) for transform

Example transform stream

var f = function(chunk, enc, cb) {
   var string = chunk.toString();
   var upper = string.toUpperCase();
  this.push(upper, enc);
  cb();
};
MyTransform.prototype._transform = f;

WTF?

• They represent the result of a pending operation (Deferred object)
• They were part of node a while back ago
• They are part of ES6
• It's a standard! (Promises/A+)
• jQuery has a broken implementation :(
• I don't like them! :(

In a nutshell

function sleep(ms) {
  var d = Q.defer();
  setTimeout(function() {
    d.resolve();
  }, ms);
  return d.promise;
}
sleep(1337).then(function() {
 console.log('done! :3');
});

But that's not magic! :(

The Magic

Chaining!

a1().then(function(val) {
  return a2(val);
}).then(function(val) {
  return a3(val);
}).then(function(val) {
  console.log('done!');
}, function(err) {
  console.log('err!');
});

try {
  var v = a1();
  v = a2(v);
  v = a3(v);
  console.log('done!');
} catch(e) {
  console.log('err!');
}

• Callback-Hell avoided!
• Errors propagate
• Libraries (Q, etc) can provide stack-trace & basic control flow (forks/join, etc)!

Too much boilerplate code :(

The idea

magic(function* (resume) {
  try {
    var result = yield fs
      .readFile('./existing.file', resume);
    console.log(result);
    var result2 = yield fs
      .readFile('./missing.file', resume);
  } catch(e) {
    console.error(e);
  }
});

The magic

simplified version of genzen (my library)

function magic(f) {
  var g = f(function(err, result) {
    if(err) {
      return g.throw(err);
    }
    g.next(result);
  });
  g.next();
}

Use a robust library, like co, galaxy, suspend, etc. They work with promises, can do basic control flow, etc...