Important, but

out of scope:

• Redundant infrastructure.
• Backups.
• Disaster recovery.

In scope:

• Application errors.
• Deploys.
• Node.js stuff:
  • Domains.
  • Cluster.
  • Express.

Keys to 100% uptime.

1. Sensibly handle

uncaught exceptions.

2. Use domains

to catch and contain errors.

3. Manage processes

with cluster.

4. Gracefully terminate connections.

1. Sensibly handle uncaught exceptions.

Uncaught exceptions happen when:

• An exception is thrown but not caught.
• An error event is emitted but nothing is listening for it.

From node/lib/events.js:

EventEmitter.prototype.emit = function(type) {
  // If there is no 'error' event listener then throw.
  if (type === 'error') {
      ...
    } else if (er instanceof Error) {
      throw er; // Unhandled 'error' event
    } else {
      ...

An uncaught exception

crashes the process.

If the process is a server:

x 100s??

It starts with...

Domains.

2. Use domains to catch and contain errors.

try/catch doesn't do async.

try {
  var f = function() {
    throw new Error("uh-oh");
  };
  setTimeout(f, 100);
} catch (ex) {
  console.log("try / catch won't catch", ex);
}

Domains are a bit like

try/catch for async.

var d = require('domain').create();

d.on('error', function (err) {
  console.log("domain caught", err);
});

var f = d.bind(function() {
  throw new Error("uh-oh");
});

setTimeout(f, 100);

The active domain is

domain.active.

var d = require('domain').create();
console.log(domain.active); // <-- null

var f = d.bind(function() {
  console.log(domain.active === d) // <-- true
  console.log(process.domain === domain.active) // <-- true
  throw new Error("uh-oh");
});

New EventEmitters bind

to the active domain.

EventEmitter.prototype.emit = function(type) {
  if (type === 'error') {
    if (this.domain) {  // This is important!
      ...
      this.domain.emit('error', er);
    } else if ...

Log the error.

Helpful additional fields:

• error.domain
• error.domainEmitter
• error.domainBound
• error.domainThrown

Then it's up to you.

• Ignore.
• Retry.
• Abort (e.g., return 500).
• Throw (becomes an unknown error).

Do I have to create a new domain

every time I do an async operation?

Use middleware.

More convenient.

In Express, this might look like:

var domainWrapper = function(req, res, next) {
  var reqDomain = domain.create();
  reqDomain.add(req);
  reqDomain.add(res);

  reqDomain.once('error', function(err) {
    res.send(500); // or next(err);
  });

  reqDomain.run(next);
};

Based on https://github.com/brianc/node-domain-middleware https://github.com/mathrawka/express-domain-errors

node-mongodb-native does not

play well with active domain.

console.log(domain.active); // a domain
AppModel.findOne(function(err, doc) {
  console.log(domain.active); // undefined
  next();
});

See https://github.com/LearnBoost/mongoose/pull/1337

Fix with explicit binding.

console.log(domain.active); // a domain
AppModel.findOne(domain.active.bind(function(err, doc) {
  console.log(domain.active); // still a domain
  next();
}));

Cluster module.

Node = one thread per process.

Most machines have multiple CPUs.

One process per CPU = cluster.

master / workers

• 1 master process forks n workers.
• Master and workers communicate state via IPC.
• When workers want to listen to a socket, master registers them for it.
• Each new connection to socket is handed off to a worker.
• No shared application state between workers.

What about when a worker

isn't working anymore?

Some coordination is needed.

Worker tells cluster master it's done accepting new connections.

Cluster master forks replacement.

Worker dies.

Zero downtime deployment.

• When master starts, give it a symlink to worker code.

• After deploy new code, update symlink.

• Send signal to master: fork new workers!

• Master tells old workers to shut down, forks new workers from new code.

• Master process never stops running.

Signals.

A way to communicate with running processes.

SIGHUP: reload workers (some like SIGUSR2).

$ kill -s HUP <pid>
$ service <node-service-name> reload

Process management options.

Forever

github.com/nodejitsu/forever

• Has been around...forever.
• No cluster awareness — used on a single process.
• Simply restarts the process when it dies.
• More comparable to Upstart or Monit.

Naught

github.com/superjoe30/naught

• Newer.
• Cluster aware.
• Zero downtime errors and deploys.
• Runs as daemon.
• Handles log compression, rotation.

Recluster

github.com/doxout/recluster

• Newer.
• Cluster aware.
• Zero downtime errors and deploys.
• Does not run as daemon.
• Log agnostic.
• Simple, relatively easy to reason about.

Need to clean up:

• In-flight requests.
• HTTP keep-alive (open TCP) connections.

Revisiting our middleware from earlier:

var domainWrapper = function(afterErrorHook) {
  return function(req, res, next) {
    var reqDomain = domain.create();
    reqDomain.add(req);
    reqDomain.add(res);

    reqDomain.once('error', function(err) {
      next(err);
      if(afterErrorHook) afterErrorHook(err);  // Hook.
    });
    reqDomain.run(next);
  };
};

3. Then call process.exit

in server.close callback.

var afterErrorHook = function(err) {
  app.set("isShuttingDown", true);
  server.close(function() {
    process.exit(1);  // <-- all clear to exit
  });
}

Set a timer.

If timeout period expires and server is still around, call process.exit.

On startup:

• Cluster master comes up (for example, via Upstart).
• Cluster master forks workers from symlink.
• Each worker's server starts accepting connections.

On deploy:

• Point symlink to new version.
• Send signal to cluster master.
• Master tells existing workers to stop accepting new connections.
• Master forks new workers from new code.
• Existing workers shut down gracefully.

On error:

• Server catches it via domain.
• Next action depends on you: retry? abort? rethrow? etc.

On uncaught exception:

• ??

// The infamous "uncaughtException" event!
process.on('uncaughtException', function(err) {
  // ??
})

Node docs say not to keep running.

An unhandled exception means your application — and by extension node.js itself — is in an undefined state. Blindly resuming means anything could happen. You have been warned.

What to do?

First, log the error so you know what happened.

Then, you've got to

kill the process.

It's not so bad. We can now do so

with minimal trouble.

On uncaught exception:

• Log error.
• Server stops accepting new connections.
• Worker tells cluster master it's done.
• Master forks a replacement worker.
• Worker exits gracefully when all connections are closed, or after timeout.

Tip:

Be able to produce errors on demand

on your dev and staging servers.

(Disable this in production.)

Tip:

Keep cluster master simple.

It needs to run for a long time without being updated.

Things change.

I've been talking about:

{
  "node": "~0.10.20",
  "express": "~3.4.0",
  "connect": "~2.9.0",
  "mongoose": "~3.6.18",
  "recluster": "=0.3.4"
}

The Future:

Node 0.11 / 0.12

For example, cluster module has some changes.

Cluster is experimental.

Domains are unstable.