Bluster

Adventures with dead simple cluster management

@matt_mcclure

Before we get started

Just to get a few things straight

Bluster solves a current problem

It's made our development life a lot easier

So...where is it?

We're holding out for bigger and better things

(Tectonic)

Let's go ahead and start talking about how we started using CoreOS and what led us to building Bluster in the first place

CoreOS is great at onboarding

Initial testing was all CloudFormation

Incredibly easy to spin up / tear down a quick test cluster.

But...

Autoscaling groups left a lot to be desired
Too much (AWS) Magic
We were considering other cloud providers ()

What we wanted in a solution

Cross Platform

Didn't need to do it all up front, but we wanted the option

Simple configuration

Needing to configuration per provider was fine

But it all needed to be human readable

Simple to add instances

We didn't require anything crazy in terms of auto scaling

We didn't require anything crazy for auto scaling, but we did need to be able to easily manually spin up new instances. When we did bring up those new instances, we wanted to be able to use the same configuration as the other machines and just specify the current CoreOS image.

Simple cluster creation

Wanted a cluster per environment

Might seem like a CoreOS anti-pattern, but we were the internal stepping stone.

Play nice with SysEng
Be able to run in the different VPCs
Be able to back out of CoreOS if we needed to

Simple cluster switching

Prepending FLEETCTL_TUNNEL gets old fast

Once we had multiple tunnels set up, needed to be able to quickly and easily switch between them.

Ok with direct interaction

Didn't mind using Etcd and Fleet directly

Lastly...

What we ended up building

$ bluster bootstrap

Specifying everything in the command itself was ok for about...never. Almost immediately switched to exclusively using --config.

Generated a custom cloud config for us based on the config used.

JSON configuration

{
  "vpc": "vpc-1234abcd",
  "securityGroup": "sg-1234abcd",
  "quay": "your-cool-quay-token",
  "iam": "paperboy-qa",
  "key": "some-keyfile",
  "advertised": "private",
  "current_machines": [
    "10.96.3.110",
    "10.96.4.184",
    "10.96.0.238"
  ],
  "complimentary_services": [
    "datadog.service",
    "redis.service",
    "smoke-qa.service",
    "sumologic-qa.service",
    "paperboy-qa-presence@.service"
  ],
  "primary_services": {
    "paperboy-qa@.service": 3
  },
  "discovery": "https://discovery.etcd.io/80af233780f9a5c66b772ccdb4772f7b"
}

If "discovery" isn't included, we go get a new one and add it to the file
"current_machines" needs to be added after the cluster is created (what we use for tunnels later)
"complimentary_services" and "primary_services" is still a little half baked

Simple Usage

$ bluster add-instance

Usage

$ bluster add-instance -c ops/foobar.json -n 1 talk-example ami-d2033bba

In the interest of time, I didn't record the terminal output. I think you can use your imagination on this one.

$ bluster etcd

Does almost nothing

Tunnels into the cluster and runs etcd commands.

Seriously. That's it. That's all we needed and it's been *immensely* useful.

Usage example

$ bluster fleet

TL;DR: Prepends FLEETCTL_TUNNEL to commands

Does even less

Literally prepends FLEETCTL_TUNNEL to commands

¯\_(ツ)_/¯

$ bluster deploy

Relies entirely on Fleet

Basically stops and starts service files (but waits for previous to start again)

We also rely on etcd for watching deployment and triggering registering / deregistering from load balancers.

Wonderful in theory

Generally feel more comfortable manually deploying

Talk about small scope of deploys, canary deploys, rollbacks, etc.

$ bluster initialize

Submits complimentary services, starts them, then deploys primary services

Easy to digest

{
  "complimentary_services": [
    "datadog.service",
    "redis.service",
    "smoke-qa.service",
    "sumologic-qa.service"
  ],
  "primary_services": {
    "paperboy-qa@.service": 3
  }
}

Kinda works

But when it doesn't, it's a bit of a bluster

Get it?

Bluster's Shortcomings

Deploys are, for the most part, manual.
It's a dumb tool. You have to know what you're doing.
Is that a feature?
No, it's a shortcoming. Easy to do dumb things (like submit template files).

Regret? Nope.

Learned a lot
Debugged a lot
Because of this, really wanted someone else to build a better solution
Because of this, really excited about Tectonic

Open sourcing it?

If you're interested in this, get in touch.

Council of Beards

Thank you

Questions? Comments? Ping me!

mmcclure@brightcove.com @matt_mcclure

Bluster – Before we get started – CoreOS is great at onboarding

mmcc