Running a production Play! service in the cloud

Scott Clasen / @scottclasen / Heroku API Team

Play! at Heroku

We run several production services based on Play! / Scala, since the days of

addSbtPlugin("play" % "sbt-plugin" % "2.0")

Play! at Heroku

This talk summarizes some of the tooling and best practices we have developed.

Many of them also codified in this template app sclasen/cloudplay

Or this library sclasen/play-extras

preface: building heroku on heroku

we try to the extent possible to build Heroku on Heroku

preface: building heroku on heroku

we call this ongoing effort: ephemeralization

preface: building heroku on heroku

we try to the extent possible to build Heroku on Heroku

we call this ongoing effort: ephemeralization

all of our Play! services fall into this category

preface: the 12 factor app

this talk touches on several of the concepts described in http://12factor.net/

The Twelve Factors

1-6

Codebase: One codebase tracked in revision control, many deploys
Dependencies: Explicitly declare and isolate dependencies
Config: Store config in the environment
Backing Services: Treat backing services as attached resources
Build, release, run: Strictly separate build and run stages
Processes: Execute the app as one or more stateless processes

The Twelve Factors

7-12

Port binding: Export services via port binding
Concurrency: Scale out via the process model
Disposability: Maximize robustness with fast startup and graceful shutdown
Dev/prod parity: Keep development, staging, and production as similar as possible
Logs: Treat logs as event streams
Admin processes: Run admin/management tasks as one-off processes

preface: basic tools

we use some basic tooling whether an app is ephemeralized, or not

Procfile / .env

foreman / forego

Procfile

web:           bin/web
continuous:    bin/continuous

$ forego start web
// starts the web process, capture stdout
$ forego start
// start web and continuous processes, capture stdout on both

.env

FOO=foooo
BAR=barrr

$ echo "foo: $FOO bar: $BAR"
foo: bar:
$ forego run echo "foo: $FOO bar: $BAR"
foo: foooo bar: barrr

preface: basic tools

heroku config plugin

heroku config:pull -o -a app-dev
//pull the config from app-dev and write it to your .env
heroku config:push -o -a app-dev
//write your .env to the config vars of the app

on with it!

Continuous Deployment
Functionality not fully provided by Play!
Visibility

Continuous Deployment

Pipelines!

Continuous Deployment

Pipelines!

https://devcenter.heroku.com/articles/labs-pipelines

backed by a Play! service as a matter of fact.

Pipelines + Jenkins + Github Heroku and Hipchat plugins

Continuous Deployment: app pipeline

mostly focused on integration testing

standard dev -> staging -> prod setup

dev + staging have app config + test config in their env

Continuous Deployment: jenkins

4 jobs per service

1: app-master-to-dev

triggered by a github webhook

heroku jenkins plugin: heroku push buildstep

build app and deploy to dev

Continuous Deployment: jenkins

4 jobs per service

2: app-test-dev

run integration tests from jenkins, hitting dev

heroku config:pull -o -a app-dev

overwrites .env file locally

foreman run play test

Continuous Deployment: jenkins

4 jobs per service

3: app-promote-dev-staging

heroku pipeline:promote -a app-dev

thats it!

Continuous Deployment: jenkins

4 jobs per service

4: app-test-staging

same pattern as dev

run integration tests from jenkins, hitting staging

heroku config:pull -o -a app-dev

overwrites .env file locally

foreman run play test

Production deploy

heroku pipeline:promote -a app-staging

thats it!

on with it!

Functionality not fully provided by Play!

Functionality not fully provided by Play!

Batch/Background Processing

Evolutions/Migrations

Credentials Handling/Encryption

Batch/Background Processing

How to keep things as simple as possible?

-Dapplication.global FTW

You have all your models, services, logic

you have play akka etc

Batch/Background Processing

web:           bin/cloudplay -Dapplication.global=Web
continuous:    bin/cloudplay -Dapplication.global=Continuous

Batch/Background Processing

trait ProcessType extends GlobalSettings {
 ...
}
object Web extends ProcessType  {
    override def onStart(app: Application) = {
        log.info("Starting ProcessType: Web")
        start(app, "web")
    }
}

object Continuous extends ProcessType {
    override def onStart(app: Application) = {
        log.info("Starting ProcessType: Continuous")
        start(app, "continuous")
        processes.ContinuousProcess.start()
    }
}

Batch/Background Processing

Ok, great whats in this processes.ContinuousProcess.start()business?

Lets call it an 'actor batch process'

Run multiple instances, using postgres advisory locks where necessary.

Lets take a look at some code!

Functionality not fully provided by Play!

Evolutions/Migrations

Seldom mentioned Evolution features in play

-DapplyDownEvolutions.database=true

-Devolutions.useLocks=true

* locks require a db that supports `select for update nowait`

Evolutions/Migrations

Evolutions: provided by play

Migrations: code/sql you write

Evolutions/Migrations

Hot Compatibility

So here’s the basic principle that allows you to avoid downtime:

any evolution or migration being deployed should be compatible with the code that is already running.

Evolutions/Migrations

Hot Compatibility

In order to do so, you’ll usually split your deploy process in two steps:

Make the code compatible with the evolution/migration you need to run

Run the evolution, run the migration, and remove any code written specifically for it

Migrations

How to write/run?

-Dapplication.global FTW again

object Migration extends ProcessType {
     override def onStart(app: Application) = {
         log.info("Starting ProcessType: Migration")
         start(app, "migration")
         processes.MigrationProcess.start()
     }
 }

 //add this to procfile
 migration: bin/cloudplay -Dapplication.global=Migration

Migrations

Run the migration with: `heroku run migration`

Lets look at the code for processes.MigrationProcess.start()

Functionality not fully provided by Play!

Credentials Handling/Encryption

Generally two types of credentials in an app.

User credentials/passwords that you dont need to know, only verify.

Other credentials that you need to know in unencrypted form.

Credentials Handling/Encryption

User's Passwords

use bcrypt

or the util methods in sclasen/play-extras

which use bcrypt

Credentials Handling/Encryption

User's Passwords

CredentialsService.hashPassword

CredentialsService.checkPasswordAgainstHash

Credentials Handling/Encryption

Credentials you need in unencrypted form

database passwords, aws keys, api keys, etc

Credentials Handling/Encryption

Credentials you need in unencrypted form

If an attacker has both your code and runtime config, you are basically hosed.

So, make it such that if the attacker has only one or the other, you are safe

Credentials Handling/Encryption

Encrypting a credential

> sbt console //outputs omitted
scala>  import com.heroku.play.api.libs.security._

scala> val secret = "the secret password"

scala> val secretKey = CredentialsService.generateKey

scala> val keyMaskThatGoesInYourCodebase = CredentialsService.generateKey
// add to application.conf   secret.key.mask=keyMaskThatGoesInYourCodebase
scala> val maskedSecretKeyThatGoesInYourENV =
    CredentialsService.maskKey(secretKey, keyMaskThatGoesInYourCodebase)
// add to env  heroku config:set SECRET_KEY=maskedSecretKeyThatGoesInYourENV
scala> val encryptedSecretThatGoesInYourENV =
    CredentialsService.doEncryptCredential(secret, secretKey)
// add to env  heroku config:set SECRET=encryptedSecretThatGoesInYourENV

Credentials Handling/Encryption

Decrypting a credential

CredentialsService.decryptCredential takes the names of 2 env vars and 1 property from applicaiton.conf

> forego run sbt console //outputs omitted
scala>  import com.heroku.play.api.libs.security._

scala>  val secret =
    CredentialsService.decryptCredential("SECRET",
                                         "SECRET_KEY",
                                         "secret.key.mask")

* there is also a doDecryptCredential form for encrypted values that arent/cant be stored in the env.

on with it!

Visibility

Visibility

Logs as Event Streams

Request-Ids

Splunk, Librato

Actor Based Logback Appender

Logs as Event Streams

Factor #11 from 12 Factors

At scale you want to optimize more for machine readability

We do this at heroku with `logfmt`

Logs as Event Streams

Factor #11 from 12 Factors

At scale you want to optimize more for machine readability

We do this at heroku with `logfmt`

Logs as Event Streams

logfmt http://godoc.org/github.com/kr/logfmt

ident_byte = any byte greater than ' ', excluding '=' and '"'
string_byte = any byte excluding '"' and '\'
garbage = !ident_byte
ident = ident_byte, { ident byte }
key = ident
value = ident | '"', { string_byte | '\', '"' }, '"'
pair = key, '=', value | key, '=' | key
message = { garbage, pair }, garbage

Logs as Event Streams

//LOGFMT
foo=bar a=14 baz="hello kitty" cool%story=bro f %^asdf

//parsed to json
{ "foo": "bar", "a": 14, "baz": "hello kitty",
    "cool%story": "bro", "f": true, "%^asdf": true }

Logs as Event Streams

Which is easier to process?

ERROR user scott@heroku.com is over the account-rate-limit


error app=an-app at=over-account-limit user=scott@heroku.com

Running a production Play! service in the cloud

sclasen

Running a production Play! service in the cloud

0 0 (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/platform.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

cloudplay-talk