scaling-scala

View Github Repository Open presentation in a new window

flurdy

See all presentation from flurdy

scaling-scala

0 0

scaling-scala

Talk about scaling scala (people) for 2016.flatmap.no

On Github flurdy / scaling-scala

Scaling Scala

not performance but scaling people

by @flurdy

  • well funded
  • scale massively
  • some practices
  • some tools
  • some scars

Scaling Scala

not performance but scaling people

  • Scaling challenges
  • Best practices
  • Architecture
  • Coding style & frameworks
  • Versions
  • Pipeline, tooling & automation
  • Battle scars & Q&A

Best practices for scaling = best practices for non scaling

  • Preaching to the converted

  • Or worse/better already doing better things

  • scaling a company or department

  • lots of funding to scale massively

  • maybe startup or big budget allocation

  • best practices

  • quick rundown

  • Architecture

  • microservices

  • MS management tool

  • code style

  • how you scale affects this

  • Versions and upgrading tricky but essential

  • will show a tool sbt-bobby we have been using

  • Tools & automation

  • essential everywhere
  • but specifically when scaling

Not covering:

  • project management,
  • Recruitment,
  • Not mandating tech, just showing examples

Who am I

2 mins

  • Hampshire with my little girls
  • not the pigs
  • studied & worked in manchester
  • oslo too icy
  • Technical Architect, Tech Lead, Dev

Scaling Scala references

  • Startup from 50 to 200 employees in 1 year
  • Public sector grew from 5 to 50+ scala only teams
  • Enterprise scaling (non scala, outsource)

3 mins

  • startup quadrupled in 1 year

  • about 20 scala developers

  • very different scaling experience

  • public sector over 200 scala developers

  • Abstract says 30 but now 50

  • ie outsourcing

  • consultancy, telco & finance) mostly by outsourcing internally

Scaling Challenges

  • Massive funding expectations
  • Recruitment, staff turnover, onboarding
  • Parallel projects bottle necks
  • Delivery pipeline bottle necks
  • Code & framework conventions & proliferation
  • Testing consistency
  • Evolving ecosystem & keeping abreast
  • Tech debt bankruptcy

4 mins

dont forget scaling references slide

  • funding via VC or government budget, mandated to be used immediately

  • parallel work

  • recruitment immediately

  • talent pool
  • metioned later - affects code
  • scala mainstream means others are also scaling
  • turnover
  • more in hotbeds
  • 2 years in london is a long time
  • onboarding
  • done right brings people up to speed quickly

  • done wrong makes people leave

  • Chaotic rushed lava lamp architecture

  • Tech pivoting
  • Tech debt bankruptcy

  • Old school, new school, hipster school

  • tech pivot

  • spring, ddd, actor

  • abreast so can recruit

  • Evolving ecosystem internal and external
  • Forsaken projects

Assumption

Agile/Lean/etc, Continuous Integration, devops, pairing, TDD, integration testing, DVCS & tools, clouds/IAAS, distributed logging, etc

Common sense

8 mins

  • splunk, elestricsearch, logstash kibana
  • AWS/GCE
  • Git, github , gitlab, bitbucket
  • some that I will cover should probably be mentioned here instead

Best practices

  • Technical platform & team communications
  • Service catalog
  • Onboarding with pairing
  • Keep all services up to date
  • Microservices are cattle not pets
  • Automate it all

9 mins

  • tech coms across platform
  • blog of upcoming changes
  • pilot projects experience

  • announce mandatory / high risk changes

  • chat rooms

  • tech chats

  • Q&A chat rooms for each team/service

  • service

  • may be automated, but wiki page works
  • Documented approach to integration, changes
  • PRs, new mandatory integrations

  • services escalation

  • weekly tech show case

  • book clubs etc

  • Cull faltering services & API

  • no fat fingers

  • note this does not cover recruitment

  • more in architecture procedures

SKIPPABLE

Architecture

  • Simple
  • Uncoupled
  • Microservices
  • Release all day long

10 mins

  • micro services begin life as monoliths
  • standardised base service framework and libraries
  • roll forward API
  • pipeline covered more in next section
  • continuous delivery

Architecture

Procedures

  • Internal technical communication
  • One team owner per service
  • Pull request across teams
  • Cross team pairing
  • Wiki on PR, backlog, planning
  • PR branches are CI tested
  • github enterprise, travis, contribution.md

Microservices

  • Evolve by divide & conquer emerging monoliths
  • Keep'em dumb
  • Roll forward releases
  • Tooling
  • Local management & orchestration

13 mins

  • microservice is a known architecture
  • divide & conquer = more teams/people
  • Rinse - repeat
  • No big rewrite
  • StranglerApplication
  • Local management for dev and test

Microservices Testing

  • Spin up all dependencies
  • Including databases
  • Real, mocks, stubs
  • Releases, snapshots, source
  • Fast

14 mins

  • requirements

  • what you look for

  • including transitive dependencies

  • systems not library dependencies
  • wiremock, mockingbird
  • Configuration override
  • feature toggles
  • Override with local systems
  • Fast bit is hard

Microservices Testing

Service Manager

Local microservices management

github.com/hmrc/service-manager

  • Service catalog
  • Profiles (stacks) of services
  • Configuration overrides
  • Automatic dynamic integration test support
  • Snapshot binary or source

15 mins

  • can spin up stubs where needed
  • profiles: spin up stacks
  • swap individual services
  • configuration feature toggeles etc

Microservices Testing

Service Manager Problems

  • Lack service dependency
  • Download the world
  • No containers
  • Not that fast

16 mins

  • far from perfect
  • principles still apply
  • not advocating using it
  • writen in python

  • Not docker

  • Lack dependency : evolved profiles solves this

  • wiremock

Coding style

Company culture & recruitment

  • Ultra selective
  • Volume selective
  • Candidate pool
  • Staff turn over
  • Outsource

17 mins

This depends on:

  • company culture
  • compage/department size
  • startup, entrerprise,
  • big pool of candidates (SF,NY,London)
  • ability to find and recruit experience
  • Startup , Enterprise, Public sector,
  • staff turnover - pick up code quickly or not
  • outsource little control over skill, mostly as cheap as possible

Coding style

ROC: Readable, Optimised, Correct

  • Don't be too clever
  • Don't be stupid

Code for the next one

Share and evolve coding style

  • Pair
  • Informal workshops

19 mins

Not going to mandate full FP or just imperative java without semi colon

ROC: Chat with ex colleague (Noel)

  • CAP Theorem
  • consistency, availability, partition tolerance

This depends on:-

  • small company - its your baby
  • big company - its 30 others baby
  • easy to read
  • easy to maintain
  • clever, challenging code,
  • KISS, continual new java converts.
  • who will extend/fix it tomorrow/next quarter
  • make it a bit better
  • take pride in your work
  • pair two way learning

Frameworks

  • Standards / Convention or not
  • Best framework for the job per team/project (startup)
  • Same framework as others (public sector) New frameworks via pilot projects

21 mins

  • standardised per team
  • self starter
  • startp Play, Akka, Spray, Unfiltered, Fineagle, Spark
  • Spray and akka services,
  • play tools

  • node.js customer facing

  • large corp = play or akka

  • Dont stagnate

Versions

JDK, Scala version, library, plugins, API

Up to date across all projects & services

  • Continuous smooth evolution
  • Roll forward
  • Deprecate & remove API
  • Don't stagnate any projects
  • common problem when collaborating across teams and projects
  • Common micro service framework dependencies
  • Puppet, Docker
  • takes a lot of time and energy when scaling
  • roll forward api support
  • Not force upgrades on every change
  • Google announce braking API as comments in API responses
  • kill non upgradable services
  • No API set in stone forever

Versions

Practices & tools

  • Monorepo
  • SBT-Bobby
  • how to inform people of version changes

Monorepo

  • One source repository to rule them all
  • Instant version changes
  • Atomic commits across projects

Problems:

  • Developer fog / bankruptcy
  • Complicated parallel delta builds
  • Atomic commits across multiple services
  • Twitter, Google.
  • scaladays
  • heroku
  • Pants, blaze
  • 1000 of commits per day
  • smaller monorepo,
  • cvs/subversion days

SKIPPABLE

sbt-bobby

  • Sbt plugin: github.com/hmrc/sbt-bobby
  • Keeps versions up to date across all sbt projects
  • Set library versions as outdated
  • Fail builds
  • Grace period to upgrade
  • Build output warning
  • Part of CI build (at least)

25 mins

  • dont forget monorepo slide

  • libraries and plugins

  • outdated at a certain date future or past
  • No artifacts without it
  • or part of local build
  • not part of every build, often commented out

Bobby problems

  • Short deadlines
  • Unrealistic deadlines for complicated upgrades
  • Non advertised deadlines
  • Perfect storm before "code freeze"
  • still deliver business value
  • Dont abuse grace period
  • deadlines affect all teams
  • announce big upgrades in other channels
  • not just bobby warnings

Install Bobby

~/.sbt/0.13/plugins/build.sbt 
resolvers += Resolver.url( "hmrc-sbt-plugin-releases",
	url("https://dl.bintray.com/hmrc/sbt-plugin-releases")
)(Resolver.ivyStylePatterns)

addSbtPlugin("uk.gov.hmrc" % "sbt-bobby" % "0.30.0")
sbt validate include validate in automated build commands

Configure Bobby

~/.sbt/bobby.conf 
deprecated-dependencies = https://some-url/bobby-deprecated.json
deprecated-dependencies = file:///~/.sbt/bobby-deprecated.json
~/.sbt/bobby-deprecated.json 
{
  "libraries":[
	 {
	  "organisation": "com.example",
	  "name": "mylibrary",
	  "range": "(,6.0.0)",
	  "reason": "Security vulnerability before v6",
	  "from": "2016-06-23"
	 }
  ],
  "plugins":[]
}

Blacklist versions

Range Meaning (,1.0.0] Has to be newer than 1.0.0 [1.0.0] Can't use v1.0.0 [1.2.0,1.3.0] 1.2.0 <= x <= 1.3.0 [1.0.0,2.0.0) 1.0.0 <= x < 2.0.0 [1.5.0,) Has to be older than 1.5.0 [*-SNAPSHOT] No SNAPSHOTs - Note difference between [ and (. SKIPPABLE

Build fail

Tools & Automation

  • Permissions
  • Bootstrap
  • Pipeline & tools
  • tools = general tooling, support tools, investigation

Tools & Automation

Permissions

Everyone view

Easy write permissions

  • Allow all (forgiveness)
  • Easy self service (permission)

30 mins

Tools & Automation

Bootstrap

  • Steps on Wiki page
  • Automated scripts
  • Git repos
  • Source scaffold application
  • CI build jobs
  • Wiki space
  • Chat channels
  • Metrics, monitoring, alert

-bootstrap

  • wiki script then automated
  • Creates repo
  • scaffold code,
  • Gitter8, Activator, Git clone
  • alerts and monitoring
  • CI job
  • Graphite, Sensu, Pagerduty,

SKIPPABLE

Tools & Automation

Pipeline & tools

Delivery pipeline

  • Clear & easy
  • Automated
  • No bottle necks

One button / automate everything

  • Quick / easy / automated / reproducible / reduce risk
  • delivery
    • thoughtworks go
    • Jenkins 2.0
    • Spinaker
    • team city on steroids
  • Integrate all tools in chat
    • CI, repos, deploy tools, monitoring in chat
    • slack hipchat
  • total time ie cost saver

SKIPPABLE

Version control everything

  • Environments
    • Configuration, Puppet, Chef, Ansible, Vagrant, Docker

  • Build and deploy tools (CI)

    • Jenkins DSL

Scalable & reproducible

And test it!

  • minimise 100 fat finger via UI screw things up accidentally
  • prefer other CI (team city)
  • but jenkins servers are then reproducible in an instant
  • duplicate.
  • multiple CI instances etc

SKIPPABLE

Tools

One button

blog.flurdy/../one-button-aim-nearly-there

CI abuse (Team City)

  • Too many tools
  • Automatic or one button
  • For everything...
  • For all environments
  • No longer chores
  • Never abused

More and more tools

32 mins

  • tools: cli, deployinator, fabric, team city, puppet, scp, sbt, maven,

  • ci abuse = anti pattern

  • build, test, deploy, smoke test, perf test, promote, tag, release, rollback, redeploy, restart server, clean db, seed db, migrate db

  • emphasise not the abuse

  • no fat fingers

  • no searching for tool

  • Migrated all to scala

  • One binary
  • Env config in Git

  • Frequent Puppet runs

  • chores became easy

  • never abused

  • for all environments

  • Tools dashboards, investigations, payment plans promotion tru envs

Jenkins DSL

import javaposse.jobdsl.dsl.DslFactory
import uk.gov.hmrc.jenkinsjobs.domain.builder.SbtMicroserviceJobBuilder

new SbtMicroserviceJobBuilder('myamazeballs-service').
    withScalaStyle().
    withSCoverage().
   build(this as DslFactory)
./gradlew clean test
Seed job to activate
  • Groovy not scala
  • Initially created at a Javaposse roundup
  • ready made builders for many purposes

  • normal builds, devops jobs, wall displays

  • rebuild, duplicate jenkins

Battle scars & Q&A

  • Perfect storm of versions
  • Release bottle necks
  • Environment bottle necks
  • No feature flag development
  • Requirements & pull Request flood
  • Pull request graveyard
  • Too many, few, bad, old frameworks & tools
  • Dumb down too much

38 mins

Might too generic

  • no flags no parallel work
  • Frameworks not fit for purpose
  • Crap tools
  • ninjas that go too often off the reservation
  • clones that dont try to improve, dont spot mistakes
  • No communication before urgent work
  • dumb down , outsourced new to scala

SKIPPABLE

More information

SKIPPABLE

Scaling Scala not performance but scaling people by @flurdy well funded scale massively some practices some tools some scars