Just Enough Openstack for Docker

View Github Repository Open presentation in a new window

paulczar

See all presentation from paulczar

Just Enough Openstack for Docker

1 0

presentation-just-enough-docker-for-openstack

slides from seattle openstack day

On Github paulczar / presentation-just-enough-docker-for-openstack

Just Enough Openstack for Docker

* Rule one of getting a CFP accepted is to mention "docker"

OpenStack is a large collection of complicated projects that join together to form a hard to deploy and harder to run distributed system. This talk will show you how you can harness the concepts of bimodal IT to increase agility in your IT organization by utilizing a minimal Openstack install and various Docker based tools to build a platform that will increase developer productivity.

Rule two is to write a excerpt that doesn't really say anything meaningful

Openstack is a large collection of complicated projects that join together to form a hard to deploy and harder to run distributed system. This talk will show you how you can harness the concepts of bimodal IT to increase agility in your IT organization by utilizing a minimal Openstack install and various Docker based tools to build a platform that will increase developer productivity.

that is little more than a set of loosely coupled buzzwords.

Just Enough Openstack for Docker

  • Docker Docker Docker
  • Hard to deploy
  • Harder to run
  • distributed system
  • bimodal IT
  • increase agility in your IT organization
  • Minimal Openstack install
  • Docker based tools
  • build a platform
  • increase developer productivity

Just Enough Openstack for Docker

  • Bimodal IT
  • Just Enough Openstack [for Docker]
  • Docker on Openstack
Once your CFP is accepted you can pretty much talk about anything you want. In this case is these three main talking points which are sort of mentioned in the CFP, but it doesn't matter because nobody is quite sure what you promised to talk about. Interestingly, just about every concept I'm going to talk about has been mentioned at some point aready today.

Let's get to know eachother

* But first, let's take a moment to get to know eachother. * my name is Paul Czarkowski, Infrastructure Engineer at Bluebox, recently acquired by ibm * I worked in large enterprises for 10 years, left to work at startup, got bought by the original tech enterprise. * I have been working with Openstack since Folsom and Docker since 0.3. * I have spoken at both Openstack Summit and Dockercon.

Let's get to know eachother

* clearly I can't ask each of you to introduce yourself, but I can run some audience participation... * show of hands: - work at enterprise - running openstack, - running docker * take a look around the room ... these are the people you should network with today.

Bimodal IT

* I want to start off by talking about bimodal IT. * believe it to be the most important thing I will talk about today. * lack of bimodal IT is in my exp the most common failing at enterprises.

Traditional IT

Pets

* mode 1 is your legacy IT business. * focus on providing reliability via business process. * Waterfall, ITIL, Change Comittees, 12am maintenance windows. * Exchange, Physical, Windows 2003, RHEL + Satellite. * Ticket driven IT infrastructure. * Focus on improving reliability of existing services. VMWare! * this is pets of the pets vs cattle analogy

Forward Thinking

Cattle

* mode 2 is your forward thinking IT business. * Completely different org to that of Mode 1 * focus on reducing bottlenecks and delivering business value * Scrum/Kanban, Devops, Infra as code, CI/CD, short feedback loops * Google Apps, Slack, Cloud ( AWS, Azure, Openstack ), Containers, *AAS * self service IT infrastructure * this is the cattle of the pets vs cattle

Forward Thinking

Cattle

* without this second mode, your business will suffer greatly as your competitors go down this path * symptoms of not doing bimodal IT include a 3 year "cloud" RFC/RFP process that end in vmware/hyperv * if you do a RFP for cloud you've already failed. * asking 'how can we do everything with this' when evaluating tech * building a goal/metric of 'how many systems did you virtualize' * having discussions that include: like P2V migrations, 1:1 virtualization, etc
* not really increasing developer productivity, we're reducing friction. * 2nd mode of IT should borrow heavily from Lean practices, Toyota Manufacturing, etc * TOC is /very/ important here, read PP and the goal
Docker and the Three Ways of DevOps by John Willis * three ways of devops * done right, 2nd mode will drag 1st mode through these transformations * 1st mode becomes cattle, 2nd mode becomes insect swarm. * eventually the two modes will be almost indistinguishable from a process/tooling perspective
* My summary here: don't worry about legacy when you're making decisions for the future. * Build seperate team for build/develop. disturbingly large no of ppl will be happy to stay in mode1 * Our focus here at the openstack day should be on the 2nd mode of IT.

Openstack is...

  • Hard to Install
  • Harder to Operate
* with that out of the way, let's talk about how terrible openstack is * which OS, Which distro, packages vs source ? * Hell even installing openstack cli clients can be hard

Which Operating System ?

  • Redhat - RHEL, CentOS
  • Debian - Debian, Ubuntu
* Before you can even start to install you need to choose an OS. * Ubuntu is predominent OS in CI, Newer kernels etc. * Support? They'll take your money if you really want to pay them... * when was last time you used and got value out of RHEL support ?

Which Openstack distro ?

  • OS native - Ubuntu, RHEL
  • from Source ... git://
  • Helion, Mirantis, ...
  • Roll your own packages

http://github.com/blueboxgroup/giftwrap

* You've chosen an OS ... now choose an openstack "distro" * Really two choices here for somebody new to openstack. * 1) use the distro packages * 2) install from source ( its python, not complicated)
http://docs.openstack.org/admin-guide-cloud/content/logical-architecture.html * Each "shape" in this diagram represents an openstack component and 100+ configurable options. 1000s! * Openstack has no opinions, expects you to have them and provide them. * Database - Mysql, Percona+Galera, Postgres, ORACLE?!?!?! * Queuing - RabbitMQ, ZeroMQ, QPID * Network - ml2, OVS, linux bridge, VXLAN, VLAN, GRE, vendor * Compute - KVM, Xen, HyperV, VMWare, Docker, LXC, etc * Cinder - LVM, Gluster, NFS, NetApp, EMC
* If you're new to openstack/cloud lets not complicate things. this is all you really need. * MySQL and RabbitMQ ( don't even worry about replication / HA that can come later ) * Horizon, Keystone, Nova, Neutron, and Glance. * Start with a single externally routed VLAN for your neutron network * tricky stuff like SDN, Floating IPs, etc, later * If you have a NetApp or decent NAS/SAN maybe add cinder after a few months

Openstack Big TentConfiguration Management

* ofc, if want sane repeatable openstack, you'll use config management * unless you roll your own, your choice here will probably dictate your OS choices. * spoiler most are Ubuntu + System Packages * are as unopinionated as openstack, therefore still need to make a lot of upfront decisions.

Ursula

  • http://github.com/blueboxgroup/ursula
  • Curated and opinionated
  • Used to manage a lot of openstack clouds by a small team
  • Feature flags to enable/disable optional items
  • Kick the tires with Vagrant.
  • ~ 1200 ansible tasks to do a simple openstack install
* bluebox's tooling for deploying and managing openstack clouds is called Urusla * opensource and freely available on github. * very opinionated and curated to be able to run a large number of medium size clouds by small team. * We use a vagrant env for development, makes it easy to kick the tires. * JEOD env ... still has 1200 tasks to install openstack!!!

Shameless Plug

<Bluebox Logo goes here>

  • For ~ $5k/mo Bluebox will host and manage your Private Cloud for you.
  • Avoid all the BS of running openstack, and spend your time learning how to consume.

WTF is Docker ?

https://www.docker.com/whatisdocker/

Process Isolation, chroot on steroids Uses linux kernel tooling ( cgroups, namespaces, etc) Shareable Immutable Artifacts - Images / Docker Hub As much or as little of an OS in the image as you want. * < 1 sec startup time, have seen < 10mb to > gigs

Docker vs VM

* Docker = Containers ! Virt. Can act like a VM * security/isolation half way between chroot and VM * Any OS capable of running docker can run your dockerized app... rhel vs deb * layered filesystem means that containers can share common data.

Dockerfile

# Base docker image
FROM paulczar/chrome

RUN apt-get install -y pepperflashplugin-nonfree \
    && rm -rf /var/lib/apt/lists/*

COPY local.conf /etc/fonts/local.conf


COPY . /slides

# Autorun chrome
ENTRYPOINT [ "/usr/bin/google-chrome-stable" ]
CMD [ "--user-data-dir=/data", "--no-sandbox", "file:///slides/index.html" ]
* This is a dockerfile. * instructions on how to create an docker image * FROM, RUN, COPY, CMD

Docker Compose

nginx:
  build: .
  command: /app/bin/boot nginx
  volumes: [ .:/app ]
  links: [ mysql, hhvm ]
  ports: [ 8081:8080 ]
hhvm:
  build: .
  command: /app/bin/boot hhvm
  volumes: [ .:/app ]
  links: [ mysql ]
  ports: [ 9000 ]
mysql:
  image: orchardup/mysql
  ports: [ 3306:3306 ]
* takes yml file and builds containers as described * volume mount CWD into container * just a few seconds to get app running assuming prebuilt. * if I run boot without argument it runs both nginx and hhvm * as you can see am using community image for mysql... aok.

Factorish

http://factorish.github.io/factorish/#/

  • Run legacy apps in a container
  • Make the container 12 factor, not the app
  • Demo apps from simple python to full ELK stack
    • http://github.com/factorish/factorish
    • http://github.com/factorish/factorish-elk
* Toolset I developed for making legacy style apps run in a docker container * ties together tooling to help with config templating, process management, service discovery and more * examples from basic python app to full ELK stack with SD, auto-clustering mysql/galera.

Openstack Images for Docker

  • Ubuntu Trusty / Fedora
  • CoreOS / RancherOS
  • Atomic / Snappy

How to Docker with Openstack ?

  • Just plain old Docker.
  • Docker Machine
  • Deis
  • CoreOS + Fleet
  • Kubernetes
  • Rancher Labs * Going to talk through a bunch of ways to docker on openstack

Docker Machine

https://github.com/docker/machine/releases

* you could just spin up an instance and ssh in and use docker, or just expose the docker port * but Docker machine makes that a little easier and more secure. * docker machine will spin up a new server, install docker, and set up a client/server key/pair. * it reads your openrc style environment variables, but still requires a bunch of settings * hide them behind an alias or a script or something.

Docker Machine

$ docker-machine create -d openstack \
  --openstack-image-id 1525c3f3-1224-4958-bd07-da9feaedf18b \
  --openstack-flavor-id 2  --openstack-net-id ba0fdd03-72b5-41eb-bb67-fef437fd6cb4 \
  --openstack-floatingip-pool external \
  --openstack-ssh-user ubuntu --openstack-sec-groups docker \
    testenv
Creating machine...
To see how to connect Docker to this machine, run: docker-machine env testenv
$
* in this case the image is ubuntu trusty and I have both an internal network and a floating ip pool. * you can see I also need to specify the ssh user and a security group. * sec-group should allow ssh and the docker port, I also added the ephemeral docker port range.

Docker Machine

$ docker-machine env testenv
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://x.x.x.x:2376"
export DOCKER_CERT_PATH="/home/pczarkowski/.docker/machine/machines/testenv"
export DOCKER_MACHINE_NAME="testenv"
$ eval "$(docker-machine env testenv)"
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS
* after docker machine succesfully runs I can get the env vars to pass to docker like this * pass it through eval and you have everything you need to securely connect to your docker machine.

Docker Machine

$ docker run -p 80 --name nginx -d nginx
$ docker inspect nginx| grep HostPort
                    "HostPort": "32769"
$ curl x.x.x.x:32769
...
...
<title>Welcome to nginx!</title>
* now we can run any randome docker command * and it will be run in the docker-machine rather than locally * for example nginx, hostport, curl

Docker Machine

  • Supports Docker-Swarm
  • Support Docker-Compose
  • No host volumes
  • Good for test/staging/demo
  • Not great for dev
* docker swarm for docker based orchestration (NEW) * docker compose can talk to docker machines * some features are not usable on some clouds, volume mounts not on most * this means it can be good for test and demo, but not great for dev.

DEIS

  • Most mature Opensource [docker] PAAS*
  • Heroku inspired workflow
  • distributed stateless applications
  • seperation of responsibilities
  • Solid docs for running on Openstack
* arguably deis is most mature and functional opensource docker based PAAS * has heroku style workflow ( and others ). UX first,chef->docker * is a stateless distribute app itself * ops manage deis and underlying infra, devs/devops manage apps in deis. * ops/DBAs/etc manage databases and state external to DEIS * docs for running on openstack are solid

DEIS

$ git clone https://github.com/deis/example-ruby-sinatra.git
$ cd example-ruby-sinatra
$ deis create && git push deis master
Creating application... done, created nimbus-insignia
Git remote deis added
...
-----> Launching...
       done, nimbus-insignia:v2 deployed to Deis
       http://nimbus-insignia.x.x.x.x.xip.io

$ curl http://nimbus-insignia.x.x.x.x.xip.io
Powered by Deis!
Running on container ID 88384adc62d4
* example of running application in deis * was going to live demo, but we're doing this on shared laptop * deis create sees we're in a git repo and sets up a remote repo to deis git server * post-commit hooks builds and runs application * deis uses wildcard dns to route to applications. xip is great for demo with this.

DEIS

$ deis config:set POWERED_BY=openstack
Creating config... done, v3

=== nimbus-insignia
DEIS_APP: nimbus-insignia
POWERED_BY: openstack

$ curl http://nimbus-insignia.x.x.x.x.xip.io
Powered by openstack
Running on container ID 6440f619dcfc
* deis has a bunch of commands to deal with your running app * for example we can change a config setting for our app ( which is done via docker envs) * deis will rebuild the final layer of your app container which contains the config * it will then mark it as a versioned release and push to docker registry * a new contaner will then be spun up and added to LB, then old spun down * rollback to previous release is easy and quick * explain example

DEIS

$ deis scale web=3
Scaling processes... but first, coffee!
...
$ curl http://nimbus-insignia.173.247.105.17.xip.io
Running on container ID 6440f619dcfc
$ curl http://nimbus-insignia.173.247.105.17.xip.io
Running on container ID cbcf905b8b34
$ curl http://nimbus-insignia.173.247.105.17.xip.io
Running on container ID 676cb76cb562
* adding more or less containers to your app is easy, simply run deis scale * scales your number of containers to the number specified. * easy to tie this into a cron style job for seasonal traffic patterns, can tie to monitoring for auto-scale * can see here it scaled to three, and I can curl and get 3 different container IDs.

CoreOS + Fleet

  • CoreOS - minimal OS for running Containers
  • etcd - distrubuted Key/Value store
  • flannel - network overlay
  • fleet - Container Orchestration
  • fleet takes systemd units with special section
  • We actually just saw it in action with DEIS
* DEIS uses coreos with fleet to deploy and orchestrate itself. * fleet combines with etcd for container orchestration. * you pass it systemd units with an extra section for fleet metadata * schedules across a coreos cluster, one host dies, restarts on another * affinity, anti-affinity, counts, etc. * great for building a platform ( like deis ) on top of

CoreOS + Fleet

[Unit]
Description=deis-router

[Service]
EnvironmentFile=/etc/environment
ExecStartPre=/bin/sh -c "IMAGE=`/run/deis/bin/get_image /deis/router` && docker history $IMAGE >/dev/null 2>&1 || docker pull $IMAGE"
ExecStartPre=/bin/sh -c "docker inspect deis-router >/dev/null 2>&1 && docker rm -f deis-router || true"
ExecStart=/bin/sh -c "IMAGE=`/run/deis/bin/get_image /deis/router` && docker run --name deis-router --rm -p 80:80 -p 2222:2222 -p 443:443 -e EXTERNAL_PORT=80 -e HOST=$COREOS_PRIVATE_IPV4 $IMAGE"
ExecStopPost=-/usr/bin/docker rm -f deis-router

[Install]
WantedBy=multi-user.target

[X-Fleet]
Conflicts=deis-router@*.service
* this is example of fleet unit, pretty easy to understand and parse * surprisingly easy to set up affinity and boot order roles * can use affinity and boot order to ensure a containers volume container starts before it for stateful apps.

Kubernetes

  • Came from Google...but is not Google.
  • IAAS but for containers.
  • You still need to tell it what to run and where.
  • Magnum!
* not a PAAS, but rather something you would build a platform on top of * Kubernetes is to containers what openstack is to VMs. * very opsy interface for deploying/running containers * is building a platform your core compentency? If not wait until somebody does it for you * if K8s, then think about Magnum. Requires Heat.

Docker Registry

  • Use private repos on Docker Hub ( $$$ )
  • run Docker Registry locally ( OS )
  • run Docker Trusted Registry ( $$$ )
  • run CoreOS Enterprise Registry ( $$$ )
* If doing much with docker will probably need private registry. * can pay docker for private repos on SAAS registry. * run opensource docker registry locally ( deis runs one for you ), can be backed by swift? * if backing with swift/s3/etc can run on every machine, localhost, no security. * Run either DTR or CoreOS ER.... pay money ... get support ?

Docker Workflow #1

  • Development on laptop (docker-compose)
  • Testing/QA on "Cloud" ( docker-compose + docker-machine )
  • Jenkins to build docker image, store in registry.
  • Staging / Production on "Cloud" ( Ansible or Chef )
* Develop locally on your laptop, use docker-compose * Deploy Testing / QA envs to Openstack using machine + compose * on commit to "deploy" branch kick of jenkins build of image, version, push to registry * Deploy to Stage/Production using existing CM tools with docker integration * this good, because probably already using CM to manage existing infra, stateful dbs, etc

Docker Workflow #2

  • Development on laptop (docker-compose)
  • Create PR on github
  • Jenkins runs unit tests on PR ( in docker )
  • Jenkins deploys test app of PR ( in DEIS )
  • Jenkins/Human performs Integration tests against PR
  • Jenkins/Human merges PR to master
  • Jenkins deploys Staging App ( in DEIS )
  • Human approves and cuts release in Github
  • Jenkins deploys Prod App ( in DEIS )
* A more advanced workflow using DEIS. * All steps beyond creating the PR can be automated * Path to full CI/CD ( if that's your bag )

Finish

Questions?

Just Enough Openstack for Docker * Rule one of getting a CFP accepted is to mention "docker"