Container Days ATX

Advanced Docker

Created by Paul Czarkowski / @pczarkowski

Who Am I ?

Cloud Engineer - BlueBox Cloud

Blue Box Cloud

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery

Optimizing Dockerfiles

Blog Post

Optimizing Dockerfiles

Docker images are “supposed” to be small and fast.
Software is complicated, languages like Python/Ruby can result in large images.
Dockerfile != $ curl | sudo bash

Optimizing Dockerfiles

FROM ubuntu:latest

ADD . /home/docker

ADD https://dist.torproject.org/torbrowser/4.0.4/tor-browser-linux64-4.0.4_en-US.tar.xz /home/docker/tor.tar.xz

RUN apt-get update && true

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y firefox

RUN localedef -v -c -i en_US -f UTF-8 en_US.UTF-8 || :

RUN useradd -m -d /home/docker docker

RUN cd /home/docker && tar xJf /home/docker/tor.tar.xz

USER docker

CMD ["/home/docker/tor-browser_en-US/start-tor-browser"]

Optimizing Dockerfiles

Order Matters

Optimizing your Dockerfile for build cache hits.

The more cache hits the faster the build
First command to change invalidates all cache
Smart ordering can save minutes

Optimizing Dockerfiles

Order Matters

Sort your commands by

Sharable with other images Frequency of change Time to run

Optimizing Dockerfiles

Know your commands

ADD always invalidates cache
WORKDIR, CMD are cheap

Optimizing Dockerfiles

Choose your base images wisely

Remember you're using a layered FS
ubuntu:trusty + busybox > 2 x Ubuntu:trusty

Optimizing Dockerfiles

Layers are your friend

Each command is a layer
Join RUN commands together with && \
A shared layer is free for subsequent containers.
Create your own common base image from when lots of shared layers.

Optimizing Dockerfiles

Cheat!

Add new changes to bottom of Dockerfile
Once locked in, then move them up.

Optimizing Dockerfiles

Volume Containers

Use the image of the application that is consuming it as the base for the volume container.

It's free because it's shared.
Volume container now contains the Data and the Application.

Optimizing Dockerfiles

FROM ubuntu:latest

ADD . /home/docker

ADD https://dist.torproject.org/torbrowser/4.0.4/tor-browser-linux64-4.0.4_en-US.tar.xz /home/docker/tor.tar.xz

RUN apt-get update && true

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y firefox

RUN localedef -v -c -i en_US -f UTF-8 en_US.UTF-8 || :

RUN useradd -m -d /home/docker docker

RUN cd /home/docker && tar xJf /home/docker/tor.tar.xz

USER docker

CMD ["/home/docker/tor-browser_en-US/start-tor-browser"]

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

Docker vs Config Management

"I don't need Config Management now that I have Docker."

Docker vs Config Management

You probably shouldn't be trusting docker [yet] for managing long term persistent data.

Volume Containers
Volume mounts from Host

Docker vs Config Management

Persistent data stores and other infrastructure services have to be managed by something.

Chef recipes to spin up databases, Key Value stores, etc.
Ansible or Cloud Formations for RDS, ELB, etc.

Docker vs Config Management

Chef and Docker

[the tools, not the companies]have a great history of working together.

https://supermarket.chef.io/cookbooks/docker

include_recipe "docker::default"

Docker vs Config Management

docker_image 'registry' do
  action [:pull]
end

docker_container 'registry' do
  detach  true
  port    '5000:5000'
  action [:run]
end

node['my_app']['images'].each do |image|
  docker_image image do
    action [:pull]
  end
  docker_image image do
    repository "#{node['my_app']['registry']}/#{image}"
    registry   node['my_app']['registry']
    action [:tag, :push]
  end
end

Docker vs Config Management

git "#{Chef::Config[:file_cache_path]}/docker-testcontainerd" do
  repository 'git@github.com:bflad/docker-testcontainerd.git'
  notifies :build, 'docker_image[tduffield/testcontainerd]', :immediately
end

docker_image 'tduffield/testcontainerd' do
  action :pull_if_missing
end

Docker vs Config Management

Almost anything that can be done with docker can be done via the docker cookbook.

Puppet and Ansible both have a similar story.

Docker vs Config Management

Config Management in the container ?

You probably shouldn't...

But if it makes sense in your use case ... go for it.

Consider Chef for Containers if you want to do this, it will help deal with managing services etc.

Run Chef from a volume mount so that it doesn't actually live in your container.

Docker vs Config Management

Config Management to build images ?

If you're already using a tool like Packer to create images with Config Management, you can probably switch it to build Docker images very easily.

Docker vs Config Management

ChefDK in a container?

If you're like me and run an OS that's not well supported by ChefDK ... run it in a container.

$ docker run -d -v ~/chef/cookbook:/cookbook --name chefdk spheromak/docker-chefdk sleep 6000000
$ docker exec -ti chefdk bundle install
$ docker exec -ti chefdk rake test

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

Logging and Metrics

This used to be hard. It's a lot less hard now.

Logging and Metrics

Try your very best not to ever log to a file in the container. Otherwise you will probably face logfile management issues.

Logging and Metrics

Logspout by @progrium

Runs in a container, watches the docker log stream
all stdout/sderr is seen by logspout
can be set to forward to a syslog server.

$ docker run -v=/var/run/docker.sock:/tmp/docker.sock \
		progrium/logspout \
		syslog://logs.papertrailapp.com:55555

Logging and Metrics

But my app doesn't log to stdout/stderr.

It does now!

# /etc/nginx/nginx.conf
worker_processes 1;
daemon off;
user app app;
pid /app/nginx.pid;
error_log /dev/stderr;
access_log /dev/stdout;
...

Logging and Metrics

But seriously I can't configure it to log to /dev/stdout

symlink /dev/stdout to the known location of logfile.
run syslog in container, push directly to your syslog server.
logspout will hopefully soon also include a syslog server.
run a volume container and mount in path for logs
log to volume mount from host.

Logging and Metrics

Logging improvements coming soon!

Logging and Metrics

Docker is a wrapper for LXC.
LXC exposes metrics in /proc
Example - /sys/fs/cgroup/memory/lxc/[longid]/
Hard to track where all of these metrics live.

Logging and Metrics

CAdvisor to the rescue.

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

Logging and Metrics

CAdvisor

Cadvisor natively support Docker and most linux containers based on LXC

Web UI - Demo Later
Rest API
InfluxDB and Prometheus outputs.
Probably needs to support Graphite.

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

Multiple Processes

There are times when you need to run multiple processes in a container.

Ignore the nay-sayers ... this is OK.

Multiple Processes

Backgrounded in a start script

#!/bin/boot

mysqld_safe &
/usr/bin/hhvm --config /etc/hhvm/server.ini --user app -m server &
apache2 -DFOREGROUND

Multiple Processes

Backgrounded in a start script

Problems

Any process could die, container would still run.
Zombie Processes ( [defunc] )
Trying to resolve a solved problem ( init systems )
use exec any time calling "pid 1" app from script in docker.

Multiple Processes

Init system ... Runit

Simple lightweight init system
place executable script in /etc/services/[NAME]/run
restarts processes on exit/crash
`$ docker exec wordpress service apache2 restart`
handles zombie processes ( I think! )

Zombie Processess

Docker and the pid 1 zombie reaping problem

When process dies, turns into zombie process.
Parent process must explicitly wait for child process termination.
Is this really a big deal? Probably not...
However it can be avoided - Phusion - my_init
myinit when called with no arguments will start runit and let runit handle service mangement.

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

Container Centric OSes

CoreOS
Atomic
Snappy ( Ubuntu Core )
RancherOS ( brand new! )

CoreOS

Free with pay for support & premium features
A fork of ChromeOS
FastPatch for updates
No package manager!
Clustering support.
Rocket

CoreOS

Clustering Support

etcd
fleet
flannel*

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

App Configuration

via ENV or service discovery

I have an application in a container ... but how do I actually configure it ?

App Configuration

During `docker build`

This is the simplest use case, if your configuration file is very static just use ADD in your `Dockerfile` to put it in the correct place.

App Configuration

via volume mount

You can also mount it in from your host ( useful if already using CM to write the config file on the host ).

docker run -d -v /my/app/apache2/httpd.conf:/etc/apache2/httpd.conf myapp

It's OK to do this, but be aware of the potential security implications.

App Configuration

As part of a start script

#!/bin/bash
PORT=${PORT:-5000}
sed -i "s/xxxPORTxxx/$PORT/" /etc/apache2/httpd.conf
exec apache2 -DFOREGROUND

App Configuration

Using confd

lightweight templating engine written in GO.
Take key/value pairs from a number of supported storage engines.
Environment Variables, ETCD, Consul, etc.

App Configuration

Using confd

FROM ubuntu:trusty
RUN \
  curl -sSL -o /usr/local/bin/etcdctl \
  		https://s3-us-west-2.amazonaws.com/opdemand/etcdctl-v0.4.6 \
  && chmod +x /usr/local/bin/etcdctl \
  && curl -sSL -o /usr/local/bin/confd \
  		https://github.com/kelseyhightower/confd/releases/download/v0.7.1/confd-0.7.1-linux-amd64 \
  && chmod +x /usr/local/bin/confd

App Configuration

Using confd

#!/bin/bash

BACKEND=${BACKEND:-env}
until confd -onetime -backend=${BACKEND} -confdir=/etc/confd	; do
  echo "echo ==> ${APP_NAME}: waiting for confd to write initial templates..."
  sleep 1
done

exec apache2 -DFOREGROUND

App Configuration

Using confd

$ docker run -d -p 8080 \
  -e APACHE_PORT=8080 -e APACHE_ROOT=/app/web \
  apache /app/bin/boot

App Configuration

Using confd

Deeper dive into confd during demo / workshop.

Agenda

Optimizing Dockerfiles
Docker vs Config Management
Logging and Metrics
Running multiple processes in a container
Container Centric OSes
App Configuration via ENV or service discovery
Demo / Workshop

Demo / Workshop

Factorish - Example App

docker pull factorish/example

Demo / Workshop

$ docker run -d -e SERVICES_EXAMPLE_TEXT=father \
  -p 8080:8080 --name father factorish/example
$ curl localhost:8080 # or IP of boot2docker

$ docker run -d -e SERVICES_EXAMPLE_TEXT=mother \
  -p 8081:8080 --name mother factorish/example
$ curl localhost:8080 # or IP of boot2docker

$ docker kill father mother
$ docker rm father mother

Demo / Workshop

$ vagrant up
.....
$ vagrant ssh core-01
$ etcd ctl / --recursive

Container Days ATX – Advanced Docker

paulczar

Container Days ATX – Advanced Docker

1 2 (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/platform.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

cdatx-advanced-docker