Containers for continuous integration (in science)

Frederick “Erick” Matsen

http://matsen.fredhutch.org/ @ematsen

Welcome to the future

• ✔ Code under version control
• ✔ Example inputs included in repository
• ✗ Code compiles
• ✗ Code runs on example inputs
• ✗ Code does the right thing on example inputs

What do the pros do?

Check out pull requests for Railsand click on the little ✔s and ✗s.

Scientific workflows are complex

Docker enables us to bring together lots of working parts, unusual languages, and heavy dependencies.

Dockerfile

FROM ubuntu:trusty

RUN apt-get update -q && \
    apt-get install -y -q --no-install-recommends \
        ca-certificates \
        cowsay \
        git \
        make

RUN ln -s /usr/games/cowsay /usr/bin

CMD git clone https://github.com/matsen/p-neq-np.git && \
    cd p-neq-np && \
    make

Try this

git clone https://github.com/matsen/cowsay-build-env.git
cd cowsay-build-env/
docker build -t $USER/cowsay-build-env .
docker images
docker run -t $USER/cowsay-build-env
docker run -it $USER/cowsay-build-env /bin/bash

Docker hub

Either push directly:

docker push $USER/cowsay-build-env

Or set up an automated build as I have for this project:

https://hub.docker.com/r/matsen/cowsay-build-env/

We could stop here!

In fact, I wrote a little shim that would post to our Slack channel with the results of an automated build.

But, there are tools that make the process smoother and more logical, such as Wercker.

wercker.yml

Specify the Docker image used for the build:

box: matsen/cowsay-build-env

Specify what the build should do:

build:
    steps:
        - script:
            name: build
            code: make

When pushed to Wercker, you get a build page like this.

More fun

Notify:

build:
    after-steps:
        - slack-notifier:
            url: $SLACK_URL
            channel: microbiome
            username: pplacer build

Upload docs to Github pages:

deploy:
    steps:
        - ematsen/gh-pages:
            token: $GITHUB_TOKEN
            repo: matsen/pplacer
            path: docs/_build/html

YAML is great*

* and a little fussy to debug. Use yamllint.com.

Related things

• Rabix: portable workflows via Docker
• Bioboxes: modularize bioinformatics tools via Docker
• Shippable: another CI solution with Docker