Tugboat – First – The Cost of Slow

View Github Repository Open presentation in a new window

blakehall

See all presentation from blakehall

Tugboat – First – The Cost of Slow

0 0

tugboat-session-slides

Slides for https://austin2014.drupal.org/session/give-bug-fixes-url-tugboat

On Github blakehall / tugboat-session-slides

Tugboat

give your bug fixes a URL.

Howdy folks, my name is Blake Hall. I'm a Senior Developer at Lullabot. We provide consulting, strategy, design and development work while trying to simultaneously kick ass and have fun (and sometime juggle). This is my colleague, Senior Architect and Development Manager James Sansbury.

  • 45 minutes talking about tugboat
  • Github Pull Request Builder / tshirt
  • Build tool, but... it's primary job is to provide increased transparency and visibility into the development process for less technical stakeholders (including project managers) while staying out of the developers way.

This is not a UI

I understand I'm oversimplifying here, but traditional continuous integration and deployment tools aren't exactly transparent for non-technical folks. With Tugboat we're trying to give them more insight into the process.
Jenkins is another popular alternative, but this UI isn't the most user friendly either, despite the prescence of happy little clouds. This probably isn't the kind of thing you'll hand over to everyone working on a project.

Full Steam Ahead

Since this isn't exactly a typical build, continuous integration or continuous deployment tool (at least one we've come across for Drupal), we're going to start by spending a bit of time talking about the problem we (James) identified, or "why" we did this. Then we'll share some of the technical details of how we've used Tugboat to streamline our process and be more efficient on projects for Intel, MSNBC and our own lullabot.com site.

Then we'll close with some thoughts about the impact this has had on our project and development processes and where we see Tugboat going in the near future. In case you need them, Dramamine and sea-sick bags have been placed under your chairs.

First

a little bit of history

Before we dive into the Why and How behind Tugboat, let's briefly reminisce about where we've come from with Drupal development workflow and best practices.
At some point, probably in the paleolithic era in technology industry terms, we figured out that it's a good idea to maintain separate environments for development and production. Most Drupal projects I've worked on, even those pre-dating my time at Lullabot, have used a workflow similar to this
  • Production
  • Staging
  • Development
  • Over time, we've evolved several methods to actually manage promoting configuration changes (via code) up from the bottom on this chart.
  • We treat the UGC that makes up the bulk of our sites as data, and that typically moves in the opposite direction from top to bottom.
  • argue about menu items
  • content deployment
  • pain associated with peer review / QA process
  • Tugboat cuts the time it takes to get features and bug fixes deployed

hook_update_N()

Before I worked at Lullabot I can remember listening to their podcasts (2007) about every new lullabot.com launch. At least as far as I can remember, eight years is a long time, these podcasts were my first exposure to hook_update_N(). Three years later, when lullabot.com was redesigned yet again (a redesign which coincidently James here was a key part of), they talked about the same problems everyone seemed to face when it came to deploying drupal sites: editing configuration on the production server (**cough Jeff Robbins cough**) without capturing those changes in code or an update hook led to possibly unintended inconsistencies between the Development, Staging and Production. (and certainly, at the very least, complicates diffs between environments).

Features

  • DrupalCon DC / Development Seed / Strongarm & Features
  • leveraged -- abused for deployment
  • May not be the tool we like, but it's the tool we have

Configuration

Management

Initiative

Drupal 8 brings the promise of more sophisticated configuration management than we've had to date. Rather than our mis-use of Features module, developers will now be able to leverage a core api to provide import / export / sync functionality for their configuration storage. While this is unquestionably **a "good thing"** CMI won't, by itself, be able to solve all the problems associated with shepherding a ticket through it's lifecycle.
All of this work makes our work as developers easier, and makes us more efficient, but we still need to improve our process, to increase the visibility non technical folks have into work as it's in progress. Tugboat has proven incredibly useful to help us answer the question "what would this code look like, deployed right now".

The Cost of Slow

  • All of this great progress Blake has described...
  • Let's talk about the cost...

Lessened Visibility

  • When things slow down, it takes longer for the people that need to see things to see them.
  • We at Lullabot often work in one or two week sprints with a demo at the end…
  • Anyone here had a demo go wrong where the project owner, or stakeholder, or designer says
  • now that I see it, it should…
  • you misunderstood…
  • Expensive. Really expensive for large teams

Release blockers

  • Back in the paleolithic era of Drupal deployment, it was pretty easy to show a project owner or stakeholder whatever thing you were working on…
  • Saavy stakeholders
  • But Drupal is much more complex now, and we are working with larger teams, that might be working on many projects at once…
  • Can't expect stakeholders to know how to do this anymore
  • Traditionally, the way around this has been to create larger releases that can be tested all together in specific QA environments.
  • Who here has at this moment had a release fail to pass QA, but one piece of that release needs to go out right away?
  • Whole release is blocked
  • All of this leads to another interesting phenomenon
  • Releases start to back into each other, and grow in size
  • While one team fixes that release…
  • New bugs from new work
  • We often start making compromises on the process in order to payoff the debt that the process incurs

Trade-offs

  • Visibility for Testability
  • Agility for Stability
  • We've got some tradeoffs we're making.
  • Stakeholder Visibility for Project Testability
  • Process Agility for Project Stability
  • These are probably wise trade-offs…
In other words, by creating a consistent and testable deployment process, we've slowed things down to where it takes much longer for our stakeholders to see what we're building. And if we slow how long it takes for our stakeholders to see what the developers are doing, *everything* suffers. We've created a bunch of amazing ways we can QA our Drupal sites, but those things all make our process slower and our project murkier.

"It worked fine on my local!"

  • Anyone heard these words?
  • All of the ways we use to get around the pain of QAing our Drupal sites make that QA process less accurate
  • Because it has become a multi-step, long process to fully deploy a Drupal site, we take short cuts
    • Skipping db re-import
    • Stage File Proxy
    • Not fully reverting features
  • All of these shortcuts are what cause inconsistent behavior across environments.

I get it.

Now what?

That's it, I hope you enjoyed our presentation!

What if…

  • Immediate visibility
  • Avoid regressions
  • Die, buggies!
  • Analyze before code gets merged
  • What if…
    • show stakeholders what we are working on right away
    • catch regressions before they happened
    • squash bugs before they ever made it to QA
    • detect performance, visual and functional regressions before…
  • This is what we built Tugboat to do…
  • About three years ago…
  • We have a super technically savvy stakeholder…
  • But Mike is a busy guy…
This is what we built Tugboat to do. About three years ago, we started work on the Tizen project for Intel, the Linux foundation, and Samsung. Tizen is a Drupal 7 multisite setup comprised of three different websites, tizen.org, developer.tizen.org, and source.tizen.org. We had a super technically savvy stakeholder and project owner named Mike Shaver who totally could have taken the time to build each branch of our code that he needed to test, to make sure you did what we said it was going to do. But Mike is a busy guy, and he doesn't need to be importing three databases five times a day in order to test everything that we were doing.
  • In order to save hime time, we wrote a set of very simple bash scripts that glued…
  • Anytime anyone created a new pull request, Jenkins would get triggered…
  • Anyone here familiar with simplytest.me?
  • It's much like that, except that it happens automatically, and it builds the Tizen sites with fresh database dumps and files…
In order to save time, we wrote a set of basic bash scripts coupled with some Jenkins jobs and he github integration plug-in to create that glue that we needed. Anytime anyone created a new pull request, Jenkins would be triggered by the plug-in, our bash trips would run, and a clone of the three Drupal sites with the new code merged into master Live on the site and deployed. These bash scripts would place a comment on the github pull request with links to these three sites so that Mike could go there and see the work that is being proposed as added, check to make sure it did what it was supposed to do, and that it didn't break anything else.

Tugboat saved the day (and a whole heck of a lot of cash) on MSNBC, too. We had a super aggressive timeline to build a big complex project with a huge team.

Tugboat hands out URLs like candy to:

  • External teams
  • Internal teams
  • Individual developers
  • MSNBC Newsvine
  • We've also got lots of Epics on MSNBC that need a URL
  • Any developers love having to trouble shoot a tricky hook_update_N function?
Thanks James. After all of James' work, putting the initial version of Tugboat together for the Tizen project, we shamelessly _stole_, er I mean **leveraged** it for use on lullabot.com. Like the typical cobbler's children the process of building the latest lullabot.com, to be polite, _took a while_. For quite a while the project trudged forward with whichever developer resources had spare time. This often led to jointly working on, or finishing each other's features, and lots of time sunk into updating / re-syncing a local development environment in order to do some peer review, or test simple bug fixes. Soliciting feedback, or getting peer reviews was a difficult process, because of the scattered nature of development on the project. Using tugboat allowed us to have shared, canonical, urls for in progress features. I know, I at least, used this a couple of times to push code changes to a development branch without actually having my local in sync, or up to date. I could simply hand edit a view configuration, push my commits and see / share the work incredibly quickly. This made it easier for us to solicit peer review, since the reviewer only needed the tugboat environment link in a github ticket rather than a full-blown local development environment. It also made it easier to solicit feedback and iterate on new features. Several times we would use the tugboat environment to share and discuss possible UI implementations throughout the project.
Then last spring, after our successful launch of lullabot.com, I started working on another project at Intel. Our client, Mike, wanted to have another instance of Tugboat set up for use with that project too. One of the challenges with the Tugboat, as it existed at that time, was in the actual setup process. While it was still worth doing, there was a day or two of getting everything installed and configured just right before we'd have a working service. There were several interdependent jobs to handle the production data downsync, keeping the staging sites up to date, and building the actual Tugboat environments. The code we had created was quite DRY, in the sense of don't repeat yourself, but that complicated the initial setup. Jenkins plugin dependency management, and discrepancies between plugin versions didn't help matters any either. In addition, because our Jenkins server was behind a VPN kicking off manual jobs required a bit of extra work (we all have at least a little bit of hate for VPN clients, right?).
I created a command line wrapper for our tugboat scripts and the Jenkins API interface. Now, once a Jenkins instance is setup and the github plugin is configured (in other words Jenkins can access the github repository of interest) a couple of simple commands setup the skeleton jobs needed to generate automated pull request builds. This wrapper code also allows us to backup, diff and update the Jenkins job configurations in much the same way we do with Feature modules now using drush. This ensures that we can keep our Tugboat settings in version control as well. We also don't have to rely on github (and cron) to kick off job builds. Tugboat can build environments for any arbitrary branch, commit hash or ticket number manually from the command line (interface coming soon-ish?). There's also, now, a basic plugin system, so the same Tugboat utility wrapper that's being used to spin up dev environments can be used for backing up existing sites, as a test runner, deployment helper, etc.

resemble.js

In addition to helping spin up environments for pull requests, and running our Behat tests, we're in the process of wrapping up development on another feature for Tizen's Tugboat instance -- visual regression testing. Using the resemble.js library we're able to spin up a phantom.js headless browser and capture screenshots -- at key urls and key breakpoints as part of the Tuboat environment build process. These image files are then run through resemble.js which does a pixel by pixel comparison, highlighting any differences. These images can be posted back to the relevant github ticket -- allowing us to quickly scan for any untended visual side effects of a particular code change.
Here are a couple of example images between the production version of developer.tizen.org and my localhost. You can see from the pink highlighting that quite a bit has changed. On closer examination though the changes fall into 3 categories: actual content difference (shame on me I need to downsync from production more often), missing images (I also apparently need to rsync files, or use stage_file_proxy), and improperly configured typekit fonts on my localhost. While in this case the diff is somewhat of a false positive, there are enough changes called out here to highlight the types of bugs.

Setting this up for yourself

While we haven't (yet?) released all of the wrapper code I mentioned earlier, Jerad Bitner has written an article on lullabot.com about our github pull request builder which was the precursor to Tugboat.

Pull Request Builder blog post

This article points to the majority of the resources we used piecing together the initial version of Tugboat, including the scripts responsible for building environments and the Jenkins Github Pull Request Builder plugin itself.

lullabot/jenkins_github_drupal

Jenkins Github Plugin

Where to next?

(this isn't really a UI either)

In practice we've found jenkins to be kind of clunky for our needs. Things like production backups and staging downsync / updates need to happen on a very reliable schedule. Crontab is often better for managing tasks like this than Jenkins -- which in our experience occasionally crashes in an un-obvious way (at least when compared to crontab). Really we're using Jenkins to poll for github pull requests, and to provide an interface for manually building environments that doesn't require the command line.

Welcome To

The Future

Rather than continue to rely on this UI, we're working on our own. In the plans so far we have a per project dashboard, that will provide at a glance all of the Tugboat environments across a particular project. This will give nontechnical stakeholders a bird's eye view to the work in progress code at any given moment in time. In addition, each tugboat build environment will also have their own landing page. This will be a handy place for us to aggregate information about the environment. When was it built? Does it have the most recent code from the relevant branch? Should we manually trigger a rebuild because some potentially conflicting code has changed elsewhere? What do the test results look like? How about performance profiling (both cachegrind and the ???? front end equivalent ???? waterfall). _https://github.com/gmetais/grunt-devperf/_ Can, or should, we use vagrant along side puppet or chef to provision environments and perhaps, in the process, convince developers to use the same stack locally to more closely match the target stack? Perhaps we will use containers, with something like docker, to make each tugboat project a bit more resource efficient while cleanly silo'ing each tugboat build environment.

Q&A

blake@lullabot.com

james@lullabot.com