git@github.com:ncareol

Erik Johnson, NCAR EOL

Boulder Ruby, 2014/08/19

http://erikj.github.io/gitgithubncareol

NCAR Earth Observing Laboratory

(EOL)

NCAR EOL Facilities

NCAR EOL Facilities

NCAR EOL Facilities

NCAR EOL Facilities

NCAR EOL Facilities

NCAR EOL CDS

https://www.eol.ucar.edu/about/our-organization/cds

• EOL computing & IT infrastructure
• Data-management services
  • real-time & post-facto access
• Field-project support
  • IT
  • collaborative tools

NCAR EOL CDS

Collaborative Technologies & Metadata (CTM) group

• Databases of metadata
• Web applications
• MapServer
• Data ingest
• IRC

Field Projects

• Campaigns to conduct atmospheric research
  • Weather
  • Climate
  • Chemistry
  • Pollution

Field Projects

• e.g.
  • DC3 (Kansas, 2012)
  • MPEX (US Southwest, 2013)
  • SAS / NOMADSS / SOAS (US Southeast, 2013)
  • CONTRAST (Guam, 2014)
  • DEEPWAVE (New Zealand, 2014)
  • FRAPPE (Colorado, 2014)

Field Catalog

• ingests, serves and archives field-project data:
  • operational
  • research
  • model
  • primarily images, text, HTML, PDF and KML
• used during and after project

Field Catalog

Suite of applications:

• zith9 database schema (MySQL)
• Catalog Models Rubygem (Ruby, ActiveRecord)
• Catalog Ingest application (Ruby, ActiveRecord, Inotify, EventMachine)
• Field Catalog web application (Ruby on Rails, Haml, CoffeeScript, Bootstrap, PDF.js, CKEditor)
• Catalog Maps web application (Ruby on Rails, Haml, CoffeeScript, OpenLayers, Bootstrap)
• Catalog Nagios monitoring utility (Ruby, ActiveRecord, Nagios)
• Catalog Stats command-line application (Ruby, ActiveRecord)
• IRCBot Bot-provided IRC utilities (Ruby, Cinch)

catalog.eol.ucar.edu

catalog.eol.ucar.edu/frappe

PDF Report

Product Navigation

Satellite Product

Catalog Maps

catalog.eol.ucar.edu/maps/deepwave

Field-Catalog Development

• Group lead is Product Owner
• developed by 2-6 active Software Engineers @ any time: CTM team & others
• administered by ~ one Software Engineer / active project
• scrum: three times / week

Git

• Distributed revision-control system
• every check out is a complete, free-standing repository

github.com:ncareol

• 17 private repositories
• 9 public repositories
• 13 teams
• 16 members
• initial plan: Bronze, 10 private repositories, $25 / month
• later: Silver, 20 private repositories, $50 / month

https://github.com/pricing

GitHub Issues

In the beginning...

There is an issue:

GitHub Issues

Issues generally fall into one of two categories:

• features: make something / make something better
• bugs: something is wrong and needs to be fixed

Issue Description

composed in plain text and / or Markdown / GFM + GitHub Hotlinking

Markdown

plain-text syntax for authoring that compiles to HTML

designed to be written quickly w/ rich-text formatting provided by HTML

Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML). https://daringfireball.net/projects/markdown/

GitHub-Flavored Markdown + Hotlinking

• handy formatting enhancements to plain-old Markdown

• code fencing and language-specific syntax highlighting via

  ```
  ```
  
  ```language
  ```

• #issues, @usernames and commit hashes are hot-linked

• checklists

https://help.github.com/articles/github-flavored-markdown

https://github.com/blog/831-issues-2-0-the-next-generation

https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

Issue Description

# Issue Heading

A **good idea** for a *new feature* from @erikj that relates to #60:

```ruby
puts 'Hello Boulder Ruby'
```

Issues

Issues can be observed and commented upon as they progress

• Comments

  • in-line w/ Issue
  • specific commits or lines of commits
  • Markdown / GFM w/ hotlinking

• Users receive notifications when their username is mentioned or comments are added to issues to which they subscribe or to commits they've made

Labels

• Issues can be assigned one or more labels

• Create / edit / delete labels as needed

• Useful for categorizing issues

Labels: CatalogMaps

CHECKED OUT

When a developer is ready to work on an issue:

• assign Issue to themselves

• add the CHECKED OUT label to the issue

Milestones

A Milestone is composed of:

• title
• description (optional)
• due date
• set of issues

Milestones

We generally assign significant, planned versions to their own milestones, e.g.

• Catalog Maps 2.0
• Catalog Maps 2.1
• Catalog Maps 2.2

Git Branches

• Branches are a fundamental part of Git
• Subversion branches are complete copies of their parent branch
• Git branches are only the commits that diverge from their parent branch

Git-Branching Workflow

• "Never commit to your master"
  • https://c9.io/site/blog/2011/11/never-commit-to-your-master/
• everything is branched from develop
• changes are made in dedicated branches
• branches are merged back in to develop when they are ready
• master is updated via merge from develop
• master is always deployable

adapted from http://nvie.com/posts/a-successful-git-branching-model/

Branch Categories

• features: feature-...
• fixes: fix-...
• experimental: xfeature-...
• hot features and fixes: hotfeature-...

http://nvie.com/posts/a-successful-git-branching-model/

Branch Names

There are two hard problems in computer science: cache invalidation, naming things and off-by-one errors. - Phil Karlton + Martin Fowler

Branch Names

e.g.

• feature-42-the-answer
• fix-314-pi-miscalculation
• hotfix-all-bugs

Commit Messages: GitHub Hotlinking

Like Issue content and comments, #issue IDs, @usernames and commit hashes will be autolinked in GitHub's web interface.

GitHub Pull Requests

• request to merge one branch into another
• can be from separate branch in same repository or a branch in a separate repository

Github Pull Requests: Features

• description
• commits to be merged
• code diff
• comments
  • in line
  • specific code lines
• can be merged locally and request will be closed when merge is pushed to origin

https://help.github.com/articles/using-pull-requests

Putting Things Together: Bundler

Bundle w/ Git

• Rubygems
• Bundler package manager
• Catalog Models gem: library of shared ActiveRecord models
• Gem loaded by Bundler via Git, specified in application's Gemfile:

gem 'catalog_models',
  git: 'git@github.com:ncareol/catalog_models.git',
  tag: 'v1.7.3'

http://bundler.io/v1.5/git.html

Vendor Everything

• Package dependencies w/ the source code into vendor/
• Eliminates third-party dependencies on deployment

$ bundle install --local

http://errtheblog.com/posts/50-vendor-everything

http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-applies/

Deployment

Easy Deploys w/ Git

• Git-based workflow that allows us to deploy to various environments by using git push

• Inspired by Heroku PaaS

$ git push heroku master

Counting objects: 63, done.
Compressing objects: 100% (49/49), done.
Writing objects: 100% (63/63), 26.35 KiB, done.
Total 63 (delta 2), reused 0 (delta 0)
-----> Ruby/Rails app detected
-----> Installing dependencies using Bundler version 1.3.0.pre.5
Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin --deployment
Fetching gem metadata from https://rubygems.org/.....
...

https://devcenter.heroku.com/articles/git

Easy Deploys w/ Git

.git/config

On Server / deployment target, enable push to non-bare repository:

[receive]
    denyCurrentBranch = ignore

Easy Deploys w/ Git

.git/hooks/post-receive

#!/bin/sh
# git stuff
cd ..
GIT_DIR='.git'
umask 002 && /usr/bin/git reset --hard
read oldcommit newcommit refname
# on-update actions
echo "bundling"
bundle install --local --path=vendor --deployment --without development test
if git diff-tree --name-status -r $newcommit $oldcommit \
  app/assets vendor/assets > /dev/null ; then
  echo "precompiling assets"
  rake RAILS_ENV=production assets:precompile
fi
echo "reloading"
/usr/local/catalog/etc/catalog-ui-reload

Easy Deploys w/ Git: Remotes

[ej@kepler] git remote -v
dev ssh://ej@ctm-dev.eol.ucar.edu/usr/local/catalog/catalog_ui (fetch)
dev ssh://ej@ctm-dev.eol.ucar.edu/usr/local/catalog/catalog_ui (push)
ops ssh://ej@sferic.eol.ucar.edu:23/usr/local/catalog/catalog_ui (fetch)
ops ssh://ej@sferic.eol.ucar.edu:23/usr/local/catalog/catalog_ui (push)
staging ssh://ej@sferic-dev.eol.ucar.edu/usr/local/catalog/catalog_ui (push)
staging ssh://ej@sferic-dev.eol.ucar.edu/usr/local/catalog/catalog_ui (fetch)

Easy-Deploy Example

$ git checkout develop && git pull
$ git merge --no-ff feature-42-awesomeness
$ git push dev develop # deploy local develop branch to dev
# test changes on dev environment
$ git checkout master && git pull
$ git merge --no-ff develop
$ git push staging master # deploy local master to staging
# test changes on staging environment
# tag release:
$ git tag -a v1.4.2 -m "add awesomeness"
$ git push ops master # deploy local master to ops
# test changes on ops environment
# push merged branches and tag to origin:
$ git push origin develop && git push origin master
$ git push --tags

Workflow Summary

Lessons Learned

• Learn and use the tools that we have / are available: Git, Markdown, languages' preferred packaging utilities, etc.
• Git can enhance your workflow
• GitHub
  • Reliable and simple but powerful
  • Great tool for communication between developers and stakeholders
  • Subscribe to GitHub's blog for updates on features
  • Issue discovery can be difficult when there are many Issues
    • => Easy to create duplicates
  • Organize w/ labels
• Use Git tags for releases
• Minimize / eliminate third-party dependencies

Room For Improvement

• Cross-repo issues: where do they issues belong?
• Deployment collision-- surprise! user friendly, can be surprising, in violation of PLOS, surprises in ops are bad
• Application and assets out of sync while precompiling on deployment
• Scrum, less useful when staff are working on isolated tasks
• Schema scattered across two repositories as Rails migrations and SQL files

Summary

• Development starts w/ an Issue
• Markdown / GFM and auto-linking features
• Git-Branching Workflow
• GitHub Pull Requests: code review and discussion
• Bundle / w/ Git, Vendor Everything
• Git-Based Easy Deploys

Future / Wish List

• More Features
• Ruby 2.1, Rails 4.1, Bootstrap 3
• Continuous Integration
• DevOps
• Docker
• Open Source

</content>

• thanks!
• erik@erikj.me
• http://github.com/erikj
• slides: http://erikj.github.io/gitgithubncareol
• source: https://github.com/erikj/gitgithubncareol
• stickers!
• comments, questions?

Extras!

Misc Git: .gitconfig

• use aliases

  [alias]
    ss = status
    s  = status -sb
    cl = config --list
    a  = add
    ci = commit
    co = checkout
    d  = diff
    ds = diff --cached # diff staged changes
    p  = push
    b  = branch
    undo    = reset --soft HEAD^

https://github.com/erikj/dotfiles/blob/master/kepler/.gitconfig

https://github.com/erikj/dotfiles/blob/master/.gitconfig.aliases

Misc Git: .gitconfig

• use global .gitignore
• keep developer-specific cruft out of project .gitignores

• ~/.gitconfig

  [core]
    excludesfile = ~/.gitignore-global

• ~/.gitignore-global

  *.DS_Store
  *.sublime-*
  /.pow*
  jmeter.log

https://github.com/erikj/dotfiles/blob/master/kepler/.gitconfig

https://github.com/erikj/dotfiles/blob/master/.gitignore-global