Motivation

The thing we made is called Birdwave, and it's primarily a data visualization. The idea of Birdwave came to me partly because of an interest and a hobby that I have.

Birding

• Observing and identifying birds, visually and by ear.
• Quiet and relaxing (when you need to get away from big crowds of technologists)
• Closely tied with migratory patterns:
  • Migration seasons are peaks
  • Non-migratory birds are all you have in the off-season
• Field guides
  • Beautiful illustrations, identifying characteristics
  • Show static maps of breeding areas during the year

Motivation

• Whimsical descriptions of songs:
  • Yellow warbler -- "sweet sweet sweet I'm so sweet"
  • Chestnut-sided warbler -- "pleased pleased pleased to meecha"
• So I thought it'd be nice to plot bird migration data on a map, as it changes month-to-month.
• Basically this...

Data for the App

eBird data set

• Data for 1 year for the US region
• 11 GB of tab-separated values
• Over 1700 species

eBird

• Collaboration between National Audubon Society and Cornell Lab of Ornithology
• Gather bird sighting reports from all over the world (an incredible amount of data)
• Have a very basic API, not good for trends
• Data available for download free for academic use

Audience Interaction

• How much data do you think that was?

Data for the App

The need for an API

• Safe, quick data import
• Too much data to load at once
• Dynamic nature of the app
• d3 handles JSON requests

• There's way too much data to not load on demand
• The event-driven nature of the app meant that it couldn't be modeled with a traditional request-response cycle. There needed to be an API capable of rendering tailored responses to different parameters. That's when John decided to use Clojure to build this API.

Clojure Syntax

/* JavaScript */
function greet(who, event) {
  return "Greetings " + who + "! Welcome to " + event "!";
}

greet("Good People", "Codemash");
              

;; Clojure
(defn greet [who event]
  (str "Greetings " who "! Welcome to " event "!"))

(greet "Good People" "Codemash")
            

• Let's compare with something familiar, javascript.
• clojure is a lisp
• function position
• lisp's simplicity is purposeful. because it's code is a data structure, a so called s-expression, it's possible to write very powerful macros
• macros - code that writes code
• clojure uses the basic lisp syntax with addition of richer data-type literals

Clojure Syntax

Clojure Syntax

• () List
• [] Vector
• {} Map

• clojure code is all made of lists
• vectors are like lists but allow constant time random access like arrays
• maps are associative datastructures, like Hashes in Ruby, or Objects in Javascript. Except you can use any type of value as a key in clojure's maps
• many more built-in types. we may point out some of them as we go along.
• now let's take a look at some birdwave code.

Parsing the Data

;; Clojure

(def fields
  [:sighting/guid                    ;; "GLOBAL UNIQUE IDENTIFIER"
   :taxon/order                      ;; "TAXONOMIC ORDER"
   nil                               ;; "CATEGORY"
   :taxon/common-name                ;; "COMMON NAME"
   :taxon/scientific-name            ;; "SCIENTIFIC NAME"
   :taxon/subspecies-common-name     ;; "SUBSPECIES COMMON NAME"
   :taxon/subspecies-scientific-name ;; "SUBSPECIES SCIENTIFIC NAME"
   :sighting/count                   ;; "OBSERVATION COUNT" ;; x indicates uncounted
   ;; ...
   ])
            

• the details of this aren't that important. Just need to ease you all into this whole clojure thing.
• Our file of bird sightings is tab separated values
• Records delimited by a newline
• Def fields defines a var called fields which identifies
• We split a line on tab characters and then compare to this vector
• If there is a keyword in the same position then we extract that value into a hashmap with this keyword as its key
• semicolons indicate comments
• keywords begin with a colon and contain an optional namespace

Parsing the Data

;; Clojure

(defn sighting-seq
  "Return a lazy sequence of lines from filename, transformed into sighting maps"
  [filename skip-rows nth-row]
  (->> (io/reader filename)
       (line-seq)
       (drop (or skip-rows 1))
       (take-nth nth-row)
       (map sighting)))
            

• This example constructs a pipeline of data transformation steps
• beginning with a lazy sequence of lines read from the file
• each step is lazy
• returning a lazy sequence means we can take 100 records, 1000, or all of them
• and we'll never run out of memory because it only does the work on demand
• the whole collection never exists in memory at the same time.

Data Storage and Query

Datomic

• Now that we have parsed our file into data structures we need some place to put them that's easily queried
• We chose to use the database conceived by Clojure's author, Rich Hickey.

Datomic

• Schema
• Transactional
• History preserving
• Query language is Clojure data
• Results are Clojure data structures
• Query is executed in application server

• We didn't particularly need transactions or history for this application
• but we were curious and wanted to experience using this new technology
• and we liked the fact that query would scale horizontally with the application.

Datomic

;; Clojure

(q '[:find (sum ?count) (count ?e)
     :where
     [?t :taxon/order "2881"]
     [?e :sighting/taxon ?t]
     [?e :sighting/state "Ohio"]
     [?e :sighting/county "Sandusky"]
     [?e :sighting/count ?count]]
   (db conn))

;;=> [[540 108]]
            

• Example of datomic query that finds the number of bald eagle sightings
• And the total eagle count for Sandusky Ohio
• Don't get into it... For flavor only.
• Easy at first -- kept schema simple with a single entity per sighting
• Slow import, leading to use of partial data set for dev
• Talking reeeally slow... many hours.
• But it was enough data to get started on a web service

Displaying the Data

• Set of libraries that enable DOM manipulation based on data bindings
• Fluent API around using functional transforms of the bound data to change the state of the DOM elements

We'll quickly cover the parts relating to mapping. d3 understands 2 kinds of mapping data: GeoJSON and TopoJSON.

GeoJSON is a subset of JSON that encodes geographic data. Using shapefiles generated from GIS mapping software, it represents geometries as connected arcs. d3 can take these arcs and plot SVG paths in the browser, complete with projections.

TopoJSON is a form of GeoJSON specific to topologies, meaning that the arcs are aware of "inside" and "outside", and can eliminate redundant arcs between adjacent geometries (such as state borders). TopoJSON payloads can be upto 80% smaller than GeoJSON for the same amount of information. We used TopoJSON which contained each county's name and state name bound to the geometry to uniquely identify each county.

By normalizing the API to use the same key for each county as the SVG, we were able to associate the county-wise sighting data with the map, and as it changed month-to-month, d3 would transition the elements' colors accordingly.

Displaying the Data

Got ClojureScript?

Immediate wins:

• Easy to integrate into the existing stack
• Same language on both client and server
• Interoperability with JavaScript

ClojureScript seemed like an easy choice, given that we had a few immediate wins:

As we began using more of it, its value became more apparent. Initially, though, it took a little getting used to.

What is ClojureScript?

Well, what is it, precious?

When you ask most people what ClojureScript is, this is the reaction you seem to get. In the world of JavaScript dev, it's not very clear where ClojureScript fits in. It's not a framework, or syntactic sugar on top of JavaScript. So what is it? It's a program that compiles Clojure into JavaScript.

What is ClojureScript?

• Compiler for Clojure to JavaScript
• Emits JS optimized for the Google Closure library
• Several benefits over vanilla JS
  • Persistent data structures
  • Object keys as opposed to only strings
  • Laziness
  • Macros
  • Function argument destructuring

Compiler

• cljsbuild plugin for Leiningen

Closure

• Minification
• Namespacing modules
• Dead code elimination

Benefits

These benefits have been spoken about in detail in other talks, and we have resources at the end which list some of them. It's really interesting work, and has come a long way since the initial thought experiment.

So, the next step was to get d3 working with ClojureScript.

ClojureScript and JavaScript Interop

Methods

// JavaScript

var activeState = function() {
  return d3.select(".active");
}
            

;; ClojureScript

(defn active-state [] (.select js/d3 ".active"))
            

ClojureScript and JavaScript Interop

Properties

// JavaScript

var target = function() {
  return d3.event.target;
}
            

;; ClojureScript

(defn target [] (.-target (.-event js/d3)))
;; OR
(defn target [] (.. js/d3 -event -target)
            

ClojureScript and JavaScript Interop

Fluent APIs and the -> Macro

// JavaScript

var months = d3.time.scale
               .domain([new Date(2012,10,15), new Date(2013,10,15)])
               .range([0, 900])
            

;; ClojureScript

(def months ( -> (js/d3.time.scale)
   (.domain (array (js/Date. 2012 10 15) (js/Date. 2013 10 15)))
   (.range (array 0 900))))
            

Progress

Progress

Client-side Components

• Map
• Date slider
• Species list

We Have a Prototype! But...

• Unpolished UI
• No structure to the data
• The database query was slow
• Differing views of end result
• Not responsive

Chandu:

The fact that d3 could do DOM manipulation and event-handling let us build a prototype UI rather quickly by using d3 for both the mapping as well as the other interactive parts of the app, such as the bird and month selectors.

I felt like we were done. We had a working version, lots of material for write-ups, some good insights. For me, ClojureScript was still just a means to get d3 working, nothing more. But John had other ideas!

John:

• Datomic improvements combination of splitting into species and sighting entities and partitioning the sightings by species
• Achieved acceptable performance
• Looking for other ways to contribute.
• About that time there was a ton of exciting news happening in the clojurescript channels about a new library for front-end development.
• Dude, Om is the new hotness.

React and Om

Imagination Land

John: What Chandu did with D3 and the interactive maps is amazing and D3 is a super powerful library for creating visualizations but when it came time to create the controls for our application we were frustrated by the same thing we're always frustrated by namely the scattering around of state and the growing complexity of dealing with events and coordination of state.

We wondered what a functional approach to UIs would look like. [Next Slide]

• Pretend you don't know anything about the DOM or browser enfironment
• Unlearn the last 15 years of experience you have manipulating the DOM
• Picture an ideal world where the DOM is just a function of your program state

An Om Component

• Here you can see a list of bird species
• The user can type into the search box to filter the list
• Up and down arrows to highlight and enter to select
• Here we'll just focus on the filtering functionality

Initial Render

Remember: Imagination Land

;; ClojureScript

(ul {:className "species-list"}
  (li {:className "species"} (a {:href "#/taxon/1"} "Abert's Towhee"))
  (li {:className "species"} (a {:href "#/taxon/2"} "Acadian Flycatcher"))
  (li {:className "species"} (a {:href "#/taxon/3"} "Acorn Woodpecker"))
  ... )
            

Building the List

;; ClojureScript

(defn species-li [species]
  (li {:className "species"}
      (a {:href (path species)} (:common-name species))))

(map species-li all-species)
            

In imagination land the most straightforward way to render this list is to iterate over our data, building up our list as we go. Then when the list is built we replace old dom with our new representation. And our job is done.

Note that this code doesn't mutate the DOM directly. It returns a datastructure that we can then ask the browser to paint for us. This makes testing our view code easy.

Interactivity

;; ClojureScript

(map species-li
     (filter (match-string "black-thr") species))
            

User types in some filter text. On each keystroke we match against the list.

Putting It Together

;; ClojureScript

(defn filter-list-items [filter-text species]
  (ul {:className "species-list"}
      (map species-li
           (filter (match-string filter-text) species))))
              

Browserland

We Don't Live in Imagination Land

• Sadly we don't live in our imaginationland
• the reality of the browser is more like south park's imaginationland
• no data structures. no progression of dom values
• Replacing entire swaths of dom is slow - nobody does this
• instead, best practice is to replace only the bits that need to change
• it is left to the application programmer to manage all of this complexity
• frameworks like ember and angular shift this complexity to KVO - also complex!

React: A Better DOM

• Render the dom you want
• React takes care of the details
• Keeps real DOM in sync with your ideal DOM

• React gets us close to imaginationland
• open source library released by facebook
• your application renders a virtual-dom, which is just data
• react renders the real dom.
• when you render a new virtual dom, react does a diff with the previous version
• then it only performs the minimal set of dom manipulations necessary to achieve the desired structure.

Om

• Builds upon react
• Leverages ClojureScript's immutability

Om is a library which uses react and extends it with some powerful concepts. Because it uses clojurescripts persistent data structures it is able to achieve better out-of-the box performance than React itself. Let's see an example.

Om Example

;; ClojureScript

(defn species-list [model owner]
  (reify
    om/IRenderState
    (render-state [this state]
      (apply dom/ul #js {:className "species-list"}
             (om/build-all species-item
                           (:filtered-list state)
                           {:state (select-keys state [:highlighted :selected :select-ch])})))))
            

• this is the om equivalent of our imaginationland component
• our parent component keeps a filtered-list of species that match the current filter
• we render a ul and then iterate over that filtered list to render each species

Handling Events In Om

;; ClojureScript

(defn species-item [model owner]
  (reify
    om/IRenderState
    (render-state [_ {:keys [highlighted selected select-ch]}]
      (dom/li #js {:className (str "taxon"
                                   (if (= model highlighted) " highlighted")
                                   (if (= model selected) " active"))}
        (dom/a #js {:href (taxon-path (:taxon/order model))
                    :onClick (fn [e]
                              (.preventDefault e)
                              (put! select-ch model))}
               (display-name model))))))
            

• here we see the species-item component
• onClick registers an event handler
• our function in this case puts our model value onto a core.async channel
• which was passed in by this components owner
• this channel is our way of passing events back up the stack to the main application controller
• I was happy with the way things were shaping up so I rewrote the existing controls and opened up a PR

Brainsplosion

Chandu: It took me a little while to warm up to Om and React. In the beginning, it seemed like needless complexity as compared to what I was used to with say something like Angular, which would just bind to whatever state you already had.

But as I began to understand that I didn't need to care about the component's view state, only the data it needed, I began to see the simplicity of the approach. I wanted to try building a component, and the perfect opportunity presented itself.

Integrating With Flickr

One of the feature requests for Birdwave was to show a photo of the selected bird where possible. I needed a service that could take the bird name, search Flickr for images with a CC license and return the first one. What I didn't need was a giant Flickr library, just something simple that could build (as it turned out) 2 API requests.

Integrating With Flickr

;; ClojureScript

(ns bird-wave.flickr (:require [cemerick.url :refer (url)]))

(def api-base-url (url "https://api.flickr.com/services/rest/"))
;;#cemerick.url.URL{:protocol "https", :username nil, :password nil, :host "api.flickr.com", :port -1, :path "/services/rest", :query nil, :anchor nil}

(str (assoc api-base-url :query {:api_key "my_super_flickr_key"}))
;; https://api.flickr.com/services/rest/?api_key=my_super_flickr_key
            

Here's a simple example of what ClojureScript allowed me to do: Query string can contain: api_key, nojsoncallback, license, search The url library allows me to take a string and create a URL instance I can associate a map of key values to this instance When I call str on it, it creates a query string with the map * It made creating the long Flickr API strings a breeze

Integrating With Flickr

Updating the Model

;; ClojureScript

(js/d3.json search-url (fn [data]
  (om/update! model :photo (first-photo data))))
;; { "photos": { "page": 1, "pages": "223", "perpage": 1, "total": "223",
;;  "photo": [
;;    { "id": "4769690133", "owner": "31064702@N05", "secret": "818406d0cd", "server": "4123", "farm": 5, "title": "Eastern Kingbird", "ispublic": 1, "isfriend": 0, "isfamily": 0, "ownername": "Dawn Huczek", "url_q": "https:\/\/farm5.staticflickr.com\/4123\/4769690133_818406d0cd_q.jpg", "height_q": "150", "width_q": "150" }
;;  ] }, "stat": "ok" }

(js/d3.json url (fn [data]
  (om/update! model :attribution (attribution data))))
;; { "photo": { "id": "4769690133", "secret": "818406d0cd", "server": "4123", "farm": 5, "dateuploaded": "1278473496", "isfavorite": 0, "license": 4, "safety_level": 0, "rotation": 0, "originalsecret": "d7072dbb9a", "originalformat": "jpg",
;; "owner": { "nsid": "31064702@N05", "username": "Dawn Huczek", "realname": "", "location": "USA", "iconserver": "2915", "iconfarm": 3, "path_alias": "" },
;; ...
;; "urls": {
;;   "url": [
;;     { "type": "photopage", "_content": "https:\/\/www.flickr.com\/photos\/31064702@N05\/4769690133\/" }
;;   ] }, "media": "photo" }, "stat": "ok" }
            

When a user selects a bird, we make the first request. search-url is the Flickr url we need to hit. The json we get back is shown under. The first-photo function extracts the nested photo info.

When a user clicks the "view attribution" link, we make the second request, and fetch the json under that. The attribution function pulls out the username and link to the Flickr photo page.

Integrating With Flickr

The Selection Image Component

;; ClojureScript
(dom/div #js {:id "selection-image"
              :className (if (seq model) "loaded" "no-photo")}
   (dom/img #js {:className "photo"
                 :src (try-with-default model :url_q "/images/loading.png")})
   (dom/div #js {:className "attribution"}
     (dom/h3 #js {:className "title"}
       (try-with-default model :title "No photo available"))
     (dom/div #js {:className "by"}
       …
       (if (seq (:attribution model))
         (dom/a #js {:className "detail fetched"
                     :href (get-in model [:attribution :url])
                     :target "_blank"}
                (get-in model [:attribution :by]))
         (dom/a #js {:className "detail"
                     :href "#"
                     :onClick #(fetch-attribution % model)}
                "view attribution")))))
            

Progress

Progress

Client-side Components

• Map
• Date slider
• Species list

• Push state
• Bird photo
• Header
• Typeahead

Birdwave with Om

This Is Way Better! But...

• Unpolished UI
• No structure to the data
• The database query was slow
• Differing views of end result
• Not responsive

Adding Responsiveness

Adding Responsiveness

• CSS is not enough
• Screen size is user input

Adding Responsiveness

Client changes

• Detect and store screen size in app state
  • xs: 0px < width <= 520px
  • sm: 520px < width <= 768px
  • md: 768px < width <= 1024px
  • lg: 1024px < width

Adding Responsiveness

;; ClojureScript

(defn watch-screen-size [model]
  (let [size-handler (fn [size]  #(swap! model assoc :screen-size size))]
    (-> js/enquire
      (.register "screen and (min-width: 0px) and (max-width: 520px)"    (size-handler "xs"))
      (.register "screen and (min-width: 521px) and (max-width: 768px)"  (size-handler "sm"))
      (.register "screen and (min-width: 769px) and (max-width: 1024px)" (size-handler "md"))
      (.register "screen and (min-width: 1025px)"                        (size-handler "lg")))))
            

Adding Responsiveness

Client changes

• Update necessary components to render accordingly
  • Map renders states vs counties on md, sm, xs
  • Photo does not render on md, sm, xs
  • Slider changes to select widget on sm, xs
• Update ajax calls
  • Map data
  • API requests

Adding Responsiveness

;; ClojureScript

;; display the slider on large screens, and the select widget on small
(if (contains? #{"lg" "md"} (:screen-size model))
  (om/build date-slider model {:state {:time-period-ch time-period-ch}})
  (om/build date-select model {:state {:time-period-ch time-period-ch}}))
            

Adding Responsiveness

Server changes

• Handle query parameters on API requests
• Add queries to return data aggregated by state vs county

Progress

Ship It!

Resources

Birdwave

• Announcing Birdwave
• Choropleths and d3js
• Using the Flickr API with ClojureScript
• Adding an Om Component -- a Walkthrough
• Responsive JavaScript with EnquireJS

Resources

Birding

• eBird.org
• Cornell Lab of Ornithology
• Birding portal by Cornell

Resources

Clojure

• clojure.org
• Are We There Yet? - Rich Hickey
• Datomic
• Pedestal
• core.async

Resources

ClojureScript

• The ClojureScript Compilation Pipeline
• David Nolen on ClojureScript and Om
• David Nolen's seminal post on Om's approach
• Running ClojureScript with Node
• React.js
• Be Predictable, Not Correct (React.js)

Resources

d3

• d3js Home page
• Scott Murray's tutorials...
• ... and his book
• Mike Bostock's simple map example
• Building a Choropleth
• Play with GeoJSON