Marcos Iglesias

• My name is Marcos
• Software Engineer at Eventbrite

El Bierzo

• I am Spanish, from a beautiful region called El Bierzo

Online

• Personal Blog
• Eventbrite Engineering Blog
• @golodhros

• Find me on my personal blog and EB's engineering blog
• I am also on twitter, here's my unpronunciable handler, where I share interesting articles and resources about dataviz, d3 and front end technologies in general.
• Let's see what bring you here tonight

• [Poll]
• Who does not know what D3 is?

D3 introduction

• Data-Driven Documents
• JavaScript library to manipulate data based documents
• Open web standards (SVG, HTML and CSS)
• Allows interactions with your graphs

• D3 stands for data-driven documents. It is a JS library with a huge API and focused on mathematical transformations of data.
• It is based on web standards as SVG, HTML and CSS, and one of it’s strong points is its integration of animations and user interaction capabilities.
• One of it’s more celebrated features is it’s seamless integration of animations and the way it allows us to create engaging interactions.
• D3 is a Front End toolbox for creating interactive visualization, but it is not a charting library. So if you are looking for a package that gives you a bunch of ready made charts, d3 by itself won’t help you, but you could choose among several projects based on d3.

How does it work?

• Loads data
• Binds data to elements
• Transforms those elements
• Transitions between states
• Example

• D3 loads data and attaches it to the DOM. Then, it binds that data to DOM elements and transforms those elements, transitioning among states if necessary. Example.
• The bubbles represent a data entry. The transformation I refer is the size and color of the bubble, so the bigger the bubble, the higher the quantity; the greener the value, the higher the rise against previous year. Smart, isn’t it?
• That’s not all, as we can click on one of the buttons and it changes the focus of the visualization, using the same data but showing another point of view. Check out the animations, they are kind of “automatic”. Magic!

D3 Niceties

• Based on attaching data to the DOM
• Styling of elements with CSS
• Transitions and animations baked in
• Total control over our graphs
• Amazing community
• Decent amount of publications

• Same tools as FE development
• Perfect toolbox for interactive data visualizations

What can you do with D3?

• Let's talk about its uses
• Lots of possibilities

Bar charts

• All kind of business charts

Pie charts

Bubble charts

Choropleth

Map projections

• All kind of maps

Dashboards

Algorithm visualization

Artistic visualizations

Interactive data explorations

Contracting story

• I was contracting for a B2B company with a data analysis dashboard product

Marketing guy: Hey, I saw this nice chart, could we do something like that?

• Nice chart for a marketing page

• It was a bump chart, but uglier than this one
• I followed my usual workflow and had something ready pretty quickly

He loved it!

Usual workflow

• That workflow starts...

Search for an example

• Several options

Read and adapt code

Add/remove features

Polish it up

Usual workflow

• Idea or requirement
• Search for an example
• Adapt the code
• Add/remove features
• Polish it up

• Looks like a good plan, right?
• The problem lays on the code we need to adapt, as it is usually written in the standard way.

The standard way

• The SW is just the way we usually find D3 code on the Internet
• I will use a different example to show it

Code example

• Simplest example
• One of my first contacts with D3 was via this article
• Although there is not a lot of code, there is a lot of stuff happening here.
• Let's dive into it

Creating container

var margin = {top: 20, right: 20, bottom: 30, left: 40},
    width = 960 - margin.left - margin.right,
    height = 500 - margin.top - margin.bottom;

var svg = d3.select("body").append("svg")
    .attr("width", width + margin.left + margin.right)
    .attr("height", height + margin.top + margin.bottom)
  .append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");
                        

• First D3 pattern: Mike Bostock Margin Convention
• It can be considered a hack, but only once per chart
• T: Before discovering this pattern, moving stuff was hard

Setting up scales and axes

var x = d3.scale.ordinal()
    .rangeRoundBands([0, width], .1);

var y = d3.scale.linear()
    .range([height, 0]);

var xAxis = d3.svg.axis()
    .scale(x)
    .orient("bottom");

var yAxis = d3.svg.axis()
    .scale(y)
    .orient("left")
    .ticks(10, "%");
                        

• Scales are functions that glue data and pixels
• T: Input domain, output range
• Axis are visual representations of those scales
• Creating the scales and the axis that depend on them

Loading data

// Loads Data
d3.tsv("data.tsv", type, function(error, data) {
  if (error) throw error;
  // Chart Code here
});

// Cleans Data
function type(d) {
  d.frequency = +d.frequency;
  return d;
}
                        

• Callback pattern
• Cleaning function that coerces the values to be a number with this JS hack

Drawing axes

// Rest of the scales
x.domain(data.map(function(d) { return d.letter; }));
y.domain([0, d3.max(data, function(d) { return d.frequency; })]);

// Draws X axis
svg.append("g")
    .attr("class", "x axis")
    .attr("transform", "translate(0," + height + ")")
    .call(xAxis);

// Draws Y axis
svg.append("g")
    .attr("class", "y axis")
    .call(yAxis)
  .append("text")
    .attr("transform", "rotate(-90)")
    .attr("y", 6)
    .attr("dy", ".71em")
    .style("text-anchor", "end")
    .text("Frequency");
                        

• Set the domain of the scales, dependent on the data
• Draws the x and y axes, labeling the latter

Drawing bars

svg.selectAll(".bar")
  .data(data)
.enter().append("rect")
  .attr("class", "bar")
  .attr("x", function(d) { return x(d.letter); })
  .attr("width", x.rangeBand())
  .attr("y", function(d) { return y(d.frequency); })
  .attr("height", function(d) { return height - y(d.frequency); });
                        

• Here we see the Enter, Update, Exit

Output

• So, now we have a bar chart that works right?
• Well… it’s true, it works, and probably you could copy this code, modify it a bit and use it somewhere in your website.
• But let's be serious, who thinks this is good-quality code?

Standard D3: drawbacks

• Monolithic functions
• Chained method calls
• Hard to change code
• Impossible to reuse
• Delicate

• Read from top to bottom to know what is doing
• Compact code that is not really flexible
• Hardcoded values and strings
• Reuse is copy/paste, not DRY
• Won’t know if it works after modifying any line of code
• See how this has to do with my story

Story continues...

• Remember we had a bump chart, and he loved it

• But you know, he was a marketing guy...

Marketing guy: What if we change this thing here...

• I would need to refactor, but this refactor was something like this:

• Because I wasn't refactoring, I was just changing things!

• So I had to deal with some magic numbers

Trial and error

• Kill some bugs

Done!

M-guy: nice, let’s change this other thing!

• You keep it cool

• Fight some nasty bugs

Done!

M-guy: Great! I love it so much I want it on the product!

M-guy: So good you have it almost ready, right?

• More quality was necessary

• I had to deal with some Impostor syndrome at that point

• More nasty bugs

• A terrible experience

I was hating myself!

• That day my usual workflow failed me!
• I know you must have felt this way sometimes
• In my mind, there are 3 different

Possible outcomes

• You take it through
• You dump it and start all over again
• You avoid refactoring

• More effort than what it’s worth
• Dump is Hard, because of the investment
• Heroic, could be the fastest at first
• First two are not really time effective
• Last option could end in something like this:

• Hard time working with this
• Next developer will hate you

Reusable Chart API

• Who knows what the reusable chart API is?
• Essentially a way of creating components for D3 charts.

jQuery VS MV*

• It’s like when we moved away from jQuery based UIs to MVC or MV* frameworks
• T: Precise analogy, not everybody would get it

• First mention in Mike Bostock's seminal post Towards Reusable Charts
• It hasn’t changed a lot since then
• T: Well known, established technique. Libraries based in d3 like NVD3 uses it. I found it on DD3E and Mastering D3.

Reusable Chart API - code

return function module(){
    // @param  {D3Selection} _selection A d3 selection that represents
    // the container(s) where the chart(s) will be rendered
    function exports(_selection){

        // @param {object} _data The data to generate the chart
        _selection.each(function(_data){
            // Assigns private variables
            // Builds chart
        });
    }

    // @param  {object} _x Margin object to get/set
    // @return { margin | module} Current margin or Bar Chart module to chain calls
    exports.margin = function(_x) {
        if (!arguments.length) return margin;
        margin = _x;
        return this;
    };

    return exports;
}
                        

• Returns a function that will accept a d3 selection as input
• It will extract the data from that selection to build a chart, using the selection as container.
• T: It will also accept some configuration, set beforehand.

Reusable Chart API - use

// Creates bar chart component and configures its margins
barChart = chart()
    .margin({top: 5, left: 10});

container = d3.select('.chart-container');

// Calls bar chart with the data-fed selector
container.datum(dataset).call(barChart);
                        

• T: We could also create different instances of the same chart

Reusable Chart API - benefits

• Modular
• Composable
• Configurable
• Consistent
• Teamwork Enabling
• Testable

• Change of mindset, from building charts to components. Create a library.
• You can build higher-order components from different elements.
• Output can change without changing the code, as we separate the core from the configuration
• Following the same strategy we can reuse methods between charts
• Easy to grab for newcomers and easy to share workload
• The most important, no legacy code

The TDD way

• Interaction
• We have seen the problems of example code and a solution, the Reusable Chart API.
• Solution allows us to test, opening a new world of possibilities. Among them is to Test Driven our Charts.
• Craft the same bar chart test-by-test
• T: I will skip some steps

The "before" block

container = d3.select('.test-container');
dataset = [
    {   letter: 'A',
        frequency: .08167
    },{
        letter: 'B',
        frequency: .01492
    },...
];
barChart = barChart();

container.datum(dataset).call(barChart);
                        

• 3 things: container, data and chart instance
• call chart with the container which has the data attached
• T: datum adds the data, but doesn't do a join

Test: basic chart

it('should render a chart with minimal requirements', function(){
    expect(containerFixture.select('.bar-chart').empty()).toBeFalsy();
});
                        

• Several ways of checking for an element

Code: basic chart

return function module(){
    var margin = {top: 20, right: 20, bottom: 30, left: 40},
        width = 960, height = 500,
        svg;

    function exports(_selection){
        _selection.each(function(_data){
            var chartWidth = width - margin.left - margin.right,
                chartHeight = height - margin.top - margin.bottom;

            if (!svg) {
                svg = d3.select(this)
                    .append('svg')
                    .classed('bar-chart', true);
            }
        });
    };
    return exports;
}
                        

Reference: Towards Reusable Charts

Test: containers

it('should render container, axis and chart groups', function(){
    expect(containerFixture.select('g.container-group').empty()).toBeFalsy();
    expect(containerFixture.select('g.chart-group').empty()).toBeFalsy();
    expect(containerFixture.select('g.x-axis-group').empty()).toBeFalsy();
    expect(containerFixture.select('g.y-axis-group').empty()).toBeFalsy();
});
                       

• Nice an tidy structure

Code: containers

function buildContainerGroups(){
    var container = svg.append("g").attr("class", "container-group");

    container.append("g").attr("class", "chart-group");
    container.append("g").attr("class", "x-axis-group axis");
    container.append("g").attr("class", "y-axis-group axis");
}
                       

Test: axes

it('should render an X and Y axes', function(){
    expect(containerFixture.select('.x-axis-group.axis').empty()).toBeFalsy();
    expect(containerFixture.select('.y-axis-group.axis').empty()).toBeFalsy();
});
                        

• Public effects are the axis elements on the DOM

Code: scales

function buildScales(){
    xScale = d3.scale.ordinal()
        .domain(data.map(function(d) { return d.letter; }))
        .rangeRoundBands([0, chartWidth], .1);

    yScale = d3.scale.linear()
        .domain([0, d3.max(data, function(d) { return d.frequency; })])
        .range([chartHeight, 0]);
}
                        

• Creating after getting data

Code: axes

function buildAxis(){
    xAxis = d3.svg.axis()
        .scale(xScale)
        .orient("bottom");

    yAxis = d3.svg.axis()
        .scale(yScale)
        .orient("left")
        .ticks(10, "%");
}
                        

Code: axes drawing

function drawAxis(){
    svg.select('.x-axis-group')
        .append("g")
        .attr("class", "x axis")
        .attr("transform", "translate(0," + chartHeight + ")")
        .call(xAxis);

    svg.select(".y-axis-group")
        .append("g")
        .attr("class", "y axis")
        .call(yAxis)
      .append("text")
        .attr("transform", "rotate(-90)")
        .attr("y", 6)
        .attr("dy", ".71em")
        .style("text-anchor", "end")
        .text("Frequency");
}
                        

• remember we first need the groups and svg

Test: bars drawing

it('should render a bar for each data entry', function(){
    var numBars = dataset.length;

    expect(containerFixture.selectAll('.bar').size()).toEqual(numBars);
});
                        

• Counting bars and data length

Code: bars drawing

function drawBars(){
    // Setup the enter, exit and update of the actual bars in the chart.
    // Select the bars, and bind the data to the .bar elements.
    var bars = svg.select('.chart-group').selectAll(".bar")
        .data(data);

    // If there aren't any bars create them
    bars.enter().append('rect')
        .attr("class", "bar")
        .attr("x", function(d) { return xScale(d.letter); })
        .attr("width", xScale.rangeBand())
        .attr("y", function(d) { return yScale(d.frequency); })
        .attr("height", function(d) { return chartHeight - yScale(d.frequency); });
}
                        

Reference: Thinking with joins, General Update Pattern

• Enter: creates elements as needed

Test: margin accessor

it('should provide margin getter and setter', function(){
    var defaultMargin = barChart.margin(),
        testMargin = {top: 4, right: 4, bottom: 4, left: 4},
        newMargin;

    barChart.margin(testMargin);
    newMargin = barChart.margin();

    expect(defaultMargin).not.toBe(testMargin);
    expect(newMargin).toBe(testMargin);
});
                        

Code: margin accessor

exports.margin = function(_x) {
    if (!arguments.length) return margin;
    margin = _x;
    return this;
};
                        

• All-in-one getter and setter

Looks the same, but is not

Final code: standard way

var margin = {top: 20, right: 20, bottom: 30, left: 40},
    width = 960 - margin.left - margin.right,
    height = 500 - margin.top - margin.bottom;

var x = d3.scale.ordinal()
    .rangeRoundBands([0, width], .1);

var y = d3.scale.linear()
    .range([height, 0]);

var xAxis = d3.svg.axis()
    .scale(x)
    .orient("bottom");

var yAxis = d3.svg.axis()
    .scale(y)
    .orient("left")
    .ticks(10, "%");

var svg = d3.select("body").append("svg")
    .attr("width", width + margin.left + margin.right)
    .attr("height", height + margin.top + margin.bottom)
  .append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

d3.tsv("data.tsv", type, function(error, data) {
  x.domain(data.map(function(d) { return d.letter; }));
  y.domain([0, d3.max(data, function(d) { return d.frequency; })]);

  svg.append("g")
      .attr("class", "x axis")
      .attr("transform", "translate(0," + height + ")")
      .call(xAxis);

  svg.append("g")
      .attr("class", "y axis")
      .call(yAxis)
    .append("text")
      .attr("transform", "rotate(-90)")
      .attr("y", 6)
      .attr("dy", ".71em")
      .style("text-anchor", "end")
      .text("Frequency");

  svg.selectAll(".bar")
      .data(data)
    .enter().append("rect")
      .attr("class", "bar")
      .attr("x", function(d) { return x(d.letter); })
      .attr("width", x.rangeBand())
      .attr("y", function(d) { return y(d.frequency); })
      .attr("height", function(d) { return height - y(d.frequency); });

});

function type(d) {
  d.frequency = +d.frequency;
  return d;
}
                        

Final code: TDD way

return function module(){
    var margin = {top: 20, right: 20, bottom: 30, left: 40},
        width = 960, height = 500,
        chartWidth, chartHeight,
        xScale, yScale,
        xAxis, yAxis,
        data, svg;

    function exports(_selection){
        _selection.each(function(_data){
            chartWidth = width - margin.left - margin.right;
            chartHeight = height - margin.top - margin.bottom;
            data = _data;

            buildScales();
            buildAxis();
            buildSVG(this);
            drawBars();
            drawAxis();
        });
    }

    function buildContainerGroups(){ ... }

    function buildScales(){ ... }

    function buildAxis(){ ... }

    function drawAxis(){ ... }

    function drawBars(){ ... }

    // Accessors to all configurable attributes
    exports.margin = function(_x) { ... };

    return exports;
};
                        

TDD way - benefits

• Stress free refactors
• Goal oriented
• Deeper understanding
• Improved communication
• Quality, production ready output

• Won’t be afraid of touching your charts anymore. First hacky approach, you can improve it now!
• As TDD forces you to first understand what you want to do
• Learn more about the D3 API.
• Create a documentation that does not get outdated. Small and specific code that can be understood easily.
• Charts will be a first-class citizen inside your repo. No rewrites.

How to get started

Some ideas

• Test something that is in production
• TDD the last chart you built
• Pick a block, refactor it
• TDD your next chart

• Try to test something you have in production, just a bit
• Check for the last easy chart you built, give it a read, and TDD it.
• If you want to see something new, pick a block, refactor it into TDD
• Write your next one with the reusable API

Repository walkthrough

• Code repository, the basement for a chart library
• Uses: Jasmine, Karma Runner
• Structure, configuration files
• Different branches
• Run tests and Debugging

What happened with my contracting gig?

I used the Reusable Chart API

• Line chart of just one value

Adding multiple dimensions?

• Added a second and third dimensions to find correlations

I had tests!

• My tests were there to back me up!

Toogle dimensions, adding more y-axis?

• New level of complexity

• I was pretty chill
• (Pause)

Books

• Super Fast introduction (<60 pg.) for web developers
• Talks about: Enter Selections, Scales, Axes
• Interactions and Transitions
• Layouts

• Intro Book for everyone
• Starting from 0 and no experience on FE
• GeoJSON and Maps

• Reusable charts in a way we could all use
• Charts are simple and test friendly
• You can create larger ones by composition
• Introduces a data loader module
• Proposes some tests with Jasmine
• All in a small size

• Same Reusable charts
• Ambitious book, almost a reference
• Integrations: Backbone, Socket.IO, Mapbox.
• Great number of different charts: Choropleth, Area, Advanced Maps
• Bower Package using Grunt for build and Vows for testing
• Less Importance to Tests, great book