Data Packages: put it in a box

A brief history of container shipping and an introduction to the Data Package.

Nick Stenning (@nickstenning).

We are living in the era of

Break-bulk data

The impact of containerisation

year,  vessel,             loadingCostPerTon
1956,  SS Ideal-X,         0.16
1956,  medium break-bulk,  5.86
          

year,  tonsPerManHour,     hoursInPort
1959,  0.627,              504
1976,  4234,               18
          

The Data Package

datapackage.json

{
  "name": "gold-prices"
}
          

datapackage.json

{
  "name": "gold-prices",
  "title": "Gold Prices (Monthly in USD)",
  "license": "odc-pddl",
  "sources": [{
    "name": "Bundesbank statistics",
    "web": "http://www.bundesbank.de/Navigation/[...]"
  }],
  "resources": [{
    "path": "data.csv",
    "format": "csv",
    "schema": {
      "fields": [
        {"type": "date", "name": "date"},
        {"type": "number", "name": "price"}
      ]
    }
  }],
  "version": "1.0.42"
}
          

datapackage.json

{
  "name": "gold-prices",                   # Identifier ([a-z0-9._-]+)
  "title": "Gold Prices (Monthly in USD)", # One-sentence description
  "license": "odc-pddl",                   # Open License identifier
  "sources": [...],                        # Array of sources (defined)
  "resources": [...],                      # Array of resources (part-defined)
  "version": "1.0.42"                      # Semver-compatible version
}
          

date,price
1950-01-01,34.730
1950-02-01,34.730
...
            

Data Package

↑

Tabular Data Package

Tabular Data Package

• ≥1 data file (AKA "resource")
• resource format == CSV
• resource schema (JSON Table Schema)

Tools

• Low-level handling tools
  • Data Package Manager
  • Python / R / Ruby libraries
• High-level authoring and management tools
  • Data Packager
  • CSVLint
  • Git Data Package viewer
  • Datacentral, the "poor man's CKAN"

http://dataprotocols.org/

https://github.com/datasets/

Just put it in a box already.

Image credits

• Workers carrying bananas: National Archives and Records Administration (NAI: 518791) on Wikimedia Commons
• Dockers, when loading bagged cargo: CC BY-SA by Buonasera on Wikimedia Commons
• Van carrier waiting for a container at HHLA Container Terminal: CC BY-SA by Derlorenz68 on Wikimedia Commons
• Container terminal from the air: CC BY-SA by Thomas Fries on Wikimedia Commons
• Empty shipping container: CC BY-NC-ND by James Good on Flickr
• Copyright-holders for remaining images could not be identified. Please contact the author to address any concerns around image rights.

Rights

This presentation is released under the terms of the Creative Commons Attribution License (International, 4.0).