Jamie Hyneman

Take a common myth

Devise an experiment

Build a test rig

Test the myth!

Make a determination

• Confirmed
• Plausible
• Busted

Cool Toys

Me too!

Ruby

Multi-Disciplinary

Lingua Franca

Shell!

Bash

The myth

"Bash isn't Webscale"

The build

A single-page web chat application, backed by a Bash script

https://github.com/avdi/walrus

Highlights

Serving HTTP

Netcat

Like cat(1) for TCP/UDP sockets.

A Simple HTTP server

$ printf "HTTP/1.1 200 OK\r\n\r\nHello world\n" |
    nc -l 8080

Client

$ curl localhost:8080
Hello world

Netcat output

$ printf "HTTP/1.1 200 OK\r\n\r\nHello world\n" |
    nc -l 8080
GET / HTTP/1.1
User-Agent: curl/7.32.0
Host: localhost:8080
Accept: */*

Serving Pages Dynamically

Netcat is Read/Write

No callbacks.

Dynamic Netcat

Branching on input.

Coprocesses

coproc NAME COMMAND

Rot13 Coprocess

$ coproc rot13 { stdbuf -oL tr '[A-Za-z]' '[N-ZA-Mn-za-m]'; }
[1] 26812

Coprocess Filehandles

$ echo ${rot13[0]}
63
$ echo ${rot13[1]}
60

Writing and Reading

$ echo "hello, world" >&${rot13[1]}
$ echo "this is a secret" >&${rot13[1]}
$ read line <&${rot13[0]}
$ echo $line
uryyb, jbeyq
$ read line <&${rot13[0]}
$ echo $line
guvf vf n frperg

Ending a Coprocess

The rude way

$ kill ${rot13_PID}

Ending a Coprocess

The polite way

somecommand 1&>-

exec 60&>-

Interpolating the coprocess STDIN

exec ${rot13[1]}&>-

(Doesn't parse correctly)

Using eval

eval "exec ${rot13[1]}&>-"

Iä! Iä! Cthulhu Fhtagn!

Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn!

Netcat as a coprocess

  # Start netcat
  coproc nc ( nc -l 8080 )

  # Plug it into a request handler
  handle_request ${nc_PID} <&${nc[0]} >&${nc[1]}

  # Close filehandles
  eval "exec ${nc[1]}>&-"     
  eval "exec ${nc[0]}<&-"

Organizing the Program

Shell Functions

Miniature Shell Scripts

• Positional params ($1, $2, $*, etc.)
• Redirectable STDIN, STDOUT, and STDERR
• Return an integer exit status

Naive ROT13 Function

rot13() {
  echo "$1" | stdbuf -oL tr '[A-Za-z]' '[N-ZA-Mn-za-m]'
}

$ rot13 "pssst"
cfffg

Pipeline ROT13 Function

rot13() {
  stdbuf -oL tr '[A-Za-z]' '[N-ZA-Mn-za-m]'
}

$ echo "pssst" | rot13
cfffg

stdbuf

stdbuf -oL tr '[A-Za-z]' '[N-ZA-Mn-za-m]'

stdbuf -oL tr '[A-Za-z]' '[N-ZA-Mn-za-m]'

Local Variables

foo() {
  bar=42 # global!
}
$ foo
$ echo ${bar}
42

foo() {
  local bar=42 # local
}
$ foo
$ echo ${bar} # no output

Serving static files

HTTP/1.1 GET /index.html

Content Type

text/html, application/json, etc.

Associative Arrays

(AKA Map, Dictionary, Hash)

declare -A content_types=([js]=text/javascript
                          [html]=text/html)

Find content type for extension

$ echo ${content_types[html]}
text/html

$ echo ${content_types[foo]-text/plain}
text/plain

Get file extension

$ file=public/index.html
$ echo ${file##*.}
html

Serving a file

local file="public/${path}"
if [ -f "${file}" ]; do
  local ext="${file##*.}"
  local type="${content_types[${ext}]}"
  printf "HTTP/1.1 200 OK\r\n"
  printf "Content-Type: ${type}\r\n\r\n"
  cat "${file}"
fi

Actor Model

But this is Bash...

John Agar

B-List Actor

B-Movie Actor Model

Create an Actor

agar() {
  # ...
}

Get actor name and args

local name=$1
local args=${@:1}

Create Pipe

mkdir -p fifos
local fifoname=fifos/${name}
mkfifo ${fifoname}

Spawn Actor

{ 
    queue=${fifoname} ${name} ${args[@]}
    rm -f ${fifoname}
} &

"Send" Helper

send() {
    local dest=$1
    local message=${*:1}
    echo "${message}" > fifos/${dest} &
}

Create an Actor

agar main

"Main" Function

main() {
  while true; do
    serve_with_coproc &
    read < ${queue}
  done
}

Signaling Main

# ...
read req_line
send main "continue"
# ...

Take that, Erlang!

Persistence

JSON Messages

{
  "name": "Avdi",
  "text": "Testing 1 2 3"
}

Database

Everyone's favorite NoSQL document database...

PostgreSQL!

Schema

CREATE TABLE messages (
    id        serial PRIMARY KEY,
    content   json,
    posted_at timestamptz DEFAULT now()
);

INSERT INTO messages (content) VALUES
('{
    "name": "Bullwinkle",
    "text": "Hey Rocky, watch me pull a rabbit out of my hat"
  }');
INSERT INTO messages (content) VALUES
('{ "name": "Rocky", "text": "Again?" }');
INSERT INTO messages (content) VALUES
('{ "name": "Bullwinkle", "text": "Presto!" }');

SELECT content->>'text' AS text
FROM messages
WHERE content->>'name' = 'Bullwinkle';

                      text                     
-------------------------------------------------
 Hey Rocky, watch me pull a rabbit out of my hat
 Presto!
(2 rows)

No need to parse JSON in Bash!

The Myth

"Bash isn't Webscale"

I got it working... barely.

Analysis

• Slow
• Unreliable
• Only works in Firefox (?!)
• Leaks Processes
• FIFOs are a pain to work with
• So is netcat

This was a terrible idea!

Myth: Confirmed

Dumb ideas

Gherkin

A Lisp in Bash

"The one dependency we can count on"

https://github.com/alandipert/gherkin

shasm

An x86 assembler in Bash

"It just continuously cracks me up."

—Rick Hohensee

"What a pointless waste of time"

—My brain

A Kernel in Rust

"If you ask questions that are really dumb, eventually you know things"

— Julia Evans

http://jvns.ca/

"How do I make the next Instagram?"

"How do I write a kernel?"

Outlier Knowledge

Comes from asking outlier questions

Experimentation, not Demonstration

When we experiment—when we try things, and we fail—we start to ask why, and that’s when we learn.

–Jamie Hyneman

A story to tell

Abstractions stripped away

• SQL
• HTTP

Greater familiarity with tools

• netcat
• psql
• stdbuf
• mitmproxy

Increased comfort with Bash scripting

• Functions
• Associative arrays
• Coprocesses

It was fun

Experiment!

Thank You!

Code: https://github.com/avdi/walrus

I make screencasts: RubyTapas.com

@avdi / avdi@avdi.org

• MythBusters: © 2014 Discovery Communications, LLC.
• Photo of Tom Cruise by Gareth Cattermole © 2010 Getty Images
• "The Mole People" poster © Universal Pictures
• All others either unknown or in the public domain