Hacking an nginx module

@donatasm

Hacking in a good way!

• nginx is modular
• add features by building a module
• source code is the only documentation not true anymore, api docs are work in progress, woohoo! https://www.nginx.com/resources/wiki/extending/api/

Why would you need a custom module?

• tracking metrics of your bussiness
• dynamic responses in case backend servers fail
• get rid of http from your backend, because nginx does it best

Nginx

An http and reverse proxy server

Nginx is fast

Nginx is scalable

• Modern I/O event notification mechanisms
  • epoll, on Linux
  • kqueue, on OS X
  • IO completion ports, on Windows

Nginx architecture

Why this is cool

Hot-swap releases!

Hot-swap releases

start_new_master() {
  echo -n $"Starting new master $prog: "
  killproc -p $PIDFILE $prog -USR2
}

kill_old_master() {
    echo -n $"Killing old master $prog: "
    old_pid_file="$pid_location/running.pid.oldbin"
    killproc -p $old_pid_file $prog -QUIT
}

upgrade() {
    start_new_master
    sleep 1
    kill_old_master
}
          

Nginx configuration

• Is made of module directives
  • A configuration entry that loads and configures a module

Nginx config contexts

• Each directive has a context
  • A hierarchical structure grouping directives

Nginx config context hierarchy

• main
  • events
  • http
    • server
      • location
        • location
        • ...
    • ...
    • upstream
    • ...

Nginx config example

worker_processes 16;
working_directory /usr/local/bidder-http/logs/;

http {
  error_log /usr/local/bidder-http/logs/error.log warn;
  log_not_found off;

  server {
    listen *:80;

    location /bidder/bids {
      bidder name handler=openrtb timeout=80 bidders;
    }
  }

  upstream bidders {
    bidder_req_limit;
  	server 11.12.13.14:8080 weight=4 max_fails=0 max_conns=6 req_limit=128;
  }
}
          

Nginx module types

• handler
• filter
• upstream

Handler module

• read the request
• generate a response

Filter modules

• enriches handler's request or response
• filter modules can be chained

Upstream modules

• chooses one backend server
• transforms an http request if needed
• sends it to backend server
• receives a response and transforms it back

Writing and compiling a module

DEMO

Nginx buffers

Writing and compiling a module

DEMO

Nginx custom modules in production

Nginx custom modules in production

• load balancing
  • choosing the node that is "alive and "fast"
• respond with default response if something wrong
• wait for backend max 80-100ms

Libs and modules

• Upstream check
  • https://github.com/yaoweibin/nginx_upstream_check_module

Upstream check

Libs and modules

• Jasmine
  • a lightweigth json tokenizer
  • http://zserge.com/jsmn.html

Libs and modules

• HdrHistogram (C)
  • histograms for tracking latencies
  • https://github.com/HdrHistogram/HdrHistogram_c

Testing

• unit test tcp stack is quite a challenge
• treating nginx process as black box:

Small Scala test framework to rescue

it should "return empty bid response " +
  "when request contains requestId field" in withNginxBidder { bidder =>
  val requestId = random.nextLong()
  val response = bidder.client
    .send(createBidRequest(requestId))
  response should equal (emptyBidResponse(requestId))
}
          

Testing, more examples

  it should "send request to the bidder" in withNginxBidder { bidder =>
    val bidRequestBody = Array[Byte](1, 2, 3, 4, 5)
    val bidRequest = createBidRequest(bidRequestBody)
    bidder.server.respondWith(EmptyBidResponse)
    val bidResponse = bidder.client.send(bidRequest)
    bidResponse should equal(EmptyBidResponse)
    bidder.server.requests should contain (bidRequest)
  }

  it should "return 413 for huge request body" in withNginxBidder { bidder =>
    val bidRequestBody = (for (i <- 1 to 10000000) yield 1.toByte).toArray
    val bidRequest = createBidRequest(bidRequestBody)
    val bidResponse = bidder.client.send(bidRequest)
    bidResponse should equal (HttpResponse(413, HttpEntity.Empty,
      List(`Content-Length`(0), `Connection`("close"))))
  }
          

Testing, nginx wrapper

def withNginxBidder(testCode: Bidder => Any): Unit = {
  var nginx: Nginx = null
  var bidder: Bidder = null
  implicit val system = ActorSystem("bidder-http", akkaConfig)

  try {
    nginx = Nginx()
    nginx.errors should not include "[emerg]"
    nginx.errors should not include "[error]"
    bidder = new Bidder
    nginx.spinUntilStarted(100)
    testCode(bidder)
  }
  finally {
    system.shutdown()
    system.awaitTermination()
    if (nginx != null) {
      Try(nginx.destroy())
      assert(nginx.exitCode != Signal.SIGSEGV,
        "nginx process exited with segmentation fault")
    }
  }
}
          

Benchmarking

#!/bin/sh

wrk -c4096 -t16 -d1h \
  --latency --script request.lua \
  http://1.2.3.4:7070/bids
          

Debugging

• ./configure --with-debug
  
  gdb /path/to/nginx core.dump
  backtrace full
  
  ngx_log_debug0(NGX_LOG_DEBUG_HTTP, request->connection->log, 0,
    "THIS IS DEBUG MESSAGE");
              

Useful links

• Emiller's Guide To Nginx Module Development
• MySQL support in NGINX
• Разработка модулей для nginx

Thank you!

QA