Twelve days of metrics – Graphite for the masses – What Graphite is NOT

View Github Repository Open presentation in a new window

SEJeff

See all presentation from SEJeff

Twelve days of metrics – Graphite for the masses – What Graphite is NOT

1 0

graphite-talk

Graphite talk for Chicago LUG

On Github SEJeff / graphite-talk

Twelve days of metrics

Graphite for the masses

Created by Jeff Schroeder / @SEJeff / github: SEJeff / Presentation Video

http://envisionlinux.org/graphite-for-the-masses

Who Am I?

  • Husband of an awesome woman
  • Co-maintainer of graphite and salt stack
  • Lover of open source, bacon, and good beer
  • GNOME SysAdmin team and foundation member
  • Professional Linux superhero for a Chicago financial firm

What the heck is graphite?

Eyecandy for devops teams A fantastic visualization tool for time-series metrics A community of metrics and graph fanatics

What Graphite is NOT

  • A replacement for keeping good logs
  • A replacement for elasticsearch / logstash
  • A replacement for logstash

A friggin' unicorn!

(no unicorns were harmed in the creation of this presentation)

Basic Architecture

  • graphite-web - web application for adhoc graphing and REST API
  • carbon - twisted daemon listening for time-series data and persisting it
  • whisper - python database library similar to RRD with a few perks

Graphite Web

A composer for adhoc graphing, a dashboard for ops, and a REST API for developers/external tools.

Composer

Used for creating and manipulating new graphs on the fly

Dashboard

A built-in dashboard creator to add graphs to a big monitoring screen

Dashboards are Useful!

Mailgun monitors their elasticsearch infrastructure with graphite.

JSON API

The REST API allows other external tools to use graphite data for creating the pretty

Carbon

The caching, relay, and storage layer. Carbon is what clients actually send metrics to.

Mention it is written in twisted and make a bad joke about twisted

Storage Schemas

Carbon uses storage schemas 

[apache_busyWorkers]
pattern = ^servers\.www.*\.workers\.busyWorkers$
retentions = 5m:1d,15m:7d,1h:30d,6h:1y

RRA Example

Feeding Carbon

Minimal carbon client

Graphite docs on sending metrics

#!/bin/bash
# Basically the best carbon client in the world
GRAPHITE_SERVER=${GRAPHITE_SERVER:-localhost}
GRAPHITE_PORT=${GRAPHITE_PORT:-2003}

echo "foo.bar.baz $(($RANDOM % 500)).$(($RANDOM % 5)) $(date +%s)" | \
    nc "$GRAPHITE_SERVER" "$GRAPHITE_PORT"

metric.schema.name value_as_float unix_epoch_time

Aggregation

The carbon aggregator buffers metrics and performs down-sampling operations. Examples could be things like summing requests per hour and only firing the counter to the cache once per hour. Many people use one of the various forms of statsd to do this as well.

TL;DNR: Use statsd, it is awesome!

Aggregating metrics lessens the I/O load on your caches quite a bit. Issues with the graphite aggregator.

Carbon Relay

Regex-based rules or consistent-hashing.

Example relay-rules.conf

[db_servers]
pattern = ^collectd\..*db.*
servers = 10.1.2.3, 10.1.2.4:2004, myserver.example.com
Describe consistent hashing and a few upsides of it

Whisper

Whisper is a python library created essentially to be a better RRD.

Better than RRD?

The internets agrees!

RRD

In RRD speak, data retention policies are configured via a RRA using reverse polish notation.

Did Someone Say Polish?

Sorry, not the yummy hot dogs us Chicagoans love!

Example RRA

Taken from stack overflow, so it has to be right

RRA:AVERAGE:0.5:1:288  \ # 5 minute avg for 1 day
RRA:AVERAGE:0.5:3:672  \ # 15 minute avg for 1 week
RRA:AVERAGE:0.5:12:744 \ # 1 hour avg for 1 month
RRA:AVERAGE:0.5:72:1480  # 6 hour avg for 1 year

storage schema example

WAT?

This assumes a step of 300 seconds. Slide just shows "steps" and "rows". This leaves out step"

Teh Pretty

Enough time blabbing, time for 12 great graphite tools

Graphene

Graphene is built for tv static dashboards

framework, some assembly required

Giraffe

Giraffe is great viewing pre-generated realtime dashboards

framework, some assembly required

cubism.js

Written by square, cubism, helps build compact and realtime dashboards.

javascript framework, some assembly required

graphite.js

If you love jQuery (I do!), graphite.js, makes it pretty trivial to pull graphs into your existing application

javascript framework, some assembly required

Full dashboard applications

For those who don't prefer a roll your own approach

Descartes

Descartes is a full featured graphing dashboard with builtin collaboration and native "composer" functionality.

obfuscurity wrote it woop woop

Gdash

Gdash uses a simple DSL for generating graphs and is easy to get going

Doesn't graph client-side graphs and uses a dsl for graphs that makes creating new dashboards quick

Tasseo

Tasseo is an alerting dashboard that can change graph colors based on thresholds.

Tattle

Tattle is a popular alerting dashboard

Sends alerts via email and allows setting thresholds in the web ui

Orion

Orion full featured dashboard application with granular access control over metrics

Oauth access control

Graph Explorer

Graph Explorer replacement for the graphite composer with tagging and other cleverness

Vimeo wrote this!

Graphsky

Graphsky is more or less a ganglia replacement dashboard built ontop of graphite

Vimeo wrote this!

Dusk

Dusk a realtime hotspot dashboard

uses cubism, amongst other things and obfuscurity wrote it woop woop

More of teh awesome

THE END