Anatomy of a Stat

Stats are represented in three parts:

name value timestamp
name value timestamp
name value timestamp
name value timestamp
...

Stats - Names

A name is a string with some periods in it, like:

some.text.with.periods.in.it

Each period will represent a directory on the filesystem, each with a subdirectory, ending in the location of the whisper database archive for the stat. Stated differently, carbon will create a tree of directories, containing subdirectories, until a stat name has no more periods.

Stats - What's in a Name?

Many query functions allow wildcards like this:

averageSeries(my.company.server.*.requests.500)

With the wildcard syntax, graphite will look in all subdirectories of my/company/server for any stat in a subdirectory at requests/500. Graphite will combine all the values and, for this function, anyway, average them.

Stats - Values

    float('nan')
    float('-inf')
    float('+inf')

These python types represent floating point numbers, ultimately platform specific implementations of IEEE 754 numbers. Although they can be stored in whisper archives, tools (graphite-web in particular) will coerce them to None.

    None

None, the python null, represents the absence of a measurement at a given timestamp.

Stats - Timestamps

Graphite timestamps are floats that represent milliseconds since the UNIX epoch

Whisper

• Whisper is essentially directories and files on a disk and some python scripts
• Whisper requires size and resolution of database files to be specified in advance
• Schemas are immutable, but Whisper can reindex from one archive into a new schema
• Reindexing or even just renaming all the time is not at all practical (source: my personal experience)

Whisper, pt. 2

• All time-slots within an archive take up space whether or not a value is stored
• Each whisper archive has a header section containing schema that describes how downsampling will occur between time resolutions
• Multiple archives in a schema can't be of arbitrary resolution - they must be even multiples
• In a whisper archive, the absence of a data point has semantic meaning

Organizing ES Data

ES Indexes

Top level container, indexes are not related to each other in any particular way unless a query specifically indicates otherwise

When using logstash, an index represents a period of time (an hour of a day, or a day); however, nothing enforces this other than the name of the index

ES Mappings and Types

Mappings define the transformation from _source, the raw document, to an indexed version with metadata, having fields, which could have more specific types than String or relationships to each other that are resolved during a query

ES Documents

Anything goes, but JSON is easiest to manage. Logstash documents look like this:

{
    '@timestamp': new Date(),
    '@version': 1,
    'message': "this text was input to logstash,
                and logstash wasn't configured to do any 
                processing on it. Logstash will simply pass it 
                through to elasticsearch in a JSON property 
                called message."
}