What is a log?

Logs are important

Large scale log management is hard

It's all about latency

Batch processing is too slow

We built Meniscus for large scale log management

Why not logstash, greylog, scribe, flume or X?

Different users have different needs.

Multiple tenants

Encryption

Durability

Compression

...

A service oriented system must address these needs directly

Designing with a clean slate

Must change client architecture as little as possible.

Must utilize already existing standards.

Must be performance oriented

Must completely automated

Python!

But wait, isn't python too slow for event stream processing?

74,387 log messages parsed per second.

That's 18.4 MB/sec per parser process with a little help from Cython.

HEY!

Python might just work

Scaling and distribution of work

The Grid

Intelligent and elastic handling of event streams

The Worker

Boxes that live in the grid and perform a specific task

Grid organization: service domains

Like workers carry the same personality and therefore can be considered route equivalent

Routing between service domains

A message is processed differently via routing between service domains

Building a grid

New workers are paired to the grid with a two line configuration

Coordinating the grid

Maintaining Grid Worker Synchronization with a little help from MongoDB

Worker Synchronization Strategies

• Coordinator synchronization broadcasts for low latency
• Worker polling for synchronization updates

So where do the events go?

MongoDB, HDFS and more! It all depends.

Indexing and more complex correlative functions

Let the downstream technologies do what they do best and get out of the way