Ceph performance

demystified

Benchmarks, Tools, and the Metrics That Matter

ceph -s

HEALTH_OK

But does it

perform?

Latency

Throughput

IOPS

RADOS

RBD

radosgw

CephFS

Block

Network

Ceph cluster

Benchmarks

Block device benchmarks

(simple)

dd if=/dev/zero of=/dev/sdh1 bs=1G count=1 oflag=direct

dd if=/dev/zero of=/dev/sdh1 bs=512 count=1000 oflag=direct

Block device benchmarks

(advanced)

fio --size=100m \
    --ioengine=libaio \
    --invalidate=1 \
    --direct=1 \
    --numjobs=10 \
    --rw=write \
    --name=fiojob \
    --blocksize_range=4K-512k \
    --iodepth=1

Network benchmarks

netperf -t TCP_STREAM -H <host>

OSD benchmark

ceph tell osd.X bench

rados benchmark

rados bench -p bench 30 write

Do this on a throwaway pool!

fio RBD benchmarks

fio --size=10G \
    --ioengine=librbd \
    --invalidate=0 \
    --direct=1 \
    --numjobs=10 \
    --rw=write \
    --name=fiojob \
    --blocksize_range=4K-512k \
    --iodepth=1 \
    --pool=bench \
    --rbdname=fio-test

Mind the cache!