Tuning corner

View Github Repository Open presentation in a new window

rluta

See all presentation from rluta

Tuning corner

0 0

network-intro

Le réseau, cet inconnu au coeur de votre application @ BreizhCamp 2016

On Github rluta / network-intro

Le réseau, cet inconnu au centre de vos applications

@raphaelluta #BzhCmp

MeowApp 1.0

Customer feedback

Network team perspective

Network characteristics

Bandwidth

Latency

Error rate

Duplex

MTU

Jitter

BDP (Bandwidth x Latency)

An alternate view of networks

Full Duplex

Bandwidth

MTU

Latency

Jitter

Error Rate

Ideal conditions

Duplex must be full

Error rate should be close to 0.0 %

MTU should be as big as possible

[...]the maximum length of an IP datagram sent over an Ethernet is 1500 octets. IETF RFC 894

Bandwidth vs Latency Evolution

Scale:
log
linear

Over the last 20 years, bandwidth has improved 1000 times faster than latency

Basics: Light speed

c = 300,000 km/s

(in vacuum)

That's about 190,000 km/s in fiber

approx. 5 ms per 1,000 km

Paris - Dublin 800 km 4 ms Paris - New York 6,000 km 30 ms Paris - San Francisco 9,000 km 45 ms

Typical networks in France

Bw avg (Mbps) RTT avg (ms) Duplex BDP (KB) Ethernet 1,000 0.5 Full 62.5 ADSL 2 5 0.5 40 Full 25 2.5 Cable 100 20 20 Full 250 50 Fibre 200 10 Full 500 EDGE 1 400 Full 50 3G 4 150 Full 75 4G 8 120 Full 120 WiFi 3-300* 2 Half 0.75 - 75 Satellite 1-1,000 500 Any* 62.5 - 62,500

Transport protocols

UDP: Fire and Forget

TCP: Reliable byte stream

Typical TCP Chat

Server
ACK
Connecting
SYN
Established
Closing
FIN
ACK
Closed (Time wait)
Closed
Messages
Hello
Hi. How are you ?
Fine
Can you show me your pussy ?
Now get lost
ok
CU
FU
Client
SYN
Connecting
ACK
Established
ACK
Closing
FIN
Closed

Windows Everywhere

  • Windows regulate flow between sender and recipient
    • Receiver window (rwnd) avoids recipient overload
    • Congestion window (cwnd) prevents network saturation
  • Max # of inflight bytes = min(rwnd, cwnd)
  • Window changes determined by :
    • Recipient available buffer (rwnd)
    • Slow start (cwnd)
    • Congestion avoidance (cwnd)

Time to Load: 200 KB

  • Adsl
D.Ack
Reset
Add
  • Adsl
  • Adsl Old
  • Cable
  • Ethernet
  • 3G
  • Ether. Far
Bandwidth (Mbps):
0
RTT (ms):
0
Rwnd (kB):
0
Cwmd (p.):
0
MTU (B):
0

Tuning corner

Linux tuning

Adjust initial congestion window $ sudo ip route change default via 192.168.1.1 dev eth0 metric 100 initcwnd 10 $ ip route | grep default default via 192.168.1.1 dev eth0 metric 100 mtu 1500 initcwnd 10
Avoid slow-start after idle connections $ sudo sysctl -w net.ipv4.tcp_slow_start_after_idle = 0
Ajust windows $ cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 6291456 # echo 4096 87380 16777216 > /proc/sys/net/ipv4/tcp_rmem $ cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 4194304 # echo 4096 65536 16777216 > /proc/sys/net/ipv4/tcp_wmem
Increase MTU $ sudo ifconfig eth0 mtu 9000 up $ sudo ip route change default via 192.168.1.1 dev eth0 metric 100 mtu 9000

Example: SOAP Request

 

Request stats

Markup size Content size Preamble size

Serialization

  • Eliminate noise
    • White spaces, comments, EXIF profiles, etc...
  • Ensure good serialization
    • Beware excessive field name lengths
    • Value dictionaries are your friend
  • Use efficient protocols
    • Protobuf, Thrift, Avro
    • JSON

Compress data

Database join query

Objective

Retrieve a list of orders for 100 customers

Assumptions

  • Average 4 orders per customer
  • Average 50 bytes of info per order
  • Average 20 bytes of info per customer
  • Network RTT is 1 ms, bandwidth 1 Gbps

Query styles

Big join

$orders = query(select c.name, o.*   from customers c join orders o on c.id = o.custId)

Lazy join

$custs = query("select id,name from customers") for ($custs as $c)   $orders = query("select * from orders where custId=${c->id}")

Query analysis

Big join

70 bytes * (100 * 4) = 28000 bytes payload 1 RTT connect + 3 RTT payload + query time Total: 4 ms + query (20 packets, 28.4 kB transfer)

Lazy join

First query:   20 bytes * 100 = 2000 bytes payload   1 RTT connect + 1 RTT payload + query time Per customer id (x100):   50 bytes * 4 = 200 bytes   1 RTT payload + query time Total: 102 ms + query (102 packets, 22 kB transfer)

Batch requestsReuse connections

Localhost

raphael$ ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
raphael$ ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.050 ms
raphael$ iperf -s -p 8888
...
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  43.6 GBytes  37.4 Gbits/sec

Don't benchmark on localhost !

Shaping Tools: GUI

Shaping Tools: command line

Add constant latency $ sudo tc qdisc add dev eth0 root netem delay 20ms $ sudo tc qdisc show dev eth0 qdisc netem 8004: root refcnt 2 limit 1000 delay 20.0ms
Add latency + jitter $ sudo tc qdisc change dev eth0 root netem delay 20ms 2ms 15% $ sudo tc qdisc show dev eth0 qdisc netem 8005: root refcnt 2 limit 1000 delay 20.0ms 2.0ms 15%
Lossy connection $ sudo tc qdisc change dev eth0 root netem loss 1 $ sudo tc qdisc show dev eth0 qdisc netem 8005: root refcnt 2 limit 1000 loss 1%
Remove shaping $ sudo tc qdisc del dev eth0 root $ sudo tc qdisc show dev eth0 qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Bandwidth shaping $ sudo tc qdisc add dev eth0 root handle 1: cbq avpkt 1000 bandwidth 10Mbit $ sudo tc qdisc show dev eth0 qdisc cbq 1: root refcnt 2 rate 10000Kbit (bounded,isolated) prio no-transmit

7 Fallacies of Distributed Systems

The network is reliable. Latency is zero. Bandwidth is infinite. The network is secure. Topology doesn't change. There is one administrator. Transport cost is zero.
Peter Deutsch 1994

Thank youQuestions ?

Raphaël Luta Freelance technical consultant, Web & Data@raphaelluta
Le réseau, cet inconnu au centre de vos applications @raphaelluta #BzhCmp