Need for Speed:Packet Edition – Primer to network performance for normal people – Tools



Need for Speed:Packet Edition – Primer to network performance for normal people – Tools

0 2


packetstory

Primer to network performance for normal people

On Github rluta / packetstory

Need for Speed:Packet Edition

Primer to network performance for normal people

Created by Raphaël Luta / @raphaelluta

#PerfUG

About me

Freelance technical consultant

Web, Mobile, (Big) Data, Performance

Twitter: @raphaelluta

Linkedin: http://fr.linkedin.com/in/raphaelluta

Github: rluta

Company: www.aptiwan.com

Mail: raphael.luta at aptiwan.com

Your new application...

...as your customers see it

...as viewed by the network guys

Their job is delivery

Network characteristics

Bandwidth

Latency

Error rate

Duplex

MTU

Jitter

An alternate view of networks

Full Duplex

Bandwidth

MTU

Latency

Jitter

Error Rate

Quizz time

Firewall

Ideal conditions

Duplex must be full

Error rate should be close to 0.0 %

MTU should be as big as possible

Ethernet MTU is normalized

...the maximum length of an IP datagram sent over an Ethernet is 1500 octets. IETF RFC 894

Bandwidth vs Latency Evolution

Over the last 20 years, bandwidth has improved 1,000 times faster than latency

Basics: Light speed

c = 300,000 km/s

(in vacuum)

That's about 190,000 km/s in fiber

approx. 5 ms per 1,000 km

Paris - Dublin 800 km 4 ms Paris - New York 6,000 km 30 ms Paris - San Francisco 9,000 km 45 ms

Common networks

Bandwidth (Mbps) Latency / RTT (ms) Ethernet 1,000 0.5 ADSL 2 20 0.7 30 Fibre/Cable 100 5 15 3G 42 50 EDGE 1.5 300 WiFi 2-342* 2* Satellite 1-1,000 500

Tools

Command line Tools

ping
traceroute
tc
Basic usage
$ ping www.aptiwan.com PING www.aptiwan.com (108.162.198.20) 56(84) bytes of data. 64 bytes from 108.162.198.20: icmp_req=1 ttl=63 time=28.3 ms 64 bytes from 108.162.198.20: icmp_req=2 ttl=63 time=27.5 ms 64 bytes from 108.162.198.20: icmp_req=3 ttl=63 time=27.1 ms ^C64 bytes from 108.162.198.20: icmp_req=4 ttl=63 time=27.9 ms --- www.aptiwan.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3008ms rtt min/avg/max/mdev = 27.104/27.747/28.359/0.505 ms
MTU testing
$ ping -s 1472 www.aptiwan.com PING www.aptiwan.com (108.162.198.20) 1472(1500) bytes of data. 1480 bytes from 108.162.198.20: icmp_req=1 ttl=63 time=40.2 ms 1480 bytes from 108.162.198.20: icmp_req=2 ttl=63 time=41.1 ms 1480 bytes from 108.162.198.20: icmp_req=3 ttl=63 time=40.8 ms ^C --- www.aptiwan.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 40.258/40.741/41.154/0.404 ms $ ping -s 1500 www.aptiwan.com PING www.aptiwan.com (108.162.198.20) 1500(1528) bytes of data. ^C --- www.aptiwan.com ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2017ms
Basic usage
$ traceroute www.aptiwan.com to www.aptiwan.com (108.162.199.20), 30 hops max, 60 byte packets 1 vss-7a-6k.fr.eu (176.31.125.253) 0.629 ms * * 2 rbx-g2-a9.fr.eu (91.121.128.118) 1.246 ms 1.241 ms 1.232 ms 3 * * * 4 195.66.225.179 (195.66.225.179) 4.246 ms 4.220 ms 4.225 ms 5 108.162.199.20 (108.162.199.20) 4.212 ms 4.203 ms 4.194 ms
Service discovery / firewall testing
$ sudo traceroute -m 7 -w 1 -T -p 80 www.aptiwan.com to www.aptiwan.com (108.162.198.20), 10 hops max, 60 byte packets 1 vss-7a-6k.fr.eu (176.31.125.253) 0.583 ms * * 2 rbx-g1-a9.fr.eu (91.121.128.114) 0.901 ms 1.143 ms 1.142 ms 3 * * * 4 * * * 5 195.66.225.179 (195.66.225.179) 4.260 ms 4.277 ms 4.261 ms 6 108.162.198.20 (108.162.198.20) 4.261 ms 4.274 ms 4.235 ms $ sudo traceroute -m 7 -w 1 -T -p 443 www.aptiwan.com to www.aptiwan.com (108.162.199.20), 10 hops max, 60 byte packets 1 vss-7a-6k.fr.eu (176.31.125.253) 0.561 ms * * 2 rbx-g2-a9.fr.eu (91.121.128.118) 0.964 ms 0.968 ms 1.279 ms 3 * * * 4 195.66.225.179 (195.66.225.179) 4.199 ms 4.206 ms 4.203 ms 5 * * * 6 * * * 7 * * *
Add constant latency
$ ping -c 1 -n www.google.fr PING www.google.fr (173.194.40.215) 56(84) bytes of data. 64 bytes from 173.194.40.215: icmp_req=1 ttl=55 time=12.6 ms --- www.google.fr ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 12.615/12.615/12.615/0.000 ms $ sudo tc qdisc add dev eth0 root netem delay 20ms $ sudo tc qdisc show dev eth0 qdisc netem 8004: root refcnt 2 limit 1000 delay 20.0ms $ ping -c 1 -n www.google.fr PING www.google.fr (173.194.40.223) 56(84) bytes of data. 64 bytes from 173.194.40.223: icmp_req=1 ttl=55 time=32.9 ms --- www.google.fr ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 32.901/32.901/32.901/0.000 ms
Add latency + jitter
$ sudo tc qdisc change dev eth0 root netem delay 20ms 2ms 15% $ sudo tc qdisc show dev eth0 qdisc netem 8005: root refcnt 2 limit 1000 delay 20.0ms 2.0ms 15%
Lossy connection
$ sudo tc qdisc change dev eth0 root netem loss 1 $ sudo tc qdisc show dev eth0 qdisc netem 8005: root refcnt 2 limit 1000 loss 1%
Remove shaping
$ sudo tc qdisc del dev eth0 root $ sudo tc qdisc show dev eth0 qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Limit bandwidth (advanced)
$ sudo tc qdisc add dev eth0 root handle 1: cbq avpkt 1000 bandwidth 10Mbit $ sudo tc qdisc show dev eth0 qdisc cbq 1: root refcnt 2 rate 10000Kbit (bounded,isolated) prio no-transmit

GUI Tools

Back to delivery

UDP: Express delivery

  • Stateless communication
  • No reception guarantees
  • Simple and efficient
  • Can easily saturate network links
  • Some well-known use cases:
    • DNS, NTP, DHCP
    • RTP, NFS (v1-3)
    • SNMP, Syslog

TCP: Reliable delivery

  • Stateful, connection oriented protocol
  • Guarantees data delivery
    • Error detection
    • Automatic retransmission
    • Preserve order
  • Sophisticated flow control
    • Prevents client overload
    • Prevents network congestion

Typical TCP Chat

Server State
Closed
Messages
Hello
Hi. How are you ?
Fine
Can you show me your pussy ?
Now get lost
ok
CU
FU
Client State
Closed

Windows Everywhere

  • Windows regulate flow between sender and recipient
    • Receiver window (rwnd) avoids recipient overload
    • Congestion window (cwnd) prevents network saturation
  • Max # of inflight bytes = min(rwnd, cwnd)
  • Window changes determined by :
    • Recipient available buffer (rwnd)
    • Slow start (cwnd)
    • Congestion avoidance (cwnd)
  • Congestion window is small in a new connection

Time to Load

ADSL
360 ms
ADSL
Fibre
Ethernet
Mobile
D. Ack
Payload (KB):
200
Bandwidth (Mbps):
10
RTT (ms):
30
Rwnd (kB):
64
Cwmd (p.):
3
MTU (B):
1500

Time to load 200 kB

Name Bw RTT Windows MTU Time Tput (kbps) ADSL 10 30 64 kB / 3 1500 360 4444.45

Tools (part 2)

Command Line Tools

iperf
ss
tcpdump
$ sudo tcpdump -i eth0 -s 1500 -w /tmp/trace.pcap "tcp and port 80" ^C1847 packets captured 1849 packets received by filter 0 packets dropped by kernel
$ ss -ti
State  Recv-Q Send-Q     Local Address:Port         Peer Address:Port
ESTAB  0      208        176.31.125.78:ssh          86.70.100.151:50669
	 cubic wscale:4,7 rto:244 rtt:35/4 ato:40
	 cwnd:10 ssthresh:7 send 3.3Mbps rcv_rtt:44 rcv_space:14480
ns397279:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
$ iperf -c ns397279.ovh.net -m -w 85kB
------------------------------------------------------------
Client connecting to ns397279.ovh.net, TCP port 5001
TCP window size:  166 KByte (WARNING: requested 83.0 KByte)
------------------------------------------------------------
[  3] local 10.0.2.15 port 39293 connected with 176.31.125.78 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-12.3 sec  1.75 MBytes  1.20 Mbits/sec
[  3] MSS size 1460 bytes (MTU 1500 bytes, ethernet)

GUI Tools: Wireshark

Tuning corner

Linux tuning

Adjust initial congestion window
$ sudo ip route change default via 192.168.1.1 dev eth0 metric 100 initcwnd 10 $ ip route | grep default default via 192.168.1.1 dev eth0 metric 100 mtu 1500 initcwnd 10
Avoid slow-start after idle connections
$ sudo sysctl -w net.ipv4.tcp_slow_start_after_idle = 0
Adjust windows
$ cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 6291456 # echo 4096 87380 16777216 > /proc/sys/net/ipv4/tcp_rmem $ cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 4194304 # echo 4096 65536 16777216 > /proc/sys/net/ipv4/tcp_wmem
Increase MTU
$ sudo ifconfig eth0 mtu 9000 up $ sudo ip route change default via 192.168.1.1 dev eth0 metric 100 mtu 9000

Evil forces

Encapsulation

Using an IPSEC VPN

Payload data
Data
TCP Packet
IP
TCP
Data
VPN/L2TPw/ NAT traversal
IP
UDP
ESP
UDP
L2TP
PPP
IP
TCP
Data
ESP
  • Total header and trailer size at least 140 bytes
  • Bandwidth loss is at least 10%, typically closer to 25-30 % for most applications

Data Bloat

Example: SOAP Request

<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">     <env:Header/>     <env:Body>         <ns2:ListerOsSouscriptiblesOut                 xmlns:ns2="http://ServiceProvider/CustomerManagement/Assistance/schemas"                 xmlns:ns3="http://ServiceProvider/schema/TVTCampaignSearch"                 xmlns:ns4="http://ServiceProvider/schema/TVTCampaignCreation"                 xmlns:ns5="http://ServiceProvider/CustomerManagement/Assistance/schemas/lff"                 xmlns:ns6="http://ServiceProvider/CustomerManagement/Assistance/schemas/cmp"                 xmlns:ns7="http://ServiceProvider/CustomerManagement/Assistance/schemas/AppairerIdentiteV1"                 xmlns:ns8="http://ServiceProvider/CustomerManagement/Assistance/schemas/pcf"                 xmlns:ns9="http://ServiceProvider/CustomerManagement/Assistance/schemas/ConsulterRechargeAccessibleV2"                 xmlns:ns10="http://ServiceProvider/CustomerManagement/Assistance/schemas/listerOs/V2"                 xmlns:ns11="http://ServiceProvider/CustomerManagement/Assistance/schemas/typeMetierOffres/V2"                 xmlns:ns12="http://ServiceProvider/CustomerManagement/Assistance/schemas/RechRecoMicro/V2"                 xmlns:ns13="http://ServiceProvider/CustomerManagement/Assistance/schemas/iiV2"                 xmlns:ns14="http://ServiceProvider/CustomerManagement/Assistance/schemas/cfrais"                 xmlns:ns15="http://ServiceProvider/CustomerManagement/Assistance/schemas/EnregistrerMailContact"                 xmlns:ns16="http://ServiceProvider/CustomerManagement/Assistance/schemas/cff"                 xmlns:ns17="http://ServiceProvider/CustomerManagement/Assistance/schemas/consultOs/V2"                 xmlns:ns18="http://ServiceProvider/CustomerManagement/Assistance/schemas/ffi"                 xmlns:ns19="http://ServiceProvider/CustomerManagement/Assistance/schemas/communRecupererInfos"                 xmlns:ns20="http://ServiceProvider/CustomerManagement/Assistance/schemas/RecupererInfosPersonneConnecteeV1"                 xmlns:ns21="http://ServiceProvider/CustomerManagement/Assistance/schemas/RecupererInfosLigneV1"                 xmlns:ns22="http://ServiceProvider/CustomerManagement/Assistance/schemas/ii2"                 xmlns:ns23="http://ServiceProvider/CustomerManagement/Assistance/schemas/mmp"                 xmlns:ns24="http://ServiceProvider/CustomerManagement/Assistance/schemas/calculImp/V2"                 xmlns:ns25="http://ServiceProvider/CustomerManagement/Assistance/schemas/InitialiserPwdV2"                 xmlns:ns26="http://ServiceProvider/CustomerManagement/Assistance/schemas/verifierLoginMotDePasse/"                 xmlns:ns27="http://ServiceProvider/CustomerManagement/Assistance/schemas/iia"                 xmlns:ns28="http://ServiceProvider/CustomerManagement/Assistance/schemas/authentificationImplicite/"                 xmlns:ns29="http://ServiceProvider/CustomerManagement/Assistance/schemas/scf"                 xmlns:ns30="http://ServiceProvider/CustomerManagement/Assistance/schemas/listerOsR/V2"                 xmlns:ns31="http://ServiceProvider/CustomerManagement/Assistance/schemas/cii"                 xmlns:ns32="http://ServiceProvider/schema/TVTCTUSearch">             <ns2:codeRetour>OK</ns2:codeRetour>             <ns2:content>                 <ns2:listeOsSouscriptibles>                     <ns2:offreCaracteristique>                         <ns2:id>1108</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1109</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11671</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11329</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1582</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11469</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11465</ns2:id>                         <ns2:noteCoherence>4</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>15</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>12</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>8</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1011</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11326</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11726</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11725</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11724</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1080</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11669</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11524</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11525</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11664</ns2:id>                         <ns2:noteCoherence>0</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>22</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11665</ns2:id>                         <ns2:noteCoherence>0</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>23</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11494</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11493</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11869</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>12227</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11288</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>12228</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11721</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11289</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11720</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11662</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>20</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>12226</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11663</ns2:id>                         <ns2:noteCoherence>0</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>21</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11479</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11499</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1078</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11477</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1079</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11002</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11476</ns2:id>                         <ns2:noteCoherence>2</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11497</ns2:id>                         <ns2:noteCoherence>0</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>3</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>11</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>18</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>13</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11498</ns2:id>                         <ns2:noteCoherence>4</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>8</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>13</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>18</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11333</ns2:id>                         <ns2:noteCoherence>4</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>12</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>8</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11145</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11336</ns2:id>                         <ns2:noteCoherence>4</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>12</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>8</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11337</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11334</ns2:id>                         <ns2:noteCoherence>4</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>12</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>8</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11148</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11147</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1437</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1435</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11658</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1453</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1075</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1783</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1077</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1076</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1073</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1911</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11441</ns2:id>                         <ns2:noteCoherence>0</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>3</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>11</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>18</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1267</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1448</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1447</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>12247</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11454</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1056</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11459</ns2:id>                         <ns2:noteCoherence>4</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>1</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>12</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>8</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11356</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1451</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1256</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11224</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11054</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11221</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11222</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11223</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>12056</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11773</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11393</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11777</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1055</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11394</ns2:id>                         <ns2:noteCoherence>0</ns2:noteCoherence>                         <ns2:listeCapacites>                             <ns2:capaciteOffre>                                 <ns2:id>3</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>15</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>11</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>6</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>7</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>5</ns2:id>                             </ns2:capaciteOffre>                             <ns2:capaciteOffre>                                 <ns2:id>18</ns2:id>                             </ns2:capaciteOffre>                         </ns2:listeCapacites>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11774</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>11428</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>12250</ns2:id>                         <ns2:noteCoherence>1</ns2:noteCoherence>                     </ns2:offreCaracteristique>                     <ns2:offreCaracteristique>                         <ns2:id>1345</ns2:id>                         <