perf_sec
NFV Performance Security
Luke Hinds
Principle Software Engineer, Red Hat
Security Risks in SRIOV
- Mac Spoofing
- Network Flow Control
- Bandwidth Exhaustion
MAC Spoofing
Guests can assign their own MAC addreses, thereby allowing MAC spoofing..
This can be negated using the IP cmd..
ip link set eth0 vf 3 mac xx:yy:xx:yy:xx:yy vlan 100 spoofchk on
It can also be toggled in OpenStack neutron:
stack@devstack:~/devstack$ neutron net-show net1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | 725ebfa1-c3c3-43fa-b8c2-cac99f1f88fb | | mtu | 0 | | name | net1 | | port_security_enabled | True | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 1001 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 1d58d120-4990-41d8-b1f2-2354df54328a | | tenant_id | ba328cf9aa72429aad0535ec4adcd882 | | vlan_transparent | False | +---------------------------+--------------------------------------+
Why would you want disable it?
802.3ad (Dynamic link aggregation)
Network Flow Control
Flow Control exists to provide lossless layer 2 network communications
This enables a receiver to send a signal to the transmitter to pause traffic
A `PAUSE` frame is sent to the transmitter whenever the receiver side runs out of buffers
SR-IOV enables multiple VFs to share a physical link on the host.
(Average 256 VFs - single port NICs)
A malicious VM could cause temporary halt to the traffic of all VMs by manipulating flow control feature.
At NIC side, the Ethernet flow control can be turned off…
ethtool -A <PF ethX> autoneg off rx off tx off
Most NEPs hardware typically allow flow-control to be disabled (sometimes per port)
Be Mindful: TCP incast (many-to-one communications)
Bandwidth Exhaustion
A single VNF consumes all bandwidth available to a physical link, denying other VFs
To avoid this, bandwidth limit can be set at the individual VF level as shown below:
ip link set eth2 vf 0 rate 100 # where rate is mentioned in Mbps
If the switch where the SR-IOV physical NIC is connected supports MAC based rate limiting, then the same restriction can be applied on switch side.
In OpenStack this can be as a QoS policy
$ neutron qos-bandwidth-limit-rule-create bw-limiter --max-kbps 3000 \ --max-burst-kbps 300 --min-kbps 1000 Created a new bandwidth_limit_rule: +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | id | 92ceb52f-170f-49d0-9528-976e2fee2d6f | | max_burst_kbps | 300 | | max_kbps | 3000 | | min_kbps | 1000 | +----------------+--------------------------------------+
DPDK
Data-Plane-Development-Kit Security considerations
Multi Process Limit in DPDK
The multi-process feature requires that the exact same hugepage memory mappings be present in all applications.
- The Linux security feature - Address-Space Layout Randomization (ASLR) can interfere with this mapping.
- VNF vendors may request to disable this feature in order to run multi-process applications.
Disabling Address-Space Layout Randomization (ASLR) may have security implications, so disable only when these have been understood.
http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html
Pause Frame Exploit
Pause frame exploits are currently not possible, due to flow controls being disabled in DPDK
Open vSwitch
Open vSwitch Security considerations
Use OVS PKI
Protect against flow table export or manipulation with TLS protection
http://openvswitch.org/support/dist-docs/INSTALL.SSL.md.html
QoS
Rate Limiting can be set for Open vSwitch instances:
- ingress-policing-rate:
Max rate (Kbps) a VM is allowed to send.
- ingress-policing-burst:
Max burst allowed.
For example:
ovs-vsctl set interface tap0 ingress_policing_rate=1000 ovs-vsctl set interface tap0 ingress_policing_burst=100
Which as we saw earlier, is also exposed to OpenStack Neutron…
$ neutron qos-bandwidth-limit-rule-create bw-limiter --max-kbps 3000 \ --max-burst-kbps 300 Created a new bandwidth_limit_rule: +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | id | 92ceb52f-170f-49d0-9528-976e2fee2d6f | | max_burst_kbps | 300 | | max_kbps | 3000 | +----------------+--------------------------------------+
SDN Controllers
Rate Limit Northbound Rest API of SDN Controllers to prevent overload of HTTP requests
Be mindful of southbound packet storms on the userplane.
Use Network Segregation / Security Zones for Control and Data Plane traffic
The End.
Questions…?