Experience with DPDK Troubleshooting Juha Kosonen - - - PowerPoint PPT Presentation

Experience with DPDK Troubleshooting

Juha Kosonen - juha.kosonen@nokia.com Ajay Simha - asimha@redhat.com

INSERT DESIGNATOR, IF NEEDED 2

Abstract

Getting near real-time performance for VNF applications requires designing the application carefully and profiling its performance under realistic conditions. OpenStack has developed many functionalities that helps in this, such as CPU and PCI topology awareness and pinning the virtual machines to dedicated cores. The host networking stack, in turn, has been optimized with features such as accelerated virtual switches and DPDK. This presentation gives an update of using DPDK in real VNF tests and tells some real life experiences from DPDK troubleshooting. What can I expect to learn?

• Overview of DPDK
• What can go wrong with installation of DPDK and how to go about troubleshooting it
• Tuning of OVS-DPDK and how to make sure you are getting the expected performance

INSERT DESIGNATOR, IF NEEDED 3

Agenda

• Introduction
• Telco Requirements
• HA and Performance
• Journey from virto to DPDK
• SR-IOV vs. DPDK (The two viable options today)
• DPDK Data Plane
• Hardware Tuning
• Throughput
• CPU core allocation
• HA
• DPDK Troubleshooting
• Install time
• HA
• Performance
• Summary

Two pillars of NFV

INSERT DESIGNATOR, IF NEEDED 4

Telco Requirements: HA and Performance

• Maximum subscribers per core
• Highly available - as close to Five 9s as possible
• If you have no HA
• Failure = Zero PPS!!
• Telco applications/services serve millions of subscribers
• Hardware based solutions provided required performance
• We need to get maximum throughput from our virtualized solution!
• Need maximum PPS (Packets per second) without drops!!
• NEPs as well as Telcos (Operators) don’t like NIC driver dependency on

VNFs

• Closer to Cloud Ready the better

Journey from VIRTIO to DPDK: virtio

Journey from VIRTIO to DPDK: PCI Passthrough

Journey from VIRTIO to DPDK: SR-IOV

Journey from VIRTIO to DPDK: PCI Passthrough

Things to consider

INSERT DESIGNATOR, IF NEEDED 9

DPDK Data Plane

• Hardware tuning
• Throughput
• CPU core allocation
• HA

INSERT DESIGNATOR, IF NEEDED 10

DPDK Data Plane: Hardware tuning

• Zero Loss requires

latest hardware generation

• Intel NICs, either

Niantic series (x520/x540) or the latest Fortville series (x710).

INSERT DESIGNATOR, IF NEEDED 11

DPDK Data Plane: Throughput

11

Host: openvswitch PCI NIC PCI NIC

de-queueing en-queueing

vhost pci

virtio- ring

polling thread

pci

VM: testpmd virtio- ring

vhost virtio virtio

[1] 59% higher than v2.5.1, [2] 34% higher than v2.5.1 10GSFP+Cu

Frame size Mpps @0.002% loss Gbps @0.002% loss Mpps/core @0.002% loss Mpps @0% loss [1] Gbps @0% loss Mpps/core @0% loss

64

12.14[1] 8.15 6.07 7.34 3.67

256

7.65[2] 16.88 3.82 4.85 2.42

1024

2.38 19.94 1.19 2.36 1.18

1500

1.63 19.90 0.81 1.61

VM L2 forwarding, intra-NUMA node, single queue

• 2 x virtio-net interfaces (node0)
• 2 x 10Gb interfaces (node0)
• Testpmd DPDK application in VM
• Bidirectional traffic
• Maximum rate while within specified loss

packet generator PCI NIC PCI NIC

Red Hat OpenStack Platform 10 - December, 2016

1 2

Resources partitioning/allocation

DPDK Data Plane: CPU Core Allocation

Red Hat OpenStack Platform 10 - December, 2016

DPDK Data Plane: DPDK HA (DPDK Bonding)

Three Things to consider

INSERT DESIGNATOR, IF NEEDED 14

DPDK Troubleshooting

• Installation
• HA
• Performance

Heat Template Parameters Setting (network-enviroment.yaml)

INSERT DESIGNATOR, IF NEEDED 15

DPDK Troubleshooting: Installation

• ComputeKernelArgs: "iommu=pt intel_iommu=on default_hugepagesz=1GB hugepagesz=1G

hugepages=12"

• NeutronDpdkCoreList: "'4,6,20,22'"
• NeutronDpdkMemoryChannels: "4"
• NeutronDpdkSocketMemory: "'1024,1024'"
• NeutronDpdkDriverType: "vfio-pci"
• NovaVcpuPinSet: "8,10,12,14,18,24,26,28,30"
• NeutronTunnelTypes: ""
• NeutronNetworkType: 'vlan'
• NeutronNetworkVLANRanges: 'datacentre:4000:4070, dpdk:4071:4071'
• HostCpusList: "'2,4,6,8,10,12,14,18,20,22,24,26,28,30'"
• NeutronDatapathType: "netdev"
• NeutronVhostuserSocketDir: "/var/run/openvswitch"
• NeutronBridgeMappings: 'datacentre:br-isolated, dpdk:br-link'

What OVS looks like in a working DPDK setup

INSERT DESIGNATOR, IF NEEDED 16

DPDK Troubleshooting: Installation

[root@overcloud-compute-0 ~]# ovs-vsctl show 9ef8aab3-0afa-4fca-9a49-7ca97d3e2ffa Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-link Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port br-link Interface br-link type: internal Port phy-br-link Interface phy-br-link type: patch

• ptions: {peer=int-br-link}

Port "dpdkbond0" Interface "dpdk1" type: dpdk Interface "dpdk0" type: dpdk

What options are being set for OVS-DPDK?

INSERT DESIGNATOR, IF NEEDED 17

DPDK Troubleshooting: Installation

[root@overcloud-compute-0 ~]# cat /etc/sysconfig/openvswitch OPTIONS="" DPDK_OPTIONS = "-l 2,3,14,15 -n 4 --socket-mem 1024,1024 -w 0000:83:00.0 -w 0000:83:00.1"

What OVS looks like in a broken DPDK setup

INSERT DESIGNATOR, IF NEEDED 18

DPDK Troubleshooting: Installation

[root@overcloud-compute-0 heat-admin]# ovs-vsctl show 857105ff-db67-41c6-812b-eaa65d224ca0 Bridge br-link fail_mode: standalone Port "dpdk0" Interface "dpdk0" type: dpdk error: "could not open network device dpdk0 (Address family not supported by protocol)" Port br-link Interface br-link type: internal

• vs_version: "2.5.0"

[root@overcloud-compute-0 heat-admin]#

Checking if physical NICs are bound to DPDK

INSERT DESIGNATOR, IF NEEDED 19

DPDK Troubleshooting: Installation

[root@overcloud-novacompute-0 ~]# dpdk-devbind --status Network devices using DPDK-compatible driver ============================================ 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe,vfio-pci Network devices using kernel driver =================================== 0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=ens255f0 drv=ixgbe unused=vfio-pci,uio_pci_generic 0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=ens255f1 drv=ixgbe unused=vfio-pci,uio_pci_generic 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=ens1f0 drv=ixgbe unused=vfio-pci,uio_pci_generic 0000:05:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=ens4f0 drv=ixgbe unused=vfio-pci,uio_pci_generic 0000:05:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=ens4f1 drv=ixgbe unused=vfio-pci,uio_pci_generic

What DPDK bond looks like

INSERT DESIGNATOR, IF NEEDED 20

DPDK Troubleshooting: HA

[root@overcloud-compute-0 ~]# ovs-appctl bond/show dpdkbond0

• --- dpdkbond0 ----

bond_mode: balance-tcp bond may use recirculation: yes, Recirc-ID : 1 bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 94 ms lacp_status: negotiated active slave mac: a0:36:9f:47:e0:62(dpdk1) slave dpdk0: enabled may_enable: true slave dpdk1: enabled active slave may_enable: true

Things to look for

INSERT DESIGNATOR, IF NEEDED 21

DPDK Troubleshooting: Performance

• Inspecting the node
• Measurements

[root@overcloud-compute-1 ~]# grep TUNED /etc/tuned/bootcmdline TUNED_BOOT_CMDLINE="nohz=on nohz_full='2,4,6,8,10,12,14,18,20,22,24,26,28,30' rcu_nocbs='2,4,6,8,10,12,14,18,20,22,24,26,28,30' intel_pstate=disable nosoftlockup" [root@overcloud-compute-0 ~]# cat /etc/tuned/cpu-partitioning-variables.conf # Examples: # isolated_cores=2,4-7 # isolated_cores=2-23 # isolated_cores='2,4,6,8,10,12,14,18,20,22,24,26,28,30' [root@overcloud-compute-1 ~]# grep vcpu /etc/nova/nova.conf vcpu_pin_set=8,10,12,14,18,24,26,28,30

22

Running tuned-adm manually - tuned HostCpuList

DPDK Troubleshooting: Inspecting the node

23

Verifying tuned is working for you (pbench): Non-tuned CPU - CPU17 Average number of Local Timer Interrupts: 1000 :-( HostCpusList: "'1,2,4,6,8,10,12,14,18,20,22,24,26,28,30'" https://github.com/distributed-system-analysis/pbench

DPDK Troubleshooting: Inspecting the node

24

Verifying tuned is working for you (pbench): tuned CPU - CPU1 Average number of Local Timer Interrupts: 2 :-)

DPDK Troubleshooting: Inspecting the node

HostCpusList: "'1,2,4,6,8,10,12,14,18,20,22,24,26,28,30'" https://github.com/distributed-system-analysis/pbench

The following command displays the packet counters as well as the packet drops

INSERT DESIGNATOR, IF NEEDED 25

DPDK Troubleshooting: Measurements

[root@overcloud-compute-0 ~]# ovs-ofctl dump-ports br-link OFPST_PORT reply (xid=0x2): 4 ports port LOCAL: rx pkts=542285, bytes=67242760, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=1, bytes=70, drop=0, errs=0, coll=0 port 1: rx pkts=153700, bytes=11011978, drop=0, errs=0, frame=?, over=?, crc=? tx pkts=271137, bytes=33620934, drop=0, errs=0, coll=? port 2: rx pkts=18084, bytes=2875028, drop=0, errs=0, frame=?, over=?, crc=? tx pkts=271135, bytes=33620740, drop=0, errs=0, coll=? port 3: rx pkts=0, bytes=0, drop=?, errs=?, frame=?, over=?, crc=? tx pkts=0, bytes=0, drop=?, errs=?, coll=?

The following command displays the correlation between port number and port name per bridge

INSERT DESIGNATOR, IF NEEDED 26

DPDK Troubleshooting: Measurements

[root@overcloud-compute-0 ~]# ovs-ofctl show br-link OFPT_FEATURES_REPLY (xid=0x2): dpid:0000a0369f47e060 n_tables:254, n_buffers:256 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst

1(dpdk1): addr:a0:36:9f:47:e0:62 config: 0 state: 0 current: 10GB-FD speed: 10000 Mbps now, 0 Mbps max 2(dpdk0): addr:a0:36:9f:47:e0:60 config: 0 state: 0 current: 10GB-FD speed: 10000 Mbps now, 0 Mbps max 3(phy-br-link): addr:ae:40:d5:6b:8e:c8 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(br-link): addr:a0:36:9f:47:e0:60 config: 0 state: 0 current: 10MB-FD COPPER speed: 10 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

The following command displays more detailed RX and TX counters

INSERT DESIGNATOR, IF NEEDED 27

DPDK Troubleshooting: Measurements

[root@overcloud-compute-0 ~]# ovs-vsctl list Interface|grep -E "^(statistics|name)" | grep -a2 dpdk name : "dpdk0" statistics : {"rx_128_to_255_packets"=18136, "rx_1_to_64_packets"=1, "rx_256_to_511_packets"=0, "rx_512_to_1023_packets"=0, "rx_65_to_127_packets"=1, rx_broadcast_packets=0, rx_bytes=2883614, rx_dropped=0, rx_errors=0, rx_jabber_errors=0, rx_packets=18138, "tx_128_to_255_packets"=271940, "tx_1_to_64_packets"=0, "tx_256_to_511_packets"=0, "tx_512_to_1023_packets"=0, "tx_65_to_127_packets"=0, tx_broadcast_packets=0, tx_bytes=33720560, tx_dropped=0, tx_errors=0, tx_multicast_packets=271940, tx_packets=271940} name : "dpdk1" < ======= SNIP ============ >

The following command displays the PMD port allocation

INSERT DESIGNATOR, IF NEEDED 28

DPDK Troubleshooting: Measurements

[root@overcloud-compute-0 ~]# ovs-appctl dpif-netdev/pmd-rxq-show pmd thread numa_id 1 core_id 14: isolated : false port: dpdk1 queue-id: 0 pmd thread : isolated : falsenuma_id 1 core_id 15 port: dpdk0 queue-id: 0

plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews

THANK YOU