netmap on cxgbe interfaces

Last year my testbed machine got the luxury of 100Gbit Chelsio T6 network interfaces with a pair of T62100-LP-CR cards. I have been using these recently for VPP development on FreeBSD and while they work well I have been having a lot of trouble doing any packet generation with them.

I got pretty miserable performance using netmaps pkt-gen on FreeBSD and trying Linux the comparative oddness of the T6 drivers seems to trip on the TRex packet generator suite on Linux.

From reading Chelsio's own documentation it seemed I was missing something, they have benchmarks using pkt-gen showing link saturation for small packets at 40Gbit/s on their T5 cards. What was up?

When I run a pkt-gen (lives in tools/tools/netmap ) benchmark I get a little under 1 Gbit/s of traffic:

$ sudo ./pkt-gen -f tx -i cc0 -S 00:07:43:74:3f:e9 -D 00:07:43:74:3f:e1
...
250.554583 main_thread [2713] 9.012 Mpps (9.587 Mpkts 4.326 Gbps in 1063799 usec) 512.00 avg_batch 0 min_space

~9 Mpps isn't that great, I paid for 100 Gbit and 4 just isn't enough.

FreeBSD has a zero copy userspace networking framework called netmap , it cleverly hooks into drivers early in their packet processing pipeline and if there is a netmap application running steals them so the host doesn't see them. Aiding with support, if the app doesn't want the given packet it can be given back to the driver and normal processing can occur.

Drivers can be natively supported with netmap or if support isn't available an emulated mode driver can be used.

Eventually after looking really really hard at the Chelsio examples I saw that the network interface names were different from mine. They were prefixed with a 'v', 'vcxl' rather than 'cxl'.

A bunch of googling later I read the top cxgbe(4) correctly:

The cxgbe driver uses different names for devices based on the associated
ASIC:

      ASIC    Port Name    Parent Device    Virtual Interface
      T4      cxgbe        t4nex            vcxgbe
      T5      cxl          t5nex            vcxl
      T6      cc           t6nex            vcc

To get a virtual interface we need to set the hw.cxgbe.num_vis sysctl to a value greater than 1. I added the following to my /boot/loader.conf :

# Add a virtual port for cxgbe                                     
hw.cxgbe.num_vis="2"            # Number of VIs per port           
hw.cxgbe.nnmrxq_vi="8"          # Number of netmap RX queues per VI
hw.cxgbe.nnmtxq_vi="8"          # Number of netmap TX queues per VI
hw.cxgbe.nrxq_vi="8"            # Number of RX queues per VI       
hw.cxgbe.ntxq_vi="8"            # Number of TX queues per VI

On a reboot I have a shiny new vcc0 device, letting pkt-gen run there I get much nicer results:

$ sudo ./pkt-gen -f tx -i vcc0 -S 00:07:43:74:3f:e9 -D 00:07:43:74:3f:e1 
...
329.706767 main_thread [2713] 68.770 Mpps (73.060 Mpkts 33.010 Gbps in 1062380 usec) 480.61 avg_batch 99999 min_space

~70 Mpps and 33 Gbps of 64 byte packets is pretty great , with an imix I can saturate the 100Gbps with pkt-gen .

I didn't find anything about this differentiator by searching, pkt-gen doesn't mention it. There are some dmesg lines, but if you don't know, you don't know.

238.184192 [1135] generic_netmap_attach     Emulated adapter for cc0 created (prev was NULL)
238.192821 [ 320] generic_netmap_register   Emulated adapter for cc0 activated
238.200244 [1889] netmap_interp_ringid      invalid ring id 1
238.206046 [ 295] generic_netmap_unregister Emulated adapter for cc0 deactivated