ixgbe: setting the number of RX/TX queues

affinity, irq, linux-device-driver, networking, nic

Solution

Some versions of ixgbe driver included into linux kernel (since 2013, 3.9 kernel, "3.11.33-k" version of ixgbe) can change RSS (queue) count in runtime even without RSS module option. There is `ethtool` tool to change parameters of network cards, and there are options to change channels:

   ethtool -l|--show-channels devname

   ethtool -L|--set-channels devname [rx N] [tx N] [other N]
          [combined N]


   -l --show-channels
          Queries the specified network device for the numbers of
          channels it has.  A channel is an IRQ and the set of queues
          that can trigger that IRQ.

   -L --set-channels
          Changes the numbers of channels of the specified network
          device.

       rx N   Changes the number of channels with only receive queues.

       tx N   Changes the number of channels with only transmit queues.

       other N
              Changes the number of channels used only for other
              purposes e.g. link interrupts or SR-IOV co-ordination.

       combined N
              Changes the number of multi-purpose channels.

Test current channel (RSS, queue) count of ixgbe eth1 with `ethtool -l eth1` and change with `ethtool -L eth1 combined 4` or `ethtool -L eth1 rx 2 tx 2`.

Implemented in `net/ethernet/intel/ixgbe/ixgbe_ethtool.c`: http://elixir.free-electrons.com/linux/v4.12/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L3442 static const struct ethtool_ops ixgbe_ethtool_ops = { ... .get_channels = ixgbe_get_channels, .set_channels = ixgbe_set_channels, ... }

`ixgbe_get_channels`: http://elixir.free-electrons.com/linux/v4.12/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L3127

`ixgbe_set_channels` to change `adapter->ring_feature[RING_F_RSS].limit`: http://elixir.free-electrons.com/linux/v4.12/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L3164

Implemented since 3.9 version of Linux kernel (around 2013): * http://elixir.free-electrons.com/linux/v3.9/ident/ixgbe_get_channels * https://patchwork.ozlabs.org/patch/211119/ "[RFC,v2,09/10] ixgbe: Add support for displaying the number of Tx/Rx channels" * https://patchwork.ozlabs.org/patch/211120/ "[RFC,v2,10/10] ixgbe: Add support for set_channels ethtool operation diffmbox"

Problem

I want to set the number of RX/TX queues used by an Intel 10G NIC. Let me explain why: I am using an Intel 10G NIC of type X520, on a Dell R720 system. I am using ixgbe version 3.6.7-k. The kernel in Ubuntu 3.2.0-59. I am running my network application on 4 out of the 24 cores on the machine. Currently the NIC is using flow-director so I've got 24 TX and RX queues, while most of the IRQs finally run on the 4 cores running the application. However, I see that some IRQs are running on the other 20 queues (this is probably happening as flow-director samples about 20% of the traffic so some traffic goes through regular RSS). Now I don't want any IRQ to be run on the other 20 cores as they are doing a different task which is damaged by the IRQs running. I tried setting the affinity of the interrupts only to the 4 cores I use, but this does not work well with flow-director. I guess a better approach will be using only 4 RX/TX queues and assigning them to the dedicated cores. But I couldn't find a way to set the number of RX/TX queue in the ixgbe driver (though this is quite simple with other 10G drivers I am familiar with, such as Broadcom's bnx2x). Any idea?

Original source