Kernel interfaces for multiqueue aware socket

From: Fenghua Yu <[email protected]>

Multiqueue and multicore provide packet parallel processing methodology. Current kernel and network drivers place one queue on one core. But the higher level socket doesn't know multiqueue. Current socket only can receive or send packets through one network interfaces. In some cases e.g. multi bpf filter tcpdump and snort, a lot of contentions come from socket operations like ring buffer. Even if the application itself has been fully parallelized and run on multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer and NIC device driver assemble packets to a single, serialized queue. Thus the application cannot actually run in parallel in high speed.

To break the serialized packets assembling bottleneck in kernel, one way is to allow socket to know multiqueue associated with a NIC interface. So each socket can handle tx/rx in one queue in parallel.

Kernel provides several interfaces by which sockets can be bound to rx/tx queues. User applications can configure socket by providing several sockets that each bound to a single queue, applications can get data from kernel in parallel. After that, competitions mentioned above can be removed.

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

This patch set provides kernel ioctl interfaces for user space. User space can either directly call the interfaces or libpcap interfaces can be further provided on the top of the kernel ioctl interfaces.

From: Fenghua Yu <[email protected]>

Multiqueue and multicore provide packet parallel processing methodology. Current kernel and network drivers place one queue on one core. But the higher level socket doesn't know multiqueue. Current socket only can receive or send packets through one network interfaces. In some cases e.g. multi bpf filter tcpdump and snort, a lot of contentions come from socket operations like ring buffer. Even if the application itself has been fully parallelized and run on multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer and NIC device driver assemble packets to a single, serialized queue. Thus the application cannot actually run in parallel in high speed.

To break the serialized packets assembling bottleneck in kernel, one way is to allow socket to know multiqueue associated with a NIC interface. So each socket can handle tx/rx in one queue in parallel.

Kernel provides several interfaces by which sockets can be bound to rx/tx queues. User applications can configure socket by providing several sockets that each bound to a single queue, applications can get data from kernel in parallel. After that, competitions mentioned above can be removed.

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

This patch set provides kernel ioctl interfaces for user space. User space can either directly call the interfaces or libpcap interfaces can be further provided on the top of the kernel ioctl interfaces.

The order of tx/rx packets is up to user application. In some cases, e.g. network monitors, ordering is not a big problem because they more care how to receive and analyze packets in highest performance in parallel.

This patch set only implements multiqueue interfaces for AF_PACKET and Intel ixgbe NIC. Other protocols and NIC's can be handled on the top of this patch set.

Signed-off-by Fenghua Yu <[email protected]>

Signed-off-by Junchang Wang <[email protected]>

Signed-off-by Xinan Tang <[email protected]>

---

From: Fenghua Yu <[email protected]>

Instead of using calculated tx queue mapping, this patch selects tx queue mapping which is specified in socket.

By doing this, tx queue mapping can be bigger than the number of cores and stressfully use multiqueue TSS. Or application can specify some of cores/queues to send packets and implement flexible load balance policies.

Signed-off-by Fenghua Yu <[email protected]>

Signed-off-by Junchang Wang <[email protected]>

Signed-off-by Xinan Tang <[email protected]>

---
 drivers/net/ixgbe/ixgbe_main.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

drivers/net/ixgbe/ixgbe_main.c +1 -1

index eee0b29..4d98928 100644
@@ -6255,7 +6255,14 @@ static int ixgbe_maybe_stop_tx(struct net_device *netdev,
 static u16 ixgbe_select_queue(struct net_device *dev, struct sk_buff *skb)
 {
 	struct ixgbe_adapter *adapter = netdev_priv(dev);
-	int txq = smp_processor_id();
+	int txq;

Quoted 11 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

Multiqueue and multicore provide packet parallel processing methodology. Current kernel and network drivers place one queue on one core. But the higher level socket doesn't know multiqueue. Current socket only can receive or send packets through one network interfaces. In some cases e.g. multi bpf filter tcpdump and snort, a lot of contentions come from socket operations like ring buffer. Even if the application itself has been fully parallelized and run on multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer and NIC device driver assemble packets to a single, serialized queue. Thus the application cannot actually run in parallel in high speed.

Quoted 21 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

Multiqueue and multicore provide packet parallel processing methodology. Current kernel and network drivers place one queue on one core. But the higher level socket doesn't know multiqueue. Current socket only can receive or send packets through one network interfaces. In some cases e.g. multi bpf filter tcpdump and snort, a lot of contentions come from socket operations like ring buffer. Even if the application itself has been fully parallelized and run on multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer and NIC device driver assemble packets to a single, serialized queue. Thus the application cannot actually run in parallel in high speed.

To break the serialized packets assembling bottleneck in kernel, one way is to allow socket to know multiqueue associated with a NIC interface. So each socket can handle tx/rx in one queue in parallel.

Kernel provides several interfaces by which sockets can be bound to rx/tx queues. User applications can configure socket by providing several sockets that each bound to a single queue, applications can get data from kernel in parallel. After that, competitions mentioned above can be removed.

Quoted 6 lines

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

??? please elaborate on these NUMA memory allocations. This should be OK after commit 564824b0c52c34692d (net: allocate skbs on local node)

Quoted 3 lines

This patch set provides kernel ioctl interfaces for user space. User space can either directly call the interfaces or libpcap interfaces can be further provided on the top of the kernel ioctl interfaces.

So, say we have 8 queues, you want libpcap opens 8 sockets, and bind them to each queue. Add a bpf filter to each one of them. This seems not generic way, because it wont work for an UDP socket for example.

And you already can do this using SKF_AD_QUEUE (added in commit d19742fb)

Also your AF_PACKET patch only address mmaped sockets.

Quoted 32 lines · Eric Dumazet

Quoted 21 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

Multiqueue and multicore provide packet parallel processing methodology. Current kernel and network drivers place one queue on one core. But the higher level socket doesn't know multiqueue. Current socket only can receive or send packets through one network interfaces. In some cases e.g. multi bpf filter tcpdump and snort, a lot of contentions come from socket operations like ring buffer. Even if the application itself has been fully parallelized and run on multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer and NIC device driver assemble packets to a single, serialized queue. Thus the application cannot actually run in parallel in high speed.

To break the serialized packets assembling bottleneck in kernel, one way is to allow socket to know multiqueue associated with a NIC interface. So each socket can handle tx/rx in one queue in parallel.

Kernel provides several interfaces by which sockets can be bound to rx/tx queues. User applications can configure socket by providing several sockets that each bound to a single queue, applications can get data from kernel in parallel. After that, competitions mentioned above can be removed.

Quoted 6 lines

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

??? please elaborate on these NUMA memory allocations. This should be OK after commit 564824b0c52c34692d (net: allocate skbs on local node)

Quoted 7 lines

Quoted 3 lines

This patch set provides kernel ioctl interfaces for user space. User space can either directly call the interfaces or libpcap interfaces can be further provided on the top of the kernel ioctl interfaces.

So, say we have 8 queues, you want libpcap opens 8 sockets, and bind them to each queue. Add a bpf filter to each one of them. This seems not generic way, because it wont work for an UDP socket for example.

This only works for AF_PACKET like this patch set shows.

Quoted 2 lines

And you already can do this using SKF_AD_QUEUE (added in commit d19742fb)

SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE.

SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx.

I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue interface and still keep interfaces of

#define SIOGNUMRXQUEUE 0x8939  /* Get number of rx queues. */
#define SIOGNUMTXQUEUE 0x893A  /* Get number of tx queues. */
#define SIOSTXQUEUEMAPPING     0x893C  /* Set tx queue mapping. */
#define SIOGRXQUEUEMAPPING     0x893D  /* Get rx queue mapping. */
#define SIOGTXQUEUEMAPPING     0x893E  /* Get tx queue mapping. */

Quoted 1 line

Quoted 2 lines

Also your AF_PACKET patch only address mmaped sockets.

The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped sockets.

SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE.

SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx.

I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue interface and still keep interfaces of
#define SIOGNUMRXQUEUE 0x8939  /* Get number of rx queues. */ #define SIOGNUMTXQUEUE 0x893A  /* Get number of tx queues. */ #define SIOSTXQUEUEMAPPING     0x893C  /* Set tx queue mapping. */ #define SIOGRXQUEUEMAPPING     0x893D  /* Get rx queue mapping. */ #define SIOGTXQUEUEMAPPING     0x893E  /* Get tx queue mapping. */

Quoted 1 line

Quoted 2 lines

Also your AF_PACKET patch only address mmaped sockets.

The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped sockets.

Quoted 6 lines

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

??? please elaborate on these NUMA memory allocations. This should be OK after commit 564824b0c52c34692d (net: allocate skbs on local node)

Quoted 2 lines · Fenghua Yu

SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE.

It is wrong. AFAIK, you can get the queue number through /sys/class/net/eth*/queues/ or /proc/interrupts

Quoted 29 lines · Eric Dumazet

Quoted 27 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

Multiqueue and multicore provide packet parallel processing methodology. Current kernel and network drivers place one queue on one core. But the higher level socket doesn't know multiqueue. Current socket only can receive or send packets through one network interfaces. In some cases e.g. multi bpf filter tcpdump and snort, a lot of contentions come from socket operations like ring buffer. Even if the application itself has been fully parallelized and run on multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer and NIC device driver assemble packets to a single, serialized queue. Thus the application cannot actually run in parallel in high speed.

To break the serialized packets assembling bottleneck in kernel, one way is to allow socket to know multiqueue associated with a NIC interface. So each socket can handle tx/rx in one queue in parallel.

Kernel provides several interfaces by which sockets can be bound to rx/tx queues. User applications can configure socket by providing several sockets that each bound to a single queue, applications can get data from kernel in parallel. After that, competitions mentioned above can be removed.

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

Quoted 3 lines

??? please elaborate on these NUMA memory allocations. This should be OK after commit 564824b0c52c34692d (net: allocate skbs on local node)

Quoted 7 lines

Quoted 3 lines

This patch set provides kernel ioctl interfaces for user space. User space can either directly call the interfaces or libpcap interfaces can be further provided on the top of the kernel ioctl interfaces.

So, say we have 8 queues, you want libpcap opens 8 sockets, and bind them to each queue. Add a bpf filter to each one of them. This seems not generic way, because it wont work for an UDP socket for example.

This only works for AF_PACKET like this patch set shows.

Quoted 2 lines

And you already can do this using SKF_AD_QUEUE (added in commit d19742fb)

SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE.

SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx.

I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue interface and still keep interfaces of

#define SIOGNUMRXQUEUE 0x8939  /* Get number of rx queues. */
#define SIOGNUMTXQUEUE 0x893A  /* Get number of tx queues. */
#define SIOSTXQUEUEMAPPING     0x893C  /* Set tx queue mapping. */
#define SIOGRXQUEUEMAPPING     0x893D  /* Get rx queue mapping. */
#define SIOGTXQUEUEMAPPING     0x893E  /* Get tx queue mapping. */

Quoted 1 line

Quoted 2 lines

Also your AF_PACKET patch only address mmaped sockets.

The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped sockets.

Quoted 10 lines · Eric Dumazet

Quoted 6 lines

With this patch, the user-space receiving speed on a Intel SR1690 server with a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed the performance penalty comes from NUMA memory allocation.

??? please elaborate on these NUMA memory allocations. This should be OK after commit 564824b0c52c34692d (net: allocate skbs on local node)

Hi Eric,
Commit 564824b0c52c34692d had been used in the experiments, but the problem remained unsolved.

SLUB was used, and both servers were equipped with 8G physical memory. Is there any
additional information I can provide?

We really need to be smarter than that, not adding raw API.

Tom Herbert added RPS, RFS, XPS, in a way applications dont have to use special API, just run normal code.

Please understand that using 8 AF_PACKET sockets bound to a given device is a total waste, because the way we loop on ptype_all before entering AF_PACKET code, and in 12% of the cases deliver the packet into a queue, and 77.5% of the case reject the packet.

This is absolutely not scalable to say... 64 queues.

I do believe we can handle that using one AF_PACKET socket for the RX side, in order to not slow down the loop we have in __netif_receive_skb()

list_for_each_entry_rcu(ptype, &ptype_all, list) {
       ...
       deliver_skb(skb, pt_prev, orig_dev); }

(Same problem with dev_queue_xmit_nit() by the way, even worse since we skb_clone() packet _before_ entering af_packet code)

And we can change af_packet to split the load to N skb queues or N ring buffers, N not being necessarly number of NIC queues, but the number needed to handle the expected load.

There is nothing preventing us changing af_packet/udp/tcp_listener to something more scalable in itself, using a set of receive queues, and NUMA friendly data set. We did multiqueue for a net_device like this, not adding N pseudo devices as we could have done.

Quoted 3 lines · Junchang Wang

Commit 564824b0c52c34692d had been used in the experiments, but the problem remained unsolved.

Quoted 4 lines

SLUB was used, and both servers were equipped with 8G physical memory. Is there any
additional information I can provide?

Yes, sure, you could provide a description of the bench you used, and data you gathered to make the conclusion that NUMA was a problem.

Quoted 3 lines · Fenghua Yu

SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE.

It is wrong. AFAIK, you can get the queue number through /sys/class/net/eth*/queues/ or /proc/interrupts

If you turn to SKF_AD_QUEUE, I think no patch for kernel is needed.

Quoted 8 lines · Changli Gao

Quoted 3 lines · Fenghua Yu

SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE.

It is wrong. AFAIK, you can get the queue number through /sys/class/net/eth*/queues/ or /proc/interrupts

Valuable comment. Thanks.

Quoted 3 lines

If you turn to SKF_AD_QUEUE, I think no patch for kernel is needed.

This patch set is about parallelization of socket interfaces to gain performance boost (say, from 1Mpps to around 5Mpps), rather than simply bounding socket to cpu/queue. Therefore, it does worth having.