Skip to contentSkip to conversation

xsk: ixgbe: solve the soft interrupt 100% CPU usage when xdp rx traffic congestion

PATCH In review Yahui Chen 2 participants 3 messages Jul 11, 2020 – Jul 13, 2020

Subsystems (4)
lore
Share & tools
lore

Yahui Chen

author
#0 lore raw

PATCH 1/1 xsk: ixgbe: solve the soft interrupt 100% CPU usage when xdp rx traffic congestion

If the user mode APP of drv-mode xsk does not process packets fast enough, the packet receiving speed is lower than the flow rate. For example, the flow rate is 1Mpps, and the APP processing speed is 10Kpss only, this will cause a series of problems:

1. If the APP doesn't use wakeup mechanism, in other world, the XDP_UMEM_USES_NEED_WAKEUP flag is not set. Then the CPU usage of the soft interrupt corresponding to the network card queue using xsk has always been 100%. The reason is that the APP is slow to receive packets, the packet desc can not return to the fill queue in time, resulting in ixgbe_alloc_rx_buffers_zc fail to obtain the desc, so ixgbe_clean_rx_irq_zc returns 64, which further causes its corresponding NAPI to be always running status, so the CPU has been busy performing soft interrupts.

2. If the wakeup mechanism is used, that is, use the XDP_UMEM_USES_NEED_WAKEUP flag. This method takes advantage of the interrupt delay function of ixgbe skillfully, thus solving the problem that the si CPU is always 100%. However, it will cause other problems. The port-level flow control will be triggered on 82599, and the pause frame will be sent to the upstream sender. This will affect the other packet receiving queues of the network card, resulting in the packet receiving rate of all queues dropping to 10Kpps.

This situation can be understood as rx traffic congestion. The reason of congestion is that XSK queue cannot get DMA address in time, so packet cannot be delivered to the host memory which result in packets congestion happening on the hardware FIFO. If the number of packets exceeds the FIFO waterline, the network card will send pause frames.

This patch design another way. When the queue corresponding to xsk is initializing, a piece of DMA memory, named congestion memory, is applied for traffic congestion. If failing to obtain the desc from the fill queue, point the xsk ring packet DMA address to the congestion memory. Let the network card hardware DMA the packet to the congetion memory, and the APP will not receive packet from the congestion memory. This way can solve the two problems mentioned above. However, the wakeup mechanism should also be retained. When the network card has only one queue, the wakeup mechanism will have advantages.

TODO:
Apply this modification to all drivers that support xsk drv mode

---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h     |  3 ++
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 61 ++++++++++++++++++++++++----
 2 files changed, 56 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe.h +3 -0
index 5ddfc83..33601c1 100644
@@ -351,6 +351,9 @@ struct ixgbe_ring {
 	};
 	struct xdp_rxq_info xdp_rxq;
 	struct xdp_umem *xsk_umem;
+#define XDP_CONGESTION_MEM_SIZE	(4096)
+	void *congestion_addr;		/* address of buffer when traffic congestion */
+	dma_addr_t congestion_dma;	/* dma address of packet when traffic congestion */
 	u16 ring_idx;		/* {rx,tx,xdp}_ring back reference idx */
 	u16 rx_buf_len;
 } ____cacheline_internodealigned_in_smp;
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +53 -8

137 lines not on this page · raw message

Signature · 1 line
1.8.3.1
maintainer
#1 lore raw
Quoted 8 lines · On Sat, 11 Jul 2020 18:10:38 +0800 Yahui Chen wrote:

2. If the wakeup mechanism is used, that is, use the XDP_UMEM_USES_NEED_WAKEUP flag. This method takes advantage of the interrupt delay function of ixgbe skillfully, thus solving the problem that the si CPU is always 100%. However, it will cause other problems. The port-level flow control will be triggered on 82599, and the pause frame will be sent to the upstream sender. This will affect the other packet receiving queues of the network card, resulting in the packet receiving rate of all queues dropping to 10Kpps.

To me the current behavior sounds correct.. if you don't want pause frames to be generated you have to disable them completely. The point of pause frames is to prevent drops.

author
#2 lore raw

xdp skb-mode does not have this problem because ixgbe_alloc_rx_buffers always returns success and can always get the DMA address for packets. I think drv-mode should do the same.

If the drv-mode app does not use the wakeup flag, the xdp queue's soft interrupt handles fewer packets but may occupy 100% of the CPU, according to the analysis of the previous email.

If the user APP uses the wakeup flag, it maybe seriously affect other queues just because the APP is not processing the packet fast enough. In extreme cases, the APP can no longer receive packets, and port-level flow control will cause no packets to be received in all queues of the network card.

As a result, the user needs to explicitly turn off flow control of network card, which is unfriendly to users.

Jakub Kicinski <[email protected]> ?2020?7?12??? ??2:14???

Quoted 14 lines · On Sat, 11 Jul 2020 18:10:38 +0800 Yahui Chen wrote:
Quoted 8 lines

2. If the wakeup mechanism is used, that is, use the XDP_UMEM_USES_NEED_WAKEUP flag. This method takes advantage of the interrupt delay function of ixgbe skillfully, thus solving the problem that the si CPU is always 100%. However, it will cause other problems. The port-level flow control will be triggered on 82599, and the pause frame will be sent to the upstream sender. This will affect the other packet receiving queues of the network card, resulting in the packet receiving rate of all queues dropping to 10Kpps.

To me the current behavior sounds correct.. if you don't want pause frames to be generated you have to disable them completely. The point of pause frames is to prevent drops.