Skip to content

staging: qlge: Fix rx stall in case of allocation failures

Benjamin Poirier lkml netdev 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v2 Latest17 patches 3 repliesIn review

Changes in v2 from the cover letter

https://lore.kernel.org/netdev/[email protected]/
* simplified QLGE_FIT16 macro down to a simple cast
* added "qlge: Fix irq masking in INTx mode"
* fixed address in pci_unmap_page() calls in "qlge: Deduplicate rx buffer queue management", no effect on end result of series
* adjusted series following move of driver to staging

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/17staging: qlge: Fix irq masking in INTx modedrivers/staging/qlge/qlge_main.c+11 -00 replies
2/17staging: qlge: Remove irq_cntdrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c drivers/staging/qlge/qlge_mpi.c+27 -790 replies
3/17staging: qlge: Remove page_chunk.last_flagdrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c+5 -90 replies
4/17staging: qlge: Deduplicate lbq_buf_sizedrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_dbg.c drivers/staging/qlge/qlge_main.c+28 -370 replies
5/17staging: qlge: Remove bq_desc.maplendrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c+15 -290 replies
6/17staging: qlge: Remove rx_ring.sbq_buf_sizedrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_dbg.c drivers/staging/qlge/qlge_main.c+10 -160 replies
7/17staging: qlge: Remove useless dma synchronization callsdrivers/staging/qlge/qlge_main.c+0 -120 replies
8/17staging: qlge: Deduplicate rx buffer queue managementdrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_dbg.c drivers/staging/qlge/qlge_main.c+335 -3940 replies
9/17staging: qlge: Fix dma_sync_single callsdrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c+22 -370 replies
10/17staging: qlge: Remove rx_ring.typedrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_dbg.c drivers/staging/qlge/qlge_main.c+19 -380 replies
11/17staging: qlge: Factor out duplicated expressiondrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c+11 -120 replies
12/17staging: qlge: Remove qlge_bq.len & sizedrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_dbg.c drivers/staging/qlge/qlge_main.c+39 -570 replies
13/17staging: qlge: Remove useless memsetdrivers/staging/qlge/qlge_main.c+0 -10 replies
14/17staging: qlge: Replace memset with assignmentdrivers/staging/qlge/qlge_main.c+1 -20 replies
15/17staging: qlge: Update buffer queue prod index despite oomdrivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_dbg.c drivers/staging/qlge/qlge_main.c+60 -630 replies
16/17staging: qlge: Refill rx buffers up to multiple of 16drivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c+19 -180 replies
17/17staging: qlge: Refill empty buffer queues from wqdrivers/staging/qlge/TODO drivers/staging/qlge/qlge.h drivers/staging/qlge/qlge_main.c+72 -190 replies

Reviews trailers per patch

4 of 17 patches have a Reviewed-by or Acked-by.

PatchManish ChopraWillem de Bruijn
1/17
2/17
3/17A-b
4/17A-b
5/17A-b
6/17R-b
7/17
8/17
9/17
10/17
11/17
12/17
13/17
14/17
15/17
16/17
17/17

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 0/17] staging: qlge: Fix rx stall in case of allocation failures

qlge refills rx buffers from napi context. In case of allocation failure, allocation will be retried the next time napi runs. If a receive queue runs out of free buffers (possibly after subsequent allocation failures), it drops all traffic, no longer raises interrupts and napi is no longer scheduled; reception is stalled until manual admin intervention.

This patch series adds a fallback mechanism for rx buffer allocation. If an rx buffer queue becomes empty, a workqueue is scheduled to refill it from process context where allocation can block until mm has freed some pages (hopefully). This approach was inspired by the virtio_net driver (commit 3161e453e496 "virtio: net refill on out-of-memory").

I've compared this with how some other devices with a similar allocation scheme handle this situation:
mlx4 relies on a periodic watchdog, sfc uses a timer, e1000e and fm10k rely on periodic hardware interrupts (IIUC). In all cases, they use this to schedule napi periodically at a fixed interval (10-250ms) until allocations succeed. This kind of approach simplifies allocations because only one context may refill buffers, however it is inefficient because of the fixed
interval: either the interval was too short, the allocation fails again and work was done without forward progress; or the interval was too long, buffers could've been allocated earlier and rx restarted earlier, instead traffic was dropped while the system was idle.

Note that the qlge driver (and device) uses two kinds of buffers for received data, so-called "small buffers" and "large buffers". The two are arranged in ring pairs, the sbq and lbq. Depending on frame size, protocol content and header splitting, data can go in either type of buffers. Because of buffer size, lbq allocations are more likely to fail and lead to stall, however I've reproduced the problem with sbq as well. The problem was originally found when running jumbo frames. In that case, qlge uses order-1 allocations for the large buffers. Although the two kinds of buffers are managed similarly, the qlge driver duplicates most data structures and code for their handling. In fact, even a casual look at the qlge driver shows it to be in a state of disrepair, to put it kindly...

Patches 1-14 are cleanups that remove, fix and deduplicate code related to sbq and lbq handling. Regarding those cleanups, patches 2 ("Remove irq_cnt") and 8 ("Deduplicate rx buffer queue management") are the most important. Finally, patches 15-17 fix the actual problem of rx stalls in case of allocation failures by implementing the fallback of allocations to a workqueue.

I've tested these patches using two different approaches:
1) A sender uses pktgen to send udp traffic. The receiver has a large swap, a large net.core.rmem_max, runs a program that dirties all free memory in a loop and runs a program that opens as many udp sockets as possible but doesn't read from them. Since received data is all queued in the sockets rather than freed, qlge is allocating receive buffers as quickly as possible and faces allocation failures if the swap is slower than the network.
2) A sender uses super_netperf. Likewise, the receiver has a large swap, a large net.core.rmem_max and runs a program that dirties all free memory in a loop. After the netperf send test is started, killall -s SIGSTOP netserver on the receiver leads to the same situation as above.

---
Changes
v1->v2
https://lore.kernel.org/netdev/[email protected]/
* simplified QLGE_FIT16 macro down to a simple cast
* added "qlge: Fix irq masking in INTx mode"
* fixed address in pci_unmap_page() calls in "qlge: Deduplicate rx buffer queue management", no effect on end result of series
* adjusted series following move of driver to staging