net: ena: Add dynamic recycling mechanism for rx buffers
Subsystems (4)
[email protected]
maintainerPATCH 1/1 net: ena: Add dynamic recycling mechanism for rx buffers
From: David Arinzon <[email protected]>
The current implementation allocates page-sized rx buffers. As traffic may consist of different types and sizes of packets, in various cases, buffers are not fully used.
This change (Dynamic RX Buffers - DRB) uses part of the allocated rx page needed for the incoming packet, and returns the rest of the unused page to be used again as an rx buffer for future packets. A threshold of 2K for unused space has been set in order to declare whether the remainder of the page can be reused again as an rx buffer.
As a page may be reused, dma_sync_single_for_cpu() is added in order to sync the memory to the CPU side after it was owned by the HW. In addition, when the rx page can no longer be reused, it is being unmapped using dma_page_unmap(), which implicitly syncs and then unmaps the entire page. In case the kernel still handles the skbs pointing to the previous buffers from that rx page, it may access garbage pointers, caused by the implicit sync overwriting them. The implicit dma sync is removed by replacing dma_page_unmap() with dma_unmap_page_attrs() with DMA_ATTR_SKIP_CPU_SYNC flag.
The functionality is disabled for XDP traffic to avoid handling several descriptors per packet.
--- .../device_drivers/ethernet/amazon/ena.rst | 32 +++++ .../net/ethernet/amazon/ena/ena_admin_defs.h | 6 +- drivers/net/ethernet/amazon/ena/ena_netdev.c | 136 ++++++++++++------ drivers/net/ethernet/amazon/ena/ena_netdev.h | 4 + 4 files changed, 136 insertions(+), 42 deletions(-)
Documentation/networking/device_drivers/ethernet/amazon/ena.rst +32 -0
@@ -205,6 +205,7 @@ Adaptive coalescing can be switched on/off through `ethtool(8)`'s More information about Adaptive Interrupt Moderation (DIM) can be found in Documentation/networking/net_dim.rst +.. _`RX copybreak`: RX copybreak ============ The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK
@@ -315,3 +316,34 @@ Rx - The new SKB is updated with the necessary information (protocol, checksum hw verify result, etc), and then passed to the network stack, using the NAPI interface function :code:`napi_gro_receive()`. + +Dynamic RX Buffers (DRB) +------------------------ + +Each RX descriptor in the RX ring is a single memory page (which is either 4KB +or 16KB long depending on system's configurations). +To reduce the memory allocations required when dealing with a high rate of small +packets, the driver tries to reuse the remaining RX descriptor's space if more +than 2KB of this page remain unused. + +A simple example of this mechanism is the following sequence of events: + +:: + + 1. Driver allocates page-sized RX buffer and passes it to hardware + +----------------------+ + |4KB RX Buffer | + +----------------------+ + + 2. A 300Bytes packet is received on this buffer + + 3. The driver increases the ref count on this page and returns it back to + HW as an RX buffer of size 4KB - 300Bytes = 3796 Bytes + +----+--------------------+ + |****|3796 Bytes RX Buffer| + +----+--------------------+ + +This mechanism isn't used when an XDP program is loaded, or when the +RX packet is less than rx_copybreak bytes (in which case the packet is +copied out of the RX buffer into the linear part of a new skb allocated +for it and the RX buffer remains the same size, see `RX copybreak`_).
drivers/net/ethernet/amazon/ena/ena_admin_defs.h +5 -1
@@ -869,7 +869,9 @@ struct ena_admin_host_info {* 2 : interrupt_moderation * 3 : rx_buf_mirroring * 4 : rss_configurable_function_key -* 31:5 : reserved +* 5 : reserved +* 6 : rx_page_reuse +* 31:7 : reserved */ u32 driver_supported_features; };
@@ -1184,6 +1186,8 @@ struct ena_admin_ena_mmio_req_read_less_resp {#define ENA_ADMIN_HOST_INFO_RX_BUF_MIRRORING_MASK BIT(3)#define ENA_ADMIN_HOST_INFO_RSS_CONFIGURABLE_FUNCTION_KEY_SHIFT 4#define ENA_ADMIN_HOST_INFO_RSS_CONFIGURABLE_FUNCTION_KEY_MASK BIT(4) +#define ENA_ADMIN_HOST_INFO_RX_PAGE_REUSE_SHIFT 6 +#define ENA_ADMIN_HOST_INFO_RX_PAGE_REUSE_MASK BIT(6)/* aenq_common_desc */ #define ENA_ADMIN_AENQ_COMMON_DESC_PHASE_MASK BIT(0)
drivers/net/ethernet/amazon/ena/ena_netdev.c +95 -41
268 lines not on this page · raw message
drivers/net/ethernet/amazon/ena/ena_netdev.h +4 -0
@@ -51,6 +51,8 @@#define ENA_DEFAULT_RING_SIZE (1024)#define ENA_MIN_RING_SIZE (256) +#define ENA_MIN_RX_BUF_SIZE (2048) +#define ENA_MIN_NUM_IO_QUEUES (1)#define ENA_TX_WAKEUP_THRESH (MAX_SKB_FRAGS + 2)
@@ -175,7 +177,9 @@ struct ena_tx_buffer {struct ena_rx_buffer {struct sk_buff *skb;struct page *page; +dma_addr_t dma_addr;u32 page_offset; +u32 buf_offset;struct ena_com_buf ena_buf; }____cacheline_aligned ;
Signature · 1 line
2.40.1
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 6 lines · On Mon, 12 Jun 2023 12:14:48 +0000 you wrote:
From: David Arinzon <[email protected]>
The current implementation allocates page-sized rx buffers. As traffic may consist of different types and sizes of packets, in various cases, buffers are not fully used.
Quoted 7 lines
This change (Dynamic RX Buffers - DRB) uses part of the allocated rx page needed for the incoming packet, and returns the rest of the unused page to be used again as an rx buffer for future packets. A threshold of 2K for unused space has been set in order to declare whether the remainder of the page can be reused again as an rx buffer.
[...]
Here is the summary with links:
- [v1,net-next] net: ena: Add dynamic recycling mechanism for rx buffers
https://git.kernel.org/netdev/net-next/c/f7d625adeb7b
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html