xsk: Support UMEM chunk_size > PAGE_SIZE
Subsystems (7)
v4 cover xsk: Support UMEM chunk_size > PAGE_SIZE
The main purpose of this patchset is to add AF_XDP support for UMEM chunk sizes > PAGE_SIZE. This is enabled for UMEMs backed by HugeTLB pages.
Changes since v3:
* Fix checkpatch.pl whitespace error.
Changes since v2:
* Related fixes/improvements included with v2 have been removed. These changes have all been resubmitted as standalone patchsets.
* Minimize uses of #ifdef CONFIG_HUGETLB_PAGE.
* Improve AF_XDP documentation.
* Update benchmark table in commit message.
Changes since v1:
* Add many fixes/improvements to the XSK selftests.
* Add check for unaligned descriptors that overrun UMEM.
* Fix compile errors when CONFIG_HUGETLB_PAGE is not set.
* Fix incorrect use of _Static_assert.
* Update AF_XDP documentation.
* Rename unaligned 9K frame size test.
* Make xp_check_dma_contiguity less conservative.
* Add more information to benchmark table.
Thanks to Magnus Karlsson for all his support!
Happy easter!
Kal Conley (3):
xsk: Support UMEM chunk_size > PAGE_SIZE
selftests: xsk: Use hugepages when umem->frame_size > PAGE_SIZE
selftests: xsk: Add tests for 8K and 9K frame sizes
Documentation/networking/af_xdp.rst | 36 ++++++++++------- include/net/xdp_sock.h | 1 + include/net/xdp_sock_drv.h | 12 ++++++ include/net/xsk_buff_pool.h | 3 +- net/xdp/xdp_umem.c | 50 ++++++++++++++++++++---- net/xdp/xsk_buff_pool.c | 28 ++++++++----- tools/testing/selftests/bpf/xskxceiver.c | 27 ++++++++++++- tools/testing/selftests/bpf/xskxceiver.h | 2 + 8 files changed, 126 insertions(+), 33 deletions(-)
Signature · 1 line
2.39.2
PATCH v4 1/3 xsk: Support UMEM chunk_size > PAGE_SIZE
Add core AF_XDP support for chunk sizes larger than PAGE_SIZE. This enables sending/receiving jumbo ethernet frames up to the theoretical maxiumum of 64 KiB. For chunk sizes > PAGE_SIZE, the UMEM is required to consist of HugeTLB VMAs (and be hugepage aligned). Initially, only SKB mode is usable pending future driver work.
For consistency, check for HugeTLB pages during UMEM registration. This implies that hugepages are required for XDP_COPY mode despite DMA not being used. This restriction is desirable since it ensures user software can take advantage of future driver support.
Even in HugeTLB mode, continue to do page accounting using order-0 (4 KiB) pages. This minimizes the size of this change and reduces the risk of impacting other code. Taking full advantage of hugepages for accounting should improve XDP performance in the general case.
No significant change in RX/TX performance was observed with this patch. A few data points are reproduced below:
Machine : Dell PowerEdge R940
CPU : Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
NIC : MT27700 Family [ConnectX-4]
+-----+------+------+-------+--------+--------+--------+ | | | | chunk | packet | rxdrop | rxdrop | | | mode | mtu | size | size | (Mpps) | (Gbps) | +-----+------+------+-------+--------+--------+--------+ | old | -z | 3498 | 4000 | 320 | 16.0 | 40.9 | | new | -z | 3498 | 4000 | 320 | 16.0 | 40.9 | +-----+------+------+-------+--------+--------+--------+ | old | -z | 3498 | 4096 | 320 | 16.4 | 42.1 | | new | -z | 3498 | 4096 | 320 | 16.5 | 42.2 | +-----+------+------+-------+--------+--------+--------+ | new | -c | 3498 | 10240 | 320 | 6.1 | 15.6 | +-----+------+------+-------+--------+--------+--------+ | new | -S | 9000 | 10240 | 9000 | 0.37 | 26.9 | +-----+------+------+-------+--------+--------+--------+
--- Documentation/networking/af_xdp.rst | 36 +++++++++++++-------- include/net/xdp_sock.h | 1 + include/net/xdp_sock_drv.h | 12 +++++++ include/net/xsk_buff_pool.h | 3 +- net/xdp/xdp_umem.c | 50 ++++++++++++++++++++++++----- net/xdp/xsk_buff_pool.c | 28 ++++++++++------ 6 files changed, 98 insertions(+), 32 deletions(-)
Documentation/networking/af_xdp.rst +22 -14
@@ -105,12 +105,13 @@ with AF_XDP". It can be found at https://lwn.net/Articles/750845/. UMEM ---- -UMEM is a region of virtual contiguous memory, divided into -equal-sized frames. An UMEM is associated to a netdev and a specific -queue id of that netdev. It is created and configured (chunk size, -headroom, start address and size) by using the XDP_UMEM_REG setsockopt -system call. A UMEM is bound to a netdev and queue id, via the bind() -system call. +UMEM is a region of virtual contiguous memory divided into equal-sized +frames. This is the area that contains all the buffers that packets can +reside in. A UMEM is associated with a netdev and a specific queue id of +that netdev. It is created and configured (start address, size, +chunk size, and headroom) by using the XDP_UMEM_REG setsockopt system +call. A UMEM is bound to a netdev and queue id via the bind() system +call. An AF_XDP is socket linked to a single UMEM, but one UMEM can have multiple AF_XDP sockets. To share an UMEM created via one socket A,
@@ -418,14 +419,21 @@ negatively impact performance. XDP_UMEM_REG setsockopt ----------------------- -This setsockopt registers a UMEM to a socket. This is the area that -contain all the buffers that packet can reside in. The call takes a -pointer to the beginning of this area and the size of it. Moreover, it -also has parameter called chunk_size that is the size that the UMEM is -divided into. It can only be 2K or 4K at the moment. If you have an -UMEM area that is 128K and a chunk size of 2K, this means that you -will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM -area and that your largest packet size can be 2K. +This setsockopt registers a UMEM to a socket. The call takes a pointer +to the beginning of this area and the size of it. Moreover, there is a +parameter called chunk_size that is the size that the UMEM is divided +into. The chunk size limits the maximum packet size that can be sent or +received. For example, if you have a UMEM area that is 128K and a chunk +size of 2K, then you will be able to hold a maximum of 128K / 2K = 64 +packets in your UMEM. In this case, the maximum packet size will be 2K. + +Valid chunk sizes range from 2K to 64K. However, in aligned mode, the +chunk size must also be a power of two. Additionally, the chunk size +must not exceed the size of a page (usually 4K). This limitation is +relaxed for UMEM areas allocated with HugeTLB pages, in which case +chunk sizes up to 64K are allowed. Note, this only works with hugepages +allocated from the kernel's persistent pool. Using Transparent Huge +Pages (THP) has no effect on the maximum chunk size. There is also an option to set the headroom of each single buffer in the UMEM. If you set this to N bytes, it means that the packet will
include/net/xdp_sock.h +1 -0
@@ -30,6 +30,7 @@ struct xdp_umem {u8 flags;bool zc;struct page **pgs; +u32 page_size;int id;struct list_head xsk_dma_list;struct work_struct work;
include/net/xdp_sock_drv.h +12 -0
@@ -12,6 +12,18 @@#define XDP_UMEM_MIN_CHUNK_SHIFT 11#define XDP_UMEM_MIN_CHUNK_SIZE (1 << XDP_UMEM_MIN_CHUNK_SHIFT) +static_assert (XDP_UMEM_MIN_CHUNK_SIZE <=PAGE_SIZE ); + +/* Allow chunk sizes up to the maximum size of an ethernet frame (64 KiB). +* Larger chunks are not guaranteed to fit in a single SKB. +*/ +#ifdef CONFIG_HUGETLB_PAGE +#define XDP_UMEM_MAX_CHUNK_SHIFT min(16, HPAGE_SHIFT) +#else +#define XDP_UMEM_MAX_CHUNK_SHIFT min(16, PAGE_SHIFT) +#endif +#define XDP_UMEM_MAX_CHUNK_SIZE (1 << XDP_UMEM_MAX_CHUNK_SHIFT) +#ifdef CONFIG_XDP_SOCKETS void xsk_tx_completed (struct xsk_buff_pool *pool,u32 nb_entries);
include/net/xsk_buff_pool.h +2 -1
@@ -75,6 +75,7 @@ struct xsk_buff_pool {u32 chunk_size;u32 chunk_shift;u32 frame_len; +u32 page_size;u8 cached_need_wakeup;bool uses_need_wakeup;bool dma_need_sync;
@@ -175,7 +176,7 @@ static inline void xp_dma_sync_for_device(struct xsk_buff_pool *pool,static inline bool xp_desc_crosses_non_contig_pg (struct xsk_buff_pool *pool,u64 addr,u32 len) { -bool cross_pg = (addr & (PAGE_SIZE -1 )) + len >PAGE_SIZE ; +bool cross_pg = (addr & (pool->page_size -1 )) + len > pool->page_size;if (likely (!cross_pg))return false;
net/xdp/xdp_umem.c +42 -8
77 lines not on this page · raw message
net/xdp/xsk_buff_pool.c +19 -9
@@ -80,9 +80,10 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs, pool->headroom = umem->headroom; pool->chunk_size = umem->chunk_size; pool->chunk_shift =ffs (umem->chunk_size) -1 ; - pool->unaligned = unaligned; pool->frame_len = umem->chunk_size - umem->headroom -XDP_PACKET_HEADROOM ; + pool->page_size = umem->page_size; + pool->unaligned = unaligned; pool->umem = umem; pool->addrs = umem->addrs;INIT_LIST_HEAD (&pool->free_list);
@@ -369,16 +370,25 @@ void xp_dma_unmap(struct xsk_buff_pool *pool, unsigned long attrs) }EXPORT_SYMBOL (xp_dma_unmap); -static void xp_check_dma_contiguity (struct xsk_dma_map *dma_map) +/* HugeTLB pools consider contiguity at hugepage granularity only. Hence, all +* order-0 pages within a hugepage have the same contiguity value. +*/ +static void xp_check_dma_contiguity (struct xsk_dma_map *dma_map,u32 page_size) { -u32 i; +u32 stride = page_size >>PAGE_SHIFT ;/* in order-0 pages */ +u32 i, j; -for (i =0 ; i < dma_map->dma_pages_cnt -1 ; i++) { -if (dma_map->dma_pages[i] +PAGE_SIZE == dma_map->dma_pages[i +1 ]) - dma_map->dma_pages[i] |=XSK_NEXT_PG_CONTIG_MASK ; -else - dma_map->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK ; +for (i =0 ; i + stride < dma_map->dma_pages_cnt;) { +if (dma_map->dma_pages[i] + page_size == dma_map->dma_pages[i + stride]) { +for (j =0 ; j < stride; i++, j++) + dma_map->dma_pages[i] |=XSK_NEXT_PG_CONTIG_MASK ; + }else { +for (j =0 ; j < stride; i++, j++) + dma_map->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK ; + } } +for (; i < dma_map->dma_pages_cnt; i++) + dma_map->dma_pages[i] &= ~XSK_NEXT_PG_CONTIG_MASK ; }static int xp_init_dma_info (struct xsk_buff_pool *pool,struct xsk_dma_map *dma_map)
@@ -441,7 +451,7 @@ int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev, }if (pool->unaligned) -xp_check_dma_contiguity (dma_map); +xp_check_dma_contiguity (dma_map, pool->page_size); err =xp_init_dma_info (pool, dma_map);if (err) {
Signature · 1 line
2.39.2
PATCH v4 2/3 selftests: xsk: Use hugepages when umem->frame_size > PAGE_SIZE
In reply to Kal ConleyHugeTLB UMEMs now support chunk_size > PAGE_SIZE. Set MAP_HUGETLB when frame_size > PAGE_SIZE for future tests.
--- tools/testing/selftests/bpf/xskxceiver.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/bpf/xskxceiver.c +1 -1
@@ -1289,7 +1289,7 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject)void *bufs;int ret; -if (ifobject->umem->unaligned_mode) +if (ifobject->umem->frame_size >sysconf (_SC_PAGESIZE) || ifobject->umem->unaligned_mode) mmap_flags |=MAP_HUGETLB ;if (ifobject->shared_umem)
Signature · 1 line
2.39.2
PATCH v4 3/3 selftests: xsk: Add tests for 8K and 9K frame sizes
In reply to Kal ConleyAdd tests:
- RUN_TO_COMPLETION_8K_FRAME_SIZE: frame_size=8192 (aligned)
- UNALIGNED_9K_FRAME_SIZE: frame_size=9000 (unaligned)
--- tools/testing/selftests/bpf/xskxceiver.c | 25 ++++++++++++++++++++++++ tools/testing/selftests/bpf/xskxceiver.h | 2 ++ 2 files changed, 27 insertions(+)
tools/testing/selftests/bpf/xskxceiver.c +25 -0
@@ -1841,6 +1841,17 @@ static void run_pkt_test(struct test_spec *test, enum test_mode mode, enum test_pkt_stream_replace (test,DEFAULT_PKT_CNT ,PKT_SIZE );testapp_validate_traffic (test);break ; +case TEST_TYPE_RUN_TO_COMPLETION_8K_FRAME : +if (!hugepages_present (test->ifobj_tx)) { +ksft_test_result_skip ("No 2M huge pages present.\n" ); +return ; + } +test_spec_set_name (test,"RUN_TO_COMPLETION_8K_FRAME_SIZE" ); + test->ifobj_tx->umem->frame_size =8192 ; + test->ifobj_rx->umem->frame_size =8192 ; +pkt_stream_replace (test,DEFAULT_PKT_CNT ,PKT_SIZE ); +testapp_validate_traffic (test); +break ;case TEST_TYPE_RX_POLL : test->ifobj_rx->use_poll = true;test_spec_set_name (test,"POLL_RX" );
@@ -1904,6 +1915,20 @@ static void run_pkt_test(struct test_spec *test, enum test_mode mode, enum test_if (!testapp_unaligned (test))return ;break ; +case TEST_TYPE_UNALIGNED_9K_FRAME : +if (!hugepages_present (test->ifobj_tx)) { +ksft_test_result_skip ("No 2M huge pages present.\n" ); +return ; + } +test_spec_set_name (test,"UNALIGNED_9K_FRAME_SIZE" ); + test->ifobj_tx->umem->frame_size =9000 ; + test->ifobj_rx->umem->frame_size =9000 ; + test->ifobj_tx->umem->unaligned_mode = true; + test->ifobj_rx->umem->unaligned_mode = true; +pkt_stream_replace (test,DEFAULT_PKT_CNT ,PKT_SIZE ); + test->ifobj_rx->pkt_stream->use_addr_for_fill = true; +testapp_validate_traffic (test); +break ;case TEST_TYPE_HEADROOM :testapp_headroom (test);break ;
tools/testing/selftests/bpf/xskxceiver.h +2 -0
@@ -69,12 +69,14 @@ enum test_mode {enum test_type {TEST_TYPE_RUN_TO_COMPLETION ,TEST_TYPE_RUN_TO_COMPLETION_2K_FRAME , +TEST_TYPE_RUN_TO_COMPLETION_8K_FRAME ,TEST_TYPE_RUN_TO_COMPLETION_SINGLE_PKT ,TEST_TYPE_RX_POLL ,TEST_TYPE_TX_POLL ,TEST_TYPE_POLL_RXQ_TMOUT ,TEST_TYPE_POLL_TXQ_TMOUT ,TEST_TYPE_UNALIGNED , +TEST_TYPE_UNALIGNED_9K_FRAME ,TEST_TYPE_ALIGNED_INV_DESC ,TEST_TYPE_ALIGNED_INV_DESC_2K_FRAME ,TEST_TYPE_UNALIGNED_INV_DESC ,
Signature · 1 line
2.39.2