XDP unaligned chunk placement support
Subsystems (8)
cover XDP unaligned chunk placement support
This patchset adds the ability to use unaligned chunks in the XDP umem.
Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are supported.
The changes in this patchset removes these restrictions, allowing XDP to be more flexible in where it can place a chunk within a umem. By relaxing where the chunks can be placed, it allows us to use an arbitrary buffer size and place that wherever we have a free address in the umem. These changes add the ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as DPDK.
Structure of the patchset:
Patch 1:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in i40e. This change is required in order for the buffer recycling to work in the unaligned chunk mode.
Patch 2:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in ixgbe. This change is required in order for the buffer recycling to work in the unaligned chunk mode.
Patch 3:
- Adds an offset parameter to zero_copy_allocator. This change will enable us to calculate the original handle in zca_free. This will be required for unaligned chunk mode since we can't easily mask back to the original handle.
Patch 4:
- Adds the offset parameter to i40e_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.
Patch 5:
- Adds the offset parameter to ixgbe_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.
Patch 6:
- Add infrastructure for unaligned chunks. Since we are dealing with unaligned chunks that could potentially cross a physical page boundary, we add checks to keep track of that information. We can later use this information to correctly handle buffers that are placed at an address where they cross a page boundary.
Patch 7:
- Add flags for umem configuration to libbpf
Patch 8:
- Modify xdpsock application to add a command line option for unaligned chunks
Patch 9:
- Addition of command line argument to pass in a desired buffer size and buffer recycling for unaligned mode. Passing in a buffer size will allow the application to use unaligned chunks with the unaligned chunk mode. Since we are now using unaligned chunks, we need to recycle our buffers in a slightly different way.
Patch 10:
- Adds hugepage support to the xdpsock application
Patch 11:
- Documentation update to include the unaligned chunk scenario. We need to explicitly state that the incoming addresses are only masked in the aligned chunk mode and not the unaligned chunk mode.
Kevin Laatz (11):
i40e: simplify Rx buffer recycle
ixgbe: simplify Rx buffer recycle
xdp: add offset param to zero_copy_allocator
i40e: add offset to zca_free
ixgbe: add offset to zca_free
xsk: add support to allow unaligned chunk placement
libbpf: add flags to umem config
samples/bpf: add unaligned chunks mode support to xdpsock samples/bpf: add buffer recycling for unaligned chunks to xdpsock samples/bpf: use hugepages in xdpsock app
doc/af_xdp: include unaligned chunk case
Documentation/networking/af_xdp.rst | 10 +- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 21 ++-- drivers/net/ethernet/intel/i40e/i40e_xsk.h | 3 +- .../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 3 +- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 21 ++-- include/net/xdp.h | 3 +- include/net/xdp_sock.h | 2 + include/uapi/linux/if_xdp.h | 4 + net/core/xdp.c | 11 ++- net/xdp/xdp_umem.c | 17 ++-- net/xdp/xsk.c | 60 +++++++++-- net/xdp/xsk_queue.h | 60 +++++++++-- samples/bpf/xdpsock_user.c | 99 ++++++++++++++----- tools/include/uapi/linux/if_xdp.h | 4 + tools/lib/bpf/xsk.c | 7 ++ tools/lib/bpf/xsk.h | 2 + 16 files changed, 241 insertions(+), 86 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/11 i40e: simplify Rx buffer recycle
Currently, the dma, addr and handle are modified when we reuse Rx buffers in zero-copy mode. However, this is not required as the inputs to the function are copies, not the original values themselves. As we use the copies within the function, we can use the original 'old_bi' values directly without having to mask and add the headroom.
--- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_xsk.c +3 -10
@@ -419,8 +419,6 @@ static void i40e_reuse_rx_buffer_zc(struct i40e_ring *rx_ring, struct i40e_rx_buffer *old_bi) {struct i40e_rx_buffer *new_bi = &rx_ring->rx_bi[rx_ring->next_to_alloc]; -unsigned long mask = (unsigned long )rx_ring->xsk_umem->chunk_mask; -u64 hr = rx_ring->xsk_umem->headroom +XDP_PACKET_HEADROOM ;u16 nta = rx_ring->next_to_alloc;/* update, and store next to alloc */
@@ -428,14 +426,9 @@ static void i40e_reuse_rx_buffer_zc(struct i40e_ring *rx_ring, rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ;/* transfer page from old buffer to new buffer */ - new_bi->dma = old_bi->dma & mask; - new_bi->dma += hr; - - new_bi->addr = (void *)((unsigned long )old_bi->addr & mask); - new_bi->addr += hr; - - new_bi->handle = old_bi->handle & mask; - new_bi->handle += rx_ring->xsk_umem->headroom; + new_bi->dma = old_bi->dma; + new_bi->addr = old_bi->addr; + new_bi->handle = old_bi->handle; old_bi->addr =NULL ; }
Signature · 1 line
2.17.1
PATCH 2/11 ixgbe: simplify Rx buffer recycle
In reply to Kevin LaatzCurrently, the dma, addr and handle are modified when we reuse Rx buffers in zero-copy mode. However, this is not required as the inputs to the function are copies, not the original values themselves. As we use the copies within the function, we can use the original 'obi' values directly without having to mask and add the headroom.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +3 -10
@@ -251,8 +251,6 @@ ixgbe_rx_buffer *ixgbe_get_rx_buffer_zc(struct ixgbe_ring *rx_ring,static void ixgbe_reuse_rx_buffer_zc (struct ixgbe_ring *rx_ring,struct ixgbe_rx_buffer *obi) { -unsigned long mask = (unsigned long )rx_ring->xsk_umem->chunk_mask; -u64 hr = rx_ring->xsk_umem->headroom +XDP_PACKET_HEADROOM ;u16 nta = rx_ring->next_to_alloc;struct ixgbe_rx_buffer *nbi;
@@ -262,14 +260,9 @@ static void ixgbe_reuse_rx_buffer_zc(struct ixgbe_ring *rx_ring, rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ;/* transfer page from old buffer to new buffer */ - nbi->dma = obi->dma & mask; - nbi->dma += hr; - - nbi->addr = (void *)((unsigned long )obi->addr & mask); - nbi->addr += hr; - - nbi->handle = obi->handle & mask; - nbi->handle += rx_ring->xsk_umem->headroom; + nbi->dma = obi->dma; + nbi->addr = obi->addr; + nbi->handle = obi->handle; obi->addr =NULL ; obi->skb =NULL ;
Signature · 1 line
2.17.1
PATCH 3/11 xdp: add offset param to zero_copy_allocator
In reply to Kevin LaatzThis patch adds an offset parameter for zero_copy_allocator.
This change is required for the unaligned chunk mode which will come later in this patch set. The offset parameter is required for calculating the original handle in unaligned mode since we can't easily mask back to it like in the aligned case.
--- include/net/xdp.h | 3 ++- net/core/xdp.c | 11 ++++++----- 2 files changed, 8 insertions(+), 6 deletions(-)
include/net/xdp.h +2 -1
@@ -53,7 +53,8 @@ struct xdp_mem_info {struct page_pool ;struct zero_copy_allocator { -void (*free)(struct zero_copy_allocator *zca,unsigned long handle); +void (*free)(struct zero_copy_allocator *zca,unsigned long handle, +off_t off); };struct xdp_rxq_info {
net/core/xdp.c +6 -5
@@ -322,7 +322,7 @@ EXPORT_SYMBOL_GPL(xdp_rxq_info_reg_mem_model);* of xdp_frames/pages in those cases. */ static void __xdp_return (void *data,struct xdp_mem_info *mem,bool napi_direct, -unsigned long handle) +unsigned long handle,off_t off) {struct xdp_mem_allocator *xa;struct page *page;
@@ -353,7 +353,7 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,rcu_read_lock ();/* mem->id is valid, checked in xdp_rxq_info_reg_mem_model() */ xa =rhashtable_lookup (mem_id_ht, &mem->id, mem_id_rht_params); - xa->zc_alloc->free (xa->zc_alloc, handle); + xa->zc_alloc->free (xa->zc_alloc, handle, off);rcu_read_unlock ();default :/* Not possible, checked in xdp_rxq_info_reg_mem_model() */
@@ -363,19 +363,20 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,void xdp_return_frame (struct xdp_frame *xdpf) { -__xdp_return (xdpf->data, &xdpf->mem, false,0 ); +__xdp_return (xdpf->data, &xdpf->mem, false,0 ,0 ); }EXPORT_SYMBOL_GPL (xdp_return_frame);void xdp_return_frame_rx_napi (struct xdp_frame *xdpf) { -__xdp_return (xdpf->data, &xdpf->mem, true,0 ); +__xdp_return (xdpf->data, &xdpf->mem, true,0 ,0 ); }EXPORT_SYMBOL_GPL (xdp_return_frame_rx_napi);void xdp_return_buff (struct xdp_buff *xdp) { -__xdp_return (xdp->data, &xdp->rxq->mem, true, xdp->handle); +__xdp_return (xdp->data, &xdp->rxq->mem, true, xdp->handle, + xdp->data - xdp->data_hard_start); }EXPORT_SYMBOL_GPL (xdp_return_buff);
Signature · 1 line
2.17.1
PATCH 4/11 i40e: add offset to zca_free
In reply to Kevin LaatzThis patch adds the offset param to for zero_copy_allocator to i40e_zca_free. This change is required to calculate the handle, otherwise, this function will not work in unaligned chunk mode since we can't easily mask back to the original handle in unaligned chunk mode.
--- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 8 ++++---- drivers/net/ethernet/intel/i40e/i40e_xsk.h | 3 ++- 2 files changed, 6 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_xsk.c +4 -4
@@ -438,16 +438,16 @@ static void i40e_reuse_rx_buffer_zc(struct i40e_ring *rx_ring,* @alloc: Zero-copy allocator * @handle: Buffer handle **/ -void i40e_zca_free (struct zero_copy_allocator *alloc,unsigned long handle) +void i40e_zca_free (struct zero_copy_allocator *alloc,unsigned long handle, +off_t off) {struct i40e_rx_buffer *bi;struct i40e_ring *rx_ring; -u64 hr, mask; +u64 hr;u16 nta; rx_ring = container_of(alloc, struct i40e_ring, zca); hr = rx_ring->xsk_umem->headroom +XDP_PACKET_HEADROOM ; - mask = rx_ring->xsk_umem->chunk_mask; nta = rx_ring->next_to_alloc; bi = &rx_ring->rx_bi[nta];
@@ -455,7 +455,7 @@ void i40e_zca_free(struct zero_copy_allocator *alloc, unsigned long handle) nta++; rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ; - handle &= mask; + handle -= off; bi->dma =xdp_umem_get_dma (rx_ring->xsk_umem, handle); bi->dma += hr;
drivers/net/ethernet/intel/i40e/i40e_xsk.h +2 -1
@@ -12,7 +12,8 @@ int i40e_queue_pair_disable(struct i40e_vsi *vsi, int queue_pair);int i40e_queue_pair_enable (struct i40e_vsi *vsi,int queue_pair);int i40e_xsk_umem_setup (struct i40e_vsi *vsi,struct xdp_umem *umem,u16 qid); -void i40e_zca_free (struct zero_copy_allocator *alloc,unsigned long handle); +void i40e_zca_free (struct zero_copy_allocator *alloc,unsigned long handle, +off_t off);bool i40e_alloc_rx_buffers_zc (struct i40e_ring *rx_ring,u16 cleaned_count);int i40e_clean_rx_irq_zc (struct i40e_ring *rx_ring,int budget);
Signature · 1 line
2.17.1
PATCH 5/11 ixgbe: add offset to zca_free
In reply to Kevin LaatzThis patch adds the offset param to for zero_copy_allocator to ixgbe_zca_free. This change is required to calculate the handle, otherwise, this function will not work in unaligned chunk mode since we can't easily mask back to the original handle in unaligned chunk mode.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h | 3 ++- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 8 ++++---- 2 files changed, 6 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h +2 -1
@@ -33,7 +33,8 @@ struct xdp_umem *ixgbe_xsk_umem(struct ixgbe_adapter *adapter,int ixgbe_xsk_umem_setup (struct ixgbe_adapter *adapter,struct xdp_umem *umem,u16 qid); -void ixgbe_zca_free (struct zero_copy_allocator *alloc,unsigned long handle); +void ixgbe_zca_free (struct zero_copy_allocator *alloc,unsigned long handle, +off_t off);void ixgbe_alloc_rx_buffers_zc (struct ixgbe_ring *rx_ring,u16 cleaned_count); int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +4 -4
@@ -268,16 +268,16 @@ static void ixgbe_reuse_rx_buffer_zc(struct ixgbe_ring *rx_ring, obi->skb =NULL ; } -void ixgbe_zca_free (struct zero_copy_allocator *alloc,unsigned long handle) +void ixgbe_zca_free (struct zero_copy_allocator *alloc,unsigned long handle, +off_t off) {struct ixgbe_rx_buffer *bi;struct ixgbe_ring *rx_ring; -u64 hr, mask; +u64 hr;u16 nta; rx_ring = container_of(alloc, struct ixgbe_ring, zca); hr = rx_ring->xsk_umem->headroom +XDP_PACKET_HEADROOM ; - mask = rx_ring->xsk_umem->chunk_mask; nta = rx_ring->next_to_alloc; bi = rx_ring->rx_buffer_info;
@@ -285,7 +285,7 @@ void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle) nta++; rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta :0 ; - handle &= mask; + handle -= off; bi->dma =xdp_umem_get_dma (rx_ring->xsk_umem, handle); bi->dma += hr;
Signature · 1 line
2.17.1
PATCH 6/11 xsk: add support to allow unaligned chunk placement
In reply to Kevin LaatzCurrently, addresses are chunk size aligned. This means, we are very restricted in terms of where we can place chunk within the umem. For example, if we have a chunk size of 2k, then our chunks can only be placed at 0,2k,4k,6k,8k... and so on (ie. every 2k starting from 0).
This patch introduces the ability to use unaligned chunks. With these changes, we are no longer bound to having to place chunks at a 2k (or whatever your chunk size is) interval. Since we are no longer dealing with aligned chunks, they can now cross page boundaries. Checks for page contiguity have been added in order to keep track of which pages are followed by a physically contiguous page.
--- include/net/xdp_sock.h | 2 ++ include/uapi/linux/if_xdp.h | 4 +++ net/xdp/xdp_umem.c | 17 +++++++---- net/xdp/xsk.c | 60 ++++++++++++++++++++++++++++++++----- net/xdp/xsk_queue.h | 60 ++++++++++++++++++++++++++++++++----- 5 files changed, 121 insertions(+), 22 deletions(-)
include/net/xdp_sock.h +2 -0
@@ -19,6 +19,7 @@ struct xsk_queue;struct xdp_umem_page {void *addr;dma_addr_t dma; +bool next_pg_contig; };struct xdp_umem_fq_reuse {
@@ -48,6 +49,7 @@ struct xdp_umem {bool zc;spinlock_t xsk_list_lock;struct list_head xsk_list; +u32 flags; };struct xdp_sock {
include/uapi/linux/if_xdp.h +4 -0
@@ -17,6 +17,9 @@#define XDP_COPY (1 << 1)/* Force copy-mode */ #define XDP_ZEROCOPY (1 << 2)/* Force zero-copy mode */ +/* Flags for xsk_umem_config flags */ +#define XDP_UMEM_UNALIGNED_CHUNKS (1 << 0) +struct sockaddr_xdp {__u16 sxdp_family;__u16 sxdp_flags;
@@ -52,6 +55,7 @@ struct xdp_umem_reg {__u64 len;/* Length of packet data area */ __u32 chunk_size;__u32 headroom; +__u32 flags; };struct xdp_statistics {
net/xdp/xdp_umem.c +11 -6
@@ -301,12 +301,14 @@ static int xdp_umem_account_pages(struct xdp_umem *umem)static int xdp_umem_reg (struct xdp_umem *umem,struct xdp_umem_reg *mr) { +bool unaligned_chunks = mr->flags &XDP_UMEM_UNALIGNED_CHUNKS ;u32 chunk_size = mr->chunk_size, headroom = mr->headroom;unsigned int chunks, chunks_per_page;u64 addr = mr->addr, size = mr->len;int size_chk, err, i; -if (chunk_size <XDP_UMEM_MIN_CHUNK_SIZE || chunk_size >PAGE_SIZE ) { +if ((!unaligned_chunks && chunk_size <XDP_UMEM_MIN_CHUNK_SIZE ) || + chunk_size >PAGE_SIZE ) {/* Strictly speaking we could support this, if: * - huge pages, or* * - using an IOMMU, or
@@ -316,7 +318,7 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)return -EINVAL ; } -if (!is_power_of_2 (chunk_size)) +if (!unaligned_chunks && !is_power_of_2 (chunk_size))return -EINVAL ;if (!PAGE_ALIGNED (addr)) {
@@ -333,9 +335,11 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)if (chunks ==0 )return -EINVAL ; - chunks_per_page =PAGE_SIZE / chunk_size; -if (chunks < chunks_per_page || chunks % chunks_per_page) -return -EINVAL ; +if (!unaligned_chunks) { + chunks_per_page =PAGE_SIZE / chunk_size; +if (chunks < chunks_per_page || chunks % chunks_per_page) +return -EINVAL ; + } headroom =ALIGN (headroom,64 );
@@ -344,13 +348,14 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)return -EINVAL ; umem->address = (unsigned long )addr; - umem->chunk_mask = ~((u64 )chunk_size -1 ); + umem->chunk_mask = unaligned_chunks ?U64_MAX : ~((u64 )chunk_size -1 ); umem->size = size; umem->headroom = headroom; umem->chunk_size_nohr = chunk_size - headroom; umem->npgs = size /PAGE_SIZE ; umem->pgs =NULL ; umem->user =NULL ; + umem->flags = mr->flags;INIT_LIST_HEAD (&umem->xsk_list);spin_lock_init (&umem->xsk_list_lock);
net/xdp/xsk.c +52 -8
114 lines not on this page · raw message
net/xdp/xsk_queue.h +52 -8
124 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 7/11 libbpf: add flags to umem config
In reply to Kevin LaatzThis patch adds a 'flags' field to the umem_config and umem_reg structs. This will allow for more options to be added for configuring umems.
The first use for the flags field is to add a flag for unaligned chunks mode. These flags can either be user-provided or filled with a default.
--- tools/include/uapi/linux/if_xdp.h | 4 ++++ tools/lib/bpf/xsk.c | 7 +++++++ tools/lib/bpf/xsk.h | 2 ++ 3 files changed, 13 insertions(+)
tools/include/uapi/linux/if_xdp.h +4 -0
@@ -17,6 +17,9 @@#define XDP_COPY (1 << 1)/* Force copy-mode */ #define XDP_ZEROCOPY (1 << 2)/* Force zero-copy mode */ +/* Flags for xsk_umem_config flags */ +#define XDP_UMEM_UNALIGNED_CHUNKS (1 << 0) +struct sockaddr_xdp {__u16 sxdp_family;__u16 sxdp_flags;
@@ -52,6 +55,7 @@ struct xdp_umem_reg {__u64 len;/* Length of packet data area */ __u32 chunk_size;__u32 headroom; +__u32 flags; };struct xdp_statistics {
tools/lib/bpf/xsk.c +7 -0
@@ -115,6 +115,7 @@ static void xsk_set_umem_config(struct xsk_umem_config *cfg, cfg->comp_size =XSK_RING_CONS__DEFAULT_NUM_DESCS ; cfg->frame_size =XSK_UMEM__DEFAULT_FRAME_SIZE ; cfg->frame_headroom =XSK_UMEM__DEFAULT_FRAME_HEADROOM ; + cfg->flags =XSK_UMEM__DEFAULT_FLAGS ;return ; }
@@ -122,6 +123,7 @@ static void xsk_set_umem_config(struct xsk_umem_config *cfg, cfg->comp_size = usr_cfg->comp_size; cfg->frame_size = usr_cfg->frame_size; cfg->frame_headroom = usr_cfg->frame_headroom; + cfg->flags = usr_cfg->flags; } static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
@@ -181,6 +183,11 @@ int xsk_umem__create(struct xsk_umem **umem_ptr, void *umem_area, __u64 size, mr.len = size; mr.chunk_size = umem->config.frame_size; mr.headroom = umem->config.frame_headroom; + mr.flags = umem->config.flags; + +/* Headroom must be 0 for unaligned chunks */ +if ((mr.flags &XDP_UMEM_UNALIGNED_CHUNKS ) && mr.headroom !=0 ) +return -EINVAL ; err =setsockopt (umem->fd,SOL_XDP ,XDP_UMEM_REG , &mr,sizeof (mr));if (err) {
tools/lib/bpf/xsk.h +2 -0
@@ -170,12 +170,14 @@ LIBBPF_API int xsk_socket__fd(const struct xsk_socket *xsk);#define XSK_UMEM__DEFAULT_FRAME_SHIFT 11/* 2048 bytes */ #define XSK_UMEM__DEFAULT_FRAME_SIZE (1 << XSK_UMEM__DEFAULT_FRAME_SHIFT)#define XSK_UMEM__DEFAULT_FRAME_HEADROOM 0 +#define XSK_UMEM__DEFAULT_FLAGS 0struct xsk_umem_config {__u32 fill_size;__u32 comp_size;__u32 frame_size;__u32 frame_headroom; +__u32 flags; };/* Flags for the libbpf_flags field. */
Signature · 1 line
2.17.1
PATCH 8/11 samples/bpf: add unaligned chunks mode support to xdpsock
In reply to Kevin LaatzThis patch adds support for the unaligned chunks mode. The addition of the unaligned chunks option will allow users to run the application with more relaxed chunk placement in the XDP umem.
Unaligned chunks mode can be used with the '-u' or '--unaligned' command line options.
--- samples/bpf/xdpsock_user.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
samples/bpf/xdpsock_user.c +18 -2
@@ -67,6 +67,8 @@ static int opt_ifindex;static int opt_queue;static int opt_poll;static int opt_interval =1 ; +static u32 opt_umem_flags; +static int opt_unaligned_chunks;static u32 opt_xdp_bind_flags;static __u32 prog_id;
@@ -276,14 +278,21 @@ static size_t gen_eth_frame(struct xsk_umem_info *umem, u64 addr)static struct xsk_umem_info *xsk_configure_umem (void *buffer,u64 size) {struct xsk_umem_info *umem; +struct xsk_umem_config umem_cfg;int ret; umem =calloc (1 ,sizeof (*umem));if (!umem)exit_with_error (errno); + umem_cfg.fill_size =XSK_RING_PROD__DEFAULT_NUM_DESCS ; + umem_cfg.comp_size =XSK_RING_CONS__DEFAULT_NUM_DESCS ; + umem_cfg.frame_size =XSK_UMEM__DEFAULT_FRAME_SIZE ; + umem_cfg.frame_headroom =XSK_UMEM__DEFAULT_FRAME_HEADROOM ; + umem_cfg.flags = opt_umem_flags; + ret =xsk_umem__create (&umem->umem, buffer, size, &umem->fq, &umem->cq, -NULL ); + &umem_cfg);if (ret)exit_with_error (-ret);
@@ -346,6 +355,7 @@ static struct option long_options[] = { {"interval", required_argument, 0, 'n'}, {"zero-copy", no_argument, 0, 'z'}, {"copy", no_argument, 0, 'c'}, + {"unaligned", no_argument, 0, 'u'}, {0, 0, 0, 0} };
@@ -365,6 +375,7 @@ static void usage(const char *prog)" -n, --interval=n Specify statistics update interval (default 1 sec).\n" " -z, --zero-copy Force zero-copy mode.\n" " -c, --copy Force copy mode.\n" +" -u, --unaligned Enable unaligned chunk placement\n" "\n" ;fprintf (stderr, str, prog);exit (EXIT_FAILURE );
@@ -377,7 +388,7 @@ static void parse_command_line(int argc, char **argv) opterr =0 ;for (;;) { - c =getopt_long (argc, argv,"Frtli:q:psSNn:cz" , long_options, + c =getopt_long (argc, argv,"Frtli:q:psSNn:czu" , long_options, &option_index);if (c ==-1 )break ;
@@ -417,9 +428,14 @@ static void parse_command_line(int argc, char **argv)case 'c' : opt_xdp_bind_flags |=XDP_COPY ;break ; +case 'u' : + opt_umem_flags |=XDP_UMEM_UNALIGNED_CHUNKS ; + opt_unaligned_chunks =1 ; +break ;case 'F' : opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST ;break ; +default :usage (basename (argv[0 ])); }
Signature · 1 line
2.17.1
PATCH 9/11 samples/bpf: add buffer recycling for unaligned chunks to xdpsock
In reply to Kevin LaatzThis patch adds buffer recycling support for unaligned buffers. Since we don't mask the addr to 2k at umem_teg in unaligned mode, we need to make sure we give back the correct, original addr to the fill queue. To do this, we need to mask the addr with the buffer size.
To pass in a buffer size, use the --buf-size=n argument.
NOTE: For xdpsock to work in aligned chunk mode, you still need to pass 'power of 2' buffer size.
--- samples/bpf/xdpsock_user.c | 71 +++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 20 deletions(-)
samples/bpf/xdpsock_user.c +51 -20
@@ -60,6 +60,10 @@ enum benchmark_type { BENCH_L2FWD = 2, }; +#define LENGTH (256UL*1024*1024) +#define ADDR (void *)(0x0UL) +#define SHMAT_FLAGS (0) +static enum benchmark_type opt_bench =BENCH_RXDROP ;static u32 opt_xdp_flags =XDP_FLAGS_UPDATE_IF_NOEXIST ;static const char *opt_if ="" ;
@@ -67,6 +71,7 @@ static int opt_ifindex;static int opt_queue;static int opt_poll;static int opt_interval =1 ; +static u64 opt_buffer_size =XSK_UMEM__DEFAULT_FRAME_SIZE ;static u32 opt_umem_flags;static int opt_unaligned_chunks;static u32 opt_xdp_bind_flags;
@@ -287,7 +292,7 @@ static struct xsk_umem_info *xsk_configure_umem(void *buffer, u64 size) umem_cfg.fill_size =XSK_RING_PROD__DEFAULT_NUM_DESCS ; umem_cfg.comp_size =XSK_RING_CONS__DEFAULT_NUM_DESCS ; - umem_cfg.frame_size =XSK_UMEM__DEFAULT_FRAME_SIZE ; + umem_cfg.frame_size = opt_buffer_size; umem_cfg.frame_headroom =XSK_UMEM__DEFAULT_FRAME_HEADROOM ; umem_cfg.flags = opt_umem_flags;
@@ -334,8 +339,8 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)exit_with_error (-ret);for (i =0 ; i <XSK_RING_PROD__DEFAULT_NUM_DESCS * -XSK_UMEM__DEFAULT_FRAME_SIZE ; - i +=XSK_UMEM__DEFAULT_FRAME_SIZE ) + opt_buffer_size; + i += opt_buffer_size) *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx++) = i;xsk_ring_prod__submit (&xsk->umem->fq,XSK_RING_PROD__DEFAULT_NUM_DESCS );
@@ -356,6 +361,7 @@ static struct option long_options[] = { {"zero-copy", no_argument, 0, 'z'}, {"copy", no_argument, 0, 'c'}, {"unaligned", no_argument, 0, 'u'}, + {"buf-size", required_argument, 0, 'b'}, {0, 0, 0, 0} };
@@ -376,6 +382,7 @@ static void usage(const char *prog)" -z, --zero-copy Force zero-copy mode.\n" " -c, --copy Force copy mode.\n" " -u, --unaligned Enable unaligned chunk placement\n" +" -b, --buf-size=n Specify the buffer size to use\n" "\n" ;fprintf (stderr, str, prog);exit (EXIT_FAILURE );
@@ -388,7 +395,7 @@ static void parse_command_line(int argc, char **argv) opterr =0 ;for (;;) { - c =getopt_long (argc, argv,"Frtli:q:psSNn:czu" , long_options, + c =getopt_long (argc, argv,"Frtli:q:psSNn:czub" , long_options, &option_index);if (c ==-1 )break ;
@@ -432,6 +439,9 @@ static void parse_command_line(int argc, char **argv) opt_umem_flags |=XDP_UMEM_UNALIGNED_CHUNKS ; opt_unaligned_chunks =1 ;break ; +case 'b' : + opt_buffer_size =atoi (optarg); +break ;case 'F' : opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST ;break ;
@@ -483,13 +493,22 @@ static inline void complete_tx_l2fwd(struct xsk_socket_info *xsk)while (ret != rcvd) {if (ret <0 )exit_with_error (-ret); - ret =xsk_ring_prod__reserve (&xsk->umem->fq, rcvd, - &idx_fq); + ret =xsk_ring_prod__reserve (&xsk->umem->fq, rcvd, &idx_fq); + } + +if (opt_umem_flags &XDP_UMEM_UNALIGNED_CHUNKS ) { +for (i =0 ; i < rcvd; i++) { +u64 comp_addr = *xsk_ring_cons__comp_addr (&xsk->umem->cq, + idx_cq++); +u64 masked_comp = (comp_addr & ~((u64 )opt_buffer_size-1 )); + *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx_fq++) = + masked_comp; + } + }else { +for (i =0 ; i < rcvd; i++) + *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx_fq++) = + *xsk_ring_cons__comp_addr (&xsk->umem->cq, idx_cq++); } -for (i =0 ; i < rcvd; i++) - *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx_fq++) = - *xsk_ring_cons__comp_addr (&xsk->umem->cq, - idx_cq++);xsk_ring_prod__submit (&xsk->umem->fq, rcvd);xsk_ring_cons__release (&xsk->umem->cq, rcvd);
@@ -533,13 +552,25 @@ static void rx_drop(struct xsk_socket_info *xsk) ret =xsk_ring_prod__reserve (&xsk->umem->fq, rcvd, &idx_fq); } -for (i =0 ; i < rcvd; i++) { -u64 addr =xsk_ring_cons__rx_desc (&xsk->rx, idx_rx)->addr; -u32 len =xsk_ring_cons__rx_desc (&xsk->rx, idx_rx++)->len; -char *pkt =xsk_umem__get_data (xsk->umem->buffer, addr); +if (opt_umem_flags &XDP_UMEM_UNALIGNED_CHUNKS ) { +for (i =0 ; i < rcvd; i++) { +u64 addr =xsk_ring_cons__rx_desc (&xsk->rx, idx_rx)->addr; +u32 len =xsk_ring_cons__rx_desc (&xsk->rx, idx_rx++)->len; +char *pkt =xsk_umem__get_data (xsk->umem->buffer, addr); +u64 masked_addr = (addr & ~((u64 )opt_buffer_size-1 )); + +hex_dump (pkt, len, addr); + *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx_fq++) = masked_addr; + } + }else { +for (i =0 ; i < rcvd; i++) { +u64 addr =xsk_ring_cons__rx_desc (&xsk->rx, idx_rx)->addr; +u32 len =xsk_ring_cons__rx_desc (&xsk->rx, idx_rx++)->len; +char *pkt =xsk_umem__get_data (xsk->umem->buffer, addr); -hex_dump (pkt, len, addr); - *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx_fq++) = addr; +hex_dump (pkt, len, addr); + *xsk_ring_prod__fill_addr (&xsk->umem->fq, idx_fq++) = addr; + } }xsk_ring_prod__submit (&xsk->umem->fq, rcvd);
@@ -677,20 +708,20 @@ int main(int argc, char **argv) } ret =posix_memalign (&bufs,getpagesize (),/* PAGE_SIZE aligned */ -NUM_FRAMES *XSK_UMEM__DEFAULT_FRAME_SIZE ); +NUM_FRAMES * opt_buffer_size);if (ret)exit_with_error (ret);/* Create sockets... */ umem =xsk_configure_umem (bufs, -NUM_FRAMES *XSK_UMEM__DEFAULT_FRAME_SIZE ); +NUM_FRAMES * opt_buffer_size); xsks[num_socks++] =xsk_configure_socket (umem);if (opt_bench ==BENCH_TXONLY ) {int i; -for (i =0 ; i <NUM_FRAMES *XSK_UMEM__DEFAULT_FRAME_SIZE ; - i +=XSK_UMEM__DEFAULT_FRAME_SIZE ) +for (i =0 ; i <NUM_FRAMES * opt_buffer_size; + i += opt_buffer_size) (void )gen_eth_frame (umem, i); }
Signature · 1 line
2.17.1
PATCH 10/11 samples/bpf: use hugepages in xdpsock app
In reply to Kevin LaatzThis patch modifies xdpsock to use mmap instead of posix_memalign. With this change, we can use hugepages when running the application in unaligned chunks mode. Using hugepages makes it more likely that we have physically contiguous memory, which supports the unaligned chunk mode better.
--- samples/bpf/xdpsock_user.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
samples/bpf/xdpsock_user.c +9 -5
@@ -74,6 +74,7 @@ static int opt_interval = 1;static u64 opt_buffer_size =XSK_UMEM__DEFAULT_FRAME_SIZE ;static u32 opt_umem_flags;static int opt_unaligned_chunks; +static int opt_mmap_flags;static u32 opt_xdp_bind_flags;static __u32 prog_id;
@@ -438,6 +439,7 @@ static void parse_command_line(int argc, char **argv)case 'u' : opt_umem_flags |=XDP_UMEM_UNALIGNED_CHUNKS ; opt_unaligned_chunks =1 ; + opt_mmap_flags =MAP_HUGETLB ;break ;case 'b' : opt_buffer_size =atoi (optarg);
@@ -707,11 +709,13 @@ int main(int argc, char **argv)exit (EXIT_FAILURE ); } - ret =posix_memalign (&bufs,getpagesize (),/* PAGE_SIZE aligned */ -NUM_FRAMES * opt_buffer_size); -if (ret) -exit_with_error (ret); - +/* Reserve memory for the umem. Use hugepages if unaligned chunk mode */ + bufs =mmap (NULL ,NUM_FRAMES * opt_buffer_size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS |opt_mmap_flags,-1 ,0 ); +if (bufs ==MAP_FAILED ) { +printf ("ERROR: mmap failed\n" ); +exit (EXIT_FAILURE ); + }/* Create sockets... */ umem =xsk_configure_umem (bufs,NUM_FRAMES * opt_buffer_size);
Signature · 1 line
2.17.1
PATCH 11/11 doc/af_xdp: include unaligned chunk case
In reply to Kevin LaatzThe addition of unaligned chunks mode, the documentation needs to be updated to indicate that the incoming addr to the fill ring will only be masked if the user application is run in the aligned chunk mode. This patch also adds a line to explicitly indicate that the incoming addr will not be masked if running the user application in the unaligned chunk mode.
--- Documentation/networking/af_xdp.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +6 -4
@@ -153,10 +153,12 @@ an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has Frames passed to the kernel are used for the ingress path (RX rings). -The user application produces UMEM addrs to this ring. Note that the -kernel will mask the incoming addr. E.g. for a chunk size of 2k, the -log2(2048) LSB of the addr will be masked off, meaning that 2048, 2050 -and 3000 refers to the same chunk. +The user application produces UMEM addrs to this ring. Note that, if +running the application with aligned chunk mode, the kernel will mask +the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of +the addr will be masked off, meaning that 2048, 2050 and 3000 refers +to the same chunk. If the user application is run in the unaligned +chunks mode, then the incoming addr will be left untouched. UMEM Completion Ring
Signature · 1 line
2.17.1
Quoted 8 lines · Kevin Laatz
This patchset adds the ability to use unaligned chunks in the XDP umem.
Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is
PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are supported.
Quoted 8 lines
The changes in this patchset removes these restrictions, allowing XDP to be more flexible in where it can place a chunk within a umem. By relaxing where the chunks can be placed, it allows us to use an arbitrary buffer size and place that wherever we have a free address in the umem. These changes add the ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as DPDK.
Thanks for working on this, Kevin and Ciara!
I have some minor comments on the series, but in general I think it's in good shape!
For some reason the series was submitted twice (at least on my side)?
Thanks,
Bj?rn
Quoted 91 lines
Structure of the patchset:
Patch 1:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in i40e. This change is required in order for the buffer recycling to work in the unaligned chunk mode.Patch 2:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in ixgbe. This change is required in order for the buffer recycling to work in the unaligned chunk mode.Patch 3:
- Adds an offset parameter tozero_copy_allocator. This change will enable us to calculate the original handle inzca_free. This will be required for unaligned chunk mode since we can't easily mask back to the original handle.Patch 4:
- Adds the offset parameter toi40e_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.Patch 5:
- Adds the offset parameter toixgbe_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.Patch 6:
- Add infrastructure for unaligned chunks. Since we are dealing with unaligned chunks that could potentially cross a physical page boundary, we add checks to keep track of that information. We can later use this information to correctly handle buffers that are placed at an address where they cross a page boundary.Patch 7:
- Add flags for umem configuration to libbpfPatch 8:
- Modify xdpsock application to add a command line option for unaligned chunksPatch 9:
- Addition of command line argument to pass in a desired buffer size and buffer recycling for unaligned mode. Passing in a buffer size will allow the application to use unaligned chunks with the unaligned chunk mode. Since we are now using unaligned chunks, we need to recycle our buffers in a slightly different way.Patch 10:
- Adds hugepage support to the xdpsock applicationPatch 11:
- Documentation update to include the unaligned chunk scenario. We need to explicitly state that the incoming addresses are only masked in the aligned chunk mode and not the unaligned chunk mode.Kevin Laatz (11):
i40e: simplify Rx buffer recycle
ixgbe: simplify Rx buffer recycle
xdp: add offset param tozero_copy_allocator
i40e: add offset tozca_free
ixgbe: add offset tozca_free
xsk: add support to allow unaligned chunk placement
libbpf: add flags to umem config
samples/bpf: add unaligned chunks mode support to xdpsocksamples/bpf: add buffer recycling for unaligned chunks to xdpsocksamples/bpf: use hugepages in xdpsock app
doc/af_xdp: include unaligned chunk caseDocumentation/networking/af_xdp.rst | 10 +- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 21 ++-- drivers/net/ethernet/intel/i40e/i40e_xsk.h | 3 +- .../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 3 +- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 21 ++-- include/net/xdp.h | 3 +- include/net/xdp_sock.h | 2 + include/uapi/linux/if_xdp.h | 4 + net/core/xdp.c | 11 ++- net/xdp/xdp_umem.c | 17 ++-- net/xdp/xsk.c | 60 +++++++++-- net/xdp/xsk_queue.h | 60 +++++++++-- samples/bpf/xdpsock_user.c | 99 ++++++++++++++----- tools/include/uapi/linux/if_xdp.h | 4 + tools/lib/bpf/xsk.c | 7 ++ tools/lib/bpf/xsk.h | 2 + 16 files changed, 241 insertions(+), 86 deletions(-)--
2.17.1
Quoted 24 lines · Bj?rn T?pel
Quoted 9 lines · Kevin Laatz
This patchset adds the ability to use unaligned chunks in the XDP umem.
Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is
PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are
supported.Quoted 13 lines
The changes in this patchset removes these restrictions, allowing XDP to be
more flexible in where it can place a chunk within a umem. By relaxing where
the chunks can be placed, it allows us to use an arbitrary buffer size and
place that wherever we have a free address in the umem. These changes add the
ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as
DPDK.
Quoted 6 lines
Thanks for working on this, Kevin and Ciara!
I have some minor comments on the series, but in general I think it's in good shape!
For some reason the series was submitted twice (at least on my side)?
Apologies for the confusion... The first set had a typo in the bpf mailing list address (.com vs .org). Will fix for the v2.
Quoted 4 lines
Thanks,
Bj?rn
Thanks for reviewing. Will address your comments in the v2.
Quoted 102 lines
Quoted 101 lines
Structure of the patchset:
Patch 1:
? - Remove unnecessary masking and headroom addition during zero-copy Rx ??? buffer recycling in i40e. This change is required in order for the ??? buffer recycling to work in the unaligned chunk mode.Patch 2:
? - Remove unnecessary masking and headroom addition during ??? zero-copy Rx buffer recycling in ixgbe. This change is required in ??? order for the? buffer recycling to work in the unaligned chunk mode.Patch 3:
? - Adds an offset parameter tozero_copy_allocator. This change will ??? enable us to calculate the original handle inzca_free. This will be ??? required for unaligned chunk mode since we can't easily mask back to ??? the original handle.Patch 4:
? - Adds the offset parameter toi40e_zca_free. This change is needed for
??? calculating the handle since we can't easily mask back to the original
??? handle like we can in the aligned case.Patch 5:
? - Adds the offset parameter toixgbe_zca_free. This change is needed for
??? calculating the handle since we can't easily mask back to the original
??? handle like we can in the aligned case.Patch 6:
? - Add infrastructure for unaligned chunks. Since we are dealing ??? with unaligned chunks that could potentially cross a physical page ??? boundary, we add checks to keep track of that information. We can ??? later use this information to correctly handle buffers that are ??? placed at an address where they cross a page boundary.Patch 7:
? - Add flags for umem configuration to libbpfPatch 8:
? - Modify xdpsock application to add a command line option for ??? unaligned chunksPatch 9:
? - Addition of command line argument to pass in a desired buffer size ??? and buffer recycling for unaligned mode. Passing in a buffer size will
??? allow the application to use unaligned chunks with the unaligned chunk
??? mode. Since we are now using unaligned chunks, we need to recycle our
??? buffers in a slightly different way.Patch 10:
? - Adds hugepage support to the xdpsock applicationPatch 11:
? - Documentation update to include the unaligned chunk scenario. We need
??? to explicitly state that the incoming addresses are only masked in the
??? aligned chunk mode and not the unaligned chunk mode.Kevin Laatz (11):
? i40e: simplify Rx buffer recycle
? ixgbe: simplify Rx buffer recycle
? xdp: add offset param tozero_copy_allocator
? i40e: add offset tozca_free
? ixgbe: add offset tozca_free
? xsk: add support to allow unaligned chunk placement
? libbpf: add flags to umem config
?samples/bpf: add unaligned chunks mode support to xdpsock ?samples/bpf: add buffer recycling for unaligned chunks to xdpsock ?samples/bpf: use hugepages in xdpsock app
? doc/af_xdp: include unaligned chunk case?Documentation/networking/af_xdp.rst?????????? | 10 +- ?drivers/net/ethernet/intel/i40e/i40e_xsk.c??? | 21 ++-- ?drivers/net/ethernet/intel/i40e/i40e_xsk.h??? |? 3 +- ?.../ethernet/intel/ixgbe/ixgbe_txrx_common.h? |? 3 +- ?drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c? | 21 ++-- ?include/net/xdp.h???????????????????????????? |? 3 +- ?include/net/xdp_sock.h??????????????????????? |? 2 + ?include/uapi/linux/if_xdp.h?????????????????? |? 4 + ?net/core/xdp.c??????????????????????????????? | 11 ++- ?net/xdp/xdp_umem.c??????????????????????????? | 17 ++-- ?net/xdp/xsk.c???????????????????????????????? | 60 +++++++++-- ?net/xdp/xsk_queue.h?????????????????????????? | 60 +++++++++-- ?samples/bpf/xdpsock_user.c??????????????????? | 99 ++++++++++++++----- ?tools/include/uapi/linux/if_xdp.h???????????? |? 4 + ?tools/lib/bpf/xsk.c?????????????????????????? |? 7 ++ ?tools/lib/bpf/xsk.h?????????????????????????? |? 2 +?16 files changed, 241 insertions(+), 86 deletions(-)
Signature · 1 line
2.17.1
Quoted 11 lines · Kevin Laatz
This patchset adds the ability to use unaligned chunks in the XDP umem.
Currently, all chunk addresses passed to the umem are masked to be chunk
size aligned (default is 2k, max isPAGE_SIZE). This limits where we can
place chunks within the umem as well as limiting the packet sizes that are
supported.
Quoted 13 lines
The changes in this patchset removes these restrictions, allowing XDP to be
more flexible in where it can place a chunk within a umem. By relaxing where
the chunks can be placed, it allows us to use an arbitrary buffer size and
place that wherever we have a free address in the umem. These changes add the
ability to support jumboframes and make it easy to integrate with other
existing frameworks that have their own memory management systems, such as
DPDK.
I'm a little unclear on how this should work, and have a few issues here:
1) There isn't any support for the user defined umem->headroom
2) When queuing RX buffers, the handle (aka umem offset) is used,
which
points to the start of the buffer area. When the buffer appears in the completion queue, handle points to the start of the received
data,
which might be different from the buffer start address.
Normally, this RX address is just put back in the fill queue, and
the
mask is used to find the buffer start address again. This no
longer
works, so my question is, how is the buffer start address
recomputed
from the actual data payload address?
Same with TX - if the TX payload isn't aligned in with the start of the buffer, what happens?
3) This appears limited to crossing a single page boundary, but there is no constraint check on chunk_size.
Signature · 1 line
Jonathan
Quoted 105 lines
Structure of the patchset:
Patch 1:
- Remove unnecessary masking and headroom addition during zero-copy
Rx
buffer recycling in i40e. This change is required in order for the buffer recycling to work in the unaligned chunk mode.Patch 2:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in ixgbe. This change is required in order for the buffer recycling to work in the unaligned chunk
mode.Patch 3:
- Adds an offset parameter tozero_copy_allocator. This change will enable us to calculate the original handle inzca_free. This will
be
required for unaligned chunk mode since we can't easily mask back
to
the original handle.Patch 4:
- Adds the offset parameter toi40e_zca_free. This change is needed
for
calculating the handle since we can't easily mask back to the
original
handle like we can in the aligned case.Patch 5:
- Adds the offset parameter toixgbe_zca_free. This change is needed
for
calculating the handle since we can't easily mask back to the
original
handle like we can in the aligned case.Patch 6:
- Add infrastructure for unaligned chunks. Since we are dealing with unaligned chunks that could potentially cross a physical page boundary, we add checks to keep track of that information. We can later use this information to correctly handle buffers that are placed at an address where they cross a page boundary.Patch 7:
- Add flags for umem configuration to libbpfPatch 8:
- Modify xdpsock application to add a command line option for unaligned chunksPatch 9:
- Addition of command line argument to pass in a desired buffer size and buffer recycling for unaligned mode. Passing in a buffer size
will
allow the application to use unaligned chunks with the unaligned
chunk
mode. Since we are now using unaligned chunks, we need to recycle
our
buffers in a slightly different way.Patch 10:
- Adds hugepage support to the xdpsock applicationPatch 11:
- Documentation update to include the unaligned chunk scenario. We
need
to explicitly state that the incoming addresses are only masked in
the
aligned chunk mode and not the unaligned chunk mode.Kevin Laatz (11):
i40e: simplify Rx buffer recycle
ixgbe: simplify Rx buffer recycle
xdp: add offset param tozero_copy_allocator
i40e: add offset tozca_free
ixgbe: add offset tozca_free
xsk: add support to allow unaligned chunk placement
libbpf: add flags to umem config
samples/bpf: add unaligned chunks mode support to xdpsocksamples/bpf: add buffer recycling for unaligned chunks to xdpsocksamples/bpf: use hugepages in xdpsock app
doc/af_xdp: include unaligned chunk caseDocumentation/networking/af_xdp.rst | 10 +- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 21 ++-- drivers/net/ethernet/intel/i40e/i40e_xsk.h | 3 +- .../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 3 +- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 21 ++-- include/net/xdp.h | 3 +- include/net/xdp_sock.h | 2 + include/uapi/linux/if_xdp.h | 4 + net/core/xdp.c | 11 ++- net/xdp/xdp_umem.c | 17 ++-- net/xdp/xsk.c | 60 +++++++++-- net/xdp/xsk_queue.h | 60 +++++++++-- samples/bpf/xdpsock_user.c | 99 ++++++++++++++----- tools/include/uapi/linux/if_xdp.h | 4 + tools/lib/bpf/xsk.c | 7 ++ tools/lib/bpf/xsk.h | 2 + 16 files changed, 241 insertions(+), 86 deletions(-)Signature · 1 line
2.17.1
Quoted 23 lines · Jonathan Lemon
Quoted 8 lines · Kevin Laatz
This patchset adds the ability to use unaligned chunks in the XDP umem.
Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is
PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are
supported.Quoted 12 lines
The changes in this patchset removes these restrictions, allowing XDP to be
more flexible in where it can place a chunk within a umem. By relaxing where
the chunks can be placed, it allows us to use an arbitrary buffer size and
place that wherever we have a free address in the umem. These changes add the
ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as
DPDK.
Quoted 4 lines
I'm a little unclear on how this should work, and have a few issues here:
?1) There isn't any support for the user defined
umem->headroom
For the unaligned chunks case, it does not make sense to to support a user defined headroom since the user can point directly to where they want the data to start via the buffer address. Therefore, for unaligned chunks, the user defined headroom should always be 0 (aka the user did not define a headroom and the default value of 0 is used). Any other value will be caught and we return an invalid argument error.
Quoted 6 lines
?2) When queuing RX buffers, the handle (aka umem offset) is used, which ??? points to the start of the buffer area.? When the buffer appears in ??? the completion queue, handle points to the start of the received data,
??? which might be different from the buffer start address.
Quoted 7 lines
??? Normally, this RX address is just put back in the fill queue, and the ??? mask is used to find the buffer start address again.? This no longer ??? works, so my question is, how is the buffer start address recomputed ??? from the actual data payload address?
??? Same with TX - if the TX payload isn't aligned in with the start of ??? the buffer, what happens?
On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default scenario). This was missing from the v1, will add this in the v2, to have xdpsock use the default value from libbpf! If the user is using another BPF program that uses a different offset, then the computation will need to be adjusted for that accordingly. In v2 we'll add support for this via command-line parameter.
However, we are also working on an "in-order" patchset, hopefully to be published soon, to guarantee the buffers returned to the application are in the same order as those provided to the kernel. Longer term, this is the best solution here as it allows the application to track itself, via a "shadow ring" or otherwise, the buffers sent to the kernel and any metadata associated with them, such as the start of buffer address.
Quoted 3 lines
?3) This appears limited to crossing a single page boundary, but there ??? is no constraint check on
chunk_size.
There is an existing check for chunk_size during xdp_umem_reg (in xdp_umem.c) The check makes sure that chunk size is at least XDP_UMEM_MIN_CHUNK_SIZE and at most PAGE_SIZE. Since the max is page size, we only need to check the immediate next page for contiguity. While this patchset allows a max of 4k sized buffers, it is still an improvement from the current state. Future enhancements could look into extending the 4k limit but for now it is a good first step towards supporting hugepages efficiently.
Best regards,
Kevin
Quoted 4 lines · Kevin
On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the
XDP_PACKET_HEADROOMwhen computing the original address (in the default scenario).
That assumes specific layout for the data inside the buffer. Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset XDP_PACKET_HEADROOM + metadata len..
I think that's very limiting. What is the challenge in providing aligned addresses, exactly?
Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · Kevin
On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the
XDP_PACKET_HEADROOMwhen computing the original address (in the default scenario).That assumes specific layout for the data inside the buffer. Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset
XDP_PACKET_HEADROOM+ metadata len..
Yes, if NICs prepend extra data to the packet that would be a problem for using this feature in isolation. However, if we also add in support for in-order
RX and TX rings, that would no longer be an issue. However, even for NICs which do prepend data, this patchset should not break anything that is currently
working.
Quoted 3 lines
I think that's very limiting. What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment. ??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the
actual writable space specified in units of 1k - therefore we need 2k + metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting
the same memory channels when accessing memory.
??? ??? - it allows the application to choose the actual buffer size it
wants to use.
??? We make use of the above to allow us to speed up processing
significantly and also reduce the packet buffer memory size.
??? Not having arbitrary buffer alignment also means an AF_XDP driver
for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Quoted 10 lines · Kevin
Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · Kevin
On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the
XDP_PACKET_HEADROOMwhen computing the original address (in the default scenario).That assumes specific layout for the data inside the buffer.? Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset
XDP_PACKET_HEADROOM+ metadata len..
Quoted 7 lines
Yes, if NICs prepend extra data to the packet that would be a problem for using this feature in isolation. However, if we also add in support for in-order
RX and TX rings, that would no longer be an issue. However, even for NICs which do prepend data, this patchset should not break anything that is currently
working.
(Late on the ball. I'm in vacation mode.)
In your example Jakub, how would this look in XDP? Wouldn't the timestamp be part of the metadata (xdp_md.data_meta)? Isn't data-data_meta (if valid) <= XDP_PACKET_HEADROOM? That was my assumption.
There were some discussion on having meta data length in the struct xdp_desc, before AF_XDP was merged, but the conclusion was that this was *not* needed, because AF_XDP and the XDP program had an implicit contract. If you're running AF_XDP, you also have an XDP program running and you can determine the meta data length (and also getting back the original buffer).
So, today in AF_XDP if XDP metadata is added, the userland application can look it up before the xdp_desc.addr (just like regular XDP), and how the XDP/AF_XDP application determines length/layout of the metadata i out-of-band/not specified.
This is a bit messy/handwavy TBH, so maybe adding the length to the descriptor *is* a good idea (extending the options part of the xdp_desc)? Less clean though. OTOH the layout of the meta data still need to be determined.
Bj?rn
Quoted 3 lines · Bj?rn T?pel
In your example Jakub, how would this look in XDP? Wouldn't the timestamp be part of the metadata (xdp_md.data_meta)? Isn't data-data_meta (if valid) <=
XDP_PACKET_HEADROOM? That was my assumption.
The driver parses the metadata and copies it outside of the prepend before XDP runs. Then XDP runs unaware of the prepend contents. That's the current situation.
XDP_PACKET_HEADROOM is before the entire frame. Like this:
buffer start
/ DMA addr given to the device
/ /
v v
| XDP_HEADROOM | meta data | packet data |
Length of meta data comes in the standard fixed size descriptor.
The metadata prepend is in TV form ("TLV with no length field", length's implied by type).
Quoted 12 lines
There were some discussion on having meta data length in the struct
xdp_desc, beforeAF_XDPwas merged, but the conclusion was that this was *not* needed, becauseAF_XDPand the XDP program had an implicit contract. If you're runningAF_XDP, you also have an XDP program running and you can determine the meta data length (and also getting back the original buffer).So, today in
AF_XDPif XDP metadata is added, the userland application can look it up before the xdp_desc.addr (just like regular XDP), and how the XDP/AF_XDP application determines length/layout of the metadata i out-of-band/not specified.
Quoted 4 lines
This is a bit messy/handwavy TBH, so maybe adding the length to the descriptor *is* a good idea (extending the options part of the
xdp_desc)? Less clean though. OTOH the layout of the meta data still need to be determined.
Right, the device prepend is not exposed as metadata to XDP.
Quoted 6 lines · Kevin
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · Kevin
On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the
XDP_PACKET_HEADROOMwhen computing the original address (in the default scenario).
Quoted 7 lines
Quoted 3 lines
That assumes specific layout for the data inside the buffer. Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset
XDP_PACKET_HEADROOM+ metadata len..Yes, if NICs prepend extra data to the packet that would be a problem for using this feature in isolation. However, if we also add in support for in-order RX and TX rings, that would no longer be an issue.
Can you shed more light on in-order rings? Do you mean that RX frames come in order buffers were placed in the fill queue? That wouldn't make practical sense, no? Even if the application does no reordering there is also XDP_DROP and XDP_TX. Please explain :)
Quoted 2 lines
However, even for NICs which do prepend data, this patchset should not break anything that is currently working.
My understanding from the beginnings of AF_XDP was that we were searching for a format flexible enough to support most if not all NICs. Creating an ABI which will preclude vendors from supporting DPDK via AF_XDP would seriously undermine the neutrality aspect.
Quoted 5 lines
Quoted 2 lines
I think that's very limiting. What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
Presumably support for jumbos would require a multi-buffer setup, and therefore extensions to the ring format. Should we perhaps look into implementing unaligned chunks by extending ring format as well?
Quoted 18 lines
2) higher level user-space frameworks which may want to use
AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment. ??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the
actual writable space specified in units of 1k - therefore we need 2k + metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting
the same memory channels when accessing memory.
??? ??? - it allows the application to choose the actual buffer size it
wants to use.
??? We make use of the above to allow us to speed up processing
significantly and also reduce the packet buffer memory size.??? Not having arbitrary buffer alignment also means an
AF_XDPdriver
for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Quoted 13 lines · Kevin
Quoted 11 lines · Jakub Kicinski
Quoted 6 lines · Kevin
On the application side (xdpsock), we don't have to worry about the user
defined headroom, since it is 0, so we only need to account for theXDP_PACKET_HEADROOMwhen computing the original address (in the default
scenario).That assumes specific layout for the data inside the buffer. Some NICs
will prepend information like timestamp to the packet, meaning the packet would start at offsetXDP_PACKET_HEADROOM+ metadata len..
Quoted 9 lines
Yes, if NICs prepend extra data to the packet that would be a problem for
using this feature in isolation. However, if we also add in support for in-order
RX and TX rings, that would no longer be an issue. However, even for NICs
which do prepend data, this patchset should not break anything that is currently
working.
I read this as "the correct buffer address is recovered from the shadow ring".
I'm not sure I'm comfortable with that, and I'm also not sold on in-order completion
for the RX/TX rings.
Quoted 19 lines
Quoted 2 lines
I think that's very limiting. What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to useAF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k + metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory.
??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.
Quoted 5 lines
??? Not having arbitrary buffer alignment also means an
AF_XDPdriver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.
Signature · 1 line
Jonathan
Laatz
cover XDP unaligned chunk placement support
Quoted 27 lines · Jonathan Lemon
Quoted 20 lines · Kevin
Quoted 2 lines · Jakub Kicinski
I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to useAF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.Quoted 5 lines
??? Not having arbitrary buffer alignment also means an
AF_XDPdriver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Quoted 5 lines
Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.
Yes, we can do this.
Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back to the original address using subtraction.
Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approach are obviously possible too.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20190701/5a471c67/attachment.html>
Quoted 28 lines · Kevin
Quoted 27 lines · Jonathan Lemon
Quoted 25 lines · Kevin
Quoted 2 lines · Jakub Kicinski
I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to useAF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.??? Not having arbitrary buffer alignment also means an
AF_XDPdriver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Quoted 7 lines
Quoted 5 lines
Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.Yes, we can do this.
That'd be best, can DPDK reasonably guarantee the slicing is uniform? E.g. it's not desperate buffer pools with different bases?
Quoted 4 lines
Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back to the original address using subtraction.
Unfortunately the metadata depends on the packet and how much info the device was able to extract. So it's variable length.
Quoted 10 lines
Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approach are obviously possible too.
Seems reasonable to me..
Quoted 43 lines
-----Original Message-----
From: Jakub Kicinski [mailto:jakub.kicinski at netronome.com]
Sent: Monday, July 1, 2019 10:20 PM
To: Laatz, Kevin <[email protected]>
Cc: Jonathan Lemon <[email protected]>; netdev at vger.kernel.org; ast at kernel.org; daniel at iogearbox.net; Topel, Bjorn <[email protected]>; Karlsson, Magnus <[email protected]>; bpf at vger.kernel.org; intel-wired-lan at lists.osuosl.org; Richardson, Bruce <[email protected]>; Loftus, Ciara <[email protected]>
Subject: Re: [PATCH 00/11] XDP unaligned chunk placement supportQuoted 29 lines · Kevin
Quoted 28 lines · Jonathan Lemon
Quoted 26 lines · Kevin
Quoted 2 lines · Jakub Kicinski
I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to useAF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.??? Not having arbitrary buffer alignment also means an
AF_XDPdriver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Since all buffers in the umem are the same chunk size, the original buffer address can be recalculated with some multiply/shift math. However, this is more expensive than just a mask operation.
Yes, we can do this.
That'd be best, can DPDK reasonably guarantee the slicing is uniform? E.g. it's not desperate buffer pools with different bases?
It's generally uniform, but handling the crossing of (huge)page boundaries complicates things a bit. Therefore I think the final option below is best as it avoids any such problems.
Quoted 20 lines
Quoted 5 lines
Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back to the original address using subtraction.
Unfortunately the metadata depends on the packet and how much info the device was able to extract. So it's variable length.
Quoted 10 lines
Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approach are obviously possible too.
Quoted 1 line
Seems reasonable to me..
I think this is probably the best solution, and also has the advantage that a buffer retains its base address the full way through the cycle of Rx and Tx.
Quoted 92 lines · Bruce
Quoted 51 lines
-----Original Message-----
From: Jakub Kicinski [mailto:jakub.kicinski at netronome.com]
Sent: Monday, July 1, 2019 10:20 PM
To: Laatz, Kevin <[email protected]>
Cc: Jonathan Lemon <[email protected]>;
netdev at vger.kernel.org;
ast at kernel.org; daniel at iogearbox.net; Topel, Bjorn <[email protected]>; Karlsson, Magnus <[email protected]>;
bpf at vger.kernel.org; intel-wired-lan at lists.osuosl.org; Richardson, Bruce
<[email protected]>; Loftus, Ciara <[email protected]>
Subject: Re: [PATCH 00/11] XDP unaligned chunk placement supportQuoted 36 lines · Kevin
Quoted 35 lines · Jonathan Lemon
Quoted 33 lines · Kevin
Quoted 3 lines · Jakub Kicinski
I think that's very limiting.? What is the challenge in providing
aligned addresses, exactly?The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue
supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to useAF_XDP,
such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need
the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly
hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer
size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.??? Not having arbitrary buffer alignment also means an
AF_XDPdriver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an
arbitrary buffer alignment, existing apps will need to be modified to use that new capability.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Since all buffers in the umem are the same chunk size, the original buffer address can be recalculated with some multiply/shift math. However, this is more expensive than just a mask operation.
Yes, we can do this.
That'd be best, can DPDK reasonably guarantee the slicing is uniform? E.g. it's not desperate buffer pools with different bases?
It's generally uniform, but handling the crossing of (huge)page boundaries
complicates things a bit. Therefore I think the final option below is best as it avoids any such problems.Quoted 27 lines
Quoted 7 lines
Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet).
We can use that information to get back to the original address using
subtraction.Unfortunately the metadata depends on the packet and how much info the
device was able to extract. So it's variable length.Quoted 14 lines
Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead
return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an
alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits
of the address as a packet offset. Other variations on these approach
are obviously possible too.
Quoted 6 lines
Quoted 1 line
Seems reasonable to me..
I think this is probably the best solution, and also has the advantage that
a buffer retains its base address the full way through the cycle of Rx and Tx.
I like this as well - it also has the advantage that drivers can keep performing adjustments on the handle, which ends up just modifying the offset.
Signature · 1 line
Jonathan
Quoted 27 lines · Jonathan Lemon
Quoted 20 lines · Kevin
Quoted 2 lines · Jakub Kicinski
I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?
The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to useAF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.Quoted 5 lines
??? Not having arbitrary buffer alignment also means an
AF_XDPdriver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.
Quoted 5 lines
Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.
Yes, we can do this.
Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back the original address using subtraction.
Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address at the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of the packet offset with that buffer. Id using 16-bits of descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approaches are obviously possible too.