Skip to contentSkip to conversation

XDP unaligned chunk placement support

PATCH In review Kevin Laatz 7 participants 27 messages Jun 20, 2019 – Jul 2, 2019

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover XDP unaligned chunk placement support

This patchset adds the ability to use unaligned chunks in the XDP umem.

Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are supported.

The changes in this patchset removes these restrictions, allowing XDP to be more flexible in where it can place a chunk within a umem. By relaxing where the chunks can be placed, it allows us to use an arbitrary buffer size and place that wherever we have a free address in the umem. These changes add the ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as DPDK.

Structure of the patchset:
Patch 1:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in i40e. This change is required in order for the buffer recycling to work in the unaligned chunk mode.

Patch 2:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in ixgbe. This change is required in order for the buffer recycling to work in the unaligned chunk mode.

Patch 3:
- Adds an offset parameter to zero_copy_allocator. This change will enable us to calculate the original handle in zca_free. This will be required for unaligned chunk mode since we can't easily mask back to the original handle.

Patch 4:
- Adds the offset parameter to i40e_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.

Patch 5:
- Adds the offset parameter to ixgbe_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.

Patch 6:
- Add infrastructure for unaligned chunks. Since we are dealing with unaligned chunks that could potentially cross a physical page boundary, we add checks to keep track of that information. We can later use this information to correctly handle buffers that are placed at an address where they cross a page boundary.

Patch 7:
- Add flags for umem configuration to libbpf

Patch 8:
- Modify xdpsock application to add a command line option for unaligned chunks

Patch 9:
- Addition of command line argument to pass in a desired buffer size and buffer recycling for unaligned mode. Passing in a buffer size will allow the application to use unaligned chunks with the unaligned chunk mode. Since we are now using unaligned chunks, we need to recycle our buffers in a slightly different way.

Patch 10:
- Adds hugepage support to the xdpsock application

Patch 11:
- Documentation update to include the unaligned chunk scenario. We need to explicitly state that the incoming addresses are only masked in the aligned chunk mode and not the unaligned chunk mode.

Kevin Laatz (11):
i40e: simplify Rx buffer recycle
ixgbe: simplify Rx buffer recycle
xdp: add offset param to zero_copy_allocator
i40e: add offset to zca_free
ixgbe: add offset to zca_free
xsk: add support to allow unaligned chunk placement
libbpf: add flags to umem config
samples/bpf: add unaligned chunks mode support to xdpsock samples/bpf: add buffer recycling for unaligned chunks to xdpsock samples/bpf: use hugepages in xdpsock app
doc/af_xdp: include unaligned chunk case

 Documentation/networking/af_xdp.rst           | 10 +-
 drivers/net/ethernet/intel/i40e/i40e_xsk.c    | 21 ++--
 drivers/net/ethernet/intel/i40e/i40e_xsk.h    |  3 +-
 .../ethernet/intel/ixgbe/ixgbe_txrx_common.h  |  3 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c  | 21 ++--
 include/net/xdp.h                             |  3 +-
 include/net/xdp_sock.h                        |  2 +
 include/uapi/linux/if_xdp.h                   |  4 +
 net/core/xdp.c                                | 11 ++-
 net/xdp/xdp_umem.c                            | 17 ++--
 net/xdp/xsk.c                                 | 60 +++++++++--
 net/xdp/xsk_queue.h                           | 60 +++++++++--
 samples/bpf/xdpsock_user.c                    | 99 ++++++++++++++-----
 tools/include/uapi/linux/if_xdp.h             |  4 +
 tools/lib/bpf/xsk.c                           |  7 ++
 tools/lib/bpf/xsk.h                           |  2 +
 16 files changed, 241 insertions(+), 86 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/11 i40e: simplify Rx buffer recycle

Currently, the dma, addr and handle are modified when we reuse Rx buffers in zero-copy mode. However, this is not required as the inputs to the function are copies, not the original values themselves. As we use the copies within the function, we can use the original 'old_bi' values directly without having to mask and add the headroom.

---
 drivers/net/ethernet/intel/i40e/i40e_xsk.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_xsk.c +3 -10
index 1b17486543ac..c89e692e8663 100644
@@ -419,8 +419,6 @@ static void i40e_reuse_rx_buffer_zc(struct i40e_ring *rx_ring,
 				    struct i40e_rx_buffer *old_bi)
 {
 	struct i40e_rx_buffer *new_bi = &rx_ring->rx_bi[rx_ring->next_to_alloc];
-	unsigned long mask = (unsigned long)rx_ring->xsk_umem->chunk_mask;
-	u64 hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;
 	u16 nta = rx_ring->next_to_alloc;
 
 	/* update, and store next to alloc */
@@ -428,14 +426,9 @@ static void i40e_reuse_rx_buffer_zc(struct i40e_ring *rx_ring,
 	rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
 
 	/* transfer page from old buffer to new buffer */
-	new_bi->dma = old_bi->dma & mask;
-	new_bi->dma += hr;
-
-	new_bi->addr = (void *)((unsigned long)old_bi->addr & mask);
-	new_bi->addr += hr;
-
-	new_bi->handle = old_bi->handle & mask;
-	new_bi->handle += rx_ring->xsk_umem->headroom;
+	new_bi->dma = old_bi->dma;
+	new_bi->addr = old_bi->addr;
+	new_bi->handle = old_bi->handle;
 
 	old_bi->addr = NULL;
 }
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/11 ixgbe: simplify Rx buffer recycle

In reply to Kevin Laatz

Currently, the dma, addr and handle are modified when we reuse Rx buffers in zero-copy mode. However, this is not required as the inputs to the function are copies, not the original values themselves. As we use the copies within the function, we can use the original 'obi' values directly without having to mask and add the headroom.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +3 -10
index bfe95ce0bd7f..49536adafe8e 100644
@@ -251,8 +251,6 @@ ixgbe_rx_buffer *ixgbe_get_rx_buffer_zc(struct ixgbe_ring *rx_ring,
 static void ixgbe_reuse_rx_buffer_zc(struct ixgbe_ring *rx_ring,
 				     struct ixgbe_rx_buffer *obi)
 {
-	unsigned long mask = (unsigned long)rx_ring->xsk_umem->chunk_mask;
-	u64 hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;
 	u16 nta = rx_ring->next_to_alloc;
 	struct ixgbe_rx_buffer *nbi;
 
@@ -262,14 +260,9 @@ static void ixgbe_reuse_rx_buffer_zc(struct ixgbe_ring *rx_ring,
 	rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
 
 	/* transfer page from old buffer to new buffer */
-	nbi->dma = obi->dma & mask;
-	nbi->dma += hr;
-
-	nbi->addr = (void *)((unsigned long)obi->addr & mask);
-	nbi->addr += hr;
-
-	nbi->handle = obi->handle & mask;
-	nbi->handle += rx_ring->xsk_umem->headroom;
+	nbi->dma = obi->dma;
+	nbi->addr = obi->addr;
+	nbi->handle = obi->handle;
 
 	obi->addr = NULL;
 	obi->skb = NULL;
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/11 xdp: add offset param to zero_copy_allocator

In reply to Kevin Laatz

This patch adds an offset parameter for zero_copy_allocator.

This change is required for the unaligned chunk mode which will come later in this patch set. The offset parameter is required for calculating the original handle in unaligned mode since we can't easily mask back to it like in the aligned case.

---
 include/net/xdp.h |  3 ++-
 net/core/xdp.c    | 11 ++++++-----
 2 files changed, 8 insertions(+), 6 deletions(-)
include/net/xdp.h +2 -1
index 0f25b3675c5c..ea801fd2bf98 100644
@@ -53,7 +53,8 @@ struct xdp_mem_info {
 struct page_pool;
 
 struct zero_copy_allocator {
-	void (*free)(struct zero_copy_allocator *zca, unsigned long handle);
+	void (*free)(struct zero_copy_allocator *zca, unsigned long handle,
+			off_t off);
 };
 
 struct xdp_rxq_info {
net/core/xdp.c +6 -5
index 4b2b194f4f1f..a77a7162d213 100644
@@ -322,7 +322,7 @@ EXPORT_SYMBOL_GPL(xdp_rxq_info_reg_mem_model);
  * of xdp_frames/pages in those cases.
  */
 static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
-			 unsigned long handle)
+			 unsigned long handle, off_t off)
 {
 	struct xdp_mem_allocator *xa;
 	struct page *page;
@@ -353,7 +353,7 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
 		rcu_read_lock();
 		/* mem->id is valid, checked in xdp_rxq_info_reg_mem_model() */
 		xa = rhashtable_lookup(mem_id_ht, &mem->id, mem_id_rht_params);
-		xa->zc_alloc->free(xa->zc_alloc, handle);
+		xa->zc_alloc->free(xa->zc_alloc, handle, off);
 		rcu_read_unlock();
 	default:
 		/* Not possible, checked in xdp_rxq_info_reg_mem_model() */
@@ -363,19 +363,20 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
 
 void xdp_return_frame(struct xdp_frame *xdpf)
 {
-	__xdp_return(xdpf->data, &xdpf->mem, false, 0);
+	__xdp_return(xdpf->data, &xdpf->mem, false, 0, 0);
 }
 EXPORT_SYMBOL_GPL(xdp_return_frame);
 
 void xdp_return_frame_rx_napi(struct xdp_frame *xdpf)
 {
-	__xdp_return(xdpf->data, &xdpf->mem, true, 0);
+	__xdp_return(xdpf->data, &xdpf->mem, true, 0, 0);
 }
 EXPORT_SYMBOL_GPL(xdp_return_frame_rx_napi);
 
 void xdp_return_buff(struct xdp_buff *xdp)
 {
-	__xdp_return(xdp->data, &xdp->rxq->mem, true, xdp->handle);
+	__xdp_return(xdp->data, &xdp->rxq->mem, true, xdp->handle,
+			xdp->data - xdp->data_hard_start);
 }
 EXPORT_SYMBOL_GPL(xdp_return_buff);
 
Signature · 1 line
2.17.1
#4 lore raw

PATCH 4/11 i40e: add offset to zca_free

In reply to Kevin Laatz

This patch adds the offset param to for zero_copy_allocator to i40e_zca_free. This change is required to calculate the handle, otherwise, this function will not work in unaligned chunk mode since we can't easily mask back to the original handle in unaligned chunk mode.

---
 drivers/net/ethernet/intel/i40e/i40e_xsk.c | 8 ++++----
 drivers/net/ethernet/intel/i40e/i40e_xsk.h | 3 ++-
 2 files changed, 6 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/i40e/i40e_xsk.c +4 -4
index c89e692e8663..8c281f356293 100644
@@ -438,16 +438,16 @@ static void i40e_reuse_rx_buffer_zc(struct i40e_ring *rx_ring,
  * @alloc: Zero-copy allocator
  * @handle: Buffer handle
  **/
-void i40e_zca_free(struct zero_copy_allocator *alloc, unsigned long handle)
+void i40e_zca_free(struct zero_copy_allocator *alloc, unsigned long handle,
+		off_t off)
 {
 	struct i40e_rx_buffer *bi;
 	struct i40e_ring *rx_ring;
-	u64 hr, mask;
+	u64 hr;
 	u16 nta;
 
 	rx_ring = container_of(alloc, struct i40e_ring, zca);
 	hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;
-	mask = rx_ring->xsk_umem->chunk_mask;
 
 	nta = rx_ring->next_to_alloc;
 	bi = &rx_ring->rx_bi[nta];
@@ -455,7 +455,7 @@ void i40e_zca_free(struct zero_copy_allocator *alloc, unsigned long handle)
 	nta++;
 	rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
 
-	handle &= mask;
+	handle -= off;
 
 	bi->dma = xdp_umem_get_dma(rx_ring->xsk_umem, handle);
 	bi->dma += hr;
drivers/net/ethernet/intel/i40e/i40e_xsk.h +2 -1
index 8cc0a2e7d9a2..85691dc9ac42 100644
@@ -12,7 +12,8 @@ int i40e_queue_pair_disable(struct i40e_vsi *vsi, int queue_pair);
 int i40e_queue_pair_enable(struct i40e_vsi *vsi, int queue_pair);
 int i40e_xsk_umem_setup(struct i40e_vsi *vsi, struct xdp_umem *umem,
 			u16 qid);
-void i40e_zca_free(struct zero_copy_allocator *alloc, unsigned long handle);
+void i40e_zca_free(struct zero_copy_allocator *alloc, unsigned long handle,
+		off_t off);
 bool i40e_alloc_rx_buffers_zc(struct i40e_ring *rx_ring, u16 cleaned_count);
 int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget);
 
Signature · 1 line
2.17.1
#5 lore raw

PATCH 5/11 ixgbe: add offset to zca_free

In reply to Kevin Laatz

This patch adds the offset param to for zero_copy_allocator to ixgbe_zca_free. This change is required to calculate the handle, otherwise, this function will not work in unaligned chunk mode since we can't easily mask back to the original handle in unaligned chunk mode.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h | 3 ++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c         | 8 ++++----
 2 files changed, 6 insertions(+), 5 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h +2 -1
index d93a690aff74..49702e2a4360 100644
@@ -33,7 +33,8 @@ struct xdp_umem *ixgbe_xsk_umem(struct ixgbe_adapter *adapter,
 int ixgbe_xsk_umem_setup(struct ixgbe_adapter *adapter, struct xdp_umem *umem,
 			 u16 qid);
 
-void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle);
+void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle,
+		off_t off);
 
 void ixgbe_alloc_rx_buffers_zc(struct ixgbe_ring *rx_ring, u16 cleaned_count);
 int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +4 -4
index 49536adafe8e..1ec02077ccb2 100644
@@ -268,16 +268,16 @@ static void ixgbe_reuse_rx_buffer_zc(struct ixgbe_ring *rx_ring,
 	obi->skb = NULL;
 }
 
-void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle)
+void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle,
+		off_t off)
 {
 	struct ixgbe_rx_buffer *bi;
 	struct ixgbe_ring *rx_ring;
-	u64 hr, mask;
+	u64 hr;
 	u16 nta;
 
 	rx_ring = container_of(alloc, struct ixgbe_ring, zca);
 	hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;
-	mask = rx_ring->xsk_umem->chunk_mask;
 
 	nta = rx_ring->next_to_alloc;
 	bi = rx_ring->rx_buffer_info;
@@ -285,7 +285,7 @@ void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle)
 	nta++;
 	rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
 
-	handle &= mask;
+	handle -= off;
 
 	bi->dma = xdp_umem_get_dma(rx_ring->xsk_umem, handle);
 	bi->dma += hr;
Signature · 1 line
2.17.1
#6 lore raw

PATCH 6/11 xsk: add support to allow unaligned chunk placement

In reply to Kevin Laatz

Currently, addresses are chunk size aligned. This means, we are very restricted in terms of where we can place chunk within the umem. For example, if we have a chunk size of 2k, then our chunks can only be placed at 0,2k,4k,6k,8k... and so on (ie. every 2k starting from 0).

This patch introduces the ability to use unaligned chunks. With these changes, we are no longer bound to having to place chunks at a 2k (or whatever your chunk size is) interval. Since we are no longer dealing with aligned chunks, they can now cross page boundaries. Checks for page contiguity have been added in order to keep track of which pages are followed by a physically contiguous page.

---
 include/net/xdp_sock.h      |  2 ++
 include/uapi/linux/if_xdp.h |  4 +++
 net/xdp/xdp_umem.c          | 17 +++++++----
 net/xdp/xsk.c               | 60 ++++++++++++++++++++++++++++++++-----
 net/xdp/xsk_queue.h         | 60 ++++++++++++++++++++++++++++++++-----
 5 files changed, 121 insertions(+), 22 deletions(-)
include/net/xdp_sock.h +2 -0
index ae0f368a62bb..c07977e271c4 100644
@@ -19,6 +19,7 @@ struct xsk_queue;
 struct xdp_umem_page {
 	void *addr;
 	dma_addr_t dma;
+	bool next_pg_contig;
 };
 
 struct xdp_umem_fq_reuse {
@@ -48,6 +49,7 @@ struct xdp_umem {
 	bool zc;
 	spinlock_t xsk_list_lock;
 	struct list_head xsk_list;
+	u32 flags;
 };
 
 struct xdp_sock {
include/uapi/linux/if_xdp.h +4 -0
index caed8b1614ff..8548f2110a77 100644
@@ -17,6 +17,9 @@
 #define XDP_COPY	(1 << 1) /* Force copy-mode */
 #define XDP_ZEROCOPY	(1 << 2) /* Force zero-copy mode */
 
+/* Flags for xsk_umem_config flags */
+#define XDP_UMEM_UNALIGNED_CHUNKS (1 << 0)
+
 struct sockaddr_xdp {
 	__u16 sxdp_family;
 	__u16 sxdp_flags;
@@ -52,6 +55,7 @@ struct xdp_umem_reg {
 	__u64 len; /* Length of packet data area */
 	__u32 chunk_size;
 	__u32 headroom;
+	__u32 flags;
 };
 
 struct xdp_statistics {
net/xdp/xdp_umem.c +11 -6
index 2b18223e7eb8..4d4782572855 100644
@@ -301,12 +301,14 @@ static int xdp_umem_account_pages(struct xdp_umem *umem)
 
 static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)
 {
+	bool unaligned_chunks = mr->flags & XDP_UMEM_UNALIGNED_CHUNKS;
 	u32 chunk_size = mr->chunk_size, headroom = mr->headroom;
 	unsigned int chunks, chunks_per_page;
 	u64 addr = mr->addr, size = mr->len;
 	int size_chk, err, i;
 
-	if (chunk_size < XDP_UMEM_MIN_CHUNK_SIZE || chunk_size > PAGE_SIZE) {
+	if ((!unaligned_chunks && chunk_size < XDP_UMEM_MIN_CHUNK_SIZE) ||
+			chunk_size > PAGE_SIZE) {
 		/* Strictly speaking we could support this, if:
 		 * - huge pages, or*
 		 * - using an IOMMU, or
@@ -316,7 +318,7 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)
 		return -EINVAL;
 	}
 
-	if (!is_power_of_2(chunk_size))
+	if (!unaligned_chunks && !is_power_of_2(chunk_size))
 		return -EINVAL;
 
 	if (!PAGE_ALIGNED(addr)) {
@@ -333,9 +335,11 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)
 	if (chunks == 0)
 		return -EINVAL;
 
-	chunks_per_page = PAGE_SIZE / chunk_size;
-	if (chunks < chunks_per_page || chunks % chunks_per_page)
-		return -EINVAL;
+	if (!unaligned_chunks) {
+		chunks_per_page = PAGE_SIZE / chunk_size;
+		if (chunks < chunks_per_page || chunks % chunks_per_page)
+			return -EINVAL;
+	}
 
 	headroom = ALIGN(headroom, 64);
 
@@ -344,13 +348,14 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr)
 		return -EINVAL;
 
 	umem->address = (unsigned long)addr;
-	umem->chunk_mask = ~((u64)chunk_size - 1);
+	umem->chunk_mask = unaligned_chunks ? U64_MAX : ~((u64)chunk_size - 1);
 	umem->size = size;
 	umem->headroom = headroom;
 	umem->chunk_size_nohr = chunk_size - headroom;
 	umem->npgs = size / PAGE_SIZE;
 	umem->pgs = NULL;
 	umem->user = NULL;
+	umem->flags = mr->flags;
 	INIT_LIST_HEAD(&umem->xsk_list);
 	spin_lock_init(&umem->xsk_list_lock);
 
net/xdp/xsk.c +52 -8

114 lines not on this page · raw message

net/xdp/xsk_queue.h +52 -8

124 lines not on this page · raw message

Signature · 1 line
2.17.1
#7 lore raw

PATCH 7/11 libbpf: add flags to umem config

In reply to Kevin Laatz

This patch adds a 'flags' field to the umem_config and umem_reg structs. This will allow for more options to be added for configuring umems.

The first use for the flags field is to add a flag for unaligned chunks mode. These flags can either be user-provided or filled with a default.

---
 tools/include/uapi/linux/if_xdp.h | 4 ++++
 tools/lib/bpf/xsk.c               | 7 +++++++
 tools/lib/bpf/xsk.h               | 2 ++
 3 files changed, 13 insertions(+)
tools/include/uapi/linux/if_xdp.h +4 -0
index caed8b1614ff..8548f2110a77 100644
@@ -17,6 +17,9 @@
 #define XDP_COPY	(1 << 1) /* Force copy-mode */
 #define XDP_ZEROCOPY	(1 << 2) /* Force zero-copy mode */
 
+/* Flags for xsk_umem_config flags */
+#define XDP_UMEM_UNALIGNED_CHUNKS (1 << 0)
+
 struct sockaddr_xdp {
 	__u16 sxdp_family;
 	__u16 sxdp_flags;
@@ -52,6 +55,7 @@ struct xdp_umem_reg {
 	__u64 len; /* Length of packet data area */
 	__u32 chunk_size;
 	__u32 headroom;
+	__u32 flags;
 };
 
 struct xdp_statistics {
tools/lib/bpf/xsk.c +7 -0
index 7ef6293b4fd7..df4207d4ff4a 100644
@@ -115,6 +115,7 @@ static void xsk_set_umem_config(struct xsk_umem_config *cfg,
 		cfg->comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
 		cfg->frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
 		cfg->frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM;
+		cfg->flags = XSK_UMEM__DEFAULT_FLAGS;
 		return;
 	}
 
@@ -122,6 +123,7 @@ static void xsk_set_umem_config(struct xsk_umem_config *cfg,
 	cfg->comp_size = usr_cfg->comp_size;
 	cfg->frame_size = usr_cfg->frame_size;
 	cfg->frame_headroom = usr_cfg->frame_headroom;
+	cfg->flags = usr_cfg->flags;
 }
 
 static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
@@ -181,6 +183,11 @@ int xsk_umem__create(struct xsk_umem **umem_ptr, void *umem_area, __u64 size,
 	mr.len = size;
 	mr.chunk_size = umem->config.frame_size;
 	mr.headroom = umem->config.frame_headroom;
+	mr.flags = umem->config.flags;
+
+	/* Headroom must be 0 for unaligned chunks */
+	if ((mr.flags & XDP_UMEM_UNALIGNED_CHUNKS) && mr.headroom != 0)
+		return -EINVAL;
 
 	err = setsockopt(umem->fd, SOL_XDP, XDP_UMEM_REG, &mr, sizeof(mr));
 	if (err) {
tools/lib/bpf/xsk.h +2 -0
index 82ea71a0f3ec..8d393873b70f 100644
@@ -170,12 +170,14 @@ LIBBPF_API int xsk_socket__fd(const struct xsk_socket *xsk);
 #define XSK_UMEM__DEFAULT_FRAME_SHIFT    11 /* 2048 bytes */
 #define XSK_UMEM__DEFAULT_FRAME_SIZE     (1 << XSK_UMEM__DEFAULT_FRAME_SHIFT)
 #define XSK_UMEM__DEFAULT_FRAME_HEADROOM 0
+#define XSK_UMEM__DEFAULT_FLAGS 0
 
 struct xsk_umem_config {
 	__u32 fill_size;
 	__u32 comp_size;
 	__u32 frame_size;
 	__u32 frame_headroom;
+	__u32 flags;
 };
 
 /* Flags for the libbpf_flags field. */
Signature · 1 line
2.17.1
#8 lore raw

PATCH 8/11 samples/bpf: add unaligned chunks mode support to xdpsock

In reply to Kevin Laatz

This patch adds support for the unaligned chunks mode. The addition of the unaligned chunks option will allow users to run the application with more relaxed chunk placement in the XDP umem.

Unaligned chunks mode can be used with the '-u' or '--unaligned' command line options.

---
 samples/bpf/xdpsock_user.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
samples/bpf/xdpsock_user.c +18 -2
index d08ee1ab7bb4..e26f43382d01 100644
@@ -67,6 +67,8 @@ static int opt_ifindex;
 static int opt_queue;
 static int opt_poll;
 static int opt_interval = 1;
+static u32 opt_umem_flags;
+static int opt_unaligned_chunks;
 static u32 opt_xdp_bind_flags;
 static __u32 prog_id;
 
@@ -276,14 +278,21 @@ static size_t gen_eth_frame(struct xsk_umem_info *umem, u64 addr)
 static struct xsk_umem_info *xsk_configure_umem(void *buffer, u64 size)
 {
 	struct xsk_umem_info *umem;
+	struct xsk_umem_config umem_cfg;
 	int ret;
 
 	umem = calloc(1, sizeof(*umem));
 	if (!umem)
 		exit_with_error(errno);
 
+	umem_cfg.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
+	umem_cfg.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
+	umem_cfg.frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
+	umem_cfg.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM;
+	umem_cfg.flags = opt_umem_flags;
+
 	ret = xsk_umem__create(&umem->umem, buffer, size, &umem->fq, &umem->cq,
-			       NULL);
+			       &umem_cfg);
 	if (ret)
 		exit_with_error(-ret);
 
@@ -346,6 +355,7 @@ static struct option long_options[] = {
 	{"interval", required_argument, 0, 'n'},
 	{"zero-copy", no_argument, 0, 'z'},
 	{"copy", no_argument, 0, 'c'},
+	{"unaligned", no_argument, 0, 'u'},
 	{0, 0, 0, 0}
 };
 
@@ -365,6 +375,7 @@ static void usage(const char *prog)
 		"  -n, --interval=n	Specify statistics update interval (default 1 sec).\n"
 		"  -z, --zero-copy      Force zero-copy mode.\n"
 		"  -c, --copy           Force copy mode.\n"
+		"  -u, --unaligned	Enable unaligned chunk placement\n"
 		"\n";
 	fprintf(stderr, str, prog);
 	exit(EXIT_FAILURE);
@@ -377,7 +388,7 @@ static void parse_command_line(int argc, char **argv)
 	opterr = 0;
 
 	for (;;) {
-		c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
+		c = getopt_long(argc, argv, "Frtli:q:psSNn:czu", long_options,
 				&option_index);
 		if (c == -1)
 			break;
@@ -417,9 +428,14 @@ static void parse_command_line(int argc, char **argv)
 		case 'c':
 			opt_xdp_bind_flags |= XDP_COPY;
 			break;
+		case 'u':
+			opt_umem_flags |= XDP_UMEM_UNALIGNED_CHUNKS;
+			opt_unaligned_chunks = 1;
+			break;
 		case 'F':
 			opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
 			break;
+
 		default:
 			usage(basename(argv[0]));
 		}
Signature · 1 line
2.17.1
#9 lore raw

PATCH 9/11 samples/bpf: add buffer recycling for unaligned chunks to xdpsock

In reply to Kevin Laatz

This patch adds buffer recycling support for unaligned buffers. Since we don't mask the addr to 2k at umem_teg in unaligned mode, we need to make sure we give back the correct, original addr to the fill queue. To do this, we need to mask the addr with the buffer size.

To pass in a buffer size, use the --buf-size=n argument.
NOTE: For xdpsock to work in aligned chunk mode, you still need to pass 'power of 2' buffer size.

---
 samples/bpf/xdpsock_user.c | 71 +++++++++++++++++++++++++++-----------
 1 file changed, 51 insertions(+), 20 deletions(-)
samples/bpf/xdpsock_user.c +51 -20
index e26f43382d01..7b4ce047deb2 100644
@@ -60,6 +60,10 @@ enum benchmark_type {
 	BENCH_L2FWD = 2,
 };
 
+#define LENGTH (256UL*1024*1024)
+#define ADDR (void *)(0x0UL)
+#define SHMAT_FLAGS (0)
+
 static enum benchmark_type opt_bench = BENCH_RXDROP;
 static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
 static const char *opt_if = "";
@@ -67,6 +71,7 @@ static int opt_ifindex;
 static int opt_queue;
 static int opt_poll;
 static int opt_interval = 1;
+static u64 opt_buffer_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
 static u32 opt_umem_flags;
 static int opt_unaligned_chunks;
 static u32 opt_xdp_bind_flags;
@@ -287,7 +292,7 @@ static struct xsk_umem_info *xsk_configure_umem(void *buffer, u64 size)
 
 	umem_cfg.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
 	umem_cfg.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
-	umem_cfg.frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
+	umem_cfg.frame_size = opt_buffer_size;
 	umem_cfg.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM;
 	umem_cfg.flags = opt_umem_flags;
 
@@ -334,8 +339,8 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
 		exit_with_error(-ret);
 	for (i = 0;
 	     i < XSK_RING_PROD__DEFAULT_NUM_DESCS *
-		     XSK_UMEM__DEFAULT_FRAME_SIZE;
-	     i += XSK_UMEM__DEFAULT_FRAME_SIZE)
+		     opt_buffer_size;
+	     i += opt_buffer_size)
 		*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx++) = i;
 	xsk_ring_prod__submit(&xsk->umem->fq,
 			      XSK_RING_PROD__DEFAULT_NUM_DESCS);
@@ -356,6 +361,7 @@ static struct option long_options[] = {
 	{"zero-copy", no_argument, 0, 'z'},
 	{"copy", no_argument, 0, 'c'},
 	{"unaligned", no_argument, 0, 'u'},
+	{"buf-size", required_argument, 0, 'b'},
 	{0, 0, 0, 0}
 };
 
@@ -376,6 +382,7 @@ static void usage(const char *prog)
 		"  -z, --zero-copy      Force zero-copy mode.\n"
 		"  -c, --copy           Force copy mode.\n"
 		"  -u, --unaligned	Enable unaligned chunk placement\n"
+		"  -b, --buf-size=n	Specify the buffer size to use\n"
 		"\n";
 	fprintf(stderr, str, prog);
 	exit(EXIT_FAILURE);
@@ -388,7 +395,7 @@ static void parse_command_line(int argc, char **argv)
 	opterr = 0;
 
 	for (;;) {
-		c = getopt_long(argc, argv, "Frtli:q:psSNn:czu", long_options,
+		c = getopt_long(argc, argv, "Frtli:q:psSNn:czub", long_options,
 				&option_index);
 		if (c == -1)
 			break;
@@ -432,6 +439,9 @@ static void parse_command_line(int argc, char **argv)
 			opt_umem_flags |= XDP_UMEM_UNALIGNED_CHUNKS;
 			opt_unaligned_chunks = 1;
 			break;
+		case 'b':
+			opt_buffer_size = atoi(optarg);
+			break;
 		case 'F':
 			opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
 			break;
@@ -483,13 +493,22 @@ static inline void complete_tx_l2fwd(struct xsk_socket_info *xsk)
 		while (ret != rcvd) {
 			if (ret < 0)
 				exit_with_error(-ret);
-			ret = xsk_ring_prod__reserve(&xsk->umem->fq, rcvd,
-						     &idx_fq);
+			ret = xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);
+		}
+
+		if (opt_umem_flags & XDP_UMEM_UNALIGNED_CHUNKS) {
+			for (i = 0; i < rcvd; i++) {
+				u64 comp_addr = *xsk_ring_cons__comp_addr(&xsk->umem->cq,
+						idx_cq++);
+				u64 masked_comp = (comp_addr & ~((u64)opt_buffer_size-1));
+				*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) =
+						masked_comp;
+			}
+		} else {
+			for (i = 0; i < rcvd; i++)
+				*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) =
+						*xsk_ring_cons__comp_addr(&xsk->umem->cq, idx_cq++);
 		}
-		for (i = 0; i < rcvd; i++)
-			*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) =
-				*xsk_ring_cons__comp_addr(&xsk->umem->cq,
-							  idx_cq++);
 
 		xsk_ring_prod__submit(&xsk->umem->fq, rcvd);
 		xsk_ring_cons__release(&xsk->umem->cq, rcvd);
@@ -533,13 +552,25 @@ static void rx_drop(struct xsk_socket_info *xsk)
 		ret = xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq);
 	}
 
-	for (i = 0; i < rcvd; i++) {
-		u64 addr = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx)->addr;
-		u32 len = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++)->len;
-		char *pkt = xsk_umem__get_data(xsk->umem->buffer, addr);
+	if (opt_umem_flags & XDP_UMEM_UNALIGNED_CHUNKS) {
+		for (i = 0; i < rcvd; i++) {
+			u64 addr = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx)->addr;
+			u32 len = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++)->len;
+			char *pkt = xsk_umem__get_data(xsk->umem->buffer, addr);
+			u64 masked_addr = (addr & ~((u64)opt_buffer_size-1));
+
+			hex_dump(pkt, len, addr);
+			*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = masked_addr;
+		}
+	} else {
+		for (i = 0; i < rcvd; i++) {
+			u64 addr = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx)->addr;
+			u32 len = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++)->len;
+			char *pkt = xsk_umem__get_data(xsk->umem->buffer, addr);
 
-		hex_dump(pkt, len, addr);
-		*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = addr;
+			hex_dump(pkt, len, addr);
+			*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = addr;
+		}
 	}
 
 	xsk_ring_prod__submit(&xsk->umem->fq, rcvd);
@@ -677,20 +708,20 @@ int main(int argc, char **argv)
 	}
 
 	ret = posix_memalign(&bufs, getpagesize(), /* PAGE_SIZE aligned */
-			     NUM_FRAMES * XSK_UMEM__DEFAULT_FRAME_SIZE);
+			     NUM_FRAMES * opt_buffer_size);
 	if (ret)
 		exit_with_error(ret);
 
        /* Create sockets... */
 	umem = xsk_configure_umem(bufs,
-				  NUM_FRAMES * XSK_UMEM__DEFAULT_FRAME_SIZE);
+				  NUM_FRAMES * opt_buffer_size);
 	xsks[num_socks++] = xsk_configure_socket(umem);
 
 	if (opt_bench == BENCH_TXONLY) {
 		int i;
 
-		for (i = 0; i < NUM_FRAMES * XSK_UMEM__DEFAULT_FRAME_SIZE;
-		     i += XSK_UMEM__DEFAULT_FRAME_SIZE)
+		for (i = 0; i < NUM_FRAMES * opt_buffer_size;
+		     i += opt_buffer_size)
 			(void)gen_eth_frame(umem, i);
 	}
 
Signature · 1 line
2.17.1
#10 lore raw

PATCH 10/11 samples/bpf: use hugepages in xdpsock app

In reply to Kevin Laatz

This patch modifies xdpsock to use mmap instead of posix_memalign. With this change, we can use hugepages when running the application in unaligned chunks mode. Using hugepages makes it more likely that we have physically contiguous memory, which supports the unaligned chunk mode better.

---
 samples/bpf/xdpsock_user.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
samples/bpf/xdpsock_user.c +9 -5
index 7b4ce047deb2..8ed63ad68428 100644
@@ -74,6 +74,7 @@ static int opt_interval = 1;
 static u64 opt_buffer_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
 static u32 opt_umem_flags;
 static int opt_unaligned_chunks;
+static int opt_mmap_flags;
 static u32 opt_xdp_bind_flags;
 static __u32 prog_id;
 
@@ -438,6 +439,7 @@ static void parse_command_line(int argc, char **argv)
 		case 'u':
 			opt_umem_flags |= XDP_UMEM_UNALIGNED_CHUNKS;
 			opt_unaligned_chunks = 1;
+			opt_mmap_flags = MAP_HUGETLB;
 			break;
 		case 'b':
 			opt_buffer_size = atoi(optarg);
@@ -707,11 +709,13 @@ int main(int argc, char **argv)
 		exit(EXIT_FAILURE);
 	}
 
-	ret = posix_memalign(&bufs, getpagesize(), /* PAGE_SIZE aligned */
-			     NUM_FRAMES * opt_buffer_size);
-	if (ret)
-		exit_with_error(ret);
-
+	/* Reserve memory for the umem. Use hugepages if unaligned chunk mode */
+	bufs = mmap(NULL, NUM_FRAMES * opt_buffer_size, PROT_READ|PROT_WRITE,
+			MAP_PRIVATE|MAP_ANONYMOUS|opt_mmap_flags, -1, 0);
+	if (bufs == MAP_FAILED) {
+		printf("ERROR: mmap failed\n");
+		exit(EXIT_FAILURE);
+	}
        /* Create sockets... */
 	umem = xsk_configure_umem(bufs,
 				  NUM_FRAMES * opt_buffer_size);
Signature · 1 line
2.17.1
#11 lore raw

PATCH 11/11 doc/af_xdp: include unaligned chunk case

In reply to Kevin Laatz

The addition of unaligned chunks mode, the documentation needs to be updated to indicate that the incoming addr to the fill ring will only be masked if the user application is run in the aligned chunk mode. This patch also adds a line to explicitly indicate that the incoming addr will not be masked if running the user application in the unaligned chunk mode.

---
 Documentation/networking/af_xdp.rst | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +6 -4
index e14d7d40fc75..16fbc68cac50 100644
@@ -153,10 +153,12 @@ an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has
 
 Frames passed to the kernel are used for the ingress path (RX rings).
 
-The user application produces UMEM addrs to this ring. Note that the
-kernel will mask the incoming addr. E.g. for a chunk size of 2k, the
-log2(2048) LSB of the addr will be masked off, meaning that 2048, 2050
-and 3000 refers to the same chunk.
+The user application produces UMEM addrs to this ring. Note that, if
+running the application with aligned chunk mode, the kernel will mask
+the incoming addr.  E.g. for a chunk size of 2k, the log2(2048) LSB of
+the addr will be masked off, meaning that 2048, 2050 and 3000 refers
+to the same chunk. If the user application is run in the unaligned
+chunks mode, then the incoming addr will be left untouched.
 
 
 UMEM Completion Ring
Signature · 1 line
2.17.1
#12 lore raw
In reply to Kevin Laatz
Quoted 8 lines · Kevin Laatz

This patchset adds the ability to use unaligned chunks in the XDP umem.

Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are supported.

Quoted 8 lines

The changes in this patchset removes these restrictions, allowing XDP to be more flexible in where it can place a chunk within a umem. By relaxing where the chunks can be placed, it allows us to use an arbitrary buffer size and place that wherever we have a free address in the umem. These changes add the ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as DPDK.

Thanks for working on this, Kevin and Ciara!

I have some minor comments on the series, but in general I think it's in good shape!

For some reason the series was submitted twice (at least on my side)?

Thanks,
Bj?rn

Quoted 91 lines

Structure of the patchset:
Patch 1:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in i40e. This change is required in order for the buffer recycling to work in the unaligned chunk mode.

Patch 2:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in ixgbe. This change is required in order for the buffer recycling to work in the unaligned chunk mode.

Patch 3:
- Adds an offset parameter to zero_copy_allocator. This change will enable us to calculate the original handle in zca_free. This will be required for unaligned chunk mode since we can't easily mask back to the original handle.

Patch 4:
- Adds the offset parameter to i40e_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.

Patch 5:
- Adds the offset parameter to ixgbe_zca_free. This change is needed for calculating the handle since we can't easily mask back to the original handle like we can in the aligned case.

Patch 6:
- Add infrastructure for unaligned chunks. Since we are dealing with unaligned chunks that could potentially cross a physical page boundary, we add checks to keep track of that information. We can later use this information to correctly handle buffers that are placed at an address where they cross a page boundary.

Patch 7:
- Add flags for umem configuration to libbpf

Patch 8:
- Modify xdpsock application to add a command line option for unaligned chunks

Patch 9:
- Addition of command line argument to pass in a desired buffer size and buffer recycling for unaligned mode. Passing in a buffer size will allow the application to use unaligned chunks with the unaligned chunk mode. Since we are now using unaligned chunks, we need to recycle our buffers in a slightly different way.

Patch 10:
- Adds hugepage support to the xdpsock application

Patch 11:
- Documentation update to include the unaligned chunk scenario. We need to explicitly state that the incoming addresses are only masked in the aligned chunk mode and not the unaligned chunk mode.

Kevin Laatz (11):
i40e: simplify Rx buffer recycle
ixgbe: simplify Rx buffer recycle
xdp: add offset param to zero_copy_allocator
i40e: add offset to zca_free
ixgbe: add offset to zca_free
xsk: add support to allow unaligned chunk placement
libbpf: add flags to umem config
samples/bpf: add unaligned chunks mode support to xdpsock samples/bpf: add buffer recycling for unaligned chunks to xdpsock samples/bpf: use hugepages in xdpsock app
doc/af_xdp: include unaligned chunk case

 Documentation/networking/af_xdp.rst           | 10 +-
 drivers/net/ethernet/intel/i40e/i40e_xsk.c    | 21 ++--
 drivers/net/ethernet/intel/i40e/i40e_xsk.h    |  3 +-
 .../ethernet/intel/ixgbe/ixgbe_txrx_common.h  |  3 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c  | 21 ++--
 include/net/xdp.h                             |  3 +-
 include/net/xdp_sock.h                        |  2 +
 include/uapi/linux/if_xdp.h                   |  4 +
 net/core/xdp.c                                | 11 ++-
 net/xdp/xdp_umem.c                            | 17 ++--
 net/xdp/xsk.c                                 | 60 +++++++++--
 net/xdp/xsk_queue.h                           | 60 +++++++++--
 samples/bpf/xdpsock_user.c                    | 99 ++++++++++++++-----
 tools/include/uapi/linux/if_xdp.h             |  4 +
 tools/lib/bpf/xsk.c                           |  7 ++
 tools/lib/bpf/xsk.h                           |  2 +
 16 files changed, 241 insertions(+), 86 deletions(-)

--
2.17.1

#13 lore raw
Quoted 24 lines · Bj?rn T?pel
Quoted 9 lines · Kevin Laatz

This patchset adds the ability to use unaligned chunks in the XDP umem.

Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are
supported.

Quoted 13 lines

The changes in this patchset removes these restrictions, allowing XDP to be
more flexible in where it can place a chunk within a umem. By relaxing where
the chunks can be placed, it allows us to use an arbitrary buffer size and
place that wherever we have a free address in the umem. These changes add the
ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as
DPDK.

Quoted 6 lines

Thanks for working on this, Kevin and Ciara!

I have some minor comments on the series, but in general I think it's in good shape!

For some reason the series was submitted twice (at least on my side)?

Apologies for the confusion... The first set had a typo in the bpf mailing list address (.com vs .org). Will fix for the v2.

Quoted 4 lines

Thanks,
Bj?rn

Thanks for reviewing. Will address your comments in the v2.

Quoted 102 lines
Quoted 101 lines

Structure of the patchset:
Patch 1:
? - Remove unnecessary masking and headroom addition during zero-copy Rx ??? buffer recycling in i40e. This change is required in order for the ??? buffer recycling to work in the unaligned chunk mode.

Patch 2:
? - Remove unnecessary masking and headroom addition during ??? zero-copy Rx buffer recycling in ixgbe. This change is required in ??? order for the? buffer recycling to work in the unaligned chunk mode.

Patch 3:
? - Adds an offset parameter to zero_copy_allocator. This change will ??? enable us to calculate the original handle in zca_free. This will be ??? required for unaligned chunk mode since we can't easily mask back to ??? the original handle.

Patch 4:
? - Adds the offset parameter to i40e_zca_free. This change is needed for
??? calculating the handle since we can't easily mask back to the original
??? handle like we can in the aligned case.

Patch 5:
? - Adds the offset parameter to ixgbe_zca_free. This change is needed for
??? calculating the handle since we can't easily mask back to the original
??? handle like we can in the aligned case.

Patch 6:
? - Add infrastructure for unaligned chunks. Since we are dealing ??? with unaligned chunks that could potentially cross a physical page ??? boundary, we add checks to keep track of that information. We can ??? later use this information to correctly handle buffers that are ??? placed at an address where they cross a page boundary.

Patch 7:
? - Add flags for umem configuration to libbpf

Patch 8:
? - Modify xdpsock application to add a command line option for ??? unaligned chunks

Patch 9:
? - Addition of command line argument to pass in a desired buffer size ??? and buffer recycling for unaligned mode. Passing in a buffer size will
??? allow the application to use unaligned chunks with the unaligned chunk
??? mode. Since we are now using unaligned chunks, we need to recycle our
??? buffers in a slightly different way.

Patch 10:
? - Adds hugepage support to the xdpsock application

Patch 11:
? - Documentation update to include the unaligned chunk scenario. We need
??? to explicitly state that the incoming addresses are only masked in the
??? aligned chunk mode and not the unaligned chunk mode.

Kevin Laatz (11):
? i40e: simplify Rx buffer recycle
? ixgbe: simplify Rx buffer recycle
? xdp: add offset param to zero_copy_allocator
? i40e: add offset to zca_free
? ixgbe: add offset to zca_free
? xsk: add support to allow unaligned chunk placement
? libbpf: add flags to umem config
? samples/bpf: add unaligned chunks mode support to xdpsock ? samples/bpf: add buffer recycling for unaligned chunks to xdpsock ? samples/bpf: use hugepages in xdpsock app
? doc/af_xdp: include unaligned chunk case

?Documentation/networking/af_xdp.rst?????????? | 10 +-
?drivers/net/ethernet/intel/i40e/i40e_xsk.c??? | 21 ++--
?drivers/net/ethernet/intel/i40e/i40e_xsk.h??? |? 3 +-
?.../ethernet/intel/ixgbe/ixgbe_txrx_common.h? |? 3 +-
?drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c? | 21 ++--
?include/net/xdp.h???????????????????????????? |? 3 +-
?include/net/xdp_sock.h??????????????????????? |? 2 +
?include/uapi/linux/if_xdp.h?????????????????? |? 4 +
?net/core/xdp.c??????????????????????????????? | 11 ++-
?net/xdp/xdp_umem.c??????????????????????????? | 17 ++--
?net/xdp/xsk.c???????????????????????????????? | 60 +++++++++--
?net/xdp/xsk_queue.h?????????????????????????? | 60 +++++++++--
?samples/bpf/xdpsock_user.c??????????????????? | 99 ++++++++++++++-----
?tools/include/uapi/linux/if_xdp.h???????????? |? 4 +
?tools/lib/bpf/xsk.c?????????????????????????? |? 7 ++
?tools/lib/bpf/xsk.h?????????????????????????? |? 2 +

?16 files changed, 241 insertions(+), 86 deletions(-)

Signature · 1 line
2.17.1
#14 lore raw
In reply to Kevin Laatz
Quoted 11 lines · Kevin Laatz

This patchset adds the ability to use unaligned chunks in the XDP umem.

Currently, all chunk addresses passed to the umem are masked to be chunk
size aligned (default is 2k, max is PAGE_SIZE). This limits where we can
place chunks within the umem as well as limiting the packet sizes that are
supported.

Quoted 13 lines

The changes in this patchset removes these restrictions, allowing XDP to be
more flexible in where it can place a chunk within a umem. By relaxing where
the chunks can be placed, it allows us to use an arbitrary buffer size and
place that wherever we have a free address in the umem. These changes add the
ability to support jumboframes and make it easy to integrate with other
existing frameworks that have their own memory management systems, such as
DPDK.

I'm a little unclear on how this should work, and have a few issues here:

1) There isn't any support for the user defined umem->headroom

2) When queuing RX buffers, the handle (aka umem offset) is used,
which
points to the start of the buffer area. When the buffer appears in the completion queue, handle points to the start of the received
data,
which might be different from the buffer start address.

Normally, this RX address is just put back in the fill queue, and
the
mask is used to find the buffer start address again. This no
longer
works, so my question is, how is the buffer start address
recomputed
from the actual data payload address?

Same with TX - if the TX payload isn't aligned in with the start of the buffer, what happens?

3) This appears limited to crossing a single page boundary, but there is no constraint check on chunk_size.

Signature · 1 line
Jonathan
Quoted 105 lines

Structure of the patchset:
Patch 1:
- Remove unnecessary masking and headroom addition during zero-copy
Rx
buffer recycling in i40e. This change is required in order for the buffer recycling to work in the unaligned chunk mode.

Patch 2:
- Remove unnecessary masking and headroom addition during zero-copy Rx buffer recycling in ixgbe. This change is required in order for the buffer recycling to work in the unaligned chunk
mode.

Patch 3:
- Adds an offset parameter to zero_copy_allocator. This change will enable us to calculate the original handle in zca_free. This will
be
required for unaligned chunk mode since we can't easily mask back
to
the original handle.

Patch 4:
- Adds the offset parameter to i40e_zca_free. This change is needed
for
calculating the handle since we can't easily mask back to the
original
handle like we can in the aligned case.

Patch 5:
- Adds the offset parameter to ixgbe_zca_free. This change is needed
for
calculating the handle since we can't easily mask back to the
original
handle like we can in the aligned case.

Patch 6:
- Add infrastructure for unaligned chunks. Since we are dealing with unaligned chunks that could potentially cross a physical page boundary, we add checks to keep track of that information. We can later use this information to correctly handle buffers that are placed at an address where they cross a page boundary.

Patch 7:
- Add flags for umem configuration to libbpf

Patch 8:
- Modify xdpsock application to add a command line option for unaligned chunks

Patch 9:
- Addition of command line argument to pass in a desired buffer size and buffer recycling for unaligned mode. Passing in a buffer size
will
allow the application to use unaligned chunks with the unaligned
chunk
mode. Since we are now using unaligned chunks, we need to recycle
our
buffers in a slightly different way.

Patch 10:
- Adds hugepage support to the xdpsock application

Patch 11:
- Documentation update to include the unaligned chunk scenario. We
need
to explicitly state that the incoming addresses are only masked in
the
aligned chunk mode and not the unaligned chunk mode.

Kevin Laatz (11):
i40e: simplify Rx buffer recycle
ixgbe: simplify Rx buffer recycle
xdp: add offset param to zero_copy_allocator
i40e: add offset to zca_free
ixgbe: add offset to zca_free
xsk: add support to allow unaligned chunk placement
libbpf: add flags to umem config
samples/bpf: add unaligned chunks mode support to xdpsock samples/bpf: add buffer recycling for unaligned chunks to xdpsock samples/bpf: use hugepages in xdpsock app
doc/af_xdp: include unaligned chunk case

 Documentation/networking/af_xdp.rst           | 10 +-
 drivers/net/ethernet/intel/i40e/i40e_xsk.c    | 21 ++--
 drivers/net/ethernet/intel/i40e/i40e_xsk.h    |  3 +-
 .../ethernet/intel/ixgbe/ixgbe_txrx_common.h  |  3 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c  | 21 ++--
 include/net/xdp.h                             |  3 +-
 include/net/xdp_sock.h                        |  2 +
 include/uapi/linux/if_xdp.h                   |  4 +
 net/core/xdp.c                                | 11 ++-
 net/xdp/xdp_umem.c                            | 17 ++--
 net/xdp/xsk.c                                 | 60 +++++++++--
 net/xdp/xsk_queue.h                           | 60 +++++++++--
 samples/bpf/xdpsock_user.c                    | 99 
++++++++++++++-----
 tools/include/uapi/linux/if_xdp.h             |  4 +
 tools/lib/bpf/xsk.c                           |  7 ++
 tools/lib/bpf/xsk.h                           |  2 +
 16 files changed, 241 insertions(+), 86 deletions(-)
Signature · 1 line
2.17.1
#15 lore raw
Quoted 23 lines · Jonathan Lemon
Quoted 8 lines · Kevin Laatz

This patchset adds the ability to use unaligned chunks in the XDP umem.

Currently, all chunk addresses passed to the umem are masked to be chunk size aligned (default is 2k, max is PAGE_SIZE). This limits where we can place chunks within the umem as well as limiting the packet sizes that are
supported.

Quoted 12 lines

The changes in this patchset removes these restrictions, allowing XDP to be
more flexible in where it can place a chunk within a umem. By relaxing where
the chunks can be placed, it allows us to use an arbitrary buffer size and
place that wherever we have a free address in the umem. These changes add the
ability to support jumboframes and make it easy to integrate with other existing frameworks that have their own memory management systems, such as
DPDK.

Quoted 4 lines

I'm a little unclear on how this should work, and have a few issues here:

?1) There isn't any support for the user defined umem->headroom

For the unaligned chunks case, it does not make sense to to support a user defined headroom since the user can point directly to where they want the data to start via the buffer address. Therefore, for unaligned chunks, the user defined headroom should always be 0 (aka the user did not define a headroom and the default value of 0 is used). Any other value will be caught and we return an invalid argument error.

Quoted 6 lines

?2) When queuing RX buffers, the handle (aka umem offset) is used, which ??? points to the start of the buffer area.? When the buffer appears in ??? the completion queue, handle points to the start of the received data,
??? which might be different from the buffer start address.

Quoted 7 lines

??? Normally, this RX address is just put back in the fill queue, and the ??? mask is used to find the buffer start address again.? This no longer ??? works, so my question is, how is the buffer start address recomputed ??? from the actual data payload address?

??? Same with TX - if the TX payload isn't aligned in with the start of ??? the buffer, what happens?

On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default scenario). This was missing from the v1, will add this in the v2, to have xdpsock use the default value from libbpf! If the user is using another BPF program that uses a different offset, then the computation will need to be adjusted for that accordingly. In v2 we'll add support for this via command-line parameter.

However, we are also working on an "in-order" patchset, hopefully to be published soon, to guarantee the buffers returned to the application are in the same order as those provided to the kernel. Longer term, this is the best solution here as it allows the application to track itself, via a "shadow ring" or otherwise, the buffers sent to the kernel and any metadata associated with them, such as the start of buffer address.

Quoted 3 lines

?3) This appears limited to crossing a single page boundary, but there ??? is no constraint check on chunk_size.

There is an existing check for chunk_size during xdp_umem_reg (in xdp_umem.c) The check makes sure that chunk size is at least XDP_UMEM_MIN_CHUNK_SIZE and at most PAGE_SIZE. Since the max is page size, we only need to check the immediate next page for contiguity. While this patchset allows a max of 4k sized buffers, it is still an improvement from the current state. Future enhancements could look into extending the 4k limit but for now it is a good first step towards supporting hugepages efficiently.

Best regards,
Kevin

#16 lore raw
Quoted 4 lines · Kevin

On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default scenario).

That assumes specific layout for the data inside the buffer. Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset XDP_PACKET_HEADROOM + metadata len..

I think that's very limiting. What is the challenge in providing aligned addresses, exactly?

#17 lore raw
Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · Kevin

On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default scenario).

That assumes specific layout for the data inside the buffer. Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset XDP_PACKET_HEADROOM + metadata len..

Yes, if NICs prepend extra data to the packet that would be a problem for using this feature in isolation. However, if we also add in support for in-order
RX and TX rings, that would no longer be an issue. However, even for NICs which do prepend data, this patchset should not break anything that is currently
working.

Quoted 3 lines

I think that's very limiting. What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment. ??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the
actual writable space specified in units of 1k - therefore we need 2k + metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting
the same memory channels when accessing memory.
??? ??? - it allows the application to choose the actual buffer size it
wants to use.
??? We make use of the above to allow us to speed up processing
significantly and also reduce the packet buffer memory size.

??? Not having arbitrary buffer alignment also means an AF_XDP driver
for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

#18 lore raw
Quoted 10 lines · Kevin
Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · Kevin

On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default scenario).

That assumes specific layout for the data inside the buffer.? Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset XDP_PACKET_HEADROOM + metadata len..

Quoted 7 lines

Yes, if NICs prepend extra data to the packet that would be a problem for using this feature in isolation. However, if we also add in support for in-order
RX and TX rings, that would no longer be an issue. However, even for NICs which do prepend data, this patchset should not break anything that is currently
working.

(Late on the ball. I'm in vacation mode.)

In your example Jakub, how would this look in XDP? Wouldn't the timestamp be part of the metadata (xdp_md.data_meta)? Isn't data-data_meta (if valid) <= XDP_PACKET_HEADROOM? That was my assumption.

There were some discussion on having meta data length in the struct xdp_desc, before AF_XDP was merged, but the conclusion was that this was *not* needed, because AF_XDP and the XDP program had an implicit contract. If you're running AF_XDP, you also have an XDP program running and you can determine the meta data length (and also getting back the original buffer).

So, today in AF_XDP if XDP metadata is added, the userland application can look it up before the xdp_desc.addr (just like regular XDP), and how the XDP/AF_XDP application determines length/layout of the metadata i out-of-band/not specified.

This is a bit messy/handwavy TBH, so maybe adding the length to the descriptor *is* a good idea (extending the options part of the xdp_desc)? Less clean though. OTOH the layout of the meta data still need to be determined.

Bj?rn

#19 lore raw
Quoted 3 lines · Bj?rn T?pel

In your example Jakub, how would this look in XDP? Wouldn't the timestamp be part of the metadata (xdp_md.data_meta)? Isn't data-data_meta (if valid) <= XDP_PACKET_HEADROOM? That was my assumption.

The driver parses the metadata and copies it outside of the prepend before XDP runs. Then XDP runs unaware of the prepend contents. That's the current situation.

XDP_PACKET_HEADROOM is before the entire frame. Like this:

buffer start
/ DMA addr given to the device
/ /
v v
| XDP_HEADROOM | meta data | packet data |

Length of meta data comes in the standard fixed size descriptor.

The metadata prepend is in TV form ("TLV with no length field", length's implied by type).

Quoted 12 lines

There were some discussion on having meta data length in the struct xdp_desc, before AF_XDP was merged, but the conclusion was that this was *not* needed, because AF_XDP and the XDP program had an implicit contract. If you're running AF_XDP, you also have an XDP program running and you can determine the meta data length (and also getting back the original buffer).

So, today in AF_XDP if XDP metadata is added, the userland application can look it up before the xdp_desc.addr (just like regular XDP), and how the XDP/AF_XDP application determines length/layout of the metadata i out-of-band/not specified.

Quoted 4 lines

This is a bit messy/handwavy TBH, so maybe adding the length to the descriptor *is* a good idea (extending the options part of the xdp_desc)? Less clean though. OTOH the layout of the meta data still need to be determined.

Right, the device prepend is not exposed as metadata to XDP.

#20 lore raw
In reply to Laatz, Kevin
Quoted 6 lines · Kevin
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · Kevin

On the application side (xdpsock), we don't have to worry about the user defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default scenario).

Quoted 7 lines
Quoted 3 lines

That assumes specific layout for the data inside the buffer. Some NICs will prepend information like timestamp to the packet, meaning the packet would start at offset XDP_PACKET_HEADROOM + metadata len..

Yes, if NICs prepend extra data to the packet that would be a problem for using this feature in isolation. However, if we also add in support for in-order RX and TX rings, that would no longer be an issue.

Can you shed more light on in-order rings? Do you mean that RX frames come in order buffers were placed in the fill queue? That wouldn't make practical sense, no? Even if the application does no reordering there is also XDP_DROP and XDP_TX. Please explain :)

Quoted 2 lines

However, even for NICs which do prepend data, this patchset should not break anything that is currently working.

My understanding from the beginnings of AF_XDP was that we were searching for a format flexible enough to support most if not all NICs. Creating an ABI which will preclude vendors from supporting DPDK via AF_XDP would seriously undermine the neutrality aspect.

Quoted 5 lines
Quoted 2 lines

I think that's very limiting. What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.

Presumably support for jumbos would require a multi-buffer setup, and therefore extensions to the ring format. Should we perhaps look into implementing unaligned chunks by extending ring format as well?

Quoted 18 lines

2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment. ??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the
actual writable space specified in units of 1k - therefore we need 2k + metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting
the same memory channels when accessing memory.
??? ??? - it allows the application to choose the actual buffer size it
wants to use.
??? We make use of the above to allow us to speed up processing
significantly and also reduce the packet buffer memory size.

??? Not having arbitrary buffer alignment also means an AF_XDP driver
for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

#21 lore raw
In reply to Laatz, Kevin
Quoted 13 lines · Kevin
Quoted 11 lines · Jakub Kicinski
Quoted 6 lines · Kevin

On the application side (xdpsock), we don't have to worry about the user
defined headroom, since it is 0, so we only need to account for the XDP_PACKET_HEADROOM when computing the original address (in the default
scenario).

That assumes specific layout for the data inside the buffer. Some NICs
will prepend information like timestamp to the packet, meaning the packet would start at offset XDP_PACKET_HEADROOM + metadata len..

Quoted 9 lines

Yes, if NICs prepend extra data to the packet that would be a problem for
using this feature in isolation. However, if we also add in support for in-order
RX and TX rings, that would no longer be an issue. However, even for NICs
which do prepend data, this patchset should not break anything that is currently
working.

I read this as "the correct buffer address is recovered from the shadow ring".
I'm not sure I'm comfortable with that, and I'm also not sold on in-order completion
for the RX/TX rings.

Quoted 19 lines
Quoted 2 lines

I think that's very limiting. What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k + metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory.
??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.

Quoted 5 lines

??? Not having arbitrary buffer alignment also means an AF_XDP driver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.

Signature · 1 line
Jonathan
#22 lore raw

cover XDP unaligned chunk placement support

Quoted 27 lines · Jonathan Lemon
Quoted 20 lines · Kevin
Quoted 2 lines · Jakub Kicinski

I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.

Quoted 5 lines

??? Not having arbitrary buffer alignment also means an AF_XDP driver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

Quoted 5 lines

Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.

Yes, we can do this.

Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back to the original address using subtraction.

Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approach are obviously possible too.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20190701/5a471c67/attachment.html>

#23 lore raw
Quoted 28 lines · Kevin
Quoted 27 lines · Jonathan Lemon
Quoted 25 lines · Kevin
Quoted 2 lines · Jakub Kicinski

I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.

??? Not having arbitrary buffer alignment also means an AF_XDP driver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

Quoted 7 lines
Quoted 5 lines

Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.

Yes, we can do this.

That'd be best, can DPDK reasonably guarantee the slicing is uniform? E.g. it's not desperate buffer pools with different bases?

Quoted 4 lines

Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back to the original address using subtraction.

Unfortunately the metadata depends on the packet and how much info the device was able to extract. So it's variable length.

Quoted 10 lines

Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approach are obviously possible too.

Seems reasonable to me..

#24 lore raw
Quoted 43 lines

-----Original Message-----
From: Jakub Kicinski [mailto:jakub.kicinski at netronome.com]
Sent: Monday, July 1, 2019 10:20 PM
To: Laatz, Kevin <[email protected]>
Cc: Jonathan Lemon <[email protected]>; netdev at vger.kernel.org; ast at kernel.org; daniel at iogearbox.net; Topel, Bjorn <[email protected]>; Karlsson, Magnus <[email protected]>; bpf at vger.kernel.org; intel-wired-lan at lists.osuosl.org; Richardson, Bruce <[email protected]>; Loftus, Ciara <[email protected]>
Subject: Re: [PATCH 00/11] XDP unaligned chunk placement support

Quoted 29 lines · Kevin
Quoted 28 lines · Jonathan Lemon
Quoted 26 lines · Kevin
Quoted 2 lines · Jakub Kicinski

I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.

??? Not having arbitrary buffer alignment also means an AF_XDP driver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Since all buffers in the umem are the same chunk size, the original buffer address can be recalculated with some multiply/shift math. However, this is more expensive than just a mask operation.

Yes, we can do this.

That'd be best, can DPDK reasonably guarantee the slicing is uniform? E.g. it's not desperate buffer pools with different bases?

It's generally uniform, but handling the crossing of (huge)page boundaries complicates things a bit. Therefore I think the final option below is best as it avoids any such problems.

Quoted 20 lines
Quoted 5 lines

Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back to the original address using subtraction.

Unfortunately the metadata depends on the packet and how much info the device was able to extract. So it's variable length.

Quoted 10 lines

Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approach are obviously possible too.

Quoted 1 line

Seems reasonable to me..

I think this is probably the best solution, and also has the advantage that a buffer retains its base address the full way through the cycle of Rx and Tx.

#25 lore raw
Quoted 92 lines · Bruce
Quoted 51 lines

-----Original Message-----
From: Jakub Kicinski [mailto:jakub.kicinski at netronome.com]
Sent: Monday, July 1, 2019 10:20 PM
To: Laatz, Kevin <[email protected]>
Cc: Jonathan Lemon <[email protected]>;
netdev at vger.kernel.org;
ast at kernel.org; daniel at iogearbox.net; Topel, Bjorn <[email protected]>; Karlsson, Magnus <[email protected]>;
bpf at vger.kernel.org; intel-wired-lan at lists.osuosl.org; Richardson, Bruce
<[email protected]>; Loftus, Ciara <[email protected]>
Subject: Re: [PATCH 00/11] XDP unaligned chunk placement support

Quoted 36 lines · Kevin
Quoted 35 lines · Jonathan Lemon
Quoted 33 lines · Kevin
Quoted 3 lines · Jakub Kicinski

I think that's very limiting.? What is the challenge in providing
aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue
supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP,
such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need
the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly
hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer
size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.

??? Not having arbitrary buffer alignment also means an AF_XDP driver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an
arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Since all buffers in the umem are the same chunk size, the original buffer address can be recalculated with some multiply/shift math. However, this is more expensive than just a mask operation.

Yes, we can do this.

That'd be best, can DPDK reasonably guarantee the slicing is uniform? E.g. it's not desperate buffer pools with different bases?

It's generally uniform, but handling the crossing of (huge)page boundaries
complicates things a bit. Therefore I think the final option below is best as it avoids any such problems.

Quoted 27 lines
Quoted 7 lines

Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet).
We can use that information to get back to the original address using
subtraction.

Unfortunately the metadata depends on the packet and how much info the
device was able to extract. So it's variable length.

Quoted 14 lines

Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address as the start of the packet, instead
return the buffer address that was passed in, and adding another 16-bit field to specify the start of packet offset with that buffer. If using another 16-bits of the descriptor space is not desirable, an
alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits
of the address as a packet offset. Other variations on these approach
are obviously possible too.

Quoted 6 lines
Quoted 1 line

Seems reasonable to me..

I think this is probably the best solution, and also has the advantage that
a buffer retains its base address the full way through the cycle of Rx and Tx.

I like this as well - it also has the advantage that drivers can keep performing adjustments on the handle, which ends up just modifying the offset.

Signature · 1 line
Jonathan
#26 lore raw
In reply to Jonathan Lemon
Quoted 27 lines · Jonathan Lemon
Quoted 20 lines · Kevin
Quoted 2 lines · Jakub Kicinski

I think that's very limiting.? What is the challenge in providing aligned addresses, exactly?

The challenges are two-fold:
1) it prevents using arbitrary buffer sizes, which will be an issue supporting e.g. jumbo frames in future.
2) higher level user-space frameworks which may want to use AF_XDP, such as DPDK, do not currently support having buffers with 'fixed' alignment.
??? The reason that DPDK uses arbitrary placement is that:
??? ??? - it would stop things working on certain NICs which need the actual writable space specified in units of 1k - therefore we need 2k
+ metadata space.
??? ??? - we place padding between buffers to avoid constantly hitting the same memory channels when accessing memory. ??? ??? - it allows the application to choose the actual buffer size it wants to use.
??? We make use of the above to allow us to speed up processing significantly and also reduce the packet buffer memory size.

Quoted 5 lines

??? Not having arbitrary buffer alignment also means an AF_XDP driver for DPDK cannot be a drop-in replacement for existing drivers in those frameworks. Even with a new capability to allow an arbitrary buffer alignment, existing apps will need to be modified to use that new capability.

Quoted 5 lines

Since all buffers in the umem are the same chunk size, the original buffer
address can be recalculated with some multiply/shift math. However, this is
more expensive than just a mask operation.

Yes, we can do this.

Another option we have is to add a socket option for querying the metadata length from the driver (assuming it doesn't vary per packet). We can use that information to get back the original address using subtraction.

Alternatively, we can change the Rx descriptor format to include the metadata length. We could do this in a couple of ways, for example, rather than returning the address at the start of the packet, instead return the buffer address that was passed in, and adding another 16-bit field to specify the start of the packet offset with that buffer. Id using 16-bits of descriptor space is not desirable, an alternative could be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be enough, right :-) ) and use the remaining 16 bits of the address as a packet offset. Other variations on these approaches are obviously possible too.