Skip to contentSkip to conversation

introduce xdp_init_buff/xdp_prepare_buff

PATCH In review Lorenzo Bianconi 8 participants 13 messages Dec 22, 2020 – Jan 18, 2021

Subsystems (25)
Networking Network drivers BPF Intel Ethernet +21 more subsystems
lore
Share & tools
lore

Lorenzo Bianconi

author
#0 lore raw

v5 cover introduce xdp_init_buff/xdp_prepare_buff

Introduce xdp_init_buff and xdp_prepare_buff utility routines to initialize xdp_buff data structure and remove duplicated code in all XDP capable drivers.

Changes since v4:
- fix xdp_init_buff/xdp_prepare_buff (natural order is xdp_init_buff() first and then xdp_prepare_buff())

Changes since v3:
- use __always_inline instead of inline for xdp_init_buff/xdp_prepare_buff
- add 'const bool meta_valid' to xdp_prepare_buff signature to avoid overwriting data_meta with xdp_set_data_meta_invalid()
- introduce removed comment in bnxt driver

Changes since v2:
- precompute xdp->data as hard_start + headroom and save it in a local variable to reuse it for xdp->data_end and xdp->data_meta in xdp_prepare_buff()

Changes since v1:
- introduce xdp_prepare_buff utility routine

Lorenzo Bianconi (2):
net: xdp: introduce xdp_init_buff utility routine
net: xdp: introduce xdp_prepare_buff utility routine

Acked-by Shay Agroskin <[email protected]>
Acked-by Martin Habets <[email protected]>
Acked-by Camelia Groza <[email protected]>
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 10 ++++----
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c |  9 +++----
 .../net/ethernet/cavium/thunder/nicvf_main.c  | 12 +++++-----
 .../net/ethernet/freescale/dpaa/dpaa_eth.c    | 10 ++++----
 .../net/ethernet/freescale/dpaa2/dpaa2-eth.c  | 14 ++++-------
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   | 18 +++++++-------
 drivers/net/ethernet/intel/ice/ice_txrx.c     | 15 ++++++------
 drivers/net/ethernet/intel/igb/igb_main.c     | 18 +++++++-------
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 19 +++++++--------
 .../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 19 +++++++--------
 drivers/net/ethernet/marvell/mvneta.c         | 10 +++-----
 .../net/ethernet/marvell/mvpp2/mvpp2_main.c   | 14 +++++------
 drivers/net/ethernet/mellanox/mlx4/en_rx.c    |  9 +++----
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  8 ++-----
 .../ethernet/netronome/nfp/nfp_net_common.c   | 12 +++++-----
 drivers/net/ethernet/qlogic/qede/qede_fp.c    |  9 +++----
 drivers/net/ethernet/sfc/rx.c                 | 10 +++-----
 drivers/net/ethernet/socionext/netsec.c       |  9 +++----
 drivers/net/ethernet/ti/cpsw.c                | 18 +++++---------
 drivers/net/ethernet/ti/cpsw_new.c            | 18 +++++---------
 drivers/net/hyperv/netvsc_bpf.c               |  8 ++-----
 drivers/net/tun.c                             | 12 ++++------
 drivers/net/veth.c                            | 14 ++++-------
 drivers/net/virtio_net.c                      | 18 +++++---------
 drivers/net/xen-netfront.c                    | 10 ++++----
 include/net/xdp.h                             | 19 +++++++++++++++
 net/bpf/test_run.c                            | 11 ++++-----
 net/core/dev.c                                | 24 +++++++++----------
 28 files changed, 163 insertions(+), 214 deletions(-)
Signature · 1 line
2.29.2
author
#1 lore raw

PATCH v5 1/2 net: xdp: introduce xdp_init_buff utility routine

Introduce xdp_init_buff utility routine to initialize xdp_buff fields const over NAPI iterations (e.g. frame_sz or rxq pointer). Rely on xdp_init_buff in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c        | 3 +--
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c       | 4 ++--
 drivers/net/ethernet/cavium/thunder/nicvf_main.c    | 4 ++--
 drivers/net/ethernet/freescale/dpaa/dpaa_eth.c      | 4 ++--
 drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c    | 8 ++++----
 drivers/net/ethernet/intel/i40e/i40e_txrx.c         | 6 +++---
 drivers/net/ethernet/intel/ice/ice_txrx.c           | 6 +++---
 drivers/net/ethernet/intel/igb/igb_main.c           | 6 +++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c       | 7 +++----
 drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c   | 7 +++----
 drivers/net/ethernet/marvell/mvneta.c               | 3 +--
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c     | 8 +++++---
 drivers/net/ethernet/mellanox/mlx4/en_rx.c          | 3 +--
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c     | 3 +--
 drivers/net/ethernet/netronome/nfp/nfp_net_common.c | 4 ++--
 drivers/net/ethernet/qlogic/qede/qede_fp.c          | 3 +--
 drivers/net/ethernet/sfc/rx.c                       | 3 +--
 drivers/net/ethernet/socionext/netsec.c             | 3 +--
 drivers/net/ethernet/ti/cpsw.c                      | 4 ++--
 drivers/net/ethernet/ti/cpsw_new.c                  | 4 ++--
 drivers/net/hyperv/netvsc_bpf.c                     | 3 +--
 drivers/net/tun.c                                   | 7 +++----
 drivers/net/veth.c                                  | 8 ++++----
 drivers/net/virtio_net.c                            | 6 ++----
 drivers/net/xen-netfront.c                          | 4 ++--
 include/net/xdp.h                                   | 7 +++++++
 net/bpf/test_run.c                                  | 4 ++--
 net/core/dev.c                                      | 8 ++++----
 28 files changed, 68 insertions(+), 72 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +1 -2
index 06596fa1f9fe..43331f6967c9 100644
@@ -1634,8 +1634,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,
 	netif_dbg(rx_ring->adapter, rx_status, rx_ring->netdev,
 		  "%s qid %d\n", __func__, rx_ring->qid);
 	res_budget = budget;
-	xdp.rxq = &rx_ring->xdp_rxq;
-	xdp.frame_sz = ENA_PAGE_SIZE;
+	xdp_init_buff(&xdp, ENA_PAGE_SIZE, &rx_ring->xdp_rxq);
 
 	do {
 		xdp_verdict = XDP_PASS;
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +2 -2
index fcc262064766..ab805d6750e5 100644
@@ -133,12 +133,12 @@ bool bnxt_rx_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons,
 	dma_sync_single_for_cpu(&pdev->dev, mapping + offset, *len, bp->rx_dir);
 
 	txr = rxr->bnapi->tx_ring;
+	/* BNXT_RX_PAGE_MODE(bp) when XDP enabled */
+	xdp_init_buff(&xdp, PAGE_SIZE, &rxr->xdp_rxq);
 	xdp.data_hard_start = *data_ptr - offset;
 	xdp.data = *data_ptr;
 	xdp_set_data_meta_invalid(&xdp);
 	xdp.data_end = *data_ptr + *len;
-	xdp.rxq = &rxr->xdp_rxq;
-	xdp.frame_sz = PAGE_SIZE; /* BNXT_RX_PAGE_MODE(bp) when XDP enabled */
 	orig_data = xdp.data;
 
 	rcu_read_lock();
drivers/net/ethernet/cavium/thunder/nicvf_main.c +2 -2
index f3b7b443f964..9fc672f075f2 100644
@@ -547,12 +547,12 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
 	cpu_addr = (u64)phys_to_virt(cpu_addr);
 	page = virt_to_page((void *)cpu_addr);
 
+	xdp_init_buff(&xdp, RCV_FRAG_LEN + XDP_PACKET_HEADROOM,
+		      &rq->xdp_rxq);
 	xdp.data_hard_start = page_address(page);
 	xdp.data = (void *)cpu_addr;
 	xdp_set_data_meta_invalid(&xdp);
 	xdp.data_end = xdp.data + len;
-	xdp.rxq = &rq->xdp_rxq;
-	xdp.frame_sz = RCV_FRAG_LEN + XDP_PACKET_HEADROOM;
 	orig_data = xdp.data;
 
 	rcu_read_lock();
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +2 -2
index 4360ce4d3fb6..26e20b96fd96 100644
@@ -2532,12 +2532,12 @@ static u32 dpaa_run_xdp(struct dpaa_priv *priv, struct qm_fd *fd, void *vaddr,
 		return XDP_PASS;
 	}
 
+	xdp_init_buff(&xdp, DPAA_BP_RAW_SIZE - DPAA_TX_PRIV_DATA_SIZE,
+		      &dpaa_fq->xdp_rxq);
 	xdp.data = vaddr + fd_off;
 	xdp.data_meta = xdp.data;
 	xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;
 	xdp.data_end = xdp.data + qm_fd_get_length(fd);
-	xdp.frame_sz = DPAA_BP_RAW_SIZE - DPAA_TX_PRIV_DATA_SIZE;
-	xdp.rxq = &dpaa_fq->xdp_rxq;
 
 	/* We reserve a fixed headroom of 256 bytes under the erratum and we
 	 * offer it all to XDP programs to use. If no room is left for the
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +4 -4
index 91cff93dbdae..a4ade0b5adb0 100644
@@ -358,14 +358,14 @@ static u32 dpaa2_eth_run_xdp(struct dpaa2_eth_priv *priv,
 	if (!xdp_prog)
 		goto out;
 
+	xdp_init_buff(&xdp,
+		      DPAA2_ETH_RX_BUF_RAW_SIZE -
+		      (dpaa2_fd_get_offset(fd) - XDP_PACKET_HEADROOM),
+		      &ch->xdp_rxq);
 	xdp.data = vaddr + dpaa2_fd_get_offset(fd);
 	xdp.data_end = xdp.data + dpaa2_fd_get_len(fd);
 	xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;
 	xdp_set_data_meta_invalid(&xdp);
-	xdp.rxq = &ch->xdp_rxq;
-
-	xdp.frame_sz = DPAA2_ETH_RX_BUF_RAW_SIZE -
-		(dpaa2_fd_get_offset(fd) - XDP_PACKET_HEADROOM);
 
 	xdp_act = bpf_prog_run_xdp(xdp_prog, &xdp);
drivers/net/ethernet/intel/i40e/i40e_txrx.c +3 -3
index 4aca637d4a23..a87fb8264d0c 100644
@@ -2344,7 +2344,7 @@ static void i40e_inc_ntc(struct i40e_ring *rx_ring)
  **/
 static int i40e_clean_rx_irq(struct i40e_ring *rx_ring, int budget)
 {
-	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
+	unsigned int total_rx_bytes = 0, total_rx_packets = 0, frame_sz = 0;
 	struct sk_buff *skb = rx_ring->skb;
 	u16 cleaned_count = I40E_DESC_UNUSED(rx_ring);
 	unsigned int xdp_xmit = 0;

@@ -2352,9 +2352,9 @@ static int i40e_clean_rx_irq(struct i40e_ring *rx_ring, int budget)
 	struct xdp_buff xdp;
 
 #if (PAGE_SIZE < 8192)
-	xdp.frame_sz = i40e_rx_frame_truesize(rx_ring, 0);
+	frame_sz = i40e_rx_frame_truesize(rx_ring, 0);
 #endif
-	xdp.rxq = &rx_ring->xdp_rxq;
+	xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq);
 
 	while (likely(total_rx_packets < (unsigned int)budget)) {
 		struct i40e_rx_buffer *rx_buffer;
drivers/net/ethernet/intel/ice/ice_txrx.c +3 -3
index a2d0aad8cfdd..500e93bf6238 100644
@@ -1089,18 +1089,18 @@ ice_is_non_eop(struct ice_ring *rx_ring, union ice_32b_rx_flex_desc *rx_desc,
  */
 int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget)
 {
-	unsigned int total_rx_bytes = 0, total_rx_pkts = 0;
+	unsigned int total_rx_bytes = 0, total_rx_pkts = 0, frame_sz = 0;
 	u16 cleaned_count = ICE_DESC_UNUSED(rx_ring);
 	unsigned int xdp_res, xdp_xmit = 0;
 	struct bpf_prog *xdp_prog = NULL;
 	struct xdp_buff xdp;
 	bool failure;
 
-	xdp.rxq = &rx_ring->xdp_rxq;
 	/* Frame size depend on rx_ring setup when PAGE_SIZE=4K */
 #if (PAGE_SIZE < 8192)
-	xdp.frame_sz = ice_rx_frame_truesize(rx_ring, 0);
+	frame_sz = ice_rx_frame_truesize(rx_ring, 0);
 #endif
+	xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq);
 
 	/* start the loop to process Rx packets bounded by 'budget' */
 	while (likely(total_rx_pkts < (unsigned int)budget)) {
drivers/net/ethernet/intel/igb/igb_main.c +3 -3
index 03f78fdb0dcd..cf9e8b7d2c70 100644
@@ -8681,13 +8681,13 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)
 	u16 cleaned_count = igb_desc_unused(rx_ring);
 	unsigned int xdp_xmit = 0;
 	struct xdp_buff xdp;
-
-	xdp.rxq = &rx_ring->xdp_rxq;
+	u32 frame_sz = 0;
 
 	/* Frame size depend on rx_ring setup when PAGE_SIZE=4K */
 #if (PAGE_SIZE < 8192)
-	xdp.frame_sz = igb_rx_frame_truesize(rx_ring, 0);
+	frame_sz = igb_rx_frame_truesize(rx_ring, 0);
 #endif
+	xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq);
 
 	while (likely(total_packets < budget)) {
 		union e1000_adv_rx_desc *rx_desc;
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3 -4
index 393d1c2cd853..7ec196589a07 100644
@@ -2291,7 +2291,7 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
 			       struct ixgbe_ring *rx_ring,
 			       const int budget)
 {
-	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
+	unsigned int total_rx_bytes = 0, total_rx_packets = 0, frame_sz = 0;
 	struct ixgbe_adapter *adapter = q_vector->adapter;
 #ifdef IXGBE_FCOE
 	int ddp_bytes;

@@ -2301,12 +2301,11 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
 	unsigned int xdp_xmit = 0;
 	struct xdp_buff xdp;
 
-	xdp.rxq = &rx_ring->xdp_rxq;
-
 	/* Frame size depend on rx_ring setup when PAGE_SIZE=4K */
 #if (PAGE_SIZE < 8192)
-	xdp.frame_sz = ixgbe_rx_frame_truesize(rx_ring, 0);
+	frame_sz = ixgbe_rx_frame_truesize(rx_ring, 0);
 #endif
+	xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq);
 
 	while (likely(total_rx_packets < budget)) {
 		union ixgbe_adv_rx_desc *rx_desc;
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +3 -4
index 4061cd7db5dd..624efcd71569 100644
@@ -1121,19 +1121,18 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector,
 				struct ixgbevf_ring *rx_ring,
 				int budget)
 {
-	unsigned int total_rx_bytes = 0, total_rx_packets = 0;
+	unsigned int total_rx_bytes = 0, total_rx_packets = 0, frame_sz = 0;
 	struct ixgbevf_adapter *adapter = q_vector->adapter;
 	u16 cleaned_count = ixgbevf_desc_unused(rx_ring);
 	struct sk_buff *skb = rx_ring->skb;
 	bool xdp_xmit = false;
 	struct xdp_buff xdp;
 
-	xdp.rxq = &rx_ring->xdp_rxq;
-
 	/* Frame size depend on rx_ring setup when PAGE_SIZE=4K */
 #if (PAGE_SIZE < 8192)
-	xdp.frame_sz = ixgbevf_rx_frame_truesize(rx_ring, 0);
+	frame_sz = ixgbevf_rx_frame_truesize(rx_ring, 0);
 #endif
+	xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq);
 
 	while (likely(total_rx_packets < budget)) {
 		struct ixgbevf_rx_buffer *rx_buffer;
drivers/net/ethernet/marvell/mvneta.c +1 -2
index 563ceac3060f..acbb9cb85ada 100644
@@ -2363,9 +2363,8 @@ static int mvneta_rx_swbm(struct napi_struct *napi,
 	u32 desc_status, frame_sz;
 	struct xdp_buff xdp_buf;
 
+	xdp_init_buff(&xdp_buf, PAGE_SIZE, &rxq->xdp_rxq);
 	xdp_buf.data_hard_start = NULL;
-	xdp_buf.frame_sz = PAGE_SIZE;
-	xdp_buf.rxq = &rxq->xdp_rxq;
 
 	sinfo.nr_frags = 0;
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +5 -3
index afdd22827223..ca05dfc05058 100644
@@ -3562,16 +3562,18 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
 			frag_size = bm_pool->frag_size;
 
 		if (xdp_prog) {
+			struct xdp_rxq_info *xdp_rxq;
+
 			xdp.data_hard_start = data;
 			xdp.data = data + MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM;
 			xdp.data_end = xdp.data + rx_bytes;
-			xdp.frame_sz = PAGE_SIZE;
 
 			if (bm_pool->pkt_size == MVPP2_BM_SHORT_PKT_SIZE)
-				xdp.rxq = &rxq->xdp_rxq_short;
+				xdp_rxq = &rxq->xdp_rxq_short;
 			else
-				xdp.rxq = &rxq->xdp_rxq_long;
+				xdp_rxq = &rxq->xdp_rxq_long;
 
+			xdp_init_buff(&xdp, PAGE_SIZE, xdp_rxq);
 			xdp_set_data_meta_invalid(&xdp);
 
 			ret = mvpp2_run_xdp(port, rxq, xdp_prog, &xdp, pp, &ps);
drivers/net/ethernet/mellanox/mlx4/en_rx.c +1 -2
index 7954c1daf2b6..815381b484ca 100644
@@ -682,8 +682,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
 	/* Protect accesses to: ring->xdp_prog, priv->mac_hash list */
 	rcu_read_lock();
 	xdp_prog = rcu_dereference(ring->xdp_prog);
-	xdp.rxq = &ring->xdp_rxq;
-	xdp.frame_sz = priv->frag_info[0].frag_stride;
+	xdp_init_buff(&xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);
 	doorbell_pending = false;
 
 	/* We assume a 1:1 mapping between CQEs and Rx descriptors, so Rx
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +1 -2
index 7f5851c61218..bc7c81f2b036 100644
@@ -1126,12 +1126,11 @@ struct sk_buff *mlx5e_build_linear_skb(struct mlx5e_rq *rq, void *va,
 static void mlx5e_fill_xdp_buff(struct mlx5e_rq *rq, void *va, u16 headroom,
 				u32 len, struct xdp_buff *xdp)
 {
+	xdp_init_buff(xdp, rq->buff.frame0_sz, &rq->xdp_rxq);
 	xdp->data_hard_start = va;
 	xdp->data = va + headroom;
 	xdp_set_data_meta_invalid(xdp);
 	xdp->data_end = xdp->data + len;
-	xdp->rxq = &rq->xdp_rxq;
-	xdp->frame_sz = rq->buff.frame0_sz;
 }
 
 static struct sk_buff *
drivers/net/ethernet/netronome/nfp/nfp_net_common.c +2 -2
index f21fb573ea3e..f66be67e8b40 100644
@@ -1822,8 +1822,8 @@ static int nfp_net_rx(struct nfp_net_rx_ring *rx_ring, int budget)
 	rcu_read_lock();
 	xdp_prog = READ_ONCE(dp->xdp_prog);
 	true_bufsz = xdp_prog ? PAGE_SIZE : dp->fl_bufsz;
-	xdp.frame_sz = PAGE_SIZE - NFP_NET_RX_BUF_HEADROOM;
-	xdp.rxq = &rx_ring->xdp_rxq;
+	xdp_init_buff(&xdp, PAGE_SIZE - NFP_NET_RX_BUF_HEADROOM,
+		      &rx_ring->xdp_rxq);
 	tx_ring = r_vec->xdp_ring;
 
 	while (pkts_polled < budget) {
drivers/net/ethernet/qlogic/qede/qede_fp.c +1 -2
index a2494bf85007..d40220043883 100644
@@ -1090,12 +1090,11 @@ static bool qede_rx_xdp(struct qede_dev *edev,
 	struct xdp_buff xdp;
 	enum xdp_action act;
 
+	xdp_init_buff(&xdp, rxq->rx_buf_seg_size, &rxq->xdp_rxq);
 	xdp.data_hard_start = page_address(bd->data);
 	xdp.data = xdp.data_hard_start + *data_offset;
 	xdp_set_data_meta_invalid(&xdp);
 	xdp.data_end = xdp.data + *len;
-	xdp.rxq = &rxq->xdp_rxq;
-	xdp.frame_sz = rxq->rx_buf_seg_size; /* PAGE_SIZE when XDP enabled */
 
 	/* Queues always have a full reset currently, so for the time
 	 * being until there's atomic program replace just mark read
drivers/net/ethernet/sfc/rx.c +1 -2
index aaa112877561..eaa6650955d1 100644
@@ -293,14 +293,13 @@ static bool efx_do_xdp(struct efx_nic *efx, struct efx_channel *channel,
 	memcpy(rx_prefix, *ehp - efx->rx_prefix_size,
 	       efx->rx_prefix_size);
 
+	xdp_init_buff(&xdp, efx->rx_page_buf_step, &rx_queue->xdp_rxq_info);
 	xdp.data = *ehp;
 	xdp.data_hard_start = xdp.data - EFX_XDP_HEADROOM;
 
 	/* No support yet for XDP metadata */
 	xdp_set_data_meta_invalid(&xdp);
 	xdp.data_end = xdp.data + rx_buf->len;
-	xdp.rxq = &rx_queue->xdp_rxq_info;
-	xdp.frame_sz = efx->rx_page_buf_step;
 
 	xdp_act = bpf_prog_run_xdp(xdp_prog, &xdp);
 	rcu_read_unlock();
drivers/net/ethernet/socionext/netsec.c +1 -2
index 19d20a6d0d44..945ca9517bf9 100644
@@ -956,8 +956,7 @@ static int netsec_process_rx(struct netsec_priv *priv, int budget)
 	u32 xdp_act = 0;
 	int done = 0;
 
-	xdp.rxq = &dring->xdp_rxq;
-	xdp.frame_sz = PAGE_SIZE;
+	xdp_init_buff(&xdp, PAGE_SIZE, &dring->xdp_rxq);
 
 	rcu_read_lock();
 	xdp_prog = READ_ONCE(priv->xdp_prog);
drivers/net/ethernet/ti/cpsw.c +2 -2
index b0f00b4edd94..78a923391828 100644
@@ -392,6 +392,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
 	}
 
 	if (priv->xdp_prog) {
+		xdp_init_buff(&xdp, PAGE_SIZE, &priv->xdp_rxq[ch]);
+
 		if (status & CPDMA_RX_VLAN_ENCAP) {
 			xdp.data = pa + CPSW_HEADROOM +
 				   CPSW_RX_VLAN_ENCAP_HDR_SIZE;

@@ -405,8 +407,6 @@ static void cpsw_rx_handler(void *token, int len, int status)
 		xdp_set_data_meta_invalid(&xdp);
 
 		xdp.data_hard_start = pa;
-		xdp.rxq = &priv->xdp_rxq[ch];
-		xdp.frame_sz = PAGE_SIZE;
 
 		port = priv->emac_port + cpsw->data.dual_emac;
 		ret = cpsw_run_xdp(priv, ch, &xdp, page, port);
drivers/net/ethernet/ti/cpsw_new.c +2 -2
index 2f5e0ad23ad7..1b3385ec9645 100644
@@ -335,6 +335,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
 	}
 
 	if (priv->xdp_prog) {
+		xdp_init_buff(&xdp, PAGE_SIZE, &priv->xdp_rxq[ch]);
+
 		if (status & CPDMA_RX_VLAN_ENCAP) {
 			xdp.data = pa + CPSW_HEADROOM +
 				   CPSW_RX_VLAN_ENCAP_HDR_SIZE;

@@ -348,8 +350,6 @@ static void cpsw_rx_handler(void *token, int len, int status)
 		xdp_set_data_meta_invalid(&xdp);
 
 		xdp.data_hard_start = pa;
-		xdp.rxq = &priv->xdp_rxq[ch];
-		xdp.frame_sz = PAGE_SIZE;
 
 		ret = cpsw_run_xdp(priv, ch, &xdp, page, priv->emac_port);
 		if (ret != CPSW_XDP_PASS)
drivers/net/hyperv/netvsc_bpf.c +1 -2
index 440486d9c999..14a7ee4c6899 100644
@@ -44,12 +44,11 @@ u32 netvsc_run_xdp(struct net_device *ndev, struct netvsc_channel *nvchan,
 		goto out;
 	}
 
+	xdp_init_buff(xdp, PAGE_SIZE, &nvchan->xdp_rxq);
 	xdp->data_hard_start = page_address(page);
 	xdp->data = xdp->data_hard_start + NETVSC_XDP_HDRM;
 	xdp_set_data_meta_invalid(xdp);
 	xdp->data_end = xdp->data + len;
-	xdp->rxq = &nvchan->xdp_rxq;
-	xdp->frame_sz = PAGE_SIZE;
 
 	memcpy(xdp->data, data, len);
drivers/net/tun.c +3 -4
index fbed05ae7b0f..a82f7823d428 100644
@@ -1599,12 +1599,11 @@ static struct sk_buff *tun_build_skb(struct tun_struct *tun,
 		struct xdp_buff xdp;
 		u32 act;
 
+		xdp_init_buff(&xdp, buflen, &tfile->xdp_rxq);
 		xdp.data_hard_start = buf;
 		xdp.data = buf + pad;
 		xdp_set_data_meta_invalid(&xdp);
 		xdp.data_end = xdp.data + len;
-		xdp.rxq = &tfile->xdp_rxq;
-		xdp.frame_sz = buflen;
 
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		if (act == XDP_REDIRECT || act == XDP_TX) {

@@ -2344,9 +2343,9 @@ static int tun_xdp_one(struct tun_struct *tun,
 			skb_xdp = true;
 			goto build;
 		}
+
+		xdp_init_buff(xdp, buflen, &tfile->xdp_rxq);
 		xdp_set_data_meta_invalid(xdp);
-		xdp->rxq = &tfile->xdp_rxq;
-		xdp->frame_sz = buflen;
 
 		act = bpf_prog_run_xdp(xdp_prog, xdp);
 		err = tun_xdp_act(tun, xdp_prog, xdp, act);
drivers/net/veth.c +4 -4
index dbacf90df2b5..6a0afce0253c 100644
@@ -642,7 +642,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
 					struct veth_xdp_tx_bq *bq,
 					struct veth_stats *stats)
 {
-	u32 pktlen, headroom, act, metalen;
+	u32 pktlen, headroom, act, metalen, frame_sz;
 	void *orig_data, *orig_data_end;
 	struct bpf_prog *xdp_prog;
 	int mac_len, delta, off;

@@ -702,11 +702,11 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
 	xdp.data = skb_mac_header(skb);
 	xdp.data_end = xdp.data + pktlen;
 	xdp.data_meta = xdp.data;
-	xdp.rxq = &rq->xdp_rxq;
 
 	/* SKB "head" area always have tailroom for skb_shared_info */
-	xdp.frame_sz = (void *)skb_end_pointer(skb) - xdp.data_hard_start;
-	xdp.frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	frame_sz = (void *)skb_end_pointer(skb) - xdp.data_hard_start;
+	frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	xdp_init_buff(&xdp, frame_sz, &rq->xdp_rxq);
 
 	orig_data = xdp.data;
 	orig_data_end = xdp.data_end;
drivers/net/virtio_net.c +2 -4
index 052975ea0af4..a22ce87bcd9c 100644
@@ -689,12 +689,11 @@ static struct sk_buff *receive_small(struct net_device *dev,
 			page = xdp_page;
 		}
 
+		xdp_init_buff(&xdp, buflen, &rq->xdp_rxq);
 		xdp.data_hard_start = buf + VIRTNET_RX_PAD + vi->hdr_len;
 		xdp.data = xdp.data_hard_start + xdp_headroom;
 		xdp.data_end = xdp.data + len;
 		xdp.data_meta = xdp.data;
-		xdp.rxq = &rq->xdp_rxq;
-		xdp.frame_sz = buflen;
 		orig_data = xdp.data;
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		stats->xdp_packets++;

@@ -859,12 +858,11 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 		 * the descriptor on if we get an XDP_TX return code.
 		 */
 		data = page_address(xdp_page) + offset;
+		xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
 		xdp.data_hard_start = data - VIRTIO_XDP_HEADROOM + vi->hdr_len;
 		xdp.data = data + vi->hdr_len;
 		xdp.data_end = xdp.data + (len - vi->hdr_len);
 		xdp.data_meta = xdp.data;
-		xdp.rxq = &rq->xdp_rxq;
-		xdp.frame_sz = frame_sz - vi->hdr_len;
 
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		stats->xdp_packets++;
drivers/net/xen-netfront.c +2 -2
index b01848ef4649..329397c60d84 100644
@@ -864,12 +864,12 @@ static u32 xennet_run_xdp(struct netfront_queue *queue, struct page *pdata,
 	u32 act;
 	int err;
 
+	xdp_init_buff(xdp, XEN_PAGE_SIZE - XDP_PACKET_HEADROOM,
+		      &queue->xdp_rxq);
 	xdp->data_hard_start = page_address(pdata);
 	xdp->data = xdp->data_hard_start + XDP_PACKET_HEADROOM;
 	xdp_set_data_meta_invalid(xdp);
 	xdp->data_end = xdp->data + len;
-	xdp->rxq = &queue->xdp_rxq;
-	xdp->frame_sz = XEN_PAGE_SIZE - XDP_PACKET_HEADROOM;
 
 	act = bpf_prog_run_xdp(prog, xdp);
 	switch (act) {
include/net/xdp.h +7 -0
index 11ec93f827c0..323340caef88 100644
@@ -76,6 +76,13 @@ struct xdp_buff {
 	u32 frame_sz; /* frame size to deduce data_hard_end/reserved tailroom*/
 };
 
+static __always_inline void
+xdp_init_buff(struct xdp_buff *xdp, u32 frame_sz, struct xdp_rxq_info *rxq)
+{
+	xdp->frame_sz = frame_sz;
+	xdp->rxq = rxq;
+}
+
 /* Reserve memory area at end-of data area.
  *
  * This macro reserves tailroom in the XDP buffer by limiting the
net/bpf/test_run.c +2 -2
index c1c30a9f76f3..a8fa5a9e4137 100644
@@ -640,10 +640,10 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
 	xdp.data = data + headroom;
 	xdp.data_meta = xdp.data;
 	xdp.data_end = xdp.data + size;
-	xdp.frame_sz = headroom + max_data_sz + tailroom;
 
 	rxqueue = __netif_get_rx_queue(current->nsproxy->net_ns->loopback_dev, 0);
-	xdp.rxq = &rxqueue->xdp_rxq;
+	xdp_init_buff(&xdp, headroom + max_data_sz + tailroom,
+		      &rxqueue->xdp_rxq);
 	bpf_prog_change_xdp(NULL, prog);
 	ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, true);
 	if (ret)
net/core/dev.c +4 -4
index a46334906c94..b1a765900c01 100644
@@ -4588,11 +4588,11 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 	struct netdev_rx_queue *rxqueue;
 	void *orig_data, *orig_data_end;
 	u32 metalen, act = XDP_DROP;
+	u32 mac_len, frame_sz;
 	__be16 orig_eth_type;
 	struct ethhdr *eth;
 	bool orig_bcast;
 	int hlen, off;
-	u32 mac_len;
 
 	/* Reinjected packets coming from act_mirred or similar should
 	 * not get XDP generic processing.

@@ -4631,8 +4631,8 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 	xdp->data_hard_start = skb->data - skb_headroom(skb);
 
 	/* SKB "head" area always have tailroom for skb_shared_info */
-	xdp->frame_sz  = (void *)skb_end_pointer(skb) - xdp->data_hard_start;
-	xdp->frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	frame_sz = (void *)skb_end_pointer(skb) - xdp->data_hard_start;
+	frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 
 	orig_data_end = xdp->data_end;
 	orig_data = xdp->data;

@@ -4641,7 +4641,7 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 	orig_eth_type = eth->h_proto;
 
 	rxqueue = netif_get_rxqueue(skb);
-	xdp->rxq = &rxqueue->xdp_rxq;
+	xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq);
 
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
Signature · 1 line
2.29.2
Acked-by on v5 1/2
#2 lore raw

Re: [PATCH v5 bpf-next 1/2] net: xdp: introduce xdp_init_buff utility routine

Quoted 27 lines · On Tue, 22 Dec 2020 22:09:28 +0100 Lorenzo Bianconi <[email protected]> wrote:

Introduce xdp_init_buff utility routine to initialize xdp_buff fields const over NAPI iterations (e.g. frame_sz or rxq pointer). Rely on xdp_init_buff in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c        | 3 +--
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c       | 4 ++--
 drivers/net/ethernet/cavium/thunder/nicvf_main.c    | 4 ++--
 drivers/net/ethernet/freescale/dpaa/dpaa_eth.c      | 4 ++--
 drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c    | 8 ++++----
 drivers/net/ethernet/intel/i40e/i40e_txrx.c         | 6 +++---
 drivers/net/ethernet/intel/ice/ice_txrx.c           | 6 +++---
 drivers/net/ethernet/intel/igb/igb_main.c           | 6 +++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c       | 7 +++----
 drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c   | 7 +++----
 drivers/net/ethernet/marvell/mvneta.c               | 3 +--
 drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c     | 8 +++++---
 drivers/net/ethernet/mellanox/mlx4/en_rx.c          | 3 +--
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c     | 3 +--
 drivers/net/ethernet/netronome/nfp/nfp_net_common.c | 4 ++--
 drivers/net/ethernet/qlogic/qede/qede_fp.c          | 3 +--
 drivers/net/ethernet/sfc/rx.c                       | 3 +--
 drivers/net/ethernet/socionext/netsec.c             | 3 +--
 drivers/net/ethernet/ti/cpsw.c                      | 4 ++--
 drivers/net/ethernet/ti/cpsw_new.c                  | 4 ++--
 drivers/net/hyperv/netvsc_bpf.c                     | 3 +--
Quoted 8 lines
 drivers/net/tun.c                                   | 7 +++----
 drivers/net/veth.c                                  | 8 ++++----
 drivers/net/virtio_net.c                            | 6 ++----
 drivers/net/xen-netfront.c                          | 4 ++--
 include/net/xdp.h                                   | 7 +++++++
 net/bpf/test_run.c                                  | 4 ++--
 net/core/dev.c                                      | 8 ++++----
 28 files changed, 68 insertions(+), 72 deletions(-)
Acked-by Jesper Dangaard Brouer <[email protected]>
Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
maintainer Acked-by on v5 1/2
#3 lore raw

RE: [PATCH v5 bpf-next 1/2] net: xdp: introduce xdp_init_buff utility routine

In reply to Lorenzo Bianconi
Quoted 6 lines · Lorenzo Bianconi

Introduce xdp_init_buff utility routine to initialize xdp_buff fields const over NAPI iterations (e.g. frame_sz or rxq pointer). Rely on xdp_init_buff in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>

---

Seems like a nice bit of cleanup. I spot checked math in a few drivers LGTM.

Acked-by John Fastabend <[email protected]>
author
#4 lore raw

PATCH v5 2/2 net: xdp: introduce xdp_prepare_buff utility routine

In reply to Lorenzo Bianconi

Introduce xdp_prepare_buff utility routine to initialize per-descriptor xdp_buff fields (e.g. xdp_buff pointers). Rely on xdp_prepare_buff() in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  7 +++----
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c |  5 +----
 .../net/ethernet/cavium/thunder/nicvf_main.c  |  8 ++++----
 .../net/ethernet/freescale/dpaa/dpaa_eth.c    |  6 ++----
 .../net/ethernet/freescale/dpaa2/dpaa2-eth.c  | 14 +++++--------
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   | 12 +++++------
 drivers/net/ethernet/intel/ice/ice_txrx.c     |  9 +++++----
 drivers/net/ethernet/intel/igb/igb_main.c     | 12 +++++------
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++------
 .../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 12 +++++------
 drivers/net/ethernet/marvell/mvneta.c         |  7 ++-----
 .../net/ethernet/marvell/mvpp2/mvpp2_main.c   |  8 +++-----
 drivers/net/ethernet/mellanox/mlx4/en_rx.c    |  6 ++----
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  5 +----
 .../ethernet/netronome/nfp/nfp_net_common.c   |  8 ++++----
 drivers/net/ethernet/qlogic/qede/qede_fp.c    |  6 ++----
 drivers/net/ethernet/sfc/rx.c                 |  7 ++-----
 drivers/net/ethernet/socionext/netsec.c       |  6 ++----
 drivers/net/ethernet/ti/cpsw.c                | 16 +++++----------
 drivers/net/ethernet/ti/cpsw_new.c            | 16 +++++----------
 drivers/net/hyperv/netvsc_bpf.c               |  5 +----
 drivers/net/tun.c                             |  5 +----
 drivers/net/veth.c                            |  8 ++------
 drivers/net/virtio_net.c                      | 12 ++++-------
 drivers/net/xen-netfront.c                    |  6 ++----
 include/net/xdp.h                             | 12 +++++++++++
 net/bpf/test_run.c                            |  7 ++-----
 net/core/dev.c                                | 20 +++++++++----------
 28 files changed, 105 insertions(+), 152 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +3 -4
index 43331f6967c9..1db6cfd2b55c 100644
@@ -1585,10 +1585,9 @@ static int ena_xdp_handle_buff(struct ena_ring *rx_ring, struct xdp_buff *xdp)
 	int ret;
 
 	rx_info = &rx_ring->rx_buffer_info[rx_ring->ena_bufs[0].req_id];
-	xdp->data = page_address(rx_info->page) + rx_info->page_offset;
-	xdp_set_data_meta_invalid(xdp);
-	xdp->data_hard_start = page_address(rx_info->page);
-	xdp->data_end = xdp->data + rx_ring->ena_bufs[0].len;
+	xdp_prepare_buff(xdp, page_address(rx_info->page),
+			 rx_info->page_offset,
+			 rx_ring->ena_bufs[0].len, false);
 	/* If for some reason we received a bigger packet than
 	 * we expect, then we simply drop it
 	 */
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +1 -4
index ab805d6750e5..641303894341 100644
@@ -135,10 +135,7 @@ bool bnxt_rx_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons,
 	txr = rxr->bnapi->tx_ring;
 	/* BNXT_RX_PAGE_MODE(bp) when XDP enabled */
 	xdp_init_buff(&xdp, PAGE_SIZE, &rxr->xdp_rxq);
-	xdp.data_hard_start = *data_ptr - offset;
-	xdp.data = *data_ptr;
-	xdp_set_data_meta_invalid(&xdp);
-	xdp.data_end = *data_ptr + *len;
+	xdp_prepare_buff(&xdp, *data_ptr - offset, offset, *len, false);
 	orig_data = xdp.data;
 
 	rcu_read_lock();
drivers/net/ethernet/cavium/thunder/nicvf_main.c +4 -4
index 9fc672f075f2..c33b4e837515 100644
@@ -530,6 +530,7 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
 				struct cqe_rx_t *cqe_rx, struct snd_queue *sq,
 				struct rcv_queue *rq, struct sk_buff **skb)
 {
+	unsigned char *hard_start, *data;
 	struct xdp_buff xdp;
 	struct page *page;
 	u32 action;

@@ -549,10 +550,9 @@ static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
 
 	xdp_init_buff(&xdp, RCV_FRAG_LEN + XDP_PACKET_HEADROOM,
 		      &rq->xdp_rxq);
-	xdp.data_hard_start = page_address(page);
-	xdp.data = (void *)cpu_addr;
-	xdp_set_data_meta_invalid(&xdp);
-	xdp.data_end = xdp.data + len;
+	hard_start = page_address(page);
+	data = (unsigned char *)cpu_addr;
+	xdp_prepare_buff(&xdp, hard_start, data - hard_start, len, false);
 	orig_data = xdp.data;
 
 	rcu_read_lock();
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +2 -4
index 26e20b96fd96..d8e568f6caf3 100644
@@ -2534,10 +2534,8 @@ static u32 dpaa_run_xdp(struct dpaa_priv *priv, struct qm_fd *fd, void *vaddr,
 
 	xdp_init_buff(&xdp, DPAA_BP_RAW_SIZE - DPAA_TX_PRIV_DATA_SIZE,
 		      &dpaa_fq->xdp_rxq);
-	xdp.data = vaddr + fd_off;
-	xdp.data_meta = xdp.data;
-	xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;
-	xdp.data_end = xdp.data + qm_fd_get_length(fd);
+	xdp_prepare_buff(&xdp, vaddr + fd_off - XDP_PACKET_HEADROOM,
+			 XDP_PACKET_HEADROOM, qm_fd_get_length(fd), true);
 
 	/* We reserve a fixed headroom of 256 bytes under the erratum and we
 	 * offer it all to XDP programs to use. If no room is left for the
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +5 -9
index a4ade0b5adb0..ca5b9be1c345 100644
@@ -350,7 +350,7 @@ static u32 dpaa2_eth_run_xdp(struct dpaa2_eth_priv *priv,
 	struct bpf_prog *xdp_prog;
 	struct xdp_buff xdp;
 	u32 xdp_act = XDP_PASS;
-	int err;
+	int err, offset;
 
 	rcu_read_lock();
 

@@ -358,14 +358,10 @@ static u32 dpaa2_eth_run_xdp(struct dpaa2_eth_priv *priv,
 	if (!xdp_prog)
 		goto out;
 
-	xdp_init_buff(&xdp,
-		      DPAA2_ETH_RX_BUF_RAW_SIZE -
-		      (dpaa2_fd_get_offset(fd) - XDP_PACKET_HEADROOM),
-		      &ch->xdp_rxq);
-	xdp.data = vaddr + dpaa2_fd_get_offset(fd);
-	xdp.data_end = xdp.data + dpaa2_fd_get_len(fd);
-	xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;
-	xdp_set_data_meta_invalid(&xdp);
+	offset = dpaa2_fd_get_offset(fd) - XDP_PACKET_HEADROOM;
+	xdp_init_buff(&xdp, DPAA2_ETH_RX_BUF_RAW_SIZE - offset, &ch->xdp_rxq);
+	xdp_prepare_buff(&xdp, vaddr + offset, XDP_PACKET_HEADROOM,
+			 dpaa2_fd_get_len(fd), false);
 
 	xdp_act = bpf_prog_run_xdp(xdp_prog, &xdp);
drivers/net/ethernet/intel/i40e/i40e_txrx.c +6 -6
index a87fb8264d0c..2574e78f7597 100644
@@ -2406,12 +2406,12 @@ static int i40e_clean_rx_irq(struct i40e_ring *rx_ring, int budget)
 
 		/* retrieve a buffer from the ring */
 		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      i40e_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = i40e_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
 #if (PAGE_SIZE > 4096)
 			/* At larger PAGE_SIZE, frame_sz depend on len size */
 			xdp.frame_sz = i40e_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/ice/ice_txrx.c +5 -4
index 500e93bf6238..422f53997c02 100644
@@ -1104,8 +1104,10 @@ int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget)
 
 	/* start the loop to process Rx packets bounded by 'budget' */
 	while (likely(total_rx_pkts < (unsigned int)budget)) {
+		unsigned int offset = ice_rx_offset(rx_ring);
 		union ice_32b_rx_flex_desc *rx_desc;
 		struct ice_rx_buf *rx_buf;
+		unsigned char *hard_start;
 		struct sk_buff *skb;
 		unsigned int size;
 		u16 stat_err_bits;

@@ -1151,10 +1153,9 @@ int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget)
 			goto construct_skb;
 		}
 
-		xdp.data = page_address(rx_buf->page) + rx_buf->page_offset;
-		xdp.data_hard_start = xdp.data - ice_rx_offset(rx_ring);
-		xdp.data_meta = xdp.data;
-		xdp.data_end = xdp.data + size;
+		hard_start = page_address(rx_buf->page) + rx_buf->page_offset -
+			     offset;
+		xdp_prepare_buff(&xdp, hard_start, offset, size, true);
 #if (PAGE_SIZE > 4096)
 		/* At larger PAGE_SIZE, frame_sz depend on len size */
 		xdp.frame_sz = ice_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/igb/igb_main.c +6 -6
index cf9e8b7d2c70..6b5adbd9660b 100644
@@ -8715,12 +8715,12 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)
 
 		/* retrieve a buffer from the ring */
 		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      igb_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = igb_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
 #if (PAGE_SIZE > 4096)
 			/* At larger PAGE_SIZE, frame_sz depend on len size */
 			xdp.frame_sz = igb_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +6 -6
index 7ec196589a07..e714db51701a 100644
@@ -2335,12 +2335,12 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
 
 		/* retrieve a buffer from the ring */
 		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      ixgbe_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = ixgbe_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
 #if (PAGE_SIZE > 4096)
 			/* At larger PAGE_SIZE, frame_sz depend on len size */
 			xdp.frame_sz = ixgbe_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +6 -6
index 624efcd71569..a534a3fb392e 100644
@@ -1160,12 +1160,12 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector,
 
 		/* retrieve a buffer from the ring */
 		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      ixgbevf_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = ixgbevf_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
 #if (PAGE_SIZE > 4096)
 			/* At larger PAGE_SIZE, frame_sz depend on len size */
 			xdp.frame_sz = ixgbevf_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/marvell/mvneta.c +2 -5
index acbb9cb85ada..d6da44b9c7ea 100644
@@ -2263,11 +2263,8 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp,
 
 	/* Prefetch header */
 	prefetch(data);
-
-	xdp->data_hard_start = data;
-	xdp->data = data + pp->rx_offset_correction + MVNETA_MH_SIZE;
-	xdp->data_end = xdp->data + data_len;
-	xdp_set_data_meta_invalid(xdp);
+	xdp_prepare_buff(xdp, data, pp->rx_offset_correction + MVNETA_MH_SIZE,
+			 data_len, false);
 
 	sinfo = xdp_get_shared_info_from_buff(xdp);
 	sinfo->nr_frags = 0;
drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +3 -5
index ca05dfc05058..a76b0adb01c6 100644
@@ -3564,17 +3564,15 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
 		if (xdp_prog) {
 			struct xdp_rxq_info *xdp_rxq;
 
-			xdp.data_hard_start = data;
-			xdp.data = data + MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM;
-			xdp.data_end = xdp.data + rx_bytes;
-
 			if (bm_pool->pkt_size == MVPP2_BM_SHORT_PKT_SIZE)
 				xdp_rxq = &rxq->xdp_rxq_short;
 			else
 				xdp_rxq = &rxq->xdp_rxq_long;
 
 			xdp_init_buff(&xdp, PAGE_SIZE, xdp_rxq);
-			xdp_set_data_meta_invalid(&xdp);
+			xdp_prepare_buff(&xdp, data,
+					 MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM,
+					 rx_bytes, false);
 
 			ret = mvpp2_run_xdp(port, rxq, xdp_prog, &xdp, pp, &ps);
drivers/net/ethernet/mellanox/mlx4/en_rx.c +2 -4
index 815381b484ca..63ff85ae0371 100644
@@ -776,10 +776,8 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
 						priv->frag_info[0].frag_size,
 						DMA_FROM_DEVICE);
 
-			xdp.data_hard_start = va - frags[0].page_offset;
-			xdp.data = va;
-			xdp_set_data_meta_invalid(&xdp);
-			xdp.data_end = xdp.data + length;
+			xdp_prepare_buff(&xdp, va - frags[0].page_offset,
+					 frags[0].page_offset, length, false);
 			orig_data = xdp.data;
 
 			act = bpf_prog_run_xdp(xdp_prog, &xdp);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +1 -4
index bc7c81f2b036..a63ce7c8b98f 100644
@@ -1127,10 +1127,7 @@ static void mlx5e_fill_xdp_buff(struct mlx5e_rq *rq, void *va, u16 headroom,
 				u32 len, struct xdp_buff *xdp)
 {
 	xdp_init_buff(xdp, rq->buff.frame0_sz, &rq->xdp_rxq);
-	xdp->data_hard_start = va;
-	xdp->data = va + headroom;
-	xdp_set_data_meta_invalid(xdp);
-	xdp->data_end = xdp->data + len;
+	xdp_prepare_buff(xdp, va, headroom, len, false);
 }
 
 static struct sk_buff *
drivers/net/ethernet/netronome/nfp/nfp_net_common.c +4 -4
index f66be67e8b40..c56889601049 100644
@@ -1914,10 +1914,10 @@ static int nfp_net_rx(struct nfp_net_rx_ring *rx_ring, int budget)
 			unsigned int dma_off;
 			int act;
 
-			xdp.data_hard_start = rxbuf->frag + NFP_NET_RX_BUF_HEADROOM;
-			xdp.data = orig_data;
-			xdp.data_meta = orig_data;
-			xdp.data_end = orig_data + pkt_len;
+			xdp_prepare_buff(&xdp,
+					 rxbuf->frag + NFP_NET_RX_BUF_HEADROOM,
+					 pkt_off - NFP_NET_RX_BUF_HEADROOM,
+					 pkt_len, true);
 
 			act = bpf_prog_run_xdp(xdp_prog, &xdp);
drivers/net/ethernet/qlogic/qede/qede_fp.c +2 -4
index d40220043883..76ef9879db17 100644
@@ -1091,10 +1091,8 @@ static bool qede_rx_xdp(struct qede_dev *edev,
 	enum xdp_action act;
 
 	xdp_init_buff(&xdp, rxq->rx_buf_seg_size, &rxq->xdp_rxq);
-	xdp.data_hard_start = page_address(bd->data);
-	xdp.data = xdp.data_hard_start + *data_offset;
-	xdp_set_data_meta_invalid(&xdp);
-	xdp.data_end = xdp.data + *len;
+	xdp_prepare_buff(&xdp, page_address(bd->data), *data_offset,
+			 *len, false);
 
 	/* Queues always have a full reset currently, so for the time
 	 * being until there's atomic program replace just mark read
drivers/net/ethernet/sfc/rx.c +2 -5
index eaa6650955d1..89c5c75f479f 100644
@@ -294,12 +294,9 @@ static bool efx_do_xdp(struct efx_nic *efx, struct efx_channel *channel,
 	       efx->rx_prefix_size);
 
 	xdp_init_buff(&xdp, efx->rx_page_buf_step, &rx_queue->xdp_rxq_info);
-	xdp.data = *ehp;
-	xdp.data_hard_start = xdp.data - EFX_XDP_HEADROOM;
-
 	/* No support yet for XDP metadata */
-	xdp_set_data_meta_invalid(&xdp);
-	xdp.data_end = xdp.data + rx_buf->len;
+	xdp_prepare_buff(&xdp, *ehp - EFX_XDP_HEADROOM, EFX_XDP_HEADROOM,
+			 rx_buf->len, false);
 
 	xdp_act = bpf_prog_run_xdp(xdp_prog, &xdp);
 	rcu_read_unlock();
drivers/net/ethernet/socionext/netsec.c +2 -4
index 945ca9517bf9..3c53051bdacf 100644
@@ -1015,10 +1015,8 @@ static int netsec_process_rx(struct netsec_priv *priv, int budget)
 					dma_dir);
 		prefetch(desc->addr);
 
-		xdp.data_hard_start = desc->addr;
-		xdp.data = desc->addr + NETSEC_RXBUF_HEADROOM;
-		xdp_set_data_meta_invalid(&xdp);
-		xdp.data_end = xdp.data + pkt_len;
+		xdp_prepare_buff(&xdp, desc->addr, NETSEC_RXBUF_HEADROOM,
+				 pkt_len, false);
 
 		if (xdp_prog) {
 			xdp_result = netsec_run_xdp(priv, xdp_prog, &xdp);
drivers/net/ethernet/ti/cpsw.c +5 -11
index 78a923391828..5239318e9686 100644
@@ -392,21 +392,15 @@ static void cpsw_rx_handler(void *token, int len, int status)
 	}
 
 	if (priv->xdp_prog) {
-		xdp_init_buff(&xdp, PAGE_SIZE, &priv->xdp_rxq[ch]);
+		int headroom = CPSW_HEADROOM, size = len;
 
+		xdp_init_buff(&xdp, PAGE_SIZE, &priv->xdp_rxq[ch]);
 		if (status & CPDMA_RX_VLAN_ENCAP) {
-			xdp.data = pa + CPSW_HEADROOM +
-				   CPSW_RX_VLAN_ENCAP_HDR_SIZE;
-			xdp.data_end = xdp.data + len -
-				       CPSW_RX_VLAN_ENCAP_HDR_SIZE;
-		} else {
-			xdp.data = pa + CPSW_HEADROOM;
-			xdp.data_end = xdp.data + len;
+			headroom += CPSW_RX_VLAN_ENCAP_HDR_SIZE;
+			size -= CPSW_RX_VLAN_ENCAP_HDR_SIZE;
 		}
 
-		xdp_set_data_meta_invalid(&xdp);
-
-		xdp.data_hard_start = pa;
+		xdp_prepare_buff(&xdp, pa, headroom, size, false);
 
 		port = priv->emac_port + cpsw->data.dual_emac;
 		ret = cpsw_run_xdp(priv, ch, &xdp, page, port);
drivers/net/ethernet/ti/cpsw_new.c +5 -11
index 1b3385ec9645..94747f82c60b 100644
@@ -335,21 +335,15 @@ static void cpsw_rx_handler(void *token, int len, int status)
 	}
 
 	if (priv->xdp_prog) {
-		xdp_init_buff(&xdp, PAGE_SIZE, &priv->xdp_rxq[ch]);
+		int headroom = CPSW_HEADROOM, size = len;
 
+		xdp_init_buff(&xdp, PAGE_SIZE, &priv->xdp_rxq[ch]);
 		if (status & CPDMA_RX_VLAN_ENCAP) {
-			xdp.data = pa + CPSW_HEADROOM +
-				   CPSW_RX_VLAN_ENCAP_HDR_SIZE;
-			xdp.data_end = xdp.data + len -
-				       CPSW_RX_VLAN_ENCAP_HDR_SIZE;
-		} else {
-			xdp.data = pa + CPSW_HEADROOM;
-			xdp.data_end = xdp.data + len;
+			headroom += CPSW_RX_VLAN_ENCAP_HDR_SIZE;
+			size -= CPSW_RX_VLAN_ENCAP_HDR_SIZE;
 		}
 
-		xdp_set_data_meta_invalid(&xdp);
-
-		xdp.data_hard_start = pa;
+		xdp_prepare_buff(&xdp, pa, headroom, size, false);
 
 		ret = cpsw_run_xdp(priv, ch, &xdp, page, priv->emac_port);
 		if (ret != CPSW_XDP_PASS)
drivers/net/hyperv/netvsc_bpf.c +1 -4
index 14a7ee4c6899..d60dcf6c9829 100644
@@ -45,10 +45,7 @@ u32 netvsc_run_xdp(struct net_device *ndev, struct netvsc_channel *nvchan,
 	}
 
 	xdp_init_buff(xdp, PAGE_SIZE, &nvchan->xdp_rxq);
-	xdp->data_hard_start = page_address(page);
-	xdp->data = xdp->data_hard_start + NETVSC_XDP_HDRM;
-	xdp_set_data_meta_invalid(xdp);
-	xdp->data_end = xdp->data + len;
+	xdp_prepare_buff(xdp, page_address(page), NETVSC_XDP_HDRM, len, false);
 
 	memcpy(xdp->data, data, len);
drivers/net/tun.c +1 -4
index a82f7823d428..1f57294cfd77 100644
@@ -1600,10 +1600,7 @@ static struct sk_buff *tun_build_skb(struct tun_struct *tun,
 		u32 act;
 
 		xdp_init_buff(&xdp, buflen, &tfile->xdp_rxq);
-		xdp.data_hard_start = buf;
-		xdp.data = buf + pad;
-		xdp_set_data_meta_invalid(&xdp);
-		xdp.data_end = xdp.data + len;
+		xdp_prepare_buff(&xdp, buf, pad, len, false);
 
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		if (act == XDP_REDIRECT || act == XDP_TX) {
drivers/net/veth.c +2 -6
index 6a0afce0253c..6e03b619c93c 100644
@@ -698,15 +698,11 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
 		skb = nskb;
 	}
 
-	xdp.data_hard_start = skb->head;
-	xdp.data = skb_mac_header(skb);
-	xdp.data_end = xdp.data + pktlen;
-	xdp.data_meta = xdp.data;
-
 	/* SKB "head" area always have tailroom for skb_shared_info */
-	frame_sz = (void *)skb_end_pointer(skb) - xdp.data_hard_start;
+	frame_sz = skb_end_pointer(skb) - skb->head;
 	frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 	xdp_init_buff(&xdp, frame_sz, &rq->xdp_rxq);
+	xdp_prepare_buff(&xdp, skb->head, skb->mac_header, pktlen, true);
 
 	orig_data = xdp.data;
 	orig_data_end = xdp.data_end;
drivers/net/virtio_net.c +4 -8
index a22ce87bcd9c..f65eea603f7d 100644
@@ -690,10 +690,8 @@ static struct sk_buff *receive_small(struct net_device *dev,
 		}
 
 		xdp_init_buff(&xdp, buflen, &rq->xdp_rxq);
-		xdp.data_hard_start = buf + VIRTNET_RX_PAD + vi->hdr_len;
-		xdp.data = xdp.data_hard_start + xdp_headroom;
-		xdp.data_end = xdp.data + len;
-		xdp.data_meta = xdp.data;
+		xdp_prepare_buff(&xdp, buf + VIRTNET_RX_PAD + vi->hdr_len,
+				 xdp_headroom, len, true);
 		orig_data = xdp.data;
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		stats->xdp_packets++;

@@ -859,10 +857,8 @@ static struct sk_buff *receive_mergeable(struct net_device *dev,
 		 */
 		data = page_address(xdp_page) + offset;
 		xdp_init_buff(&xdp, frame_sz - vi->hdr_len, &rq->xdp_rxq);
-		xdp.data_hard_start = data - VIRTIO_XDP_HEADROOM + vi->hdr_len;
-		xdp.data = data + vi->hdr_len;
-		xdp.data_end = xdp.data + (len - vi->hdr_len);
-		xdp.data_meta = xdp.data;
+		xdp_prepare_buff(&xdp, data - VIRTIO_XDP_HEADROOM + vi->hdr_len,
+				 VIRTIO_XDP_HEADROOM, len - vi->hdr_len, true);
 
 		act = bpf_prog_run_xdp(xdp_prog, &xdp);
 		stats->xdp_packets++;
drivers/net/xen-netfront.c +2 -4
index 329397c60d84..c20b78120bb4 100644
@@ -866,10 +866,8 @@ static u32 xennet_run_xdp(struct netfront_queue *queue, struct page *pdata,
 
 	xdp_init_buff(xdp, XEN_PAGE_SIZE - XDP_PACKET_HEADROOM,
 		      &queue->xdp_rxq);
-	xdp->data_hard_start = page_address(pdata);
-	xdp->data = xdp->data_hard_start + XDP_PACKET_HEADROOM;
-	xdp_set_data_meta_invalid(xdp);
-	xdp->data_end = xdp->data + len;
+	xdp_prepare_buff(xdp, page_address(pdata), XDP_PACKET_HEADROOM,
+			 len, false);
 
 	act = bpf_prog_run_xdp(prog, xdp);
 	switch (act) {
include/net/xdp.h +12 -0
index 323340caef88..c4bfdc9a8b79 100644
@@ -83,6 +83,18 @@ xdp_init_buff(struct xdp_buff *xdp, u32 frame_sz, struct xdp_rxq_info *rxq)
 	xdp->rxq = rxq;
 }
 
+static __always_inline void
+xdp_prepare_buff(struct xdp_buff *xdp, unsigned char *hard_start,
+		 int headroom, int data_len, const bool meta_valid)
+{
+	unsigned char *data = hard_start + headroom;
+
+	xdp->data_hard_start = hard_start;
+	xdp->data = data;
+	xdp->data_end = data + data_len;
+	xdp->data_meta = meta_valid ? data : data + 1;
+}
+
 /* Reserve memory area at end-of data area.
  *
  * This macro reserves tailroom in the XDP buffer by limiting the
net/bpf/test_run.c +2 -5
index a8fa5a9e4137..23dfb2010ba6 100644
@@ -636,14 +636,11 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
 	if (IS_ERR(data))
 		return PTR_ERR(data);
 
-	xdp.data_hard_start = data;
-	xdp.data = data + headroom;
-	xdp.data_meta = xdp.data;
-	xdp.data_end = xdp.data + size;
-
 	rxqueue = __netif_get_rx_queue(current->nsproxy->net_ns->loopback_dev, 0);
 	xdp_init_buff(&xdp, headroom + max_data_sz + tailroom,
 		      &rxqueue->xdp_rxq);
+	xdp_prepare_buff(&xdp, data, headroom, size, true);
+
 	bpf_prog_change_xdp(NULL, prog);
 	ret = bpf_test_run(prog, &xdp, repeat, &retval, &duration, true);
 	if (ret)
net/core/dev.c +9 -11
index b1a765900c01..2c99c48ee180 100644
@@ -4585,14 +4585,14 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 				     struct xdp_buff *xdp,
 				     struct bpf_prog *xdp_prog)
 {
+	void *orig_data, *orig_data_end, *hard_start;
 	struct netdev_rx_queue *rxqueue;
-	void *orig_data, *orig_data_end;
 	u32 metalen, act = XDP_DROP;
 	u32 mac_len, frame_sz;
 	__be16 orig_eth_type;
 	struct ethhdr *eth;
 	bool orig_bcast;
-	int hlen, off;
+	int off;
 
 	/* Reinjected packets coming from act_mirred or similar should
 	 * not get XDP generic processing.

@@ -4624,25 +4624,23 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 	 * header.
 	 */
 	mac_len = skb->data - skb_mac_header(skb);
-	hlen = skb_headlen(skb) + mac_len;
-	xdp->data = skb->data - mac_len;
-	xdp->data_meta = xdp->data;
-	xdp->data_end = xdp->data + hlen;
-	xdp->data_hard_start = skb->data - skb_headroom(skb);
+	hard_start = skb->data - skb_headroom(skb);
 
 	/* SKB "head" area always have tailroom for skb_shared_info */
-	frame_sz = (void *)skb_end_pointer(skb) - xdp->data_hard_start;
+	frame_sz = (void *)skb_end_pointer(skb) - hard_start;
 	frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
 
+	rxqueue = netif_get_rxqueue(skb);
+	xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq);
+	xdp_prepare_buff(xdp, hard_start, skb_headroom(skb) - mac_len,
+			 skb_headlen(skb) + mac_len, true);
+
 	orig_data_end = xdp->data_end;
 	orig_data = xdp->data;
 	eth = (struct ethhdr *)xdp->data;
 	orig_bcast = is_multicast_ether_addr_64bits(eth->h_dest);
 	orig_eth_type = eth->h_proto;
 
-	rxqueue = netif_get_rxqueue(skb);
-	xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq);
-
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
 
 	/* check if bpf_xdp_adjust_head was used */
Signature · 1 line
2.29.2
Acked-by on v5 2/2
#5 lore raw

Re: [PATCH v5 bpf-next 2/2] net: xdp: introduce xdp_prepare_buff utility routine

Quoted 27 lines · On Tue, 22 Dec 2020 22:09:29 +0100 Lorenzo Bianconi <[email protected]> wrote:

Introduce xdp_prepare_buff utility routine to initialize per-descriptor xdp_buff fields (e.g. xdp_buff pointers). Rely on xdp_prepare_buff() in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  7 +++----
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c |  5 +----
 .../net/ethernet/cavium/thunder/nicvf_main.c  |  8 ++++----
 .../net/ethernet/freescale/dpaa/dpaa_eth.c    |  6 ++----
 .../net/ethernet/freescale/dpaa2/dpaa2-eth.c  | 14 +++++--------
 drivers/net/ethernet/intel/i40e/i40e_txrx.c   | 12 +++++------
 drivers/net/ethernet/intel/ice/ice_txrx.c     |  9 +++++----
 drivers/net/ethernet/intel/igb/igb_main.c     | 12 +++++------
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 12 +++++------
 .../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 12 +++++------
 drivers/net/ethernet/marvell/mvneta.c         |  7 ++-----
 .../net/ethernet/marvell/mvpp2/mvpp2_main.c   |  8 +++-----
 drivers/net/ethernet/mellanox/mlx4/en_rx.c    |  6 ++----
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  5 +----
 .../ethernet/netronome/nfp/nfp_net_common.c   |  8 ++++----
 drivers/net/ethernet/qlogic/qede/qede_fp.c    |  6 ++----
 drivers/net/ethernet/sfc/rx.c                 |  7 ++-----
 drivers/net/ethernet/socionext/netsec.c       |  6 ++----
 drivers/net/ethernet/ti/cpsw.c                | 16 +++++----------
 drivers/net/ethernet/ti/cpsw_new.c            | 16 +++++----------
 drivers/net/hyperv/netvsc_bpf.c               |  5 +----
Quoted 8 lines
 drivers/net/tun.c                             |  5 +----
 drivers/net/veth.c                            |  8 ++------
 drivers/net/virtio_net.c                      | 12 ++++-------
 drivers/net/xen-netfront.c                    |  6 ++----
 include/net/xdp.h                             | 12 +++++++++++
 net/bpf/test_run.c                            |  7 ++-----
 net/core/dev.c                                | 20 +++++++++----------
 28 files changed, 105 insertions(+), 152 deletions(-)
Acked-by Jesper Dangaard Brouer <[email protected]>
Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
maintainer Acked-by on v5 2/2
#6 lore raw

RE: [PATCH v5 bpf-next 2/2] net: xdp: introduce xdp_prepare_buff utility routine

In reply to Lorenzo Bianconi
Quoted 6 lines · Lorenzo Bianconi

Introduce xdp_prepare_buff utility routine to initialize per-descriptor xdp_buff fields (e.g. xdp_buff pointers). Rely on xdp_prepare_buff() in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>

---

Same as 1, looks like a reasonable cleanup and spot checked math in a few drivers.

Acked-by John Fastabend <[email protected]>
maintainer
#7 lore raw

Re: [PATCH v5 bpf-next 2/2] net: xdp: introduce xdp_prepare_buff utility routine

In reply to Lorenzo Bianconi
Quoted 5 lines · Lorenzo Bianconi

Introduce xdp_prepare_buff utility routine to initialize per-descriptor xdp_buff fields (e.g. xdp_buff pointers). Rely on xdp_prepare_buff() in all XDP capable drivers.

Signed-off-by Lorenzo Bianconi <[email protected]>

Overall looks good to me, just one small nit in the Intel drivers below (cc Maciej):

[...]

Quoted 111 lines
drivers/net/ethernet/intel/i40e/i40e_txrx.c +6 -6
index a87fb8264d0c..2574e78f7597 100644
@@ -2406,12 +2406,12 @@ static int i40e_clean_rx_irq(struct i40e_ring *rx_ring, int budget)
 
  		/* retrieve a buffer from the ring */
  		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      i40e_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = i40e_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
  #if (PAGE_SIZE > 4096)
  			/* At larger PAGE_SIZE, frame_sz depend on len size */
  			xdp.frame_sz = i40e_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/ice/ice_txrx.c +5 -4
index 500e93bf6238..422f53997c02 100644
@@ -1104,8 +1104,10 @@ int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget)
 
  	/* start the loop to process Rx packets bounded by 'budget' */
  	while (likely(total_rx_pkts < (unsigned int)budget)) {
+		unsigned int offset = ice_rx_offset(rx_ring);
  		union ice_32b_rx_flex_desc *rx_desc;
  		struct ice_rx_buf *rx_buf;
+		unsigned char *hard_start;
  		struct sk_buff *skb;
  		unsigned int size;
  		u16 stat_err_bits;

@@ -1151,10 +1153,9 @@ int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget)
  			goto construct_skb;
  		}
 
-		xdp.data = page_address(rx_buf->page) + rx_buf->page_offset;
-		xdp.data_hard_start = xdp.data - ice_rx_offset(rx_ring);
-		xdp.data_meta = xdp.data;
-		xdp.data_end = xdp.data + size;
+		hard_start = page_address(rx_buf->page) + rx_buf->page_offset -
+			     offset;
+		xdp_prepare_buff(&xdp, hard_start, offset, size, true);
  #if (PAGE_SIZE > 4096)
  		/* At larger PAGE_SIZE, frame_sz depend on len size */
  		xdp.frame_sz = ice_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/igb/igb_main.c +6 -6
index cf9e8b7d2c70..6b5adbd9660b 100644
@@ -8715,12 +8715,12 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)
 
  		/* retrieve a buffer from the ring */
  		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      igb_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = igb_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
  #if (PAGE_SIZE > 4096)
  			/* At larger PAGE_SIZE, frame_sz depend on len size */
  			xdp.frame_sz = igb_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +6 -6
index 7ec196589a07..e714db51701a 100644
@@ -2335,12 +2335,12 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
 
  		/* retrieve a buffer from the ring */
  		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      ixgbe_rx_offset(rx_ring);
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = ixgbe_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
  #if (PAGE_SIZE > 4096)
  			/* At larger PAGE_SIZE, frame_sz depend on len size */
  			xdp.frame_sz = ixgbe_rx_frame_truesize(rx_ring, size);
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +0 -5
index 624efcd71569..a534a3fb392e 100644
@@ -1160,12 +1160,12 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector,
 
  		/* retrieve a buffer from the ring */
  		if (!skb) {
-			xdp.data = page_address(rx_buffer->page) +
-				   rx_buffer->page_offset;
-			xdp.data_meta = xdp.data;
-			xdp.data_hard_start = xdp.data -
-					      ixgbevf_rx_offset(rx_ring);

drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, · line 1171

Quoted 10 lines
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +6 -1
-			xdp.data_end = xdp.data + size;
+			unsigned int offset = ixgbevf_rx_offset(rx_ring);
+			unsigned char *hard_start;
+
+			hard_start = page_address(rx_buffer->page) +
+				     rx_buffer->page_offset - offset;
+			xdp_prepare_buff(&xdp, hard_start, offset, size, true);
  #if (PAGE_SIZE > 4096)
  			/* At larger PAGE_SIZE, frame_sz depend on len size */
  			xdp.frame_sz = ixgbevf_rx_frame_truesize(rx_ring, size);

[...]
The design is very similar for most of the Intel drivers. Why the inconsistency on ice driver compared to the rest, what's the rationale there to do it in one but not the others? Generated code better there?

Couldn't you even move the 'unsigned int offset = xyz_rx_offset(rx_ring)' out of the while loop altogether for all of them? (You already use the xyz_rx_offset() implicitly for most of them when setting xdp.frame_sz.)

Thanks,
Daniel

#8 lore raw

Re: [PATCH v5 bpf-next 2/2] net: xdp: introduce xdp_prepare_buff utility routine

Quoted 6 lines

drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, · line 1171

Quoted 6 lines
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +3 -0
+                     hard_start = page_address(rx_buffer->page) +
+                                  rx_buffer->page_offset - offset;
+                     xdp_prepare_buff(&xdp, hard_start, offset, size, true);
  #if (PAGE_SIZE > 4096)
                      /* At larger PAGE_SIZE, frame_sz depend on len size */
                      xdp.frame_sz = ixgbevf_rx_frame_truesize(rx_ring, size);

Hi Daniel,

thx for the review.

Quoted 4 lines

[...]
The design is very similar for most of the Intel drivers. Why the inconsistency on ice driver compared to the rest, what's the rationale there to do it in one but not the others? Generated code better there?

I applied the same logic for the ice driver but the code is just slightly different.

Quoted 5 lines

Couldn't you even move the 'unsigned int offset = xyz_rx_offset(rx_ring)' out of the while loop altogether for all of them? (You already use the xyz_rx_offset() implicitly for most of them when setting xdp.frame_sz.)

We discussed moving "offset = xyz_rx_offset(rx_ring)" out of the while loop before but Saeed asked to address it in a dedicated series since it is a little bit out of the scope. I have no strong opinion on it, do you prefer to address it directly here?

Regards,
Lorenzo

Quoted 3 lines

Thanks,
Daniel

maintainer
#9 lore raw

Re: [PATCH v5 bpf-next 2/2] net: xdp: introduce xdp_prepare_buff utility routine

Quoted 19 lines · Lorenzo Bianconi
Quoted 6 lines

drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, · line 1171

Quoted 6 lines
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +3 -0
+                     hard_start = page_address(rx_buffer->page) +
+                                  rx_buffer->page_offset - offset;
+                     xdp_prepare_buff(&xdp, hard_start, offset, size, true);
   #if (PAGE_SIZE > 4096)
                       /* At larger PAGE_SIZE, frame_sz depend on len size */
                       xdp.frame_sz = ixgbevf_rx_frame_truesize(rx_ring, size);

Hi Daniel,

thx for the review.

Quoted 4 lines

[...]
The design is very similar for most of the Intel drivers. Why the inconsistency on ice driver compared to the rest, what's the rationale there to do it in one but not the others? Generated code better there?

I applied the same logic for the ice driver but the code is just slightly different.

Quoted 8 lines
Quoted 3 lines

Couldn't you even move the 'unsigned int offset = xyz_rx_offset(rx_ring)' out of the while loop altogether for all of them? (You already use the xyz_rx_offset() implicitly for most of them when setting xdp.frame_sz.)

We discussed moving "offset = xyz_rx_offset(rx_ring)" out of the while loop before but Saeed asked to address it in a dedicated series since it is a little bit out of the scope. I have no strong opinion on it, do you prefer to address it directly here?

Fair enough, I might have preferred it in this series as part of the overall cleanup, but if you plan to follow up on this then this is also fine by me. Applied the v5 to bpf-next in that case, thanks!

#10 lore raw

Re: [PATCH v5 bpf-next 2/2] net: xdp: introduce xdp_prepare_buff utility routine

Quoted 24 lines · Daniel Borkmann
Quoted 23 lines · Lorenzo Bianconi
Quoted 6 lines

drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, · line 1171

Quoted 6 lines
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +3 -0
+                     hard_start = page_address(rx_buffer->page) +
+                                  rx_buffer->page_offset - offset;
+                     xdp_prepare_buff(&xdp, hard_start, offset, size, true);
   #if (PAGE_SIZE > 4096)
                       /* At larger PAGE_SIZE, frame_sz depend on len size */
                       xdp.frame_sz = ixgbevf_rx_frame_truesize(rx_ring, size);

Hi Daniel,

thx for the review.

Quoted 4 lines

[...]
The design is very similar for most of the Intel drivers. Why the inconsistency on ice driver compared to the rest, what's the rationale there to do it in one but not the others? Generated code better there?

I applied the same logic for the ice driver but the code is just slightly different.

Quoted 3 lines

Couldn't you even move the 'unsigned int offset = xyz_rx_offset(rx_ring)' out of the while loop altogether for all of them? (You already use the xyz_rx_offset() implicitly for most of them when setting xdp.frame_sz.)

Quoted 8 lines
Quoted 4 lines

We discussed moving "offset = xyz_rx_offset(rx_ring)" out of the while loop before but Saeed asked to address it in a dedicated series since it is a little bit out of the scope. I have no strong opinion on it, do you prefer to address it directly here?

Fair enough, I might have preferred it in this series as part of the overall cleanup, but if you plan to follow up on this then this is also fine by me. Applied the v5 to bpf-next in that case, thanks!

I initially pointed out the fact that we could store the output of xyz_rx_offset(rx_ring) onto a variable rather than call it per each processed buffer because value returned by that func can not change throughout the napi execution. It is based on ethtool priv flag which if changed, resets the PF (so disables napi, frees irqs, loads different Rx mem model, etc).

I realised that there is yet another place where we have unnecessary call to xyz_rx_offset() in hot path which is xyz_alloc_mapped_page(), so I expanded the idea of this optimization and I store the offset directly onto Rx ring and refer to that value.

I am including the patches that do what described above onto pending series of fixes that I had back in december '20 I suppose.

Reviewed-by on the v5 series
#11 lore raw
In reply to Lorenzo Bianconi
Quoted 23 lines · Lorenzo Bianconi

Introduce xdp_init_buff and xdp_prepare_buff utility routines to initialize xdp_buff data structure and remove duplicated code in all XDP capable drivers.

Changes since v4:
- fix xdp_init_buff/xdp_prepare_buff (natural order is xdp_init_buff() first and then xdp_prepare_buff())

Changes since v3:
- use __always_inline instead of inline for xdp_init_buff/xdp_prepare_buff
- add 'const bool meta_valid' to xdp_prepare_buff signature to avoid overwriting data_meta with xdp_set_data_meta_invalid()
- introduce removed comment in bnxt driver

Changes since v2:
- precompute xdp->data as hard_start + headroom and save it in a local variable to reuse it for xdp->data_end and xdp->data_meta in xdp_prepare_buff()

Changes since v1:
- introduce xdp_prepare_buff utility routine

Quoted 7 lines

Lorenzo Bianconi (2):
net: xdp: introduce xdp_init_buff utility routine
net: xdp: introduce xdp_prepare_buff utility routine

Acked-by Shay Agroskin <[email protected]>
Acked-by Martin Habets <[email protected]>
Acked-by Camelia Groza <[email protected]>

The changes to the Intel drivers look fine to me, although it might be nice to have someone from Intel provide a review/ack. I've added intel-wired-lan to the thread so that someone from Intel can hopefully review and also ack this.

Reviewed-by Alexander Duyck <[email protected]>
Acked-by on the v5 series
#12 lore raw
In reply to Lorenzo Bianconi

Hi Lorenzo,

wt., 22 gru 2020 o 22:13 Lorenzo Bianconi <[email protected]> napisał(a):

Quoted 23 lines

Introduce xdp_init_buff and xdp_prepare_buff utility routines to initialize xdp_buff data structure and remove duplicated code in all XDP capable drivers.

Changes since v4:
- fix xdp_init_buff/xdp_prepare_buff (natural order is xdp_init_buff() first and then xdp_prepare_buff())

Changes since v3:
- use __always_inline instead of inline for xdp_init_buff/xdp_prepare_buff
- add 'const bool meta_valid' to xdp_prepare_buff signature to avoid overwriting data_meta with xdp_set_data_meta_invalid()
- introduce removed comment in bnxt driver

Changes since v2:
- precompute xdp->data as hard_start + headroom and save it in a local variable to reuse it for xdp->data_end and xdp->data_meta in xdp_prepare_buff()

Changes since v1:
- introduce xdp_prepare_buff utility routine

Quoted 8 lines

Lorenzo Bianconi (2):
net: xdp: introduce xdp_init_buff utility routine
net: xdp: introduce xdp_prepare_buff utility routine

Acked-by Shay Agroskin <[email protected]>
Acked-by Martin Habets <[email protected]>
Acked-by Camelia Groza <[email protected]>

For Marvell mvpp2:

Acked-by Marcin Wojtas <[email protected]>

Thanks,
Marcin